[Bug]: Apply pod-kill during concurrent upsert and search, count(*) is less than expected after recovery

[ThreadDao]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.4-20250116-622af571-amd64
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server

• config

    config:
    dataCoord:
      enableActiveStandby: true
    indexCoord:
      enableActiveStandby: true
    log:
      level: debug
    queryCoord:
      enableActiveStandby: true
    rootCoord:
      enableActiveStandby: true
    trace:
      exporter: jaeger
      jaeger:
        url: http://tempo-distributor.tempo:14268/api/traces
      sampleFraction: 1

test

1. collection has 2 fields: int64 pk + 128dim vector

{'auto_id': False, 'description': '', 'fields': [{'name': 'id', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float_vector', 'description': '', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128}}], 'enable_dynamic_field': False}

2. crrate index for vector fields
3. insert 20million data, it's pk range from 0 to 20,000,000 -> flush -> index again -> load
4. Apply pod-kill on standalone when concurrent requests: upsert and search

 'concurrent_params': {'concurrent_number': 50, 'during_time': '2h', 'interval': 20, 'spawn_rate': None},
 'concurrent_tasks': [{'type': 'search',
                       'weight': 20,
                       'params': {'nq': 100, 'top_k': 100, 'search_param': {'ef': 128}, 'timeout': 30}},
                      {'type': 'flush', 'weight': 1, 'params': {'timeout': 120}},
                      {'type': 'upsert',
                       'weight': 29,
                       'params': {'nb': 200,
                                  'timeout': 60,
                                  'start_id': 0,
                                  'random_id': True,
                                  'random_vector': True,
                                  'shuffle_id': True}}]},

5. upsert the pk from 0 to 5,517,326. And count(*) all should return 20,000,000 but 19,997,800

c.query("id > 5517326", output_fields=["count(*)"], consistency_level="Strong")
data: ["{'count(*)': 14482673}"] , extra_info: {'cost': 0}
c.query("id <= 5517326", output_fields=["count(*)"], consistency_level="Strong")
data: ["{'count(*)': 5515127}"] , extra_info: {'cost': 0}
14482673 + 5515127
19997800
c.query("", output_fields=["count(*)"], consistency_level="Strong")
data: ["{'count(*)': 19997800}"] , extra_info: {'cost': 0}

Expected Behavior

No response

Steps To Reproduce

https://argo-workflows.zilliz.cc/archived-workflows/qa/ae412cad-807e-4f07-b199-2a11a9df3792?nodeId=zong-chaos-stand-244

Milvus Log

pods:

zong-chaos-stand-244-etcd-0                                       1/1     Running            0                3h17m   10.104.26.84    4am-node32   <none>           <none>
zong-chaos-stand-244-kafka-0                                      1/1     Running            2 (3h16m ago)    3h17m   10.104.34.240   4am-node37   <none>           <none>
zong-chaos-stand-244-kafka-1                                      1/1     Running            2 (3h16m ago)    3h17m   10.104.25.12    4am-node30   <none>           <none>
zong-chaos-stand-244-kafka-2                                      1/1     Running            2 (3h16m ago)    3h17m   10.104.20.38    4am-node22   <none>           <none>
zong-chaos-stand-244-kafka-zookeeper-0                            1/1     Running            0                3h17m   10.104.18.234   4am-node25   <none>           <none>
zong-chaos-stand-244-kafka-zookeeper-1                            1/1     Running            0                3h17m   10.104.26.85    4am-node32   <none>           <none>
zong-chaos-stand-244-kafka-zookeeper-2                            1/1     Running            0                3h17m   10.104.25.13    4am-node30   <none>           <none>
zong-chaos-stand-244-milvus-standalone-649cc79c94-4s59r           1/1     Running            0                123m    10.104.6.157    4am-node13   <none>           <none>
zong-chaos-stand-244-minio-64f59b65df-g7twk                       1/1     Running            0                3h17m   10.104.18.233   4am-node25   <none>           <none>

Anything else?

No response

[ThreadDao]

c.query("1000000 <= id < 2000000", output_fields=["count(*)"], consistency_level="Strong")
data: ["{'count(*)': 997800}"] , extra_info: {'cost': 0}

[yanliang567]

/assign @XuanYang-cn
/unassign