[Bug]: [Nightly] The deleted entities can be queried after compaction and release load

[NicoYuan1986]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 032292a
- Deployment mode(standalone or cluster):cluster
- MQ type(rocksmq, pulsar or kafka):    kafka
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

The deleted entities can be queried after compaction and release load.

2025-01-12T21:32:48Z {container="step-test"} ________________ TestCompactionParams.test_compact_after_delete ________________
2025-01-12T21:32:48Z {container="step-test"} [gw3] linux -- Python 3.10.12 /usr/local/bin/python3
2025-01-12T21:32:48Z {container="step-test"} 
2025-01-12T21:32:48Z {container="step-test"} self = <test_compaction.TestCompactionParams object at 0x7f4156c53ac0>
2025-01-12T21:32:48Z {container="step-test"} 
2025-01-12T21:32:48Z {container="step-test"} @pytest.mark.tags(CaseLabel.L2)
2025-01-12T21:32:48Z {container="step-test"} def test_compact_after_delete(self):
2025-01-12T21:32:48Z {container="step-test"} """
2025-01-12T21:32:48Z {container="step-test"} target: test delete and then compact
2025-01-12T21:32:48Z {container="step-test"} method: 1. create a collection and insert data
2025-01-12T21:32:48Z {container="step-test"} 2. delete all data and compact
2025-01-12T21:32:48Z {container="step-test"} 3. load query and release
2025-01-12T21:32:48Z {container="step-test"} expected: can't find the deleted data
2025-01-12T21:32:48Z {container="step-test"} """
2025-01-12T21:32:48Z {container="step-test"} nb = 50000
2025-01-12T21:32:48Z {container="step-test"} collection_w = self.init_collection_wrap(name=cf.gen_unique_str(prefix))
2025-01-12T21:32:48Z {container="step-test"} df = cf.gen_default_dataframe_data(nb=nb)
2025-01-12T21:32:48Z {container="step-test"} res, _ = collection_w.insert(df)
2025-01-12T21:32:48Z {container="step-test"} assert collection_w.num_entities == nb
2025-01-12T21:32:48Z {container="step-test"} 
2025-01-12T21:32:48Z {container="step-test"} expr = f'{ct.default_int64_field_name} in {res.primary_keys}'
2025-01-12T21:32:48Z {container="step-test"} collection_w.create_index(ct.default_float_vec_field_name, index_params=ct.default_flat_index)
2025-01-12T21:32:48Z {container="step-test"} collection_w.delete(expr)
2025-01-12T21:32:48Z {container="step-test"} assert collection_w.num_entities == nb
2025-01-12T21:32:48Z {container="step-test"} collection_w.compact()
2025-01-12T21:32:48Z {container="step-test"} collection_w.wait_for_compaction_completed()
2025-01-12T21:32:48Z {container="step-test"} # collection_w.get_compaction_plans(check_task=CheckTasks.check_delete_compact)
2025-01-12T21:32:48Z {container="step-test"} 
2025-01-12T21:32:48Z {container="step-test"} collection_w.load()
2025-01-12T21:32:48Z {container="step-test"} res = collection_w.query(expr)[0]
2025-01-12T21:32:48Z {container="step-test"} assert len(res) == 0
2025-01-12T21:32:48Z {container="step-test"} collection_w.release()
2025-01-12T21:32:48Z {container="step-test"} 
2025-01-12T21:32:48Z {container="step-test"} sleep(30)
2025-01-12T21:32:48Z {container="step-test"} collection_w.load()
2025-01-12T21:32:48Z {container="step-test"} res = collection_w.query(expr)[0]
2025-01-12T21:32:48Z {container="step-test"} assert len(res) == 0
2025-01-12T21:32:48Z {container="step-test"} collection_w.release()
2025-01-12T21:32:48Z {container="step-test"} 
2025-01-12T21:32:48Z {container="step-test"} sleep(40)
2025-01-12T21:32:48Z {container="step-test"} collection_w.load()
2025-01-12T21:32:48Z {container="step-test"} res = collection_w.query(expr)[0]
2025-01-12T21:32:48Z {container="step-test"} >       assert len(res) == 0
2025-01-12T21:32:48Z {container="step-test"} E       assert 50000 == 0
2025-01-12T21:32:48Z {container="step-test"} E        +  where 50000 = len(data: ["{'int64': 0}", "{'int64': 1}", "{'int64': 2}", "{'int64': 3}", "{'int64': 4}", "{'int64': 5}", "{'int64': 6}", "{'int64': 7}", "{'int64': 8}", "{'int64': 9}"] ...)


2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:00 - DEBUG - ci_test]: (api_request)  : [Collection.delete] args: ['int64 in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74......, kwargs: {} (api_request.py:52)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:00 - DEBUG - ci_test]: (api_response) : (insert count: 0, delete count: 50000, upsert count: 0, timestamp: 455268331789484039, success count: 0, err count: 0  (api_request.py:27)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:00 - DEBUG - ci_test]: (api_request)  : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:52)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:07 - DEBUG - ci_test]: (api_response) : None  (api_request.py:27)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:07 - DEBUG - ci_test]: (api_request)  : [Collection.compact] args: [False, 180], kwargs: {} (api_request.py:52)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:07 - DEBUG - ci_test]: (api_response) : None  (api_request.py:27)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:19 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:52)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:20 - DEBUG - ci_test]: (api_response) : None  (api_request.py:27)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:20 - DEBUG - ci_test]: (api_request)  : [Collection.query] args: ['int64 in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74......, kwargs: {} (api_request.py:52)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:20 - DEBUG - ci_test]: (api_response) : data: []   (api_request.py:27)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:20 - DEBUG - ci_test]: (api_request)  : [Collection.release] args: [], kwargs: {'timeout': 180} (api_request.py:52)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:21 - DEBUG - ci_test]: (api_response) : None  (api_request.py:27)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:51 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:52)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:52 - DEBUG - ci_test]: (api_response) : None  (api_request.py:27)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:52 - DEBUG - ci_test]: (api_request)  : [Collection.query] args: ['int64 in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74......, kwargs: {} (api_request.py:52)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:52 - DEBUG - ci_test]: (api_response) : data: []   (api_request.py:27)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:52 - DEBUG - ci_test]: (api_request)  : [Collection.release] args: [], kwargs: {'timeout': 180} (api_request.py:52)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:41:53 - DEBUG - ci_test]: (api_response) : None  (api_request.py:27)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:42:33 - DEBUG - ci_test]: (api_request)  : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:52)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:42:34 - DEBUG - ci_test]: (api_response) : None  (api_request.py:27)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:42:34 - DEBUG - ci_test]: (api_request)  : [Collection.query] args: ['int64 in [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74......, kwargs: {} (api_request.py:52)
2025-01-12T21:32:48Z {container="step-test"} [2025-01-12 19:42:34 - DEBUG - ci_test]: (api_response) : data: ["{'int64': 0}", "{'int64': 1}", "{'int64': 2}", "{'int64': 3}", "{'int64': 4}", "{'int64': 5}", "{'int64': 6}", "{'int64': 7}", "{'int64': 8}", "{'int64': 9}"] ...  (api_request.py:27)

Expected Behavior

query result is empty

Steps To Reproduce

1. create a collection and insert data
2. delete all data and compact
3. load query and release
4. wait 30s, load query and release
5. wait 40s, load query and release -> ❌

Milvus Log

1. link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI(new)/detail/master/241/pipeline/122/
2. 2025-01-12T19:42:34Z {container="step-test"} [gw3] [ 13%] FAILED testcases/test_compaction.py::TestCompactionParams::test_compact_after_delete

Anything else?

No response

[yanliang567]

/assign @XuanYang-cn
/unassign