Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: [json-inverted] Standalone terminated with error reason when inserting and querying json data #39300

Open
1 task done
ThreadDao opened this issue Jan 15, 2025 · 0 comments
Assignees
Labels
kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Milestone

Comments

@ThreadDao
Copy link
Contributor

ThreadDao commented Jan 15, 2025

Is there an existing issue for this?

  • I have searched the existing issues

Environment

- Milvus version: JsDove-optimization_json-987a55c-20250114
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

server

  • config
  config:
    dataCoord:
      enableActiveStandby: true
    indexCoord:
      enableActiveStandby: true
    log:
      level: debug
    mixCoord:
      enableActiveStandby: true
    queryCoord:
      enableActiveStandby: true
    queryNode:
      mmap:
        jsonIndexCommitInterval: 200 
        scalarIndex: false
    rootCoord:
      enableActiveStandby: true
    trace:
      exporter: jaeger
      jaeger:
        url: http://tempo-distributor.tempo:14268/api/traces
      sampleFraction: 1

test

  1. collection has 3 fields: int64 pk + 128dim vector + json fields
  2. crrate index for vector fields
  3. insert 10w data, each json entity has more than 1000+ json keys
        json_key_num = 100
        values = [{f"key_{k}": i for k in range(json_key_num)} for i in ids]
  1. flush -> vector index again -> load
  2. concurrent requests: insert + query
  • each user inserts 100 pieces of data each time, and the data generation method is the same as above
  • Each user gets 50 ids from the inserted id queue and constructs an expr "json_1['key_0'] in {}".format(inserted_ids) to query
  'concurrent_params': {'concurrent_number': 2, 'during_time': '5h', 'interval': 60, 'spawn_rate': None},
   'concurrent_tasks': [{'type': 'insert',
                         'weight': 1,
                         'params': {'nb': 100, 'start_id': 100000, 'random_id': True, 'random_vector': True, 'timeout': 30}},
                        {'type': 'scene_query', 'weight': 1, 'params': {'delete_length': 50, 'timeout': 60}}]},
  1. standalone terminated with 134 error code, the coredump file located in /tmp/cores
root@json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr:/tmp/cores# ll | grep json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr
-rw------- 1 root root  6288130048 Jan 14 18:03 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-MILVUS_FUTURE_C-8-1736877784
-rw------- 1 root root  9534926848 Jan 14 18:18 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-MILVUS_FUTURE_C-8-1736878706
-rw------- 1 root root 14802169856 Jan 14 18:42 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-MILVUS_FUTURE_C-8-1736880145
-rw------- 1 root root 20678729728 Jan 14 19:03 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-MILVUS_FUTURE_C-8-1736881397
-rw------- 1 root root 13061574656 Jan 14 19:10 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-MILVUS_FUTURE_C-8-1736881825
-rw------- 1 root root 24932749312 Jan 14 20:13 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-MILVUS_FUTURE_C-8-1736885623
-rw------- 1 root root 20996120576 Jan 14 21:03 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-MILVUS_FUTURE_C-8-1736888622
-rw------- 1 root root 23170756608 Jan 14 21:21 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-MILVUS_FUTURE_C-8-1736889685
-rw------- 1 root root 25130926080 Jan 14 22:06 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-MILVUS_FUTURE_C-8-1736892369
-rw------- 1 root root 26282323968 Jan 14 22:16 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-MILVUS_FUTURE_C-8-1736892974
-rw------- 1 root root 23943954432 Jan 14 20:43 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-MILVUS_FUTURE_C-9-1736887421
-rw------- 1 root root 27974369280 Jan 14 22:45 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-MILVUS_FUTURE_C-9-1736894732
-rw------- 1 root root  8674439168 Jan 14 18:13 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-milvus-8-1736878379
-rw------- 1 root root 24915259392 Jan 14 20:53 core-json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr-milvus-8-1736888007

Expected Behavior

No response

Steps To Reproduce

https://argo-workflows.zilliz.cc/archived-workflows/qa/3453624a-be3e-4cd3-b58a-c83ce091a182?nodeId=zong-json-growing-json-a

Milvus Log

pods:

json-inverted-mmap-1-2-milvus-standalone-7bdc487669-cmrk7       1/1     Running            1 (15h ago)       16h     10.104.9.160    4am-node14   <none>           <none>
json-inverted-mmap-200-2-milvus-standalone-75d48767c4-sw5hf     1/1     Running            0                 16h     10.104.30.215   4am-node38   <none>           <none>
json-inverted-ram-1-2-milvus-standalone-6db7988dc4-7fsc9        1/1     Running            9 (11h ago)       16h     10.104.33.230   4am-node36   <none>           <none>
json-inverted-ram-200-2-milvus-standalone-d75b496dc-sdtmr       1/1     Running            14 (11h ago)      16h     10.104.13.135   4am-node16   <none>           <none>

Anything else?

No response

@ThreadDao ThreadDao added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 15, 2025
@ThreadDao ThreadDao added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Jan 15, 2025
@ThreadDao ThreadDao modified the milestones: 2.5.3, 2.5.4 Jan 15, 2025
@yanliang567 yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 16, 2025
@yanliang567 yanliang567 removed their assignment Jan 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Issues or changes related a bug priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

No branches or pull requests

3 participants