[Bug]: Failed to import data using RemoteBulkWriter #39348

lihuiping1994 · 2025-01-16T09:17:16Z

Is there an existing issue for this?

I have searched the existing issues

Environment

- Milvus version: v2.5.3
- Deployment mode(standalone or cluster): standalone 
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus v2.5.1
- OS(Ubuntu or CentOS): Windows
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

When I imported 300w data using bulkwritten and checked the status of the collection task using the list_iimport_jobs function, I saw that progress was stuck at 80 in the returned response, "state": "Importing"， But the task details are all "state": "Completed" (when the imported data volume is only 1000, there is no situation of being uploaded)
{
"code": 0,
"data": {
"collectionName": "clue_data",
"completeTime": "",
"details": [
{
"completeTime": "2025-01-16T08:42:09Z",
"fileName": "[5.parquet]",
"fileSize": 6776153,
"importedRows": 520161,
"progress": 100,
"state": "Completed",
"totalRows": 520161
},
{
"completeTime": "2025-01-16T08:42:09Z",
"fileName": "[6.parquet]",
"fileSize": 5201131,
"importedRows": 399060,
"progress": 100,
"state": "Completed",
"totalRows": 399060
},
{
"completeTime": "2025-01-16T08:42:09Z",
"fileName": "[2.parquet]",
"fileSize": 6774659,
"importedRows": 520219,
"progress": 100,
"state": "Completed",
"totalRows": 520219
},
{
"completeTime": "2025-01-16T08:42:09Z",
"fileName": "[1.parquet]",
"fileSize": 6773335,
"importedRows": 520238,
"progress": 100,
"state": "Completed",
"totalRows": 520238
},
{
"completeTime": "2025-01-16T08:42:09Z",
"fileName": "[3.parquet]",
"fileSize": 6776574,
"importedRows": 520161,
"progress": 100,
"state": "Completed",
"totalRows": 520161
},
{
"completeTime": "2025-01-16T08:41:55Z",
"fileName": "[4.parquet]",
"fileSize": 6778112,
"importedRows": 520161,
"progress": 100,
"state": "Completed",
"totalRows": 520161
}
],
"fileSize": 39079964,
"importedRows": 3000000,
"jobId": "455347834366209865",
"progress": 80,
"state": "Importing",
"totalRows": 3000000
}
}
{
"code": 0,
"data": {
"records": [
{
"collectionName": "clue_data",
"jobId": "455347834366209865",
"progress": 80,
"state": "Importing"
}
]
}
}

Expected Behavior

After importing 300w of data using bulkwritten, use the list_iimporter function to check the status of the collection task. The progress should be 100

Steps To Reproduce

Milvus Log

milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210528]
milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210529]
milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210530]
milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210546]
milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210547]
milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210548]
milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210549]
milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210550]
milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210556]
milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210557]
milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210558]
milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210559]
milvus-standalone | [2025/01/16 09:16:29.213 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210560]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210516]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210517]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210518]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210519]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210520]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210507]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210508]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210509]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210510]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210536]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210537]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210538]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/task_index.go:316] ["query task index info successfully"] [taskID=455347834369816525] ["result state"=InProgress] [failReason=]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210539]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455347834369836027] ["task type"=JobTypeIndexJob] [state=JobStateInit]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=6]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/task_index.go:261] ["index task pre check successfully"] [taskID=455347834369836027] [segID=455347834366210507]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455347834366210540]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [indexnode/indexnode_service.go:210] ["Get Index Job Stats"] [traceID=b34136e54011f7c4396c5e1629385366] [unissued=0] [active=1] [slot=0]
milvus-standalone | [2025/01/16 09:16:29.214 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, waiting for retry..."]
milvus-standalone | [2025/01/16 09:16:30.213 +00:00] [INFO] [datacoord/task_scheduler.go:248] ["task scheduler"] ["task num"=108]
milvus-standalone | [2025/01/16 09:16:30.213 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455347834369816525] ["task type"=JobTypeIndexJob] [state=JobStateInProgress]
milvus-standalone | [2025/01/16 09:16:30.214 +00:00] [INFO] [datacoord/task_index.go:316] ["query task index info successfully"] [taskID=455347834369816525] ["result state"=InProgress] [failReason=]
milvus-standalone | [2025/01/16 09:16:30.215 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455347834369836027] ["task type"=JobTypeIndexJob] [state=JobStateInit]
milvus-standalone | [2025/01/16 09:16:30.216 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=6]
milvus-standalone | [2025/01/16 09:16:30.216 +00:00] [INFO] [datacoord/task_index.go:261] ["index task pre check successfully"] [taskID=455347834369836027] [segID=455347834366210507]
milvus-standalone | [2025/01/16 09:16:30.216 +00:00] [INFO] [indexnode/indexnode_service.go:210] ["Get Index Job Stats"] [traceID=86d2c09ad877eda7b158e6569efc5965] [unissued=0] [active=1] [slot=0]
milvus-standalone | [2025/01/16 09:16:30.216 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, waiting for retry..."]

Anything else?

No response

yanliang567 · 2025-01-17T02:19:18Z

/assign @xiaocai2333
/unassign

xiaocai2333 · 2025-01-17T06:36:24Z

The import operation will wait for the index to be built. The current progress shows 80%, which indicates that it is waiting for the index build. Based on the logs you provided, the system is indeed in the index-building phase.

xiaocai2333 · 2025-01-17T06:40:10Z

/assign @lihuiping1994

lihuiping1994 · 2025-01-17T06:44:05Z

The waiting time is very long, usually tens or even tens of hours, and the progress still hasn't changed. According to what you said, how long should we wait for 300w data to complete building the index

lihuiping1994 · 2025-01-17T06:50:52Z

/assign @xiaocai2333

lihuiping1994 · 2025-01-17T06:53:09Z

During the waiting period for the index to be established, after a long time, when using the get_iint_deprogress function to check the task status, there may be an inability to connect to the database and a crash phenomenon

xiaocai2333 · 2025-01-17T07:39:25Z

Can you provide the indexnode log during importing? @lihuiping1994

lihuiping1994 · 2025-01-17T08:24:57Z

milvus-standalone | [2025/01/17 08:24:05.895 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530384487] ["queueing time(ms)"=3228983]
milvus-standalone | [2025/01/17 08:24:05.895 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530531521] ["queueing time(ms)"=3223985]
milvus-standalone | [2025/01/17 08:24:05.895 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530711561] ["queueing time(ms)"=3217988]
milvus-standalone | [2025/01/17 08:24:05.895 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529857894] ["queueing time(ms)"=3239976]
milvus-standalone | [2025/01/17 08:24:05.895 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529880396] ["queueing time(ms)"=3239973]
milvus-standalone | [2025/01/17 08:24:05.895 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530240447] ["queueing time(ms)"=3234986]
milvus-standalone | [2025/01/17 08:24:05.895 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530366485] ["queueing time(ms)"=3228987]
milvus-standalone | [2025/01/17 08:24:05.895 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529721374] ["queueing time(ms)"=3240988]
milvus-standalone | [2025/01/17 08:24:05.895 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530510519] ["queueing time(ms)"=3223989]
milvus-standalone | [2025/01/17 08:24:05.895 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529812889] ["queueing time(ms)"=3239986]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798531002636] ["queueing time(ms)"=3206978]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529947902] ["queueing time(ms)"=3239963]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530060412] ["queueing time(ms)"=3239943]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530837600] ["queueing time(ms)"=3211983]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529925400] ["queueing time(ms)"=3239965]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529970404] ["queueing time(ms)"=3239960]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530689059] ["queueing time(ms)"=3217991]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530855602] ["queueing time(ms)"=3211979]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529745376] ["queueing time(ms)"=3240984]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530082914] ["queueing time(ms)"=3239940]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798531002666] ["queueing time(ms)"=3200990]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530037910] ["queueing time(ms)"=3239947]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530981634] ["queueing time(ms)"=3206982]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529790385] ["queueing time(ms)"=3239992]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529902898] ["queueing time(ms)"=3239969]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530015408] ["queueing time(ms)"=3239952]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529835392] ["queueing time(ms)"=3239980]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/compaction_trigger.go:306] ["the length of SegmentsChanPart is 0, skip to handle compaction"] [compactionID=455369798531018806] [signal.collectionID=0] [signal.partitionID=0] [signal.segmentID=0]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798531002667] ["queueing time(ms)"=3200988]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/task_index.go:316] ["query task index info successfully"] [taskID=455369798529532334] ["result state"=InProgress] [failReason=]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275218]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529553336] ["task type"=JobTypeIndexJob] [state=JobStateInit]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798530217945] ["queueing time(ms)"=3234990]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [WARN] [datacoord/task_scheduler.go:346] ["task running time is too long"] [taskID=455369798529532334] ["running time(ms)"=3200976]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275219]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275220]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275226]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275227]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275228]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275229]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275230]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=6]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/task_index.go:261] ["index task pre check successfully"] [taskID=455369798529553336] [segID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=PreImportTask] [pending=0] [inProgress=0] [completed=1] [failed=0]
milvus-standalone | [2025/01/17 08:24:05.896 +00:00] [INFO] [datacoord/import_checker.go:137] ["import task stats"] [type=ImportTask] [pending=0] [inProgress=0] [completed=2] [failed=0]
milvus-standalone | [2025/01/17 08:24:05.897 +00:00] [INFO] [indexnode/indexnode_service.go:210] ["Get Index Job Stats"] [traceID=db3bf4e8202d2635b492c1f843ce8958] [unissued=0] [active=1] [slot=0]
milvus-standalone | [2025/01/17 08:24:05.897 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, waiting for retry..."]
milvus-standalone | [2025/01/17 08:24:05.897 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529553336] ["queueing time(ms)"=3246987]
milvus-standalone | [2025/01/17 08:24:05.897 +00:00] [WARN] [datacoord/task_scheduler.go:335] ["task queueing time is too long"] [taskID=455369798529992906] ["queueing time(ms)"=3239958]
milvus-standalone | [2025/01/17 08:24:06.896 +00:00] [INFO] [datacoord/task_scheduler.go:248] ["task scheduler"] ["task num"=32]
milvus-standalone | [2025/01/17 08:24:06.896 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529532334] ["task type"=JobTypeIndexJob] [state=JobStateInProgress]
milvus-standalone | [2025/01/17 08:24:06.897 +00:00] [INFO] [datacoord/task_index.go:316] ["query task index info successfully"] [taskID=455369798529532334] ["result state"=InProgress] [failReason=]
milvus-standalone | [2025/01/17 08:24:06.897 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529553336] ["task type"=JobTypeIndexJob] [state=JobStateInit]
milvus-standalone | [2025/01/17 08:24:06.897 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=6]
milvus-standalone | [2025/01/17 08:24:06.897 +00:00] [INFO] [datacoord/task_index.go:261] ["index task pre check successfully"] [taskID=455369798529553336] [segID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:06.897 +00:00] [INFO] [indexnode/indexnode_service.go:210] ["Get Index Job Stats"] [traceID=c0338dea2bbf053f848196063ec831d3] [unissued=0] [active=1] [slot=0]
milvus-standalone | [2025/01/17 08:24:06.897 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, waiting for retry..."]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/task_scheduler.go:248] ["task scheduler"] ["task num"=32]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529532334] ["task type"=JobTypeIndexJob] [state=JobStateInProgress]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275218]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275219]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275220]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275226]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275227]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275228]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275229]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275230]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/task_index.go:316] ["query task index info successfully"] [taskID=455369798529532334] ["result state"=InProgress] [failReason=]
milvus-standalone | [2025/01/17 08:24:07.896 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529553336] ["task type"=JobTypeIndexJob] [state=JobStateInit]
milvus-standalone | [2025/01/17 08:24:07.897 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=6]
milvus-standalone | [2025/01/17 08:24:07.897 +00:00] [INFO] [datacoord/task_index.go:261] ["index task pre check successfully"] [taskID=455369798529553336] [segID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:07.897 +00:00] [INFO] [indexnode/indexnode_service.go:210] ["Get Index Job Stats"] [traceID=b83869ea7331bf265fa5bd218805f70a] [unissued=0] [active=1] [slot=0]
milvus-standalone | [2025/01/17 08:24:07.897 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, waiting for retry..."]
milvus-standalone | [2025/01/17 08:24:08.897 +00:00] [INFO] [datacoord/task_scheduler.go:248] ["task scheduler"] ["task num"=32]
milvus-standalone | [2025/01/17 08:24:08.897 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529532334] ["task type"=JobTypeIndexJob] [state=JobStateInProgress]
milvus-standalone | [2025/01/17 08:24:08.898 +00:00] [INFO] [datacoord/task_index.go:316] ["query task index info successfully"] [taskID=455369798529532334] ["result state"=InProgress] [failReason=]
milvus-standalone | [2025/01/17 08:24:08.898 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529553336] ["task type"=JobTypeIndexJob] [state=JobStateInit]
milvus-standalone | [2025/01/17 08:24:08.898 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=6]
milvus-standalone | [2025/01/17 08:24:08.898 +00:00] [INFO] [datacoord/task_index.go:261] ["index task pre check successfully"] [taskID=455369798529553336] [segID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:08.899 +00:00] [INFO] [indexnode/indexnode_service.go:210] ["Get Index Job Stats"] [traceID=7c3b70f5f57e0bdcc91d3e278035f78f] [unissued=0] [active=1] [slot=0]
milvus-standalone | [2025/01/17 08:24:08.899 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, waiting for retry..."]
milvus-standalone | [2025/01/17 08:24:09.896 +00:00] [INFO] [datacoord/task_scheduler.go:248] ["task scheduler"] ["task num"=32]
milvus-standalone | [2025/01/17 08:24:09.896 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529532334] ["task type"=JobTypeIndexJob] [state=JobStateInProgress]
milvus-standalone | [2025/01/17 08:24:09.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:09.897 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275218]
milvus-standalone | [2025/01/17 08:24:09.897 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275219]
milvus-standalone | [2025/01/17 08:24:09.897 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275220]
milvus-standalone | [2025/01/17 08:24:09.897 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275226]
milvus-standalone | [2025/01/17 08:24:09.897 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275227]
milvus-standalone | [2025/01/17 08:24:09.897 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275228]
milvus-standalone | [2025/01/17 08:24:09.897 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275229]
milvus-standalone | [2025/01/17 08:24:09.897 +00:00] [INFO] [datacoord/task_index.go:316] ["query task index info successfully"] [taskID=455369798529532334] ["result state"=InProgress] [failReason=]
milvus-standalone | [2025/01/17 08:24:09.897 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275230]
milvus-standalone | [2025/01/17 08:24:09.897 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529553336] ["task type"=JobTypeIndexJob] [state=JobStateInit]
milvus-standalone | [2025/01/17 08:24:09.897 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=6]
milvus-standalone | [2025/01/17 08:24:09.897 +00:00] [INFO] [datacoord/task_index.go:261] ["index task pre check successfully"] [taskID=455369798529553336] [segID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:09.900 +00:00] [INFO] [indexnode/indexnode_service.go:210] ["Get Index Job Stats"] [traceID=49110a824a80931389d5c3624544d099] [unissued=0] [active=1] [slot=0]
milvus-standalone | [2025/01/17 08:24:09.900 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, waiting for retry..."]
milvus-standalone | [2025/01/17 08:24:10.895 +00:00] [INFO] [datacoord/task_scheduler.go:248] ["task scheduler"] ["task num"=32]
milvus-standalone | [2025/01/17 08:24:10.895 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529532334] ["task type"=JobTypeIndexJob] [state=JobStateInProgress]
milvus-standalone | [2025/01/17 08:24:10.896 +00:00] [INFO] [datacoord/task_index.go:316] ["query task index info successfully"] [taskID=455369798529532334] ["result state"=InProgress] [failReason=]
milvus-standalone | [2025/01/17 08:24:10.896 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529553336] ["task type"=JobTypeIndexJob] [state=JobStateInit]
milvus-standalone | [2025/01/17 08:24:10.897 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=6]
milvus-standalone | [2025/01/17 08:24:10.897 +00:00] [INFO] [datacoord/task_index.go:261] ["index task pre check successfully"] [taskID=455369798529553336] [segID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:10.898 +00:00] [INFO] [indexnode/indexnode_service.go:210] ["Get Index Job Stats"] [traceID=e2f3d92699865cc383ae5ed46013b491] [unissued=0] [active=1] [slot=0]
milvus-standalone | [2025/01/17 08:24:10.898 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, waiting for retry..."]
milvus-standalone | [2025/01/17 08:24:11.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:11.898 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275218]
milvus-standalone | [2025/01/17 08:24:11.897 +00:00] [INFO] [datacoord/task_scheduler.go:248] ["task scheduler"] ["task num"=32]
milvus-standalone | [2025/01/17 08:24:11.899 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275219]
milvus-standalone | [2025/01/17 08:24:11.900 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275220]
milvus-standalone | [2025/01/17 08:24:11.900 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275226]
milvus-standalone | [2025/01/17 08:24:11.901 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275227]
milvus-standalone | [2025/01/17 08:24:11.901 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275228]
milvus-standalone | [2025/01/17 08:24:11.902 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275229]
milvus-standalone | [2025/01/17 08:24:11.902 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275230]
milvus-standalone | [2025/01/17 08:24:11.900 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529532334] ["task type"=JobTypeIndexJob] [state=JobStateInProgress]
milvus-standalone | [2025/01/17 08:24:11.904 +00:00] [INFO] [datacoord/task_index.go:316] ["query task index info successfully"] [taskID=455369798529532334] ["result state"=InProgress] [failReason=]
milvus-standalone | [2025/01/17 08:24:11.904 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529553336] ["task type"=JobTypeIndexJob] [state=JobStateInit]
milvus-standalone | [2025/01/17 08:24:11.905 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=6]
milvus-standalone | [2025/01/17 08:24:11.906 +00:00] [INFO] [datacoord/task_index.go:261] ["index task pre check successfully"] [taskID=455369798529553336] [segID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:11.907 +00:00] [INFO] [indexnode/indexnode_service.go:210] ["Get Index Job Stats"] [traceID=ef8b2ae28b0a2aad81422c359f15086f] [unissued=0] [active=1] [slot=0]
milvus-standalone | [2025/01/17 08:24:11.909 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, waiting for retry..."]
milvus-standalone | [2025/01/17 08:24:12.896 +00:00] [INFO] [datacoord/task_scheduler.go:248] ["task scheduler"] ["task num"=32]
milvus-standalone | [2025/01/17 08:24:12.896 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529532334] ["task type"=JobTypeIndexJob] [state=JobStateInProgress]
milvus-standalone | [2025/01/17 08:24:12.898 +00:00] [INFO] [datacoord/task_index.go:316] ["query task index info successfully"] [taskID=455369798529532334] ["result state"=InProgress] [failReason=]
milvus-standalone | [2025/01/17 08:24:12.898 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529553336] ["task type"=JobTypeIndexJob] [state=JobStateInit]
milvus-standalone | [2025/01/17 08:24:12.899 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=6]
milvus-standalone | [2025/01/17 08:24:12.900 +00:00] [INFO] [datacoord/task_index.go:261] ["index task pre check successfully"] [taskID=455369798529553336] [segID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:12.901 +00:00] [INFO] [indexnode/indexnode_service.go:210] ["Get Index Job Stats"] [traceID=f20b6cfb16b8a3c57858c8fcdc794fab] [unissued=0] [active=1] [slot=0]
milvus-standalone | [2025/01/17 08:24:12.902 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, waiting for retry..."]
milvus-standalone | [2025/01/17 08:24:13.896 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:13.898 +00:00] [INFO] [datacoord/task_scheduler.go:248] ["task scheduler"] ["task num"=32]
milvus-standalone | [2025/01/17 08:24:13.898 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529532334] ["task type"=JobTypeIndexJob] [state=JobStateInProgress]
milvus-standalone | [2025/01/17 08:24:13.899 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275218]
milvus-standalone | [2025/01/17 08:24:13.900 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275219]
milvus-standalone | [2025/01/17 08:24:13.901 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275220]
milvus-standalone | [2025/01/17 08:24:13.902 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275226]
milvus-standalone | [2025/01/17 08:24:13.902 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275227]
milvus-standalone | [2025/01/17 08:24:13.903 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275228]
milvus-standalone | [2025/01/17 08:24:13.903 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275229]
milvus-standalone | [2025/01/17 08:24:13.905 +00:00] [INFO] [datacoord/index_service.go:153] ["receive new flushed segment"] [segmentID=455369798528275230]
milvus-standalone | [2025/01/17 08:24:13.907 +00:00] [INFO] [datacoord/task_index.go:316] ["query task index info successfully"] [taskID=455369798529532334] ["result state"=InProgress] [failReason=]
milvus-standalone | [2025/01/17 08:24:13.908 +00:00] [INFO] [datacoord/task_scheduler.go:279] ["task is processing"] [taskID=455369798529553336] ["task type"=JobTypeIndexJob] [state=JobStateInit]
milvus-standalone | [2025/01/17 08:24:13.908 +00:00] [INFO] [datacoord/index_engine_version_manager.go:84] ["Merged current version"] [current=6]
milvus-standalone | [2025/01/17 08:24:13.908 +00:00] [INFO] [datacoord/task_index.go:261] ["index task pre check successfully"] [taskID=455369798529553336] [segID=455369798528275217]
milvus-standalone | [2025/01/17 08:24:13.908 +00:00] [INFO] [indexnode/indexnode_service.go:210] ["Get Index Job Stats"] [traceID=72fb5212144387cf0ae915a853cc5775] [unissued=0] [active=1] [slot=0]
milvus-standalone | [2025/01/17 08:24:13.909 +00:00] [INFO] [datacoord/task_scheduler.go:258] ["there is no idle indexing node, waiting for retry..."]

lihuiping1994 · 2025-01-17T08:35:42Z

/assign @xiaocai2333

xiaofan-luan · 2025-01-19T10:11:43Z

@lihuiping1994
I think you simlpy need large datanode and indexnode for your use case.

for 3 million vectors, it should't be too slow, unless milvus flush many smaller files when importing. due to lack of memories.

Suggestions:

create at least 1 datanode with 4core 16G memory or above
create at least 2 index node with 4core 16G memory or above.

There seems to be a lot of smaller segments, I guess it is due to lack of memory and force flush

xiaofan-luan · 2025-01-19T10:12:45Z

also for full log
Could you please refer this doc to export the whole Milvus logs for investigation? For Milvus installed with docker-compose, you can use docker-compose logs > milvus.log to export the logs.

lihuiping1994 · 2025-01-19T10:23:15Z

Windows 16GB memory, 12 core CPU, running V2.5.3 version on Docker desktop without any parameter restrictions on the container. I tried several times afterwards, with a cycle of about 20 hours, but the progress still did not change. We have now deleted the Docker mapping directory and cannot provide milvus.log

lihuiping1994 · 2025-01-19T10:23:23Z

/assign @xiaocai2333

xiaofan-luan · 2025-01-19T10:28:38Z

Windows 16GB memory, 12 core CPU, running V2.5.3 version on Docker desktop without any parameter restrictions on the container. I tried several times afterwards, with a cycle of about 20 hours, but the progress still did not change. We have now deleted the Docker mapping directory and cannot provide milvus.log

16G Might be not enough for 3million data, it might be ok with 768dim ,but definitely not 1024 or 1536 dim.

please offer more logs and we can do nothing without all the detailed logs

xiaofan-luan · 2025-01-19T10:31:18Z

if you are using standalone, I would recommend you to start from at least 12core 32GB memory.

lihuiping1994 · 2025-01-19T11:25:58Z

I mainly want to test the use of RemoteBulkWriter to import data. The native Docker standalone version did not import successfully, so I did not run the import program on a server with better performance, as production environment resources are limited

xiaofan-luan · 2025-01-19T13:01:34Z

the remote bulk writer only prepare the file and write it to S3.
Bulk import and index still only happens on the standalone server so it need some resources as well.

xiaocai2333 · 2025-01-20T03:29:49Z

also for full log Could you please refer this doc to export the whole Milvus logs for investigation? For Milvus installed with docker-compose, you can use docker-compose logs > milvus.log to export the logs.

From the portion of logs you provided, it is evident that there was an issue during the index-building phase, but the exact cause cannot be determined. A complete log is needed for further investigation. Could you provide the full log following this approach? @lihuiping1994

lihuiping1994 · 2025-01-20T03:34:59Z

When I import 300w simulated data again using the local Docker environment, I will provide complete logs for easy troubleshooting

lihuiping1994 added kind/bug Issues or changes related a bug needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 16, 2025

lihuiping1994 assigned yanliang567 Jan 16, 2025

sre-ci-robot assigned xiaocai2333 and unassigned yanliang567 Jan 17, 2025

yanliang567 added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 17, 2025

sre-ci-robot assigned lihuiping1994 Jan 17, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Failed to import data using RemoteBulkWriter #39348

[Bug]: Failed to import data using RemoteBulkWriter #39348

lihuiping1994 commented Jan 16, 2025

yanliang567 commented Jan 17, 2025

xiaocai2333 commented Jan 17, 2025

xiaocai2333 commented Jan 17, 2025

lihuiping1994 commented Jan 17, 2025

lihuiping1994 commented Jan 17, 2025

lihuiping1994 commented Jan 17, 2025

xiaocai2333 commented Jan 17, 2025 •

edited

Loading

lihuiping1994 commented Jan 17, 2025

lihuiping1994 commented Jan 17, 2025

xiaofan-luan commented Jan 19, 2025

xiaofan-luan commented Jan 19, 2025

lihuiping1994 commented Jan 19, 2025

lihuiping1994 commented Jan 19, 2025

xiaofan-luan commented Jan 19, 2025

xiaofan-luan commented Jan 19, 2025

lihuiping1994 commented Jan 19, 2025

xiaofan-luan commented Jan 19, 2025

xiaocai2333 commented Jan 20, 2025

lihuiping1994 commented Jan 20, 2025

[Bug]: Failed to import data using RemoteBulkWriter #39348

[Bug]: Failed to import data using RemoteBulkWriter #39348

Comments

lihuiping1994 commented Jan 16, 2025

Is there an existing issue for this?

Environment

Current Behavior

Expected Behavior

Steps To Reproduce

Milvus Log

Anything else?

yanliang567 commented Jan 17, 2025

xiaocai2333 commented Jan 17, 2025

xiaocai2333 commented Jan 17, 2025

lihuiping1994 commented Jan 17, 2025

lihuiping1994 commented Jan 17, 2025

lihuiping1994 commented Jan 17, 2025

xiaocai2333 commented Jan 17, 2025 • edited Loading

lihuiping1994 commented Jan 17, 2025

lihuiping1994 commented Jan 17, 2025

xiaofan-luan commented Jan 19, 2025

xiaofan-luan commented Jan 19, 2025

lihuiping1994 commented Jan 19, 2025

lihuiping1994 commented Jan 19, 2025

xiaofan-luan commented Jan 19, 2025

xiaofan-luan commented Jan 19, 2025

lihuiping1994 commented Jan 19, 2025

xiaofan-luan commented Jan 19, 2025

xiaocai2333 commented Jan 20, 2025

lihuiping1994 commented Jan 20, 2025

xiaocai2333 commented Jan 17, 2025 •

edited

Loading