[Bug]: Encountered a LoadSegment: stoi error during the query node restart.

[jaime0815]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: 2.4.5
- Deployment mode(standalone or cluster):
- MQ type(rocksmq, pulsar or kafka):    
- SDK version(e.g. pymilvus v2.0.0rc2):
- OS(Ubuntu or CentOS): 
- CPU/Memory: 
- GPU: 
- Others:

Current Behavior

An error at LoadSegment: stoi occurs during the restart of querynode. Due to the lack of a detailed error message, we cannot identify which numeric string failed to convert.

[2024/09/13 01:42:22.135 +00:00] [WARN] [segments/load_index_info.go:233] ["CStatus returns err"] [traceID=467f2db43e3ef7f5dfce17f44c7b75a1] [error=stoi] [extra="AppendIndex failed"]
[2024/09/13 01:42:22.144 +00:00] [WARN] [segments/segment_loader.go:695] ["load segment failed when load data into memory"] [traceID=467f2db43e3ef7f5dfce17f44c7b75a1] [collectionID=451632464416316619] [segmentType=Sealed] [requestSegments="[451632464418111575]"] [preparedSegments="[451632464418111575]"] [partitionID=451632464416318926] [segmentID=451632464418111575] [segmentType=L1] [error="At LoadSegment: stoi"] [errorVerbose="At LoadSegment: stoi\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) At LoadSegment\nWraps: (3) stoi\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/09/13 01:42:22.144 +00:00] [INFO] [segments/segment_loader.go:697] ["load segment done"] [traceID=467f2db43e3ef7f5dfce17f44c7b75a1] [collectionID=451632464416316619] [segmentType=Sealed] [requestSegments="[451632464418111575]"] [preparedSegments="[451632464418111575]"] [partitionID=451632464416318926] [segmentID=451632464418111575] [segmentType=L1]
[2024/09/13 01:42:22.144 +00:00] [ERROR] [funcutil/parallel.go:88] [loadSegmentFunc] [error="At LoadSegment: stoi"] [errorVerbose="At LoadSegment: stoi\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) At LoadSegment\nWraps: (3) stoi\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:88"]
[2024/09/13 01:42:22.144 +00:00] [WARN] [segments/segment_loader.go:734] ["failed to load some segments"] [traceID=467f2db43e3ef7f5dfce17f44c7b75a1] [collectionID=451632464416316619] [segmentType=Sealed] [requestSegments="[451632464418111575]"] [preparedSegments="[451632464418111575]"] [error="At LoadSegment: stoi"] [errorVerbose="At LoadSegment: stoi\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) At LoadSegment\nWraps: (3) stoi\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]
[2024/09/13 01:42:22.144 +00:00] [WARN] [segments/segment_loader.go:643] ["release new segment created due to load failure"] [traceID=467f2db43e3ef7f5dfce17f44c7b75a1] [collectionID=451632464416316619] [segmentType=Sealed] [requestSegments="[451632464418111575]"] [preparedSegments="[451632464418111575]"] [segmentID=451632464418111575] [error="At LoadSegment: stoi"] [errorVerbose="At LoadSegment: stoi\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/go/src/github.com/milvus-io/milvus/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/go/src/github.com/milvus-io/milvus/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1598\nWraps: (2) At LoadSegment\nWraps: (3) stoi\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"]

Expected Behavior

No response

Steps To Reproduce

No response

Milvus Log

No response

Anything else?

No response

[become-nice]

I also encountered the same error, the error message is as follows.

[2024/12/03 13:12:06.370 +08:00] [WARN] [task/executor.go:232] ["failed to load segment"] [taskID=1733197089983] [collectionID=453083736594493726] [replicaID=454347199065882627] [segmentID=454254090268447366] [node=161] [source=segment_checker] [shardLeader=161] [error="The specified key does not exist.: key not found[key=milvus-xxx-56789/delta_log/453083736594493726/453083736594493727/454254090268447366/454254090268866948]"]
[2024/12/03 13:12:06.370 +08:00] [ERROR] [funcutil/parallel.go:88] [loadSegmentFunc] [error="At LoadSegment: stoi"] [errorVerbose="At LoadSegment: stoi\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/workspace/source/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) At LoadSegment\nWraps: (3) stoi\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:88"]
[2024/12/03 13:12:06.370 +08:00] [ERROR] [funcutil/parallel.go:88] [loadSegmentFunc] [error="At LoadSegment: stoi"] [errorVerbose="At LoadSegment: stoi\n(1) attached stack trace\n  -- stack trace:\n  | github.com/milvus-io/milvus/internal/querynodev2/segments.(*segmentLoader).Load.func4\n  | \t/workspace/source/internal/querynodev2/segments/segment_loader.go:708\n  | github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n  | \t/workspace/source/pkg/util/funcutil/parallel.go:86\n  | runtime.goexit\n  | \t/usr/local/go/src/runtime/asm_amd64.s:1650\nWraps: (2) At LoadSegment\nWraps: (3) stoi\nError types: (1) *withstack.withStack (2) *errutil.withPrefix (3) merr.milvusError"] [idx=0] [stack="github.com/milvus-io/milvus/pkg/util/funcutil.ProcessFuncParallel.func3\n\t/workspace/source/pkg/util/funcutil/parallel.go:88"]

[xiaofan-luan]

upgrade to 2.4.20 and you should the actual error message is.

The problem is the real error is swallowed