Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enhance: [cherry-pick] clean failed/timeout compaction task in compactionHandler #35758

Closed
wants to merge 503 commits into from
Closed

enhance: [cherry-pick] clean failed/timeout compaction task in compactionHandler #35758

wants to merge 503 commits into from

Conversation

wayblink
Copy link
Contributor

issue: #35711
master pr:#34991

zhuwenxing and others added 30 commits July 8, 2024 12:54
@zhuwenxing
test: [cherry-pick]add testcase for count query (milvus-io#34471)
f2d0517 
pr: milvus-io#34453

Signed-off-by: zhuwenxing <[email protected]>
@congqixia
fix: [2.4] Add nbits parameter check for IVF_PQ (milvus-io#34451) (m…
7c3c9c2 
…ilvus-io#34473)

Cherry-pick from master
pr: milvus-io#34451
See also milvus-io#34426

Signed-off-by: Congqi Xia <[email protected]>
@XuanYang-cn
fix: Accidently exit the check loop (milvus-io#34480)
26ac769 
See also: milvus-io#34460
pr: milvus-io#34481

Signed-off-by: yangxuan <[email protected]>
@bigsheeper
fix: Fix incorrect segment num rows (milvus-io#34441) (milvus-io#34474)
0732167 
Repeated calls to UpdateStatistics, this PR correct it.

issue: milvus-io#34440

pr: milvus-io#34441

Signed-off-by: bigsheeper <[email protected]>
@bigsheeper
enhance: Check segment existence when FlushSegments and add some key …
0d7ba81 
…logs (milvus-io#34438) (milvus-io#34472)

Check if the segment exists during FlushSegments and add some key logs
in write path.

issue: milvus-io#34255

pr: milvus-io#34438

Signed-off-by: bigsheeper <[email protected]>
@wayblink
fix: [cherry-pick]fix can't enqueue when compaction queue is full(mil…
56a74e7 
…vus-io#34445) (milvus-io#34469)

issue: milvus-io#30633
pr: milvus-io#34445

Signed-off-by: wayblink <[email protected]>
@MrPresent-Han
fix: lose partitionIDs when scalar pruning and refine segment prune r…
2f38483 
…atio metrics(milvus-io#30376) (milvus-io#34475)

related: milvus-io#30376
pr: milvus-io#34477

Signed-off-by: MrPresent-Han <[email protected]>
@chasingegg
enhance: ensure autoindex default metric type compatibility (milvus-i…
f6cd841 
…o#34479)

issue: milvus-io#34304 
pr: milvus-io#34261

Signed-off-by: chasingegg <[email protected]>
@weiliu1031
fix: Query may return deleted records (milvus-io#34502)
7034260 
issue: milvus-io#34500
pr: milvus-io#34501

cause the sort in `GetLevel0Deletions` will broken the corresponed order
between pks and tss, then the pks and tss will be sorted in
segment.Delete() interface.

This PR remove this uncessary and incorrect sort progress to avoid query
may return deleted records.

Signed-off-by: Wei Liu <[email protected]>
@bigsheeper
fix: Fix unstable ut of msgdispatcher (milvus-io#34485) (milvus-io#34486
42b38fb 
)

issue: milvus-io#34484

pr: milvus-io#34485

Signed-off-by: bigsheeper <[email protected]>
@wayblink
fix: [cherry-pick] Fix compaction start time (milvus-io#34505)
7788dec 
bug: milvus-io#34497 milvus-io#34498
pr: milvus-io#34496

---------

Signed-off-by: wayblink <[email protected]>
@SimFG
enhance: [2.4] release the record in delete codec and add some log fo…
737bd7c 
…r compaction (milvus-io#34506)

/kind improvement
- pr: milvus-io#34454

Signed-off-by: SimFG <[email protected]>
@congqixia
fix: [2.4] Check err is ErrKeyNotFound when CASCachedValue (milvus-io…
ddb1015 
…#34489)

Cherry-pick from master
pr: milvus-io#34488
See also milvus-io#33785

When config item is not present in paramtable, CAS fails due to
GetConfig returns error.

This PR make this returned err instance of ErrKeyNotFound and check
error type in \`CASCachedValue\` methods.

Signed-off-by: Congqi Xia <[email protected]>
@jaime0815
enhance: support setting properties in create database request (milvu…
60aab15 
…s-io#34511)

issue: milvus-io#34493
pr: milvus-io#34510

Signed-off-by: jaime <[email protected]>
@congqixia
fix: [2.4] Write padding at end of mmap file not chunk (milvus-io#34530)
0d8defb 
Cherry-pick from master
pr: milvus-io#34529
Related to milvus-io#34508

The padding bytes shall be written only at the end of the mmap file not
the chunk of each field data file.

Signed-off-by: Congqi Xia <[email protected]>
@weiliu1031
fix: RootCoord panic during check health (milvus-io#34542)
f0d12aa 
issue: milvus-io#34540
pr: milvus-io#34541

Signed-off-by: Wei Liu <[email protected]>
@bigsheeper
fix: Fix no L0 compaction tasks are executed (milvus-io#34537) (milvu…
1589d68 
…s-io#34538)

The nodeID for compaction task initialization is 0. This PR adjusts the
task reassignment conditions to allow new compaction tasks to be
reassigned and executed.

issue: milvus-io#34460

pr: milvus-io#34537

Signed-off-by: bigsheeper <[email protected]>
@weiliu1031
fix: Indexnode stuck at stopping progress cause by wrong lifetime con…
a7190b0 
…trol (milvus-io#34559)

issue: milvus-io#34557
pr: milvus-io#34558

Signed-off-by: Wei Liu <[email protected]>
@SimFG
enhance: [2.4] add the seal segment when dispatch delete msgs (milvus…
15adb2f 
…-io#34566)

/kind improvement
- pr: milvus-io#34565

Signed-off-by: SimFG <[email protected]>
@bigsheeper
fix: Prevent dispatcher merging if curTs is 0 (milvus-io#34562) (milv…
cbba548 
…us-io#34563)

When the main dispatcher has not yet consumed data, curTs is 0. During
this time, merging dispatchers should not be allowed; otherwise, the
data of the solo dispatcher will be skipped.

issue: milvus-io#34255

pr: milvus-io#34562

---------

Signed-off-by: bigsheeper <[email protected]>
@XuanYang-cn
enhance: Add back compactionTaskNum metrics (milvus-io#34534)
2bef9c3 
Fix L0 compaction task recover unable to set segment
not isCompacting

See also: milvus-io#34460
pr: milvus-io#34583

Signed-off-by: yangxuan <[email protected]>

Signed-off-by: yangxuan <[email protected]>
@jaime0815
enhance: skip orphan channel cp meta when checking cp lag (milvus-io#…
073a501 
…34556)

issue: milvus-io#34545
pr: milvus-io#34555

Signed-off-by: jaime <[email protected]>
@jaime0815
fix: unstable UT for level0 deletion (milvus-io#34525)
bfd386a 
issue: milvus-io#34533
pr: milvus-io#34524

Signed-off-by: jaime <[email protected]>
@aoiasd
fix: [Cherry-pick] Restful API use deprecate error code cause access …
d6fc6a9 
…log panic. (milvus-io#34579)

Avoid panic when use deprecate error code and use merr code replace
deprecate common code.
relate: milvus-io#34578
pr： milvus-io#34576

---------

Signed-off-by: aoiasd <[email protected]>
@XuanYang-cn
fix: Milvus panic when compaction disabled and dropping a collection (m…
7e2a9d6 
…ilvus-io#34206)

See also: milvus-io#31059
pr: milvus-io#34103

---------

Signed-off-by: yangxuan <[email protected]>
@LoveEachDay
enhance: Refine default histogram metric bucket (milvus-io#34592)
3d29ebf 
/kind improvement
pr: milvus-io#34580

Signed-off-by: Edward Zeng <[email protected]>
@wayblink
enhance: [cherry-pick] Refine clustering compaction latency metrics (m…
9dc98ac 
…ilvus-io#34539)

issue: milvus-io#34527
pr: milvus-io#34528

Signed-off-by: wayblink <[email protected]>
@XuanYang-cn
fix: Compaction task l0 state transfer wrong (milvus-io#34599)
be41892 
This bug caused failed L0 compaction tasks never end.

See also: milvus-io#34460
pr: milvus-io#34597

---------

Signed-off-by: yangxuan <[email protected]>
@jaime0815
fix: encountering orphan channel-cp meta after GC (milvus-io#34609)
1d2062a 
issue: milvus-io#34545
pr: milvus-io#34612

Signed-off-by: jaime <[email protected]>
@wayblink
enhance: [cherry-pick] Alter compactTo segments before compactFrom to…
86b57b7 
… avoid data loss if crash (milvus-io#34532)

issue: milvus-io#34512 
pr: milvus-io#34513

Signed-off-by: wayblink <[email protected]>
weiliu1031 and others added 18 commits August 24, 2024 10:28
@weiliu1031 @sunby
fix: Fix index memory estimation (milvus-io#35225) (milvus-io#35670)
35d2f9b 
issue: milvus-io#35229
pr: milvus-io#32525

Signed-off-by: Wei Liu <[email protected]>
Co-authored-by: Bingyi Sun <[email protected]>
@PwzXxm
enhance: [2.4] enable MV on binary vectors (milvus-io#35669)
4b5bd07 
pr: milvus-io#35668

Signed-off-by: Patrick Weizhi Xu <[email protected]>
(cherry picked from commit 5404228)
@jaime0815
fix: the incorrect number of partitions in rootcoord meta (milvus-io#…
5c1a380 
…35601)

pr: milvus-io#35600

Signed-off-by: jaime <[email protected]>
@xiaofan-luan
enhance: [2.4] reduce the log level of frequent log (milvus-io#35653)
7269d5e 
pr: milvus-io#35651

Signed-off-by: xiaofanluan <[email protected]>
@chyezh
enhance: enable asan for milvus (milvus-io#35687)
7062e6f 
issue: milvus-io#35626
pr: milvus-io#35627

Signed-off-by: chyezh <[email protected]>
@XuanYang-cn
fix: [skip e2e]unstable l0 it (milvus-io#35613)
ff68eb5 
See also: milvus-io#35617 
pr: milvus-io#35612

Signed-off-by: yangxuan <[email protected]>
@XuanYang-cn
fix: [cp]Fill nil schema so that Milvus can watch channel for those u…
f2caecc 
…pgraded from 2.2 to 2.4 (milvus-io#35695)

See also: milvus-io#35701 
pr: milvus-io#35694

---------

Signed-off-by: yangxuan <[email protected]>
@yellow-shine
enhance: turnoff USE_ASAN in ci (milvus-io#35723)
33fbc1f 
Signed-off-by: Yellow Shine <[email protected]>
@congqixia
feat: [2.4] Support field partial load collection (milvus-io#35416) (m…
ab261d0 
…ilvus-io#35696)

Cherry-pick from master
pr: milvus-io#35416
Related to milvus-io#35415

---------

---------

Signed-off-by: Congqi Xia <[email protected]>
@yellow-shine
enhance new jenkinsfile for nightly and e2e (milvus-io#35715)
0362715 
milvus-io#35672

Signed-off-by: Yellow Shine <[email protected]>
@SimFG
feat: [2.4] add the rbac msg and send them to the replicate channel (m…
fc324b4 
…ilvus-io#35562)

- issue: milvus-io#35391
- pr: milvus-io#35392

Signed-off-by: SimFG <[email protected]>
@chyezh
fix: munmap deallocate too much memory (milvus-io#35726)
69e8d5e 
issue: milvus-io#35693
pr: milvus-io#35725

Signed-off-by: chyezh <[email protected]>
@bigsheeper
fix: Fix panic due to empty candidate import segments (milvus-io#35673)…
be17cf6 
… (milvus-io#35674)

issue: milvus-io#35662

pr: milvus-io#35673

---------

Signed-off-by: bigsheeper <[email protected]>
@chyezh
enhance: optimize milvus core building (milvus-io#35660)
a4533f1 
issue: milvus-io#35549,milvus-io#35611,milvus-io#35633
pr: milvus-io#35610

- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr

---------

Signed-off-by: chyezh <[email protected]>
@jaime0815
fix: mistaken deletions may occur during GC channel checkpoints (milv…
eecc598 
…us-io#35708)

issue: milvus-io#35706
pr: milvus-io#35707

Signed-off-by: jaime <[email protected]>
@bigsheeper
fix: Fix rate wasn't limited to the expected value (milvus-io#35699) (m…
8e6ec58 
…ilvus-io#35700)

Each time the rate is reset, the token bucket is fully refilled, causing
the rate wasn't limited to the expected value. This PR addresses the
issue by preventing the token reset.

issue: milvus-io#35675,
milvus-io#35702

pr: milvus-io#35699

---------

Signed-off-by: bigsheeper <[email protected]>
@jaime0815
fix: inconsistent meta view causes rate limit invalid (milvus-io#35664)
c75f556 
issue: milvus-io#35663
pr: milvus-io#35665

Signed-off-by: jaime <[email protected]>
@wayblink
enhance: clean failed/timeout compaction task in compactionHandler
603c4aa 
Signed-off-by: wayblink <[email protected]>
@wayblink wayblink closed this Aug 28, 2024
@sre-ci-robot sre-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines. label Aug 28, 2024
@sre-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: wayblink
To complete the pull request process, please assign xiaofan-luan after the PR has been reviewed.
You can assign the PR to them by writing /assign @xiaofan-luan in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@mergify Mergify
Copy link
Contributor

mergify bot commented Aug 28, 2024

⚠️ The sha of the head commit of this PR conflicts with #35759. Mergify cannot evaluate rules on this PR. ⚠️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aoiasd aoiasd Awaiting requested review from aoiasd

@bigsheeper bigsheeper Awaiting requested review from bigsheeper

Assignees
No one assigned
Labels
area/compilation area/dependency Pull requests that update a dependency file area/internal-api area/test sig/testing size/XXL Denotes a PR that changes 1000+ lines. test/integration integration test
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.