enhance: Reduce mutex contention in datacoord meta #38219

bigsheeper · 2024-12-04T13:13:36Z

Using secondary index to avoid retrieving all segments at GetSegmentsChanPart.
Perform batch SetAllocations to reduce the number of times the meta lock is acquired.

issue: #37630

mergify · 2024-12-04T13:41:32Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-04T14:10:33Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

codecov · 2024-12-04T14:30:08Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.13%. Comparing base (d89768f) to head (faa9d9c).
Report is 2 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master   #38219       +/-   ##
===========================================
+ Coverage   69.71%   81.13%   +11.41%     
===========================================
  Files         299     1404     +1105     
  Lines       26768   197748   +170980     
===========================================
+ Hits        18661   160436   +141775     
- Misses       8107    31676    +23569     
- Partials        0     5636     +5636

Components	Coverage Δ
Client	`79.53% <ø> (∅)`
Core	`69.65% <ø> (-0.06%)`	⬇️
Go	`83.06% <100.00%> (∅)`

Files with missing lines	Coverage Δ
internal/datacoord/compaction_policy_clustering.go	`75.48% <100.00%> (ø)`
internal/datacoord/compaction_policy_single.go	`61.00% <100.00%> (ø)`
internal/datacoord/compaction_trigger.go	`85.53% <100.00%> (ø)`
internal/datacoord/meta.go	`89.66% <100.00%> (ø)`
internal/datacoord/segment_info.go	`84.72% <100.00%> (ø)`

... and 1107 files with indirect coverage changes

Signed-off-by: bigsheeper <[email protected]>

1. Using secondary index to avoid retrieving all segments at GetSegmentsChanPart. 2. Perform batch SetAllocations to reduce the number of times the meta lock is acquired. issue: #37630 pr: #38219 --------- Signed-off-by: bigsheeper <[email protected]>

…2-opt-meta-mutex

Signed-off-by: bigsheeper <[email protected]>

…2-opt-meta-mutex

mergify · 2024-12-05T11:14:51Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-05T11:21:31Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

…2-opt-meta-mutex

Signed-off-by: bigsheeper <[email protected]>

mergify · 2024-12-13T14:07:13Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

…2-opt-meta-mutex

mergify · 2024-12-17T03:24:13Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

bigsheeper · 2024-12-17T03:27:21Z

rerun go-sdk

…2-opt-meta-mutex

Signed-off-by: bigsheeper <[email protected]>

…2-opt-meta-mutex

mergify · 2024-12-31T09:03:35Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

bigsheeper · 2024-12-31T09:29:10Z

rerun go-sdk

mergify · 2024-12-31T10:07:03Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

bigsheeper · 2025-01-01T06:42:42Z

rerun go-sdk

…2-opt-meta-mutex

internal/datacoord/meta.go

…2-opt-meta-mutex

Signed-off-by: bigsheeper <[email protected]>

xiaofan-luan · 2025-01-14T13:41:06Z

/lgtm
/approve

sre-ci-robot · 2025-01-14T13:41:23Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bigsheeper, xiaofan-luan

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~internal/datacoord/OWNERS~~ [xiaofan-luan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

1. Using secondary index to avoid retrieving all segments at GetSegmentsChanPart. 2. Perform batch SetAllocations to reduce the number of times the meta lock is acquired. issue: #37630 pr: #38219 --------- Signed-off-by: bigsheeper <[email protected]>

1. Using secondary index to avoid retrieving all segments at `GetSegmentsChanPart`. 2. Perform batch SetAllocations to reduce the number of times the meta lock is acquired. issue: milvus-io#37630 --------- Signed-off-by: bigsheeper <[email protected]>

sre-ci-robot added the size/M Denotes a PR that changes 30-99 lines. label Dec 4, 2024

sre-ci-robot requested review from congqixia and cydrain December 4, 2024 13:14

mergify bot added dco-passed DCO check passed. kind/enhancement Issues or changes related to enhancement labels Dec 4, 2024

Reduce mutex contention in datacoord meta

bae195c

Signed-off-by: bigsheeper <[email protected]>

sre-ci-robot added size/L Denotes a PR that changes 100-499 lines. and removed size/M Denotes a PR that changes 30-99 lines. labels Dec 5, 2024

bigsheeper force-pushed the 2412-opt-meta-mutex branch from 43ec0aa to bae195c Compare December 5, 2024 03:18

This was referenced Dec 5, 2024

enhance: [10kcp] Reduce mutex contention in datacoord meta #38229

Merged

enhance: [2.4] Reduce mutex contention in datacoord meta #38230

Open

bigsheeper added 3 commits December 5, 2024 17:52

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

3b180f5

…2-opt-meta-mutex

fix ut

590efdb

Signed-off-by: bigsheeper <[email protected]>

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

bc47595

…2-opt-meta-mutex

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

4d823d7

…2-opt-meta-mutex

mergify bot added the ci-passed label Dec 5, 2024

fix conflicts

99f6f8f

Signed-off-by: bigsheeper <[email protected]>

mergify bot removed the ci-passed label Dec 13, 2024

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

1f8c0b5

…2-opt-meta-mutex

mergify bot added the ci-passed label Dec 17, 2024

bigsheeper mentioned this pull request Dec 27, 2024

[Bug]: Test 10000 collections and 1000 partitions #37630

Open

41 tasks

bigsheeper added the 10kcp PR related to 10k collection partitions Project label Dec 30, 2024

bigsheeper added 3 commits December 31, 2024 11:33

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

4e62fba

…2-opt-meta-mutex

opt global compaction

f14e8e9

Signed-off-by: bigsheeper <[email protected]>

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

8edad7d

…2-opt-meta-mutex

mergify bot removed the ci-passed label Dec 31, 2024

mergify bot added the ci-passed label Jan 1, 2025

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

b701da2

…2-opt-meta-mutex

This was referenced Jan 1, 2025

enhance: [2.5] Reduce mutex contention in datacoord meta #38904

Merged

fix: Fix checkGeneralCapacity slowly #37976

Merged

xiaofan-luan reviewed Jan 14, 2025

View reviewed changes

internal/datacoord/meta.go Outdated Show resolved Hide resolved

bigsheeper added 2 commits January 14, 2025 19:43

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

8768861

…2-opt-meta-mutex

remove unused

faa9d9c

Signed-off-by: bigsheeper <[email protected]>

mergify bot removed the ci-passed label Jan 14, 2025

czs007 assigned xiaofan-luan Jan 14, 2025

sre-ci-robot added the lgtm label Jan 14, 2025

sre-ci-robot added the approved label Jan 14, 2025

mergify bot added the ci-passed label Jan 14, 2025

sre-ci-robot merged commit 272d95a into milvus-io:master Jan 14, 2025
19 of 20 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance: Reduce mutex contention in datacoord meta #38219

enhance: Reduce mutex contention in datacoord meta #38219

bigsheeper commented Dec 4, 2024

mergify bot commented Dec 4, 2024

mergify bot commented Dec 4, 2024

codecov bot commented Dec 4, 2024 •

edited

Loading

mergify bot commented Dec 5, 2024

mergify bot commented Dec 5, 2024

mergify bot commented Dec 13, 2024

mergify bot commented Dec 17, 2024

bigsheeper commented Dec 17, 2024

mergify bot commented Dec 31, 2024

bigsheeper commented Dec 31, 2024

mergify bot commented Dec 31, 2024

bigsheeper commented Jan 1, 2025

xiaofan-luan commented Jan 14, 2025

sre-ci-robot commented Jan 14, 2025

enhance: Reduce mutex contention in datacoord meta #38219

enhance: Reduce mutex contention in datacoord meta #38219

Conversation

bigsheeper commented Dec 4, 2024

mergify bot commented Dec 4, 2024

mergify bot commented Dec 4, 2024

codecov bot commented Dec 4, 2024 • edited Loading

Codecov Report

mergify bot commented Dec 5, 2024

mergify bot commented Dec 5, 2024

mergify bot commented Dec 13, 2024

mergify bot commented Dec 17, 2024

bigsheeper commented Dec 17, 2024

mergify bot commented Dec 31, 2024

bigsheeper commented Dec 31, 2024

mergify bot commented Dec 31, 2024

bigsheeper commented Jan 1, 2025

xiaofan-luan commented Jan 14, 2025

sre-ci-robot commented Jan 14, 2025

codecov bot commented Dec 4, 2024 •

edited

Loading