fix: Fix slow dist handle and slow observe #38566

bigsheeper · 2024-12-18T12:47:46Z

Provide partition&channel level indexing in the collection target.
Make SegmentAction not wait for distribution.
Remove scheduler and target manager mutex.
Optimize logging to reduce CPU overhead.

Signed-off-by: bigsheeper <[email protected]>

bigsheeper · 2024-12-18T12:48:37Z

slow dist handling:

slow observation:

1. Provide partition-level indexing in the collection target. 2. Make SegmentAction not wait for distribution. 3. Optimize logging to reduce CPU overhead. issue: #37630 pr: #38566 --------- Signed-off-by: bigsheeper <[email protected]>

codecov · 2024-12-18T14:15:40Z

Codecov Report

Attention: Patch coverage is 91.73554% with 20 lines in your changes missing coverage. Please review.

Project coverage is 81.08%. Comparing base (3a6408b) to head (e76bcab).
Report is 13 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/querycoordv2/task/scheduler.go	88.72%	13 Missing and 2 partials ⚠️
...rnal/querycoordv2/observers/collection_observer.go	83.33%	4 Missing ⚠️
internal/querycoordv2/meta/target.go	98.30%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #38566      +/-   ##
==========================================
+ Coverage   81.07%   81.08%   +0.01%     
==========================================
  Files        1404     1404              
  Lines      198272   198238      -34     
==========================================
- Hits       160752   160749       -3     
+ Misses      31867    31848      -19     
+ Partials     5653     5641      -12

Components	Coverage Δ
Client	`79.50% <ø> (ø)`
Core	`69.65% <ø> (ø)`
Go	`83.01% <91.73%> (+0.01%)`	⬆️

Files with missing lines	Coverage Δ
internal/querycoordv2/dist/dist_handler.go	`96.13% <100.00%> (+0.10%)`	⬆️
internal/querycoordv2/meta/target_manager.go	`87.93% <100.00%> (-0.64%)`	⬇️
internal/querycoordv2/task/action.go	`96.66% <100.00%> (+2.17%)`	⬆️
internal/querycoordv2/task/task.go	`94.22% <100.00%> (+0.05%)`	⬆️
internal/querycoordv2/utils/util.go	`85.54% <100.00%> (ø)`
pkg/metrics/querycoord_metrics.go	`100.00% <ø> (ø)`
internal/querycoordv2/meta/target.go	`93.06% <98.30%> (+2.01%)`	⬆️
...rnal/querycoordv2/observers/collection_observer.go	`86.52% <83.33%> (-0.39%)`	⬇️
internal/querycoordv2/task/scheduler.go	`88.47% <88.72%> (+1.16%)`	⬆️

... and 25 files with indirect coverage changes

internal/querycoordv2/meta/target.go

Signed-off-by: bigsheeper <[email protected]>

Print observe, dist handing and schedule time. issue: #37630 pr: #38566 Signed-off-by: bigsheeper <[email protected]>

mergify · 2024-12-19T04:15:52Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

Signed-off-by: bigsheeper <[email protected]>

supplement to pr: #38566 Signed-off-by: bigsheeper <[email protected]>

mergify · 2024-12-26T09:34:22Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-26T09:36:32Z

@bigsheeper cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

…2-opt-pull-dist

Signed-off-by: bigsheeper <[email protected]>

supplement to pr: #38566 Signed-off-by: bigsheeper <[email protected]>

mergify · 2024-12-27T02:57:14Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

mergify · 2024-12-27T03:08:53Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

mergify · 2024-12-27T03:08:54Z

@bigsheeper cpp-unit-test check failed, comment rerun cpp-unit-test can trigger the job again.

mergify · 2024-12-27T10:51:17Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

bigsheeper · 2025-01-08T07:04:50Z

rerun go-sdk

bigsheeper · 2025-01-08T07:04:55Z

/run-cpu-e2e

weiliu1031 · 2025-01-09T09:14:59Z

/lgtm

internal/querynodev2/services.go

internal/querycoordv2/task/scheduler.go

Signed-off-by: bigsheeper <[email protected]>

mergify · 2025-01-14T11:01:23Z

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

czs007 · 2025-01-14T14:06:48Z

/approve

sre-ci-robot · 2025-01-14T14:06:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bigsheeper, czs007

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

bigsheeper · 2025-01-15T02:57:58Z

/run-cpu-e2e

bigsheeper · 2025-01-15T02:58:06Z

rerun ut

…2-opt-pull-dist

Signed-off-by: bigsheeper <[email protected]>

mergify · 2025-01-15T03:54:24Z

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

bigsheeper · 2025-01-15T06:22:22Z

rerun go-sdk

czs007 · 2025-01-15T12:15:53Z

/lgtm

1. Provide partition&channel level indexing in the collection target. 2. Make SegmentAction not wait for distribution. 3. Remove scheduler and target manager mutex 4. Optimize logging to reduce CPU overhead. issue: #37630 pr: #38566 --------- Signed-off-by: bigsheeper <[email protected]>

1. Provide partition&channel level indexing in the collection target. 2. Make `SegmentAction` not wait for distribution. 3. Remove scheduler and target manager mutex. 4. Optimize logging to reduce CPU overhead. issue: milvus-io#37630 --------- Signed-off-by: bigsheeper <[email protected]>

fix: Fix slow dist handle and slow observe

d10908a

Signed-off-by: bigsheeper <[email protected]>

sre-ci-robot added the size/L Denotes a PR that changes 100-499 lines. label Dec 18, 2024

sre-ci-robot requested review from aoiasd and congqixia December 18, 2024 12:48

mergify bot added dco-passed DCO check passed. kind/bug Issues or changes related a bug labels Dec 18, 2024

bigsheeper mentioned this pull request Dec 18, 2024

fix: [10kcp] Fix slow dist handle and slow observe #38567

Merged

weiliu1031 reviewed Dec 19, 2024

View reviewed changes

internal/querycoordv2/meta/target.go Show resolved Hide resolved

print observe and dist handler time

f4b2487

Signed-off-by: bigsheeper <[email protected]>

bigsheeper mentioned this pull request Dec 19, 2024

enhance: Print observe time #38575

Merged

jaime0815 pushed a commit that referenced this pull request Dec 19, 2024

enhance: Print observe time (#38575)

c3d4469

Print observe, dist handing and schedule time. issue: #37630 pr: #38566 Signed-off-by: bigsheeper <[email protected]>

bigsheeper added 2 commits December 26, 2024 16:44

fix conflicts

86abc77

Signed-off-by: bigsheeper <[email protected]>

opt release

609039d

Signed-off-by: bigsheeper <[email protected]>

bigsheeper mentioned this pull request Dec 26, 2024

fix: [10kcp] Fix slow preprocess in qc scheduler #38784

Merged

czs007 pushed a commit that referenced this pull request Dec 26, 2024

fix: [10kcp] Fix slow preprocess in qc scheduler (#38784)

05f50b1

supplement to pr: #38566 Signed-off-by: bigsheeper <[email protected]>

bigsheeper added 2 commits December 27, 2024 10:19

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

f0e0aae

…2-opt-pull-dist

enhance: Add channel index in target, optimize logs

8802f4e

Signed-off-by: bigsheeper <[email protected]>

bigsheeper mentioned this pull request Dec 27, 2024

enhance: [10kcp] Add channel index in target, optimize logs #38804

Merged

czs007 pushed a commit that referenced this pull request Dec 27, 2024

enhance: [10kcp] Add channel index in target, optimize logs (#38804)

15b9f51

supplement to pr: #38566 Signed-off-by: bigsheeper <[email protected]>

bigsheeper mentioned this pull request Dec 27, 2024

[Bug]: Test 10000 collections and 1000 partitions #37630

Open

41 tasks

mergify bot added the ci-passed label Jan 8, 2025

sre-ci-robot assigned weiliu1031 Jan 9, 2025

sre-ci-robot added the lgtm label Jan 9, 2025

czs007 reviewed Jan 9, 2025

View reviewed changes

internal/querynodev2/services.go Outdated Show resolved Hide resolved

czs007 reviewed Jan 9, 2025

View reviewed changes

internal/querycoordv2/task/scheduler.go Show resolved Hide resolved

bigsheeper added 2 commits January 14, 2025 16:20

fix conflicts

086bbf4

Signed-off-by: bigsheeper <[email protected]>

update

9d4ee7f

Signed-off-by: bigsheeper <[email protected]>

sre-ci-robot removed the lgtm label Jan 14, 2025

mergify bot removed the ci-passed label Jan 14, 2025

czs007 assigned czs007 and unassigned weiliu1031 Jan 14, 2025

sre-ci-robot added the approved label Jan 14, 2025

bigsheeper added 2 commits January 15, 2025 11:13

Merge branch 'master' of https://github.com/milvus-io/milvus into 241…

16cac14

…2-opt-pull-dist

fix data rce

e76bcab

Signed-off-by: bigsheeper <[email protected]>

mergify bot added the ci-passed label Jan 15, 2025

sre-ci-robot added the lgtm label Jan 15, 2025

sre-ci-robot merged commit 657550c into milvus-io:master Jan 15, 2025
20 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: Fix slow dist handle and slow observe #38566

fix: Fix slow dist handle and slow observe #38566

bigsheeper commented Dec 18, 2024 •

edited

Loading

bigsheeper commented Dec 18, 2024

codecov bot commented Dec 18, 2024 •

edited

Loading

mergify bot commented Dec 19, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 27, 2024

mergify bot commented Dec 27, 2024

mergify bot commented Dec 27, 2024

mergify bot commented Dec 27, 2024

bigsheeper commented Jan 8, 2025

bigsheeper commented Jan 8, 2025

weiliu1031 commented Jan 9, 2025

mergify bot commented Jan 14, 2025

czs007 commented Jan 14, 2025

sre-ci-robot commented Jan 14, 2025

bigsheeper commented Jan 15, 2025

bigsheeper commented Jan 15, 2025

mergify bot commented Jan 15, 2025

bigsheeper commented Jan 15, 2025

czs007 commented Jan 15, 2025

fix: Fix slow dist handle and slow observe #38566

fix: Fix slow dist handle and slow observe #38566

Conversation

bigsheeper commented Dec 18, 2024 • edited Loading

bigsheeper commented Dec 18, 2024

codecov bot commented Dec 18, 2024 • edited Loading

Codecov Report

mergify bot commented Dec 19, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 26, 2024

mergify bot commented Dec 27, 2024

mergify bot commented Dec 27, 2024

mergify bot commented Dec 27, 2024

mergify bot commented Dec 27, 2024

bigsheeper commented Jan 8, 2025

bigsheeper commented Jan 8, 2025

weiliu1031 commented Jan 9, 2025

mergify bot commented Jan 14, 2025

czs007 commented Jan 14, 2025

sre-ci-robot commented Jan 14, 2025

bigsheeper commented Jan 15, 2025

bigsheeper commented Jan 15, 2025

mergify bot commented Jan 15, 2025

bigsheeper commented Jan 15, 2025

czs007 commented Jan 15, 2025

bigsheeper commented Dec 18, 2024 •

edited

Loading

codecov bot commented Dec 18, 2024 •

edited

Loading