fix: [2.5] Fix slow dist handle and slow observe

[bigsheeper]

1. Provide partition&channel level indexing in the collection target.
2. Make SegmentAction not wait for distribution.
3. Remove scheduler and target manager mutex
4. Optimize logging to reduce CPU overhead.

issue: #37630

pr: #38566

[codecov]

Codecov Report

Attention: Patch coverage is 88.42975% with 28 lines in your changes missing coverage. Please review.

Project coverage is 81.03%. Comparing base (4cb47b3) to head (d62b8c5).
Report is 4 commits behind head on 2.5.

Files with missing lines	Patch %	Lines
internal/querycoordv2/task/scheduler.go	82.70%	22 Missing and 1 partial ⚠️
...rnal/querycoordv2/observers/collection_observer.go	83.33%	4 Missing ⚠️
internal/querycoordv2/meta/target.go	98.30%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##              2.5   #38905       +/-   ##
===========================================
+ Coverage   69.50%   81.03%   +11.52%     
===========================================
  Files         300     1404     +1104     
  Lines       26763   197416   +170653     
===========================================
+ Hits        18602   159972   +141370     
- Misses       8161    31815    +23654     
- Partials        0     5629     +5629     

Components	Coverage Δ
Client	78.26% <ø> (∅)
Core	69.50% <ø> (ø)
Go	83.00% <88.42%> (∅)

Files with missing lines	Coverage Δ
internal/querycoordv2/dist/dist_handler.go	96.13% <100.00%> (ø)
internal/querycoordv2/meta/target_manager.go	86.93% <100.00%> (ø)
internal/querycoordv2/task/action.go	96.66% <100.00%> (ø)
internal/querycoordv2/task/task.go	94.22% <100.00%> (ø)
internal/querycoordv2/utils/util.go	85.54% <100.00%> (ø)
pkg/metrics/querycoord_metrics.go	100.00% <ø> (ø)
internal/querycoordv2/meta/target.go	93.06% <98.30%> (ø)
...rnal/querycoordv2/observers/collection_observer.go	86.52% <83.33%> (ø)
internal/querycoordv2/task/scheduler.go	86.69% <82.70%> (ø)

... and 1095 files with indirect coverage changes

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify]

@bigsheeper E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[bigsheeper]

rerun go-sdk

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[bigsheeper]

rerun go-sdk

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[bigsheeper]

rerun go-sdk

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[bigsheeper]

rerun go-sdk

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[czs007]

rerun go-sdk

[bigsheeper]

rerun go-sdk

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[czs007]

rerun go-sdk

[mergify]

@bigsheeper go-sdk check failed, comment rerun go-sdk can trigger the job again.

[bigsheeper]

rerun go-sdk

[czs007]

/approve
/lgtm

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bigsheeper, czs007

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [czs007]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment