fix: [2.4] Fix slow dist handle and slow observe

[bigsheeper]

1. Provide partition&channel level indexing in the collection target.
2. Make SegmentAction not wait for distribution.
3. Remove scheduler and target manager mutex
4. Optimize logging to reduce CPU overhead.

issue: #37630

pr: #38566

[codecov]

Codecov Report

Attention: Patch coverage is 91.13924% with 21 lines in your changes missing coverage. Please review.

Project coverage is 80.40%. Comparing base (b6be11c) to head (f9fdd0b).
Report is 1 commits behind head on 2.4.

Files with missing lines	Patch %	Lines
internal/querycoordv2/task/scheduler.go	88.14%	15 Missing and 1 partial ⚠️
...rnal/querycoordv2/observers/collection_observer.go	84.00%	4 Missing ⚠️
internal/querycoordv2/meta/target.go	98.03%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##              2.4   #38906      +/-   ##
==========================================
- Coverage   80.41%   80.40%   -0.01%     
==========================================
  Files        1080     1080              
  Lines      169974   169943      -31     
==========================================
- Hits       136687   136648      -39     
- Misses      28784    28786       +2     
- Partials     4503     4509       +6     

Files with missing lines	Coverage Δ
internal/querycoordv2/dist/dist_handler.go	96.12% <100.00%> (+0.10%)	⬆️
internal/querycoordv2/meta/target_manager.go	89.06% <100.00%> (-1.31%)	⬇️
internal/querycoordv2/task/action.go	100.00% <100.00%> (+0.83%)	⬆️
internal/querycoordv2/task/task.go	97.42% <100.00%> (+0.02%)	⬆️
internal/querycoordv2/utils/util.go	86.22% <100.00%> (-2.40%)	⬇️
pkg/metrics/querycoord_metrics.go	100.00% <ø> (ø)
internal/querycoordv2/meta/target.go	92.15% <98.03%> (+2.41%)	⬆️
...rnal/querycoordv2/observers/collection_observer.go	86.52% <84.00%> (-0.39%)	⬇️
internal/querycoordv2/task/scheduler.go	88.20% <88.14%> (-1.26%)	⬇️

... and 29 files with indirect coverage changes

[bigsheeper]

rerun ut

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bigsheeper
To complete the pull request process, please ask for approval from czs007 after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment