enhance: Support tracing for async watch channels

[tinswzy]

issue: #35917
Support tracing for async watch channels

[sre-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tinswzy
To complete the pull request process, please assign congqixia after the PR has been reviewed.
You can assign the PR to them by writing /assign @congqixia in a comment when ready.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mergify]

@tinswzy go-sdk check failed, comment rerun go-sdk can trigger the job again.

[codecov]

Codecov Report

Attention: Patch coverage is 93.15068% with 20 lines in your changes missing coverage. Please review.

Project coverage is 81.16%. Comparing base (702347b) to head (e165d9c).
Report is 14 commits behind head on master.

Files with missing lines	Patch %	Lines
internal/datacoord/channel_manager.go	87.50%	5 Missing and 2 partials ⚠️
internal/datacoord/channel_store.go	83.33%	2 Missing and 1 partial ⚠️
internal/flushcommon/pipeline/data_sync_service.go	76.92%	2 Missing and 1 partial ⚠️
internal/datacoord/channel.go	83.33%	1 Missing ⚠️
internal/datacoord/services.go	91.66%	0 Missing and 1 partial ⚠️
internal/flushcommon/broker/datacoord.go	66.66%	1 Missing ⚠️
internal/flushcommon/syncmgr/meta_writer.go	0.00%	1 Missing ⚠️
internal/querycoordv2/services.go	75.00%	1 Missing ⚠️
pkg/config/etcd_source.go	0.00%	1 Missing ⚠️
pkg/mq/msgstream/mq_msgstream.go	96.55%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #38444      +/-   ##
==========================================
+ Coverage   81.14%   81.16%   +0.01%     
==========================================
  Files        1395     1395              
  Lines      197488   197659     +171     
==========================================
+ Hits       160256   160434     +178     
+ Misses      31605    31600       -5     
+ Partials     5627     5625       -2     

Components	Coverage Δ
Client	79.53% <ø> (ø)
Core	69.70% <ø> (+0.02%)	⬆️
Go	83.09% <93.15%> (+0.01%)	⬆️

Files with missing lines	Coverage Δ
internal/datacoord/cluster.go	97.22% <100.00%> (ø)
internal/datacoord/meta.go	89.68% <100.00%> (+0.01%)	⬆️
internal/datacoord/metrics_info.go	84.69% <100.00%> (+0.97%)	⬆️
internal/datacoord/policy.go	85.71% <100.00%> (ø)
internal/datacoord/sync_segments_scheduler.go	100.00% <100.00%> (ø)
internal/datacoord/util.go	98.68% <100.00%> (ø)
internal/datanode/channel/channel_manager.go	89.54% <100.00%> (+0.36%)	⬆️
internal/datanode/services.go	85.01% <100.00%> (ø)
...hcommon/pipeline/flow_graph_dmstream_input_node.go	94.11% <100.00%> (ø)
internal/kv/etcd/etcd_kv.go	80.68% <100.00%> (+0.38%)	⬆️
... and 36 more

... and 35 files with indirect coverage changes

[mergify]

@tinswzy go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify]

@tinswzy E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@tinswzy go-sdk check failed, comment rerun go-sdk can trigger the job again.

[mergify]

@tinswzy go-sdk check failed, comment rerun go-sdk can trigger the job again.

[tinswzy]

rerun go-sdk

[mergify]

@tinswzy E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[mergify]

@tinswzy go-sdk check failed, comment rerun go-sdk can trigger the job again.

[tinswzy]

rerun go-sdk

[mergify]

@tinswzy E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[tinswzy]

/run-cpu-e2e

[mergify]

@tinswzy E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[tinswzy]

/run-cpu-e2e

[mergify]

@tinswzy E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.

[tinswzy]

/run-cpu-e2e

[mergify]

@tinswzy E2e jenkins job failed, comment /run-cpu-e2e can trigger the job again.