Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Migrate to vise metrics #2

Merged
merged 15 commits into from
Sep 29, 2023
Merged

feat: Migrate to vise metrics #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
964029b
Rework `concurrency` crate metrics
slowli Sep 27, 2023
b801cdf
Rework `network` crate metrics
slowli Sep 27, 2023
2d7caf4
Rework `consensus` crate metrics
slowli Sep 27, 2023
ba4855c
Rework `executor` crate metrics
slowli Sep 27, 2023
d69d96a
Update workspace lockfile
slowli Sep 27, 2023
41a3f88
Update Grafana dashboard
slowli Sep 27, 2023
97ced60
Fix latency reporting for RPC metrics
slowli Sep 27, 2023
ff60661
Rename `CallResult` to `ResultLabel`
slowli Sep 27, 2023
9a66eb2
Fix manifest path for coverage job
slowli Sep 27, 2023
e136bad
Try fixing linker settings
slowli Sep 27, 2023
6acfd28
Configure `sources` section for `cargo-deny`
slowli Sep 28, 2023
e5685d3
Move TCP metrics to `network` crate
slowli Sep 28, 2023
c3bf0ae
Do not make `canceled_owned()` context-aware
slowli Sep 28, 2023
e7c72ff
Refactor `MeteredStream` construction
slowli Sep 29, 2023
0efe78a
Change TCP metrics prefix to `network_tcp`
slowli Sep 29, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/coverage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ jobs:
uses: actions-rs/[email protected]
with:
version: '0.15.0'
args: '--manifest-path /home/runner/work/zksync-bft/zksync-bft/node/Cargo.toml -- --test-threads 1'
args: '--manifest-path node/Cargo.toml -- --test-threads 1'
env:
RUST_LOG: 'network=trace,sync_blocks=trace,consensus=trace'

Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/rust.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ on:
env:
CARGO_TERM_COLOR: always
CARGO_INCREMENTAL: "0"
RUSTFLAGS: "-Dwarnings -C linker=clang -C link-arg=-fuse-ld=lld"
RUSTFLAGS: "-Dwarnings -C linker=clang -C link-arg=-fuse-ld=lld -C link-arg=-Wl,-z,nostart-stop-gc"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stupid question: What's the reason of using the ld.lld linker? It requires specifying this additional arg to circumvent an issue with the library that powers metrics registration. In the meanwhile, I'll think about potential registration alternatives.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make CI faster. We can switch to mold if that helps.

RUSTC_WRAPPER: "sccache"
SCCACHE_GHA_ENABLED: "true"
RUST_BACKTRACE: "1"
Expand Down
44 changes: 22 additions & 22 deletions grafana_dashboard.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -341,7 +341,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "avg(rate(network_rpc_message_size_sum{test_id=\"$test_id\",type=~\"req_sent|resp_sent\"}[1m])) by (type,method,submethod)",
"expr": "avg(rate(network_rpc_message_size_bytes_sum{test_id=\"$test_id\",type=~\"req_sent|resp_sent\"}[1m])) by (type,method,submethod)",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've edited this file manually, so not 100% sure if it works.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To verify that it works, you can upload this dashboard to grafana, rerun the loadtest from CI and see if this metrics is collected properly.

"legendFormat": "__auto",
"range": true,
"refId": "A"
Expand All @@ -352,7 +352,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "quantile(0.1, rate(concurrency_net_tcp__bytes_sent{test_id=\"$test_id\"}[1m]))",
"expr": "quantile(0.1, rate(network_tcp_sent_bytes{test_id=\"$test_id\"}[1m]))",
"hide": false,
"legendFormat": "total, 10th percentile",
"range": true,
Expand All @@ -364,7 +364,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "quantile(0.5, rate(concurrency_net_tcp__bytes_sent{test_id=\"$test_id\"}[1m]))",
"expr": "quantile(0.5, rate(network_tcp_sent_bytes{test_id=\"$test_id\"}[1m]))",
"hide": false,
"legendFormat": "total, 50th percentile",
"range": true,
Expand All @@ -376,7 +376,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "quantile(0.9, rate(concurrency_net_tcp__bytes_sent{test_id=\"$test_id\"}[1m]))",
"expr": "quantile(0.9, rate(network_tcp_sent_bytes{test_id=\"$test_id\"}[1m]))",
"hide": false,
"legendFormat": "total, 90th percentile",
"range": true,
Expand All @@ -388,7 +388,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "quantile(1., rate(concurrency_net_tcp__bytes_sent{test_id=\"$test_id\"}[1m]))",
"expr": "quantile(1., rate(network_tcp_sent_bytes{test_id=\"$test_id\"}[1m]))",
"hide": false,
"legendFormat": "total, max",
"range": true,
Expand Down Expand Up @@ -482,7 +482,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "rate(avg(network_rpc_latency_count{test_id=\"$test_id\",type=\"client_send_recv\"}) by (method,submethod) [1m])",
"expr": "rate(avg(network_rpc_latency_seconds_count{test_id=\"$test_id\",type=\"client_send_recv\"}) by (method,submethod) [1m])",
"legendFormat": "__auto",
"range": true,
"refId": "A"
Expand Down Expand Up @@ -569,7 +569,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "sum by(le) (rate(network_rpc_latency_bucket{test_id=\"$test_id\",method=\"consensus\",type=\"client_send_recv\",submethod=\"LeaderPrepare\"}[1m]))",
"expr": "sum by(le) (rate(network_rpc_latency_seconds_bucket{test_id=\"$test_id\",method=\"consensus\",type=\"client_send_recv\",submethod=\"LeaderPrepare\"}[1m]))",
"format": "heatmap",
"instant": false,
"interval": "",
Expand Down Expand Up @@ -659,7 +659,7 @@
},
"editorMode": "code",
"exemplar": false,
"expr": "sum by(le) (rate(network_rpc_latency_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m]))",
"expr": "sum by(le) (rate(network_rpc_latency_seconds_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m]))",
"format": "heatmap",
"instant": false,
"interval": "",
Expand Down Expand Up @@ -755,7 +755,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "rate(max(executor_lib_io__finalized_block_number{test_id=\"$test_id\"})[1m])",
"expr": "rate(max(executor_finalized_block_number{test_id=\"$test_id\"})[1m])",
"legendFormat": "__auto",
"range": true,
"refId": "A"
Expand Down Expand Up @@ -847,7 +847,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "max(executor_lib_io__finalized_block_number{test_id=\"$test_id\"})",
"expr": "max(executor_finalized_block_number{test_id=\"$test_id\"})",
"legendFormat": "__auto",
"range": true,
"refId": "max"
Expand All @@ -858,7 +858,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "min(executor_lib_io__finalized_block_number{test_id=\"$test_id\"})",
"expr": "min(executor_finalized_block_number{test_id=\"$test_id\"})",
"hide": false,
"legendFormat": "__auto",
"range": true,
Expand Down Expand Up @@ -953,7 +953,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "avg(consensus_replica__view_timeout{test_id=\"$test_id\"})",
"expr": "avg(consensus_replica_view_timeout_seconds{test_id=\"$test_id\"})",
"legendFormat": "avg",
"range": true,
"refId": "A"
Expand All @@ -964,7 +964,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "max(consensus_replica__view_timeout{test_id=\"$test_id\"})",
"expr": "max(consensus_replica_view_timeout_seconds{test_id=\"$test_id\"})",
"hide": false,
"legendFormat": "max",
"range": true,
Expand Down Expand Up @@ -1044,7 +1044,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "sum(rate(consensus_leader__commit_phase_latency_bucket{test_id=\"$test_id\"}[1m])) by (le)",
"expr": "sum(rate(consensus_leader_commit_phase_latency_seconds_bucket{test_id=\"$test_id\"}[1m])) by (le)",
"format": "heatmap",
"legendFormat": "__auto",
"range": true,
Expand Down Expand Up @@ -1139,7 +1139,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "rate(sum (consensus_leader__processing_latency_sum{test_id=\"$test_id\"}) by (type)[1m])/rate(sum (consensus_leader__processing_latency_count{test_id=\"$test_id\"}) by (type)[1m])",
"expr": "rate(sum (consensus_leader_processing_latency_seconds_sum{test_id=\"$test_id\"}) by (type)[1m])/rate(sum (consensus_leader_processing_latency_seconds_count{test_id=\"$test_id\"}) by (type)[1m])",
"legendFormat": "__auto",
"range": true,
"refId": "A"
Expand All @@ -1150,7 +1150,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "rate(sum (consensus_replica__processing_latency_sum{test_id=\"$test_id\"}) by (type)[1m])/rate(sum (consensus_replica__processing_latency_count{test_id=\"$test_id\"}) by (type)[1m])",
"expr": "rate(sum (consensus_replica_processing_latency_seconds_sum{test_id=\"$test_id\"}) by (type)[1m])/rate(sum (consensus_replica_processing_latency_seconds_count{test_id=\"$test_id\"}) by (type)[1m])",
"hide": false,
"legendFormat": "__auto",
"range": true,
Expand Down Expand Up @@ -1527,7 +1527,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "sum(rate(concurrency_net_tcp__established{test_id=\"$test_id\",direction=\"inbound\"}[1m]))",
"expr": "sum(rate(network_tcp_established{test_id=\"$test_id\",direction=\"inbound\"}[1m]))",
"legendFormat": "new inbound",
"range": true,
"refId": "A"
Expand Down Expand Up @@ -1646,7 +1646,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "avg((min(histogram_quantile(0.8,rate(network_rpc_latency_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m]))) by (instance))/(min(rate(consensus_leader__commit_phase_latency_sum{test_id=\"$test_id\"}[1m])/rate(consensus_leader__commit_phase_latency_count{test_id=\"$test_id\"}[1m])) by (instance)))",
"expr": "avg((min(histogram_quantile(0.8,rate(network_rpc_latency_seconds_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m]))) by (instance))/(min(rate(consensus_leader_commit_phase_latency_seconds_sum{test_id=\"$test_id\"}[1m])/rate(consensus_leader_commit_phase_latency_seconds_count{test_id=\"$test_id\"}[1m])) by (instance)))",
"hide": false,
"legendFormat": "actual commit phase rate",
"range": true,
Expand All @@ -1658,7 +1658,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "2*avg(histogram_quantile(0.8,rate(network_rpc_latency_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m])))*rate(max(executor_lib_io__finalized_block_number{test_id=\"$test_id\"})[1m])",
"expr": "2*avg(histogram_quantile(0.8,rate(network_rpc_latency_seconds_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m])))*rate(max(executor_finalized_block_number{test_id=\"$test_id\"})[1m])",
"hide": false,
"legendFormat": "actual block rate",
"range": true,
Expand All @@ -1670,7 +1670,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "avg(2*histogram_quantile(0.8,rate(network_rpc_latency_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m]))/(3*histogram_quantile(0.66,rate(network_rpc_latency_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m]))))",
"expr": "avg(2*histogram_quantile(0.8,rate(network_rpc_latency_seconds_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m]))/(3*histogram_quantile(0.66,rate(network_rpc_latency_seconds_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m]))))",
"hide": false,
"legendFormat": "theoretical optimum for 2/3 signatures 3 roundtrips",
"range": true,
Expand All @@ -1694,7 +1694,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "1/(\n 1/(avg(histogram_quantile(0.8,rate(network_rpc_latency_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m])))*rate(max(executor_lib_io__finalized_block_number{test_id=\"$test_id\"})[1m]))\n-\n 1/avg((min(histogram_quantile(0.8,rate(network_rpc_latency_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m]))) by (instance))/(min(rate(consensus_leader__commit_phase_latency_sum{test_id=\"$test_id\"}[1m])/rate(consensus_leader__commit_phase_latency_count{test_id=\"$test_id\"}[1m])) by (instance)))\n)",
"expr": "1/(\n 1/(avg(histogram_quantile(0.8,rate(network_rpc_latency_seconds_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m])))*rate(max(executor_finalized_block_number{test_id=\"$test_id\"})[1m]))\n-\n 1/avg((min(histogram_quantile(0.8,rate(network_rpc_latency_seconds_bucket{test_id=\"$test_id\",method=\"ping\",type=\"client_send_recv\"}[1m]))) by (instance))/(min(rate(consensus_leader_commit_phase_latency_seconds_sum{test_id=\"$test_id\"}[1m])/rate(consensus_leader_commit_phase_latency_seconds_count{test_id=\"$test_id\"}[1m])) by (instance)))\n)",
"hide": false,
"legendFormat": "actual prepare phase rate (approx)",
"range": true,
Expand Down Expand Up @@ -1789,7 +1789,7 @@
"uid": "ebcc7fad-20b5-44f0-a8ab-7ba2195ef2c0"
},
"editorMode": "code",
"expr": "(avg(rate(network_rpc_latency_sum{test_id=\"$test_id\",type=~\"client_send_recv\"}[30s])/rate(network_rpc_latency_count{test_id=\"$test_id\",type=~\"client_send_recv\",method!=\"sync_validator_addrs\"}[30s])) by (method,submethod,type))",
"expr": "(avg(rate(network_rpc_latency_seconds_sum{test_id=\"$test_id\",type=~\"client_send_recv\"}[30s])/rate(network_rpc_latency_seconds_count{test_id=\"$test_id\",type=~\"client_send_recv\",method!=\"sync_validator_addrs\"}[30s])) by (method,submethod,type))",
"legendFormat": "__auto",
"range": true,
"refId": "A"
Expand Down
Loading
Loading