Ztunnel does not scale up with number of worker threads in expected way due to h2 locking #1174
Comments
You don't see the same behavior where? |
🤦 I updated the comment |
|
I would totally suspect connection pooling if you didn't rule it out. We only use HBONE for TLS right? Maybe some weird default in the hyper client/server? |
|
Ah.. I think I just disabled pooling improperly. The benchmark doesn't pool due to using different client network namespaces. I can reproduce full now and see good scaling with number of connections. At 2 workers (default): So looks pretty good and align with TCP numbers |
|
Its still pretty interesting that 1->2 yields no improvement, but we do still scale up to some extent. 16c gives 15Gb/s which is consistent with the non-pooling... |
|
Should I be looking at the most recent numbers you posted? I see improvement from 1-2 but it looks about the same as the first set you posted in the description |
Actually kind of interesting that we see inverse behavior
|
|
That feels somewhat expected: with connection pooling you get a smaller but monotonic throughput improvement with increased concurrency. Without pooling, you're more susceptible to longer running streams. I wonder if the delay is on client or server processing....I assume the concurrency increase affects both |
|
When you say "workers" you mean you're constraining the Tokio runtime to X threads? How are you doing that, exactly? Also - if ztunnel is connecting to itself, I wouldn't expect pooling to help at all, really? EDIT: Oh, I see, you did cross node too.
Yeah I wouldn't expect pooling to by-itself offer much of a boost at either end of the graph, it's more that it (among other things) mitigates starving the local node as the conn count increases (or would, if we didn't pool). |
|
Ztunnel is only connecting to itself in some of the tests, the ones that are meaningful (the ones running in k8s) is cross-node. |
|
Also note that the one place pooling does lock is when it hits the conn limit to Dest A and needs to open another conn to Dest A. We sort of have to lock there, as the alternative is we let clients use us to eagerly race connections to Dest A, and then we prune them after-the-fact - which is worse. It's an intentional ratelimit for those scenarios, where a client is spamming new conns to us. We also don't consult worker thread count when deciding to schedule new pool conns at all (we could, but do not) |
|
These tests are "open connection once" then "send a ton of information" on them. The connection establishment is out of the measurement path entirely basically. |
Oh ok, if the clients are opening A Single connection to ZT then H2 multiplexing doesn't matter at all then. If connection establishment is controlled for fully then it's got to be crypto overhead on the wire or It's possible that the h2 server code just isn't very good at handling multiple connections with the same src/dest tuple, since the http/2 spec doesn't exactly encourage that. |
howardjohn
changed the title
|
Root caused to hyperium/h2#531 (comment) a while back, forgot to update this issue |
Some testinging:
Note: I have a 16c machine.
The weird thing is I do not see the same behavior in our benchmarks (with #1171)
Things I have tried with success:
I would expect HBONE to behave like TCP: with 2 workers, we see ~2x throughput with 2 connections as 1. 2x is the theoretical best, of course, so something more like 1.5x-1.8x may be more realistic. We don't see that. We see virtually zero increase. We do see improvements at very high connection counts, though.
We need to find out what is holding up the processing
The text was updated successfully, but these errors were encountered: