WIP: Feature/double hbone #1429

Stevenjin8 · 2025-01-17T16:51:37Z

Initial double HBONE implementation

Right now, inner HBONE will only hold one connect tunnel. Once the inner tunnel terminates, so will the outer tunnel (but not the outer HBONE). So when ztunnel receives its first connection to a double HBONE host (E/W gateway), it will perform two TLS handshakes. Subsequent connections to the same host will perform one TLS handshake.

This behavior is not great, but if we put the inner HBONE in the connection pool, then we pin ourselves to a pod in the remote cluster since ztunnel performs connection pooling, but is not aware of the E/W gateway's routing decision.

That being said, I think this is a good place to stop and think about control plane implementation and get some feedback on how I'm approaching this.

NOTE: The TLS/certificate related code changes are just for me to tests.

Tasks:

Implement double HBONE
Implement graceful shutdowns
Fix TLS code (I don't think I can do this without more control plane info since SANs are set for services)
Implement proper inner HBONE connection pooling
Tests

Some open questions:

Do I need to make any changes to metrics?
How do should we do inner HBONE pooling? My suggestion is to have up to N inner HBONE connections per E/W or per remote cluster.
Good ways to test for race conditions in connection terminations? Right now, it seems that connections end gracefully without race conditions, but that's just on my machine.

References:

istio-testing · 2025-01-17T16:51:41Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Stevenjin8 · 2025-01-17T17:20:46Z

src/proxy/outbound.rs

@@ -107,7 +112,7 @@ impl Outbound {
                                debug!(component="outbound", dur=?start.elapsed(), "connection completed");
                            }).instrument(span);

-                            assertions::size_between_ref(1000, 1750, &serve_outbound_connection);


How did we get these numbers?

by looking a the current size and adding a small amount of buffer.

Stevenjin8 · 2025-01-17T17:56:31Z

src/proxy/pool.rs

@@ -83,6 +83,26 @@ struct ConnSpawner {

 // Does nothing but spawn new conns when asked
 impl ConnSpawner {
+    async fn new_unpooled_conn(


Anything here we can do higher up I think, but things might change if we decide to implement pooling in this PR

Yeah if we want double-hbone conns to be unpooled and thus need ~none of this surrounding machinery, then I'd be inclined to just start proxy/double-hbone.rs and use that directly, rather than complicating the purpose of this file.

(Could also just have a common HboneConnMgr trait or something too)

Stevenjin8 · 2025-01-17T19:52:12Z

src/proxy/outbound.rs

+        // This always drops ungracefully
+        // drop(conn_client);
+        // tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+        // drain_tx.send(true).unwrap();
+        // tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+        drain_tx.send(true).unwrap();
+        let _ = driver_task.await;
+        // this sleep is important, so we have a race condition somewhere
+        // tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+        res


Does anybody have any info on how to properly drop/terminate H2 connections over stream with nontrivial drops (e.g. shutting down TLS over HTTP2 CONNECT). Right now, I'm just dropping things/aborting tasks randomly until something works

Are you asking about how to cleanup after, for example, a RST_STREAM to the inner tunnel? Or something else

Kinda. I mostly mean the outer TLS stream because that's what I've looked at. It seems like if I drop conn_client before termination driver_task the TCP connection will close without sending close notifies. So yes, I'm asking if there is a way to explicitly do cleanup rather than relying on implicit drops.

I see the code changed; do you still need help figuring this out?

Im still not confident in it. It works (on my machine), but I couldn't find any docs on proper connection termination/dropping.

Stevenjin8 · 2025-01-17T19:52:39Z

src/proxy/outbound.rs

@@ -217,12 +267,12 @@ impl OutboundConnection {
        copy::copy_bidirectional(copy::TcpStreamSplitter(stream), upgraded, connection_stats).await
    }

-    async fn send_hbone_request(
+    fn create_hbone_request(


Git merge is getting confused here

keithmattix · 2025-01-17T19:13:20Z

src/config.rs

@@ -70,7 +70,7 @@ const IPV6_ENABLED: &str = "IPV6_ENABLED";

 const UNSTABLE_ENABLE_SOCKS5: &str = "UNSTABLE_ENABLE_SOCKS5";

-const DEFAULT_WORKER_THREADS: u16 = 2;
+const DEFAULT_WORKER_THREADS: u16 = 40;


I may have missed in the description, but why the change here?

I was hoping it would making debugging async rust easier (it didn't)

keithmattix · 2025-01-17T20:17:00Z

src/proxy/outbound.rs

+        // This always drops ungracefully
+        // drop(conn_client);
+        // tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+        // drain_tx.send(true).unwrap();
+        // tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+        drain_tx.send(true).unwrap();
+        let _ = driver_task.await;
+        // this sleep is important, so we have a race condition somewhere
+        // tokio::time::sleep(std::time::Duration::from_secs(1)).await;
+        res


Are you asking about how to cleanup after, for example, a RST_STREAM to the inner tunnel? Or something else

Stevenjin8 · 2025-01-21T21:17:31Z

src/proxy/outbound.rs

-
-        // Inner HBONE
-        let upgraded = TokioH2Stream::new(upgraded);
+        // TODO: dst should take a hostname? and upstream_sans currently contains E/W Gateway certs
        let inner_workload = pool::WorkloadKey {


Will reorganize later.

Stevenjin8 · 2025-01-21T21:24:37Z

src/proxy/outbound.rs

            Protocol::TCP => None,
        };
+        let (upstream_sans, final_sans) = match us.workload.protocol {


My understanding from talking to @keithmattix is that Upstream.service_sans will be repurposed to contain the identities of remote pods/waypoints, so I should change the logic of the other protocols to only use us.workload.identity instead of us.workload_and_services_san.

Yes, I think this is correct; only the double hbone codepath needs to be added/changed because there are two sans being considered: the e/w gateway SAN and the SANs of the backends. So what you have looks right to me

Stevenjin8 · 2025-01-21T21:26:22Z

src/proxy/pool.rs

@@ -511,10 +578,10 @@ impl WorkloadHBONEPool {
 #[derive(Debug, Clone)]
 // A sort of faux-client, that represents a single checked-out 'request sender' which might
 // send requests over some underlying stream using some underlying http/2 client
-struct ConnClient {
-    sender: H2ConnectClient,
+pub struct ConnClient {


istio-testing added do-not-merge/work-in-progress Block merging of a PR because it isn't ready yet. needs-rebase Indicates a PR needs to be rebased before being merged size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 17, 2025

Stevenjin8 force-pushed the feature/double-hbone branch from e27a4a8 to 40bbcd0 Compare January 17, 2025 17:00

istio-testing removed the needs-rebase Indicates a PR needs to be rebased before being merged label Jan 17, 2025

Stevenjin8 changed the title ~~Feature/double hbone~~ WIP: Feature/double hbone Jan 17, 2025

Stevenjin8 added the do-not-merge/hold Block automatic merging of a PR. label Jan 17, 2025

Stevenjin8 force-pushed the feature/double-hbone branch from 66db8ae to 99d622f Compare January 17, 2025 17:51

Stevenjin8 commented Jan 17, 2025

View reviewed changes

Stevenjin8 force-pushed the feature/double-hbone branch from 99d622f to 96bb4de Compare January 17, 2025 17:55

Stevenjin8 commented Jan 17, 2025

View reviewed changes

Stevenjin8 requested review from howardjohn and bleggett January 17, 2025 17:59

Stevenjin8 commented Jan 17, 2025

View reviewed changes

keithmattix reviewed Jan 17, 2025

View reviewed changes

Stevenjin8 added 6 commits January 21, 2025 12:07

double hbone (no inner encryption) working!!

8702bd2

hbone-ish

e340d8c

cleanup

b381e34

graceful shutdowns

2192d67

Some cleanup

dff6c92

Add some auth/tls logic back in

f1cc535

Stevenjin8 force-pushed the feature/double-hbone branch from 1ea75fb to f1cc535 Compare January 21, 2025 17:07

inline double hbone code

cab4849

Stevenjin8 commented Jan 21, 2025

View reviewed changes

Use correct(?) identities

565f41f

Stevenjin8 force-pushed the feature/double-hbone branch from a8856a4 to 565f41f Compare January 21, 2025 21:19

Stevenjin8 commented Jan 21, 2025

View reviewed changes

Stevenjin8 mentioned this pull request Jan 21, 2025

Add Stevenjin8 as networking maintainer istio/community#1548

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Feature/double hbone #1429

WIP: Feature/double hbone #1429

Stevenjin8 commented Jan 17, 2025 •

edited

Loading

istio-testing commented Jan 17, 2025

Stevenjin8 Jan 17, 2025

bleggett Jan 17, 2025

Stevenjin8 Jan 17, 2025

bleggett Jan 17, 2025 •

edited

Loading

Stevenjin8 Jan 17, 2025

keithmattix Jan 17, 2025

Stevenjin8 Jan 21, 2025

keithmattix Jan 21, 2025

Stevenjin8 Jan 21, 2025

Stevenjin8 Jan 17, 2025

keithmattix Jan 17, 2025

Stevenjin8 Jan 21, 2025

keithmattix Jan 17, 2025

Stevenjin8 Jan 21, 2025

Stevenjin8 Jan 21, 2025 •

edited

Loading

keithmattix Jan 21, 2025

Stevenjin8 Jan 21, 2025

WIP: Feature/double hbone #1429

Are you sure you want to change the base?

WIP: Feature/double hbone #1429

Conversation

Stevenjin8 commented Jan 17, 2025 • edited Loading

Initial double HBONE implementation

istio-testing commented Jan 17, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bleggett Jan 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stevenjin8 Jan 21, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Stevenjin8 commented Jan 17, 2025 •

edited

Loading

bleggett Jan 17, 2025 •

edited

Loading

Stevenjin8 Jan 21, 2025 •

edited

Loading