Fix Blackhole implementation for e2e tests #17985

henrybear327 · 2024-05-12T11:29:35Z

Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow.

For each proxy, we remove the reverse proxy and keep only the forward proxy.

PoC-level code quality for now...

k8s-ci-robot · 2024-05-12T11:29:44Z

Hi @henrybear327. Thanks for your PR.

I'm waiting for a etcd-io member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

henrybear327 · 2024-05-12T12:49:48Z

/cc @fuweid @serathius @siyuanfoundation @ahrtr

pkg/proxy/server.go

Signed-off-by: Siyuan Zhang <[email protected]>

Shorten the wait time for the open connection to expire to 5s Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy ----- B's reverse proxy - B ^ newly introduced ^ in the original codebase (renamed) ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's forward proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [Issues] - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <[email protected]>

Signed-off-by: Chun-Hung Tseng <[email protected]>

Signed-off-by: Chun-Hung Tseng <[email protected]> Co-authored-by: Iván Valdés Castillo <[email protected]>

Signed-off-by: Chun-Hung Tseng <[email protected]>

Very ugly PoC for now Reference: - etcd-io#17985 (comment)

Very ugly PoC for now Reference: - etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <[email protected]>

Due to forward proxy's need to parse the CONNECT header, which is a L7 layer feature, thus we are splitting the proxy into 2 types, for better maintainability. Reference: - etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on Fu Wei's idea discussed in the [issue](etcd-io#17737), we employ the blocking on L7 but without using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. The modified architecture will look something like this: ``` A -- A's forward proxy ----- B's reverse proxy - B ^ newly introduced ^ in the original codebase (renamed) ``` By adding this forward proxy, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [Issues] - I run into `context deadline exceeded` sometimes ``` etcd_mix_versions_test.go:175: Error Trace: /Users/henrybear327/go/src/etcd/tests/e2e/etcd_mix_versions_test.go:175 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:75 /Users/henrybear327/go/src/etcd/tests/e2e/blackhole_test.go:31 Error: Received unexpected error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE Test: TestBlackholeByMockingPartitionLeader Messages: failed to put "key-0", error: [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0] match not found. Set EXPECT_DEBUG for more info Errs: [unexpected exit code [1] after running [/Users/henrybear327/go/src/etcd/bin/etcdctl --endpoints=http://localhost:20006 put key-0 value-0]], last lines: {"level":"warn","ts":"2024-05-05T23:02:36.809726+0800","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:65","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0x140001ee960/localhost:20006","method":"/etcdserverpb.KV/Put","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: context deadline exceeded (expected "OK", got []). Try EXPECT_DEBUG=TRUE ``` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 Signed-off-by: Chun-Hung Tseng <[email protected]> It's verified that the blocking of traffic is complete, compared to previous solutions [2][3]. [Implementation] The main subtasks are - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement forward proxy by extending the existing proxy server code - implement enable/disable of the forward proxy in the e2e test The result is that for every peer, we will have the arch like this ``` A -- A's forward proxy (connections initiated from A will be forwarded from this proxy) | ^ covers case (b) | --- A's (currently existing) reverse proxy (advertised to other peers where the connection should come in from) ^ covers case (a) ``` Implement determineHTTPTransportProxyParsingFunc Signed-off-by: Chun-Hung Tseng <[email protected]> Tidy up net.SplitHostPort Signed-off-by: Chun-Hung Tseng <[email protected]> Remove reverse proxy and keep only forward proxy for all peers There is no need of reverse proxy after the introduction of forward proxy, as the forward proxy holds the information of the destination, we can filter connection traffic from there, and this can reduce 1 hop when the proxy is turned on. Clearly separate L4 and L7 connection handling logic Due to forward proxy's need to parse the CONNECT header, which is a L7 layer feature, thus we are splitting the proxy into 2 types, for better maintainability. Use a custom Listener wrapper to implement existing L4 features but switch to L7 proxy design If the first request is not a CONNECT, we will need to receive perfectly parsable HTTP packets, not just random byte streams, as Serve() will leverage ReadRequest() under the hood. Transition to forward proxy - Remove To and From fields, as this is only used in reverse proxy - Add Listen field, to indicate which URL that the forward is listening for requests - Update the tests as required TODO - Figure out why DelayTx() isn't working anymore - Doesn't support unix socket (L7 http transport proxy only support http/https/and socks5) - It's L7 so we need to send perfectly crafted http request - Bug fix: Doing var httpTransportProxyParsingFunc = determineHTTPTransportProxyParsingFunc() will initialize the function once only (env var read on program init) If the traffic to the destination is HTTPS, a CONNECT request will be sent first (containing the intended destination HOST). If the traffic to the destination is HTTP, no CONNECT request will be sent first, but normal HTTP request, with the HOST set to the final destination, will be sent. Reference: - etcd-io#17938 (comment) - etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <[email protected]> Signed-off-by: Siyuan Zhang <[email protected]> Co-authored-by: Iván Valdés Castillo <[email protected]>

Based on Fu Wei's idea discussed in the issue [1], we employ the network traffic blocking on L7, using a forward proxy, without the need of using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing reverse proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. We will remove the current use of the reverse proxy, as the forward proxy holds the information of the destination, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. The modified architecture will look something like this: ``` A --- A's forward proxy ----- B ^ newly introduced ``` It's verified that the blocking of traffic is complete, compared to previous solutions attempted in PRs [2][3]. [Implementation] The main subtasks are - transition to L7 forward proxy - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement L7 forward proxy by drastically reducing the the existing proxy server code and design to use blocking traffic - implement enable/disable of the L7 forward proxy in the e2e test Known limitations - Doesn't support unix socket (L7 http transport proxy only support http/https/and socks5) - It's L7 so we need to send perfectly crafted http request -Doesn’t support reordering, dropping, etc. packets on-the-fly [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] etcd-io#17938 (comment) [5] etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <[email protected]> Signed-off-by: Siyuan Zhang <[email protected]> Co-authored-by: Iván Valdés Castillo <[email protected]>

Based on Fu Wei's idea discussed in the issue [1], we employ the network traffic blocking on L7, using a forward proxy, without the need of using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing reverse proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. We will remove the current use of the reverse proxy, as the forward proxy holds the information of the destination, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. The modified architecture will look something like this: ``` A --- A's forward proxy ----- B ^ newly introduced ``` It's verified that the blocking of traffic is complete, compared to previous solutions attempted in PRs [2][3]. [Implementation] The main subtasks are - transition to L7 forward proxy - set up an environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement L7 forward proxy by drastically reducing the the existing proxy server code and design to use blocking traffic - implement enable/disable of the L7 forward proxy in the e2e test Known limitations are - Doesn't support unix socket (L7 http transport proxy only support http/https/and socks5) - It's L7 so we need to send perfectly crafted http request -Doesn’t support reordering, dropping, etc. packets on-the-fly [Testing] - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && \ go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] etcd-io#17938 (comment) [5] etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <[email protected]> Signed-off-by: Siyuan Zhang <[email protected]> Co-authored-by: Iván Valdés Castillo <[email protected]>

Based on Fu Wei's idea discussed in the issue [1], we employ the network traffic blocking on L7, using a forward proxy, without the need of using external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing reverse proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce an forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. We will remove the current use of the reverse proxy, as the forward proxy holds the information of the destination, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. The modified architecture will look something like this: ``` A --- A's forward proxy ----- B ^ newly introduced ``` It's verified that the blocking of traffic is complete, compared to previous solutions attempted in PRs [2][3]. [Implementation] The main subtasks are - redesigned as L7 forward proxy - unix socket support is dropped: e2e test supports unix sockets for peer communication, but only several e2e test cases use unix socket as majority of e2e test cases use http/https - introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement L7 forward proxy by drastically reducing the the existing proxy server code and design to use blocking traffic Known limitations are - Doesn't support unix socket (L7 http transport proxy only support http/https/and socks5) - It's L7 so we need to send perfectly crafted http request -Doesn’t support reordering, dropping, etc. packets on-the-fly [Testing] - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] etcd-io#17938 (comment) [5] etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <[email protected]> Signed-off-by: Siyuan Zhang <[email protected]> Co-authored-by: Iván Valdés Castillo <[email protected]> Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on Fu Wei's idea discussed in the issue [1], we employ the network traffic blocking on L7, using a forward proxy, without the need to use external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing reverse proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce a forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. We will remove the current use of the reverse proxy, as the forward proxy holds the information of the destination, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. The modified architecture will look something like this: ``` A --- A's forward proxy ----- B ^ newly introduced ``` It's verified that the blocking of traffic is complete, compared to previous solutions attempted in PRs [2][3]. [Implementation] The main subtasks are - redesigned as an L7 forward proxy - Unix socket support is dropped: e2e test supports unix sockets for peer communication, but only several e2e test cases use Unix sockets as majority of e2e test cases use HTTP/HTTPS - introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement L7 forward proxy by drastically reducing the existing proxy server code and design to use blocking traffic Known limitations are - Doesn't support unix socket (L7 HTTP transport proxy only supports HTTP/HTTPS/and socks5) - It's L7 so we need to send a perfectly crafted HTTP request -Doesn’t support reordering, dropping, etc. packets on-the-fly [Testing] - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] etcd-io#17938 (comment) [5] etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <[email protected]> Signed-off-by: Siyuan Zhang <[email protected]> Co-authored-by: Iván Valdés Castillo <[email protected]> Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on Fu Wei's idea discussed in the issue [1], we employ the network traffic blocking on L7, using a forward proxy, without the need to use external tools. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing reverse proxy is insufficient, since only scenario (a) is handled, and scenario (b) is not blocked at all. [Proposed solution] We introduce a forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. We will remove the current use of the reverse proxy, as the forward proxy holds the information of the destination, we can block all in and out traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. The modified architecture will look something like this: ``` A --- A's forward proxy ----- B ^ newly introduced ``` It's verified that the blocking of traffic is complete, compared to previous solutions attempted in PRs [2][3]. [Implementation] The main subtasks are - redesigned as an L7 forward proxy - Unix socket support is dropped: e2e test supports unix sockets for peer communication, but only several e2e test cases use Unix sockets as majority of e2e test cases use HTTP/HTTPS - introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` - implement L7 forward proxy by drastically reducing the existing proxy server code and design to use blocking traffic Known limitations are - Doesn't support unix socket (L7 HTTP transport proxy only supports HTTP/HTTPS/and socks5) - It's L7 so we need to send a perfectly crafted HTTP request -Doesn’t support reordering, dropping, etc. packets on-the-fly [Testing] - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] etcd-io#17938 (comment) [5] etcd-io#17985 (comment) Signed-off-by: Siyuan Zhang <[email protected]> Co-authored-by: Iván Valdés Castillo <[email protected]> Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on the ideas discussed in the issues [1] and PRs [2][3][6], we switch from using a L4 reverse proxy to a L7 forward proxy to properly block peer network traffic, without the need to use external tools. The design aims to implement only the minimal required features that satisfies blocking incoming and outgoing peer traffic. Complicated features such as packet reordering, packet delivery delay, etc. to a future container-based solution. [Background] A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing reverse proxy is insufficient, since only scenario (a) is handled, and network traffic in scenario (b) is not blocked at all. [Proposed solution] We introduce a L7 forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. We will remove the current use of the L4 reverse proxy, as the L7 forward proxy holds the information of the destination, we can block all incoming and outgoing traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. The modified architecture will look something like this: ``` A --- A's forward proxy --- B ^ newly introduced ``` [Implementation] The main subtasks are - redesigned as an L7 forward proxy - introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` to bypass the limitation of with http.ProxyFromEnvironment - implement a L7 forward proxy Known limitations are - Doesn't support unix socket, as L7 HTTP transport proxy only support HTTP/HTTPS/and socks5 -> although e2e test supports unix sockets for peer communication, but only few of the e2e test cases use unix sockets as majority of e2e test cases use HTTP/HTTPS. It's been discussed and decided that without the unix socket support is ok for now. - it's L7 so we need to send a perfectly crafted HTTP request - doesn’t support reordering, dropping, manipulating packets on-the-fly [Testing] - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast` [References] [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] etcd-io#17938 (comment) [5] etcd-io#17985 (comment) [6] etcd-io#17938 Signed-off-by: Siyuan Zhang <[email protected]> Signed-off-by: Iván Valdés Castillo <[email protected]> Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on the ideas discussed in the issues [1] and PRs [2][3][6], we switch from using an L4 reverse proxy to an L7 forward proxy to properly block peer network traffic, without the need to use external tools. The design aims to implement only the minimal required features that satisfy blocking incoming and outgoing peer traffic. Complicated features such as packet reordering, packet delivery delay, etc. to a future container-based solution. A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing reverse proxy is insufficient, since only scenario (a) is handled, and network traffic in scenario (b) is not blocked at all. We introduce an L7 forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. We will remove the current use of the L4 reverse proxy, as the L7 forward proxy holds the information of the destination, we can block all incoming and outgoing traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. The modified architecture will look something like this: ``` A --- A's forward proxy --- B ^ newly introduced ``` The main subtasks are - redesigned as an L7 forward proxy - introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` to bypass the limitation of `http.ProxyFromEnvironment` - implement an L7 forward proxy Known limitations are - Doesn't support unix socket, as L7 HTTP transport proxy only supports HTTP/HTTPS/and socks5 -> Although the e2e test supports unix sockets for peer communication, only a few of the e2e test cases use unix sockets as the majority of e2e test cases use HTTP/HTTPS. It's been discussed and decided that it is ok for now without the unix socket support. - it's L7 so we need to send a perfectly crafted HTTP request - doesn’t support reordering, dropping, or manipulating packets on the fly - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast` [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] etcd-io#17938 (comment) [5] etcd-io#17985 (comment) [6] etcd-io#17938 Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow. Signed-off-by: Siyuan Zhang <[email protected]> Signed-off-by: Iván Valdés Castillo <[email protected]> Signed-off-by: Chun-Hung Tseng <[email protected]>

Based on the ideas discussed in the issues [1] and PRs [2][3][6], it's been decided to switch from using an L4 reverse proxy to an L7 forward proxy to properly block peer network traffic, without the need to use external tools. The design aims to implement only the minimal required features that satisfy blocking incoming and outgoing peer traffic. Complicated features such as packet reordering, packet delivery delay, etc. to a future container-based solution. A peer will (a) receive traffic from its peers (b) initiate connections to its peers (via stream and pipeline). Thus, the current mechanism of only blocking peer traffic via the peer's existing reverse proxy is insufficient, since only scenario (a) is handled, and network traffic in scenario (b) is not blocked at all. We introduce an L7 forward proxy for each peer, which will be proxying all the connections initiated from a peer to its peers. We will remove the current use of the L4 reverse proxy, as the L7 forward proxy holds the information of the destination, we can block all incoming and outgoing traffic that is initiated from a peer to others, without having to resort to external tools, such as iptables. The modified architecture will look something like this: ``` A --- A's forward proxy --- B ^ newly introduced ``` The main subtasks are - redesigned as an L7 forward proxy - introduce a new environment variable `E2E_TEST_FORWARD_PROXY_IP` to bypass the limitation of `http.ProxyFromEnvironment` - implement an L7 forward proxy Known limitations are - Doesn't support unix socket, as L7 HTTP transport proxy only supports HTTP/HTTPS/and socks5 -> Although the e2e test supports unix sockets for peer communication, only a few of the e2e test cases use unix sockets as the majority of e2e test cases use HTTP/HTTPS. It's been discussed and decided that it is ok for now without the unix socket support. - it's L7 so we need to send a perfectly crafted HTTP request - doesn’t support reordering, dropping, or manipulating packets on the fly - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionLeader$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `make gofail-enable && make build && make gofail-disable && go test -timeout 60s -run ^TestBlackholeByMockingPartitionFollower$ go.etcd.io/etcd/tests/v3/e2e -v -count=1` - `go test -timeout 30s -run ^TestServer_ go.etcd.io/etcd/pkg/v3/proxy -v -failfast` [1] issue etcd-io#17737 [2] PR (V1) https://github.com/henrybear327/etcd/tree/fix/e2e_blackhole [3] PR (V2) etcd-io#17891 [4] etcd-io#17938 (comment) [5] etcd-io#17985 (comment) [6] etcd-io#17938 Please read https://github.com/etcd-io/etcd/blob/main/CONTRIBUTING.md#contribution-flow. Signed-off-by: Siyuan Zhang <[email protected]> Signed-off-by: Iván Valdés Castillo <[email protected]> Signed-off-by: Chun-Hung Tseng <[email protected]>

henrybear327 marked this pull request as draft May 12, 2024 11:29

k8s-ci-robot added the do-not-merge/work-in-progress label May 12, 2024

k8s-ci-robot added the needs-ok-to-test label May 12, 2024

henrybear327 mentioned this pull request May 12, 2024

[RFC] Fix Blackhole implementation for e2e tests #17950

Closed

henrybear327 force-pushed the fix/e2e_blackhole_forward_proxy_per_peer_review branch from 0e061d2 to a714d40 Compare May 12, 2024 11:30

henrybear327 mentioned this pull request May 12, 2024

Fix Blackhole implementation for e2e tests #17938

Closed

henrybear327 force-pushed the fix/e2e_blackhole_forward_proxy_per_peer_review branch 2 times, most recently from 1f1cd51 to 01eab3b Compare May 12, 2024 12:24

k8s-ci-robot requested review from ahrtr, fuweid, serathius and siyuanfoundation May 12, 2024 12:49

fuweid reviewed May 13, 2024

View reviewed changes

pkg/proxy/server.go Show resolved Hide resolved

siyuanfoundation and others added 7 commits May 16, 2024 00:14

blackhole test

9fbddd4

Signed-off-by: Siyuan Zhang <[email protected]>

Add a 5s delay after unblackholing (for the catch-up to work)

2c26aca

Shorten the wait time for the open connection to expire to 5s Signed-off-by: Chun-Hung Tseng <[email protected]>

Implement determineHTTPTransportProxyParsingFunc

5707766

Signed-off-by: Chun-Hung Tseng <[email protected]>

Tidy up net.SplitHostPort

938bc60

Signed-off-by: Chun-Hung Tseng <[email protected]> Co-authored-by: Iván Valdés Castillo <[email protected]>

Attempt to fix "No data available for forward proxy to work on"

e8fd61d

Signed-off-by: Chun-Hung Tseng <[email protected]>

Remove reverse proxy and keep only forward proxy for all peers

51f3475

Signed-off-by: Chun-Hung Tseng <[email protected]>

henrybear327 force-pushed the fix/e2e_blackhole_forward_proxy_per_peer_review branch from 01eab3b to 51f3475 Compare May 15, 2024 22:16

k8s-ci-robot added area/robustness-testing area/testing labels May 15, 2024

henrybear327 closed this May 15, 2024

henrybear327 added a commit to henrybear327/etcd that referenced this pull request May 15, 2024

Initial porting attempt using fuweid's idea

9bfc097

Very ugly PoC for now Reference: - etcd-io#17985 (comment)

henrybear327 added a commit to henrybear327/etcd that referenced this pull request May 15, 2024

Initial porting attempt using fuweid's idea

c37801f

Very ugly PoC for now Reference: - etcd-io#17985 (comment) Signed-off-by: Chun-Hung Tseng <[email protected]>

henrybear327 mentioned this pull request Sep 25, 2024

Fix Blackhole implementation for e2e tests #18641

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Blackhole implementation for e2e tests #17985

Fix Blackhole implementation for e2e tests #17985

henrybear327 commented May 12, 2024 •

edited

Loading

k8s-ci-robot commented May 12, 2024

henrybear327 commented May 12, 2024

Fix Blackhole implementation for e2e tests #17985

Fix Blackhole implementation for e2e tests #17985

Conversation

henrybear327 commented May 12, 2024 • edited Loading

k8s-ci-robot commented May 12, 2024

henrybear327 commented May 12, 2024

henrybear327 commented May 12, 2024 •

edited

Loading