Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ICMP Type 3 messages being blocked on an already open connection. #207

Open
boatmisser opened this issue Feb 8, 2024 · 2 comments
Open

Comments

@boatmisser
Copy link

What happened:
A call to an external SOAP service kept timing out; when using curl it returned a no response error message.
Once I worked out how to enable the flow logs (instructions missing from the online configuration guide) I saw this:

{"level":"info","ts":"2024-02-07T19:07:00.273Z","logger":"ebpf-client","msg":"Flow Info:  ","Src IP":"100.64.44.16","Src Port":48530,"Dest IP":"57.20.37.65","Dest Port":8443,"Proto":"TCP","Verdict":"ACCEPT"}
{"level":"info","ts":"2024-02-07T19:07:00.513Z","logger":"ebpf-client","msg":"Flow Info:  ","Src IP":"57.20.37.65","Src Port":0,"Dest IP":"100.64.44.16","Dest Port":0,"Proto":"ICMP","Verdict":"DENY"}

Once I saw the ICMP DENY message I was confused as this is not an Ingress connection; it is on the response to the already established Egress connection over which the SOAP request was sent.
When playing around with the NetworkPolicy definition for this app I found that the ICMP is accepted when I open an Ingress rule for the IP with no protocol or port restriction.

When using wireshark to take a network packet snapshot I saw the ICMP message is a Type 3 - Fragmentation needed.
I have been assured by our Networking team that such packets should be allowed over an open connection.

So, my issue is why is this ICMP Type 3 message being treated as a new Ingress when it is not; and why these messages are blocked when it seems they should be allowed?

Attach logs

I could not get that Log Collection tool to run on our Bottlerocket node.

What you expected to happen:
The ICMP Type 3 message should be allowed to reach the pod as it is coming in on an already established Egress connection.

How to reproduce it (as minimally and precisely as possible):
Make a call to a service that sends the following ICMP Type 3 message:

Internet Control Message Protocol
    Type: 3 (Destination unreachable)
    Code: 4 (Fragmentation needed)
    Checksum: 0xc2e1 [correct]
    [Checksum Status: Good]
    Unused: 0000
    MTU of next hop: 1372

Anything else we need to know?:
Previously we used Calico and with the exact same Network Policy definition and it works; using wireshark we can see the ICMP message reaching the pod. So Calico's implementation allows ICMP Type 3 messages on an already open Egress connection.

I also have an open AWS support case about this; they were the ones who suggested raising a bug report here.

Environment:

  • Kubernetes version (use kubectl version): v1.28.5-eks-5e0fdde
  • CNI Version: v1.16.2-eksbuild.1
  • Network Policy Agent Version: v1.0.7-eksbuild.1
  • OS (e.g: cat /etc/os-release): "Amazon Linux 2"
  • Kernel (e.g. uname -a): 6.1.61
@boatmisser boatmisser added the bug Something isn't working label Feb 8, 2024
@achevuru
Copy link
Contributor

@boatmisser Right now, there is no way to explicitly allow ICMP traffic via K8S Network policy resource. List of protocols are limited to TCP, UDP and SCTP. For reference ..

For other protocols, such as ARP or ICMP, the behaviour is undefined. The same applies to allow rules: when a specific pod is allowed as ingress source or egress destination, it is undefined what happens with (for example) ICMP packets. Protocols such as ICMP may be allowed by some network plugins and denied by others.

We decided to limit it to the above list of allowed protocols in our implementation and to allow ICMP one needs to allow all protocols. But, I believe your scenario is a very valid use case and I think we can allow ICMP for already established connections. We will address this use case and will keep this thread updated once we've a potential solution.

@jdn5126
Copy link
Contributor

jdn5126 commented Feb 16, 2024

xref: #90

@sjastis sjastis added feature request and removed bug Something isn't working labels Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants