Webhook issues: ipaddresspoolvalidationwebhook.metallb.io

[gastondc]

MetalLB Version

0.14.8

Deployment method

Charts

Main CNI

Flannel

Kubernetes Version

1.30.3

Cluster Distribution

Talos v1.8.1

Describe the bug

When creating the IP address pool with the L2 configuration, I encountered an issue. Below is the configuration I used:

I followed the steps to install Talos Linux using the GOVC method as outlined in the documentation:

Talos Linux Installation Guide

After K8s ready, only install MelalB with HELM:

kubectl apply -f labels.yaml

helm install metallb metallb/metallb -n metallb-system

kubectl apply -f pools.yaml -n metallb-system

pool.yaml:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.130.191-192.168.130.193

---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: example
  namespace: metallb-system
spec:
  ipAddressPools:
  - first-pool

kubectl apply -f pools.yaml -n metallb-system

Error from server (InternalError): error when creating "pools.yaml": Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://metallb-webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": context deadline exceeded
Error from server (InternalError): error when creating "pools.yaml": Internal error occurred: failed calling webhook "l2advertisementvalidationwebhook.metallb.io": failed to call webhook: Post "https://metallb-webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-l2advertisement?timeout=10s": context deadline exceeded

K8s events:

kube-apiserver-talos-s7k-is6 kube-apiserver W1016 19:31:55.657193       1 dispatcher.go:217] Failed calling webhook, failing closed l2advertisementvalidationwebhook.metallb.io: failed calling webhook "l2advertisementvalidationwebhook.metallb.io": failed to call webhook: Post "https://metallb-webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-l2advertisement?timeout=10s": context deadline exceeded
kube-apiserver-talos-s7k-is6 kube-apiserver I1016 19:31:55.657460       1 trace.go:236] Trace[1674680560]: "Create" accept:application/json,audit-id:367a54ff-4dfc-4773-9fe5-598c4c9104d0,client:10.212.134.9,api-group:metallb.io,api-version:v1beta1,name:,subresource:,namespace:metallb-system,protocol:HTTP/2.0,resource:l2advertisements,scope:resource,url:/apis/metallb.io/v1beta1/namespaces/metallb-system/l2advertisements,user-agent:kubectl/v1.30.3 (linux/amd64) kubernetes/6fc0a69,verb:POST (16-Oct-2024 19:31:45.655) (total time: 10001ms):
kube-apiserver-talos-s7k-is6 kube-apiserver Trace[1674680560]: ["Call validating webhook" configuration:metallb-webhook-configuration,webhook:l2advertisementvalidationwebhook.metallb.io,resource:metallb.io/v1beta1, Resource=l2advertisements,subresource:,operation:CREATE,UID:90aa508f-e12d-4893-96ca-bbada5d2d3da 10001ms (19:31:45.656)]
kube-apiserver-talos-s7k-is6 kube-apiserver Trace[1674680560]: [10.001793067s] [10.001793067s] END

We were able to resolve the issue thanks to this previous issue. In our case, we resolved it by adding the nodeName.

#1597

Additionally, we reproduced this error on a Proxmox environment with other images but the same version, and we were unable to replicate the error.

To Reproduce

Install Talos Linux using the GOVC method.
Install Metallb with Helm
Create the IP address pool and L2 advertisement as shown above.
Observe the error.

Expected Behavior

The IP address pool should be created without issues.

Additional Context

In Proxmox, with the same version but using a different image compatible with QEMU, we did not encounter any issues.

I've read and agree with the following

• I've checked all open and closed issues and my request is not there.
• I've checked all open and closed pull requests and my request is not there.

I've read and agree with the following

• I've checked all open and closed issues and my issue is not there.
• This bug is reproducible when deploying MetalLB from the main branch
• I have read the troubleshooting guide and I am still not able to make it work
• I checked the logs and MetalLB is not discarding the configuration as not valid
• I enabled the debug logs, collected the information required from the cluster using the collect script and will attach them to the issue
• I will provide the definition of my service and the related endpoint slices and attach them to this issue

[cyclinder]

Error from server (InternalError): error when creating "pools.yaml": Internal error occurred: failed calling webhook "ipaddresspoolvalidationwebhook.metallb.io": failed to call webhook: Post "https://metallb-webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-ipaddresspool?timeout=10s": context deadline exceeded
Error from server (InternalError): error when creating "pools.yaml": Internal error occurred: failed calling webhook "l2advertisementvalidationwebhook.metallb.io": failed to call webhook: Post "https://metallb-webhook-service.metallb-system.svc:443/validate-metallb-io-v1beta1-l2advertisement?timeout=10s": context deadline exceeded

context deadline exceeded means that the api-server failed to connect your metallb controller with the timeout error, do you check the state of the controller pods?