Label the node with the Talos configured virtual IP

[nogweii]

Feature Request

Basically, implement siderolabs/talos#7166 in Talos-CCM.

Description

It would be handy to have a label that gets moved around as the VIP changes nodes. I'd expect it to only support the built-in vip rather than any other implementation's. I'm not sure how to determine what the VIP is configured to be, though.

[sergelogvinov]

Hi, do you use Talos VIP (float-ip in control plane) for EgressGateway ?

[nogweii]

Nope. My current planned use-case is to dynamically decide which endpoint to use in my shell scripts wrapping talosctl by setting -e to whichever node has the VIP. But not to the VIP itself as per the documentation's recommendation.

[sergelogvinov]

To be honest, i did not get your idea.
talosctl get addesses can show you there the VIP (control plane float ip) currently exists.
talosctl config can contain multiple IP endpoints, and it will use one that is active (alive).

[smira]

Don't use Talos VIP for Talos API endpoint - it will break, as it depends on etcd quorum, and Talos API access you need all the time.

[ccureau]

I'm using kube-vip for my own cluster vip, and things all seem good, except for the certificate approval process. Whichever node currently is serving the VIP does not get its certificates approved automatically.

E1009 19:00:03.845312       1 controller.go:98] "CertificateSigningRequestReconciler: failed to reconcile CSR" err="providerChecks has an error: csrNodeChecks: CSR talos-cp-2 Node IP addresses don't match corresponding Node IP addresses [\"192.168.100.6\" \"192.168.100.6\" \"talos-cp-2\"], got \"192.168.100.102\"" name="csr-qml8g"
E1009 19:00:03.887570       1 controller.go:98] "CertificateSigningRequestReconciler: failed to reconcile CSR" err="providerChecks has an error: csrNodeChecks: CSR talos-cp-2 Node IP addresses don't match corresponding Node IP addresses [\"192.168.100.6\" \"192.168.100.6\" \"talos-cp-2\"], got \"192.168.100.102\"" name="csr-sw2cs"
E1009 19:00:04.202785       1 controller.go:98] "CertificateSigningRequestReconciler: failed to reconcile CSR" err="providerChecks has an error: csrNodeChecks: CSR talos-cp-2 Node IP addresses don't match corresponding Node IP addresses [\"192.168.100.6\" \"192.168.100.6\" \"talos-cp-2\"], got \"192.168.100.102\"" name="csr-w97mh"
E1009 19:00:04.252855       1 controller.go:98] "CertificateSigningRequestReconciler: failed to reconcile CSR" err="providerChecks has an error: csrNodeChecks: CSR talos-cp-2 Node IP addresses don't match corresponding Node IP addresses [\"192.168.100.6\" \"192.168.100.6\" \"talos-cp-2\"], got \"192.168.100.102\"" name="csr-xhpc2"

[sergelogvinov]

Hi, sorry for delay.

It looks like the kubelet may have announced the wrong IP and is using the kube-vip IP as the node IP. If you are using kube-vip or other floating IP solutions, you should set the node IP in the machine configuration to avoid problems:

machine:
  kubelet:
    nodeIP:
      validSubnets: ["192.168.100.6/32"]

[samos667]

IDK if it's related but in when using GUA IPV6 for talos VIP + the CCM (but without pod IPAM), node* hosting the VIP have the external IP defined to the VIP address. This happens in single-stack or dual-stack setup:

NAME     STATUS   ROLES           AGE   VERSION   INTERNAL-IP                EXTERNAL-IP               OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
loulou   Ready    control-plane   23m   v1.32.0   xxxx:xxxx:xxxx:1886:669::1   xxxx:xxxx:xxxx:1886:669::   Talos (v1.9.1)   6.12.6-talos     containerd://2.0.1

    interfaces:
      - deviceSelector:
          physical: true
        addresses:
          - xxxx:xxxx:xxxx:1886:669:0:0:1
        dhcp: false
        vip:
          ip: xxxx:xxxx:xxxx:1886:669:0:0:0

[sergelogvinov]

Hello! From what I understand, the issue seems to be that the VIP is appearing as the EXTERNAL-IP.
Could you please share the output of talosctl get addresses?

Thank you!

IDK if it's related but in when using GUA IPV6 for talos VIP + the CCM (but without pod IPAM), pod hosting the VIP have the external IP defined to the VIP address. This happens in single-stack or dual-stack setup:

NAME     STATUS   ROLES           AGE   VERSION   INTERNAL-IP                EXTERNAL-IP               OS-IMAGE         KERNEL-VERSION   CONTAINER-RUNTIME
loulou   Ready    control-plane   23m   v1.32.0   xxxx:xxxx:xxxx:1886:669::1   xxxx:xxxx:xxxx:1886:669::   Talos (v1.9.1)   6.12.6-talos     containerd://2.0.1

    interfaces:
      - deviceSelector:
          physical: true
        addresses:
          - xxxx:xxxx:xxxx:1886:669:0:0:1
        dhcp: false
        vip:
          ip: xxxx:xxxx:xxxx:1886:669:0:0:0

[samos667]

NODE             NAMESPACE   TYPE            ID                                                  VERSION   ADDRESS                                    LINK
picsou.xxxx.xx   network     AddressStatus   ens18/xxxx:xxxx:xxxx:1886:669::/128                 2         xxxx:xxxx:xxxx:1886:669::/128              ens18
picsou.xxxx.xx   network     AddressStatus   ens18/xxxx:xxxx:xxxx:1886:669::1/128                2         xxxx:xxxx:xxxx:1886:669::1/128             ens18
picsou.xxxx.xx   network     AddressStatus   ens18/fe80::30fc:e1ff:fe86:4907/64                  2         fe80::30fc:e1ff:fe86:4907/64               ens18
picsou.xxxx.xx   network     AddressStatus   lo/127.0.0.1/8                                      1         127.0.0.1/8                                lo
picsou.xxxx.xx   network     AddressStatus   lo/::1/128                                          1         ::1/128                                    lo

So I have 'give up' the VIP to a proper HA proxy. Because I can't use VIP as kube-api endpoint with kubespan following this 'limitation'.

[sergelogvinov]

Oh, I see. You have all /128 IPs. The TalosCCM doesn’t know about VIP, but I believe this can be fix in a future update. Thanks for bringing up this case.

[samos667]

BTW about IPv6 /128 usage, it's seems to break TalosCCM node IPAM. I think I will open a issue/PR when identifying what can be wrong.
But with dual or single stack IPv6, CCM failed to identifying a pod IPv6 CIDR for a node because he try to assign a nil value.

[sergelogvinov]

TalosCCM node IPAM works only with IPv6-subnet, because we need extra IPs to assign to the pods.

nil value - yeah it is a bug. Would you mind creating an issue about it, please?