How to pass external cloud provider arguments to k3s

[wanGiB]

I need to pass the following to k3s during the installation, but I'm not sure how best to go about it.

[Service]
Environment="KUBELET_EXTRA_ARGS=--cloud-provider=external"

I have the following set up for the control planes:

The first etcd server:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--kubelet-arg=cloud-provider=external --node-name=$(hostname -f) --disable=traefik --disable=servicelb --disable-cloud-controller --prefer-bundled-bin --cluster-init --node-ip=$INTERNAL_IP --tls-san $PUBLIC_IP" sh -s - server

And the following when joining additional servers

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--kubelet-arg=cloud-provider=external --node-name=$(hostname -f) --disable=traefik --disable=servicelb --disable-cloud-controller --prefer-bundled-bin --node-ip=$INTERNAL_IP --server https://$FIRST_SERVER_IP:6443 --tls-san $PUBLIC_IP" K3S_TOKEN=$FIRST_SERVER_JOIN_SECRET sh -s - server

And here is the equivalent script when joining worker nodes to the cluster:

export K3S_NODE_NAME=$(hostname -f)
export K3S_KUBELET_EXTRA_ARGS="--cloud-provider=external"
curl -sfL https://get.k3s.io | K3S_TOKEN=$FIRST_SERVER_JOIN_SECRET sh -s - agent --server https://$FIRST_SERVER_IP:6443

I just want to be sure if this way of setting the cloud provider external flag is correct in k3s?

[brandond]

K3s already starts the kubelet with --cloud-provider=external by default. What specifically are you trying to accomplish? If you want to deploy your own CCM, you should disable the one built in to K3s. This is covered in the docs: https://docs.k3s.io/networking/networking-services#deploying-an-external-cloud-controller-manager

[brandond]

Read the page I linked above to see what the k3s built in ccm does. It will work fine, but it certainly will not talk to hetzner or any other cloud infra provider if that's what you're looking for.

[wanGiB]

I've seen the page and I think I'm getting a hang of what it does.

My next question is, is there an enforced limit on the maximum number of concurrent connections to the ServiceLb that the inbuilt controller manager will host?

My entire cluster will have just a single Service of type Load Balancer which is to be provisioned by traefik using the Gateway API

[brandond]

its just iptables forwarding, so no there are no limits other than what your node can handle.

[wanGiB]

I see. Alright, unto my last question.

Now that I wish to use the inbuilt ccm, do you think the following is in order when I wish to join an agent node to the cluster?

export K3S_NODE_NAME=$(hostname -f)
export K3S_NODE_IP="$INTERNAL_IP"
export K3S_NODE_EXTERNAL_IP="$PUBLIC_IP"

curl -sfL https://get.k3s.io | K3S_TOKEN=$FIRST_SERVER_JOIN_SECRET sh -s - agent --server https://$FIRST_SERVER_IP:6443

[wanGiB]

Oh, I got this from the documentation just now:

curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent --server https://$FIRST_SERVER_IP:6443 --token $FIRST_SERVER_JOIN_SECRET --node-name=$(hostname -f) --node-ip=$INTERNAL_IP --node-external-ip=$PUBLIC_IP" sh -s -

Is this correct?