How would I setup a cluster including a bare metal server?

[if-mar]

Hi, I'm currently taking a closer look at hetzner-k3s and have already created and used a few test clusters. For productive use, however, I would like to add a bare-metal worker node with more resources than the cloud instances offer to the cluster.
I can reach the bare-metal server from the cloud network via a vSwitch.
How would I have to adapt the cluster-config yaml file to realise this? Thanks for any support!

[vitobotta]

Hey, currently, this feature isn’t supported out of the box, but I’m thinking about adding it in the future since it could be really useful.

For now, you can try to manually connect your dedicated server to the cluster. To do this, you need to have updated to hetzner-k3s version 2.0.9 (which is the latest one).

Here’s how you can do it:

1. First, clone the hetzner-k3s repository to your local machine and open it as a development container in VS Code. This is easier because it already has Crystal and other dependencies, and you’ll need Docker for this to work (see https://github.com/vitobotta/hetzner-k3s/blob/main/docs/Contributing_and_support.md#developing-with-vscode).
2. Then, open the src/kubernetes/installer.cr file and add the line puts install_script right after line 147.
3. After that, run the create command using hetzner-k3s like this:

crystal run src/hetzner-k3s.cr -- create --config <path to your cluster's yaml file>

The create command will not change anything in the cluster unless you have made changes to the configuration file. However, because of the line you added in step 2, it will print out the full command that hetzner-k3s uses to join a node to the cluster.

Lastly, run the join command on your dedicated server. If the dedicated server is in the same network as the cluster, it should work. Keep in mind that I haven’t tried this myself yet. Good luck, and let me know how it goes!

[if-mar]

Thank you for your instructions! I will try and keep you updated.

[ledniy]

@if-mar Could you please let us know if you succeeded?

[if-mar]

Unfortunately, I hadn't yet found the time to try it out as other projects had higher priority. If you test it, please let us know if it worked.

[ledniy]

Unfortunately it's not enough to run the commands from the install_script, after that the node has following taints:

│ Taints:             node.cloudprovider.kubernetes.io/uninitialized=true:NoSchedule                                                                                                       │
│                     node.kubernetes.io/network-unavailable:NoSchedule   

and such events:

│   Warning  FailedToCreateRoute       7m12s                  route_controller  Could not create route 1a015b70-7786-49fa-8b09-c9616a50cde1 10.244.6.0/24 for node bm-node-1 after 329.995 │
│ 066ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: bm-node-1 hcops/AllServersCache.getCache: not found                                                                             │
│   Warning  FailedToCreateRoute       6m42s                  route_controller  Could not create route 1a015b70-7786-49fa-8b09-c9616a50cde1 10.244.6.0/24 for node bm-node-1 after 312.246 │
│ 744ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: bm-node-1 hcops/AllServersCache.getCache: not found                                                                             │
│   Warning  FailedToCreateRoute       6m12s                  route_controller  Could not create route 1a015b70-7786-49fa-8b09-c9616a50cde1 10.244.6.0/24 for node bm-node-1 after 68.208� │
│ �s: hcloud/CreateRoute: hcops/AllServersCache.ByName: bm-node-1 hcops/AllServersCache.getCache: not found                                                                                │
│   Warning  FailedToCreateRoute       5m42s                  route_controller  Could not create route 1a015b70-7786-49fa-8b09-c9616a50cde1 10.244.6.0/24 for node bm-node-1 after 306.375 │
│ 643ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: bm-node-1 hcops/AllServersCache.getCache: not found                                                                             │
│   Warning  FailedToCreateRoute       5m12s                  route_controller  Could not create route 1a015b70-7786-49fa-8b09-c9616a50cde1 10.244.6.0/24 for node bm-node-1 after 48.912� │
│ �s: hcloud/CreateRoute: hcops/AllServersCache.ByName: bm-node-1 hcops/AllServersCache.getCache: not found                                                                                │
│   Warning  FailedToCreateRoute       4m42s                  route_controller  Could not create route 1a015b70-7786-49fa-8b09-c9616a50cde1 10.244.6.0/24 for node bm-node-1 after 316.188 │
│ 852ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: bm-node-1 hcops/AllServersCache.getCache: not found                                                                             │
│   Warning  FailedToCreateRoute       4m12s                  route_controller  Could not create route 1a015b70-7786-49fa-8b09-c9616a50cde1 10.244.6.0/24 for node bm-node-1 after 104.155 │
│ µs: hcloud/CreateRoute: hcops/AllServersCache.ByName: bm-node-1 hcops/AllServersCache.getCache: not found                                                                                │
│   Warning  FailedToCreateRoute       3m42s                  route_controller  Could not create route 1a015b70-7786-49fa-8b09-c9616a50cde1 10.244.6.0/24 for node bm-node-1 after 290.382 │
│ 223ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: bm-node-1 hcops/AllServersCache.getCache: not found                                                                             │
│   Warning  FailedToCreateRoute       3m12s                  route_controller  Could not create route 1a015b70-7786-49fa-8b09-c9616a50cde1 10.244.6.0/24 for node bm-node-1 after 299.132 │
│ 136ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: bm-node-1 hcops/AllServersCache.getCache: not found                                                                             │
│   Warning  FailedToCreateRoute       12s (x6 over 2m42s)    route_controller  (combined from similar events): Could not create route 1a015b70-7786-49fa-8b09-c9616a50cde1 10.244.6.0/24  │
│ for node bm-node-1 after 310.252463ms: hcloud/CreateRoute: hcops/AllServersCache.ByName: bm-node-1 hcops/AllServersCache.getCache: not found 

[ledniy]

@vitobotta Did you think about adding support for such nodes? It would be nice even if it will be half-manual, so we can have something like this:

- name: my-bare-metal-1
  instance_type: bare-metal
  ip: 1.1.1.1

So, to make it work we'll need to manually order dedicated server and do manual setup (install correct image, add vSwitch and correct config for it), after that I believe the tool can setup everything else?

[vitobotta]

Like I mentioned in my previous post in this thread, yes, adding support for dedicated servers is on my radar, but I don't have an ETA for now. But it's on my list