This is a mono repository for my home infrastructure and Kubernetes cluster. I try to adhere to Infrastructure as Code (IaC) and GitOps practices using Kubernetes, Flux, Renovate, and GitHub Actions.
My Kubernetes cluster is deployed with Talos. This is a semi-hyper-converged cluster, workloads and block storage are sharing the same available resources on my nodes while I have a NAS for NFS/SMB shares, bulk file storage and backups.
There is a template over at onedr0p/cluster-template if you want to try and follow along with some of the practices I use here.
- actions-runner-controller: Self-hosted Github runners.
- cert-manager: Creates SSL certificates for services in my cluster.
- cilium: Internal Kubernetes container networking interface.
- cloudflared: Enables Cloudflare secure access to certain ingresses.
- external-dns: Automatically syncs ingress DNS records to a DNS provider.
- external-secrets: Managed Kubernetes secrets using 1Password Connect.
- ingress-nginx: Kubernetes ingress controller using NGINX as a reverse proxy and load balancer.
- rook: Distributed block storage for peristent storage.
- sops: Managed secrets for Kubernetes and Terraform which are commited to Git.
- spegel: Stateless cluster local OCI registry mirror.
- volsync: Backup and recovery of persistent volume claims.
Flux watches the clusters in my kubernetes folder (see Directories below) and makes the changes to my clusters based on the state of my Git repository.
The way Flux works for me here is it will recursively search the kubernetes/${cluster}/apps folder until it finds the most top level kustomization.yaml per directory and then apply all the resources listed in it. That aforementioned kustomization.yaml will generally only have a namespace resource and one or many Flux kustomizations (ks.yaml). Under the control of those Flux kustomizations there will be a HelmRelease or other resources related to the application which will be applied.
Renovate watches my entire repository looking for dependency updates, when they are found a PR is automatically created. When some PRs are merged Flux applies the changes to my cluster.
This Git repository contains the following directories under Kubernetes.
📁 kubernetes
├── 📁 main # main cluster
├── 📁 apps # applications
├── 📁 bootstrap # bootstrap procedures
├── 📁 flux # core flux configuration
└── 📁 templates # re-useable componentsThis is a high-level look how Flux deploys my applications with dependencies. Below there are 3 apps postgres, lldap and authelia. postgres is the first app that needs to be running and healthy before lldap and authelia. Once postgres and lldap are healthy authelia will be deployed.
graph TD;
id1>Kustomization: cluster] -->|Creates| id2>Kustomization: cluster-apps];
id2>Kustomization: cluster-apps] -->|Creates| id3>Kustomization: postgres];
id2>Kustomization: cluster-apps] -->|Creates| id6>Kustomization: lldap]
id2>Kustomization: cluster-apps] -->|Creates| id8>Kustomization: authelia]
id2>Kustomization: cluster-apps] -->|Creates| id5>Kustomization: postgres-cluster]
id3>Kustomization: postgres] -->|Creates| id4[HelmRelease: postgres];
id5>Kustomization: postgres-cluster] -->|Depends on| id3>Kustomization: postgres];
id5>Kustomization: postgres-cluster] -->|Creates| id10[Postgres Cluster];
id6>Kustomization: lldap] -->|Creates| id7(HelmRelease: lldap);
id8>Kustomization: authelia] -->|Creates| id9(HelmRelease: authelia);
id8>Kustomization: authelia] -->|Depends on| id5>Kustomization: postgres-cluster];
id9(HelmRelease: authelia) -->|Depends on| id7(HelmRelease: lldap);
The Talos nodes in my cluster, by default, are located on VLAN 80 (10.0.80.10-10.0.80.14) in my network. They also get an additional network interface located on my trusted VLAN (VLAN 10). This additional interface is used by Multus to expose additional interfaces in certain apps like HomeAssistant. See example.
| Service | Use | Cost |
|---|---|---|
| 1Password | Secrets with External Secrets | ~$100/yr |
| Cloudflare | DNS & Public Tunnels | Free |
| GitHub | Hosting this repository and continuous integration/deployments | Free |
| Total: ~$9/mo |
All my DNS records are hosted publically in Cloudflare, including DNS entries that point to a local IP address.
Outside the external-dns instance mentioned above another instance is deployed in my cluster and configured to sync DNS records to Cloudflare. The only ingress this external-dns instance looks at to gather DNS records to put in Cloudflare are ones that have an ingress class name of external and contain an ingress annotation external-dns.alpha.kubernetes.io/target.
| Device | Count | OS Disk Size | Data Disk Size | Ram | Operating System | Purpose |
|---|---|---|---|---|---|---|
| Intel NUC12WSHi7 | 2 | 1TB SSD | 1TB NVMe (rook-ceph) | 64GB | Talos | Kubernetes Nodes |
| Intel NUC11PAHi7 | 2 | 1TB SSD | 1TB NVMe (rook-ceph) | 64GB | Talos | Kubernetes Nodes |
| Synology RS1219+ | 1 | - | 6x 16TB Seagate Exos | 4GB | Synology DSM | NAS - NFS Share |
| Synology DVA1622 | 1 | - | 2x 4TB Seagate Skyhawk | 4GB | Synology DSM | NVR - Cameras |
| Unifi UXG-Pro | 1 | - | - | - | - | Router / Gateway |
| Unifi US-48-500W | 1 | - | - | - | - | PoE Rackmount Switch |
| APC SMC1000I-2UC | 1 | - | - | - | - | UPS |