... managed by Flux and serviced with RenovateBot 🤖
This repository is my homelab Kubernetes cluster in a declarative state. Flux watches my cluster folder and makes the changes to my cluster based on the YAML manifests.
Feel free to open a Github issue or join the k8s@home Discord if you have any questions.
My cluster is k3s provisioned overtop Ubuntu 20.10 using the Ansible galaxy role ansible-role-k3s. This is a semi hyper-converged cluster, workloads and block storage are sharing the same available resources on my nodes.
See my server/ansible directory for my playbooks and roles.
- calico: For internal cluster networking using BGP configured on Opnsense.
- rook-ceph: Provides persistent volumes, allowing any application to consume RBD block storage.
- sealed-secrets: Encrypts Secrets into a SealedSecret, which is safe to store - even to a public repository.
- external-secrets: Allows usage of external secret management systems like AWS Secrets Manager or HashiCorp Vault, to securely add secrets in Kubernetes.
- external-dns: Creates DNS entries in a separate coredns deployment which is backed by a separate etcd deployment.
- cert-manager: Configured to create TLS certs for all ingress services automatically using LetsEncrypt.
- Github Actions for running automated Ansible tests and checking code formatting
- Rancher System Upgrade Controller to apply updates to k3s
- Renovate with the help of the k8s-at-home/renovate-helm-releases Github action keeps my application charts and container images up-to-date
In my network Calico is configured with BGP on my Opnsense router. With BGP enabled, I advertize a load balancer using externalIPs on my Kubernetes services. This makes it so I do not need Metallb. Another benefit to this is that I can directly hit any pods IP directly from any device on my local network.
| Name | CIDR |
|---|---|
| Management | 192.168.1.0/24 |
| Servers | 192.168.42.0/24 |
| k8s external services (BGP) | 192.168.69.0/24 |
| k8s pods | 10.69.0.0/16 |
| k8s services | 10.96.0.0/16 |
To prefix this, I should mention that I only use one domain name for internal and externally facing applications. Also this is the most complicated thing to explain but I will try to sum it up.
On Opnsense under Services: Unbound DNS: Overrides I have a Domain Override set to my domain with the address pointing to my non-cluster service CoreDNS load balancer IP. This allows me to use Split-horizon DNS. external-dns reads my clusters Ingress's and inserts DNS records containing the sub-domain and load balancer IP (of ingress-nginx) into the non-cluster service CoreDNS service. The records are stored into a non-cluster service etcd instance without persistence.
| Device | Count | OS Disk Size | Data Disk Size | Ram | Purpose |
|---|---|---|---|---|---|
| Intel NUC8i3BEK | 1 | 256GB NVMe | N/A | 16GB | k3s Master |
| Intel NUC8i5BEH | 3 | 120GB SSD | 1TB NVMe (rook-ceph) | 32GB | k3s Workers |
| Intel NUC8i7BEH | 2 | 750GB SSD | 1TB NVMe (rook-ceph) | 64GB | k3s Workers |
| Qnap NAS (rocinante) | 1 | N/A | 8x12TB RAID6 | 16GB | Media and shared file storage |
| Synology NAS (serenity) | 1 | N/A | 8x12TB RAID6 | 4GB | Media and shared file storage |
| Tool | Purpose |
|---|---|
| direnv | Set KUBECONFIG environment variable based on present working directory |
| git-crypt | Encrypt certain files in my repository that can only be decrypted with a key on my computers |
| go-task | Replacement for make and makefiles, who honestly likes that? |
| pre-commit | Ensure the YAML and shell script in my repo are consistent |
| kubetail | Tail logs in Kubernetes, also check out stern (which fork? good luck) |
A lot of inspiration for my cluster came from the people that have shared their clusters over at awesome-home-kubernetes