Skip to content

seblaporte/k3s-at-home

BranchesTags

Repository files navigation

🚀  k3s cluster at 🏠

This repository is dedicated to my personal k3s cluster deployed with k3sup backed by Flux, SOPS and maintained by Renovate.

k3s-cluster

📦  Components

📝  Prerequisites

💻  Nodes

  • Enable packet forwarding on the hosts and increase max_user_watches
cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-iptables = 1
net.ipv4.ip_forward=1
fs.inotify.max_user_watches=65536
EOF
sysctl --system

  • Configure DNS on nodes to use an upstream provider (e.g. 1.1.1.1, 9.9.9.9), or router's IP (Ad-blockers should only be used on devices with a web browser) and set a static IP with /etc/dhcpcd.conf file.

  • Disable swap with sudo dphys-swapfile swapoff && sudo dphys-swapfile uninstall && sudo update-rc.d dphys-swapfile remove && sudo rm -f /etc/init.d/dphys-swapfile && sudo service dphys-swapfile stop && sudo systemctl disable dphys-swapfile.service

  • Disable syslog to reduce pressure on SD card

sudo service rsyslog stop
sudo systemctl disable rsyslog
  • Configure logrotate for all logs with /etc/logrotate.conf
# use daily rotation
daily

# keep 3 days worth of backlogs
rotate 3

copytruncate
  • Disable leds all on Raspberry Pi with /boot/config.txt
[pi4]
# Disable the PWR LED
dtparam=pwr_led_trigger=none
dtparam=pwr_led_activelow=off
# Disable the Activity LED
dtparam=act_led_trigger=none
dtparam=act_led_activelow=off
# Disable ethernet port LEDs
dtparam=eth_led0=4
dtparam=eth_led1=4

Boot NFS

# Create local directory to mount
sudo mkdir -p /nfs/rpi-tftpboot
# Mount NFS directory
sudo mount -t nfs -O proto=tcp,port=2049,rw,all_squash,anonuid=1001,anongid=1001 192.168.1.10:/volume1/rpi-tftpboot /nfs/rpi-tftpboot -vvv
# Get node UUID
vcgencmd otp_dump | grep 28: | sed s/.*://g
# Create directory for each node (replace b676264b with node UUID)
sudo mkdir -p /nfs/rpi-tftpboot/b676264b
# Copy /boot to NFS directory
sudo cp -r /boot/* /nfs/rpi-tftpboot/b676264b/
# Change /boot entry in /etc/fstab :
# 192.168.1.10:/volume1/rpi-tftpboot/b676264b /boot nfs defaults,vers=3,proto=tcp 0 0
sudo nano /etc/fstab

Root on iSCSI target

🔧  Tools

📍 CLI tools required on workstation.

Tool Purpose
k3sup Tool to install k3s on your nodes
kubectl Allows you to run commands against Kubernetes clusters
flux Operator that manages your k8s cluster based on your Git repository
SOPS Encrypts k8s secrets with GnuPG
GnuPG Encrypts and signs your data
pinentry Allows GnuPG to read passphrases and PIN numbers
direnv Exports env vars based on present working directory
pre-commit Runs checks pre git commit
kustomize Template-free way to customize application configuration
helm Manage Kubernetes applications
go-task A task runner / simpler Make alternative written in Go
prettier Prettier is an opinionated code formatter.

⚠️  pre-commit

sops-pre-commit will check to make sure we are not by accident commiting secrets un-encrypted.

After pre-commit is installed :

pre-commit install-hooks

📂  Repository structure

The Git repository contains the following directories under cluster and are ordered below by how Flux will apply them.

  • base directory is the entrypoint to Flux
  • crds directory contains custom resource definitions (CRDs) that need to exist globally in your cluster before anything else exists
  • core directory (depends on crds) are important infrastructure applications (grouped by namespace) that should never be pruned by Flux
  • apps directory (depends on core) is where applications (grouped by namespace) are placed. Flux will prune resources here if they are not tracked by Git anymore.
cluster
├── apps
│   ├── default
│   ├── home-assistant
│   ├── mosquitto
│   ├── networking
│   ├── system-upgrade
│   ├── unifi
│   └── vaultwarden
├── base
│   └── flux-system
├── core
│   ├── cert-manager
│   ├── log-management
│   ├── metallb-system
│   ├── monitoring
│   ├── monitoring-config
│   ├── namespaces
│   ├── nfs-provisioner
│   └── system-upgrade
└── crds
    └── cert-manager

🚀  Deployment

🔐  Setting up GnuPG keys

📍 SOPS with GnuPG permits to encrypt and decrypt secrets.

  1. Creation of a Personal GPG Key, password protected, and export of the fingerprint.
export GPG_TTY=$(tty)
export PERSONAL_KEY_NAME="First name Last name (location) <email>"

gpg --batch --full-generate-key <<EOF
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Real: ${PERSONAL_KEY_NAME}
EOF

gpg --list-secret-keys "${PERSONAL_KEY_NAME}"
# pub   rsa4096 2021-03-11 [SC]
#       772154FFF783DE317KLCA0EC77149AC618D75581
# uid           [ultimate] k8s@home (Macbook) <[email protected]>
# sub   rsa4096 2021-03-11 [E]

export PERSONAL_KEY_FP=772154FFF783DE317KLCA0EC77149AC618D75581
  1. Creation of a Flux GPG Key and export of the fingerprint
export GPG_TTY=$(tty)
export FLUX_KEY_NAME="Cluster name (Flux) <email>"

gpg --batch --full-generate-key <<EOF
%no-protection
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Real: ${FLUX_KEY_NAME}
EOF

gpg --list-secret-keys "${FLUX_KEY_NAME}"
# pub   rsa4096 2021-03-11 [SC]
#       AB675CE4CC64251G3S9AE1DAA88ARRTY2C009E2D
# uid           [ultimate] Home cluster (Flux) <[email protected]>
# sub   rsa4096 2021-03-11 [E]

export FLUX_KEY_FP=AB675CE4CC64251G3S9AE1DAA88ARRTY2C009E2D

⛵  Installing k3s

  1. Installation of the master node

Servicelb is replaced by metallb, traefik and metrics-server will be installed with Flux.

curl -sfL https://get.k3s.io | \
INSTALL_K3S_VERSION=v1.21.4+k3s1 sh -s - server \
--node-taint CriticalAddonsOnly=true:NoExecute \
--disable servicelb \
--disable traefik \
--disable metrics-server \
--tls-san <master-ip> \
--datastore-endpoint="postgres://<user>:<password>@<ip>:<port>/<db>?sslmode=disable"
  1. Join worker nodes
curl -sfL https://get.k3s.io | \
INSTALL_K3S_VERSION=v1.21.4+k3s1 \
K3S_TOKEN=<token>  \
K3S_URL=https://<master-ip>:6443 sh -
  1. Check nodes are online
kubectl get nodes
# NAME         STATUS   ROLES                  AGE   VERSION
# k3s-master   Ready    control-plane,master   46d   v1.21.4+k3s1
# k3s-node-3   Ready    <none>                 46d   v1.21.4+k3s1
# k3s-node-2   Ready    <none>                 46d   v1.21.4+k3s1
# k3s-node-1   Ready    <none>                 46d   v1.21.4+k3s1

🔹  GitOps with Flux

  1. Verify Flux can be installed
flux --kubeconfig=./.kube/config check --pre
# ► checking prerequisites
# ✔ kubectl 1.21.4 >=1.18.0-0
# ✔ Kubernetes 1.21.4+k3s1 >=1.16.0-0
# ✔ prerequisites checks passed
  1. Pre-create the flux-system namespace
kubectl create namespace flux-system --dry-run=client -o yaml | kubectl apply -f -
  1. Add the Flux GPG key in-order for Flux to decrypt SOPS secrets
gpg --export-secret-keys --armor "${FLUX_KEY_FP}" |
kubectl create secret generic sops-gpg \
    --namespace=flux-system \
    --from-file=sops.asc=/dev/stdin
  1. Export more environment variables for application configuration
source .env
  1. Create required files based on ALL exported environment variables.
envsubst < ./tmpl/.sops.yaml > ./.sops.yaml
envsubst < ./tmpl/cluster-secrets.sops.yaml > ./cluster/base/cluster-secrets.sops.yaml
envsubst < ./tmpl/cluster-settings.yaml > ./cluster/base/cluster-settings.yaml
envsubst < ./tmpl/gotk-sync.yaml > ./cluster/base/flux-system/gotk-sync.yaml
envsubst < ./tmpl/secret.sops.yaml > ./cluster/core/cert-manager/secret.sops.yaml

  1. Verify all the above files have the correct information present

  2. Encrypt cluster/cluster-secrets.sops.yaml and cert-manager/secret.sops.yaml with SOPS

export GPG_TTY=$(tty)
sops --encrypt --in-place ./cluster/base/cluster-secrets.sops.yaml
sops --encrypt --in-place ./cluster/core/cert-manager/secret.sops.yaml

📍 Variables defined in cluster-secrets.sops.yaml and cluster-settings.sops.yaml will be usable anywhere in your YAML manifests under ./cluster

  1. Verify all the above files are encrypted with SOPS

  2. Push you changes to git

git add -A
git commit -m "initial commit"
git push
  1. Install Flux

📍 Due to race conditions with the Flux CRDs this command needs to be run twice. There should be no errors on this second run.

kubectl apply --kustomize=./cluster/base/flux-system
# namespace/flux-system configured
# customresourcedefinition.apiextensions.k8s.io/alerts.notification.toolkit.fluxcd.io created
# ...
# unable to recognize "./cluster/base/flux-system": no matches for kind "Kustomization" in version "kustomize.toolkit.fluxcd.io/v1beta1"
# unable to recognize "./cluster/base/flux-system": no matches for kind "GitRepository" in version "source.toolkit.fluxcd.io/v1beta1"
# unable to recognize "./cluster/base/flux-system": no matches for kind "HelmRepository" in version "source.toolkit.fluxcd.io/v1beta1"
# unable to recognize "./cluster/base/flux-system": no matches for kind "HelmRepository" in version "source.toolkit.fluxcd.io/v1beta1"
# unable to recognize "./cluster/base/flux-system": no matches for kind "HelmRepository" in version "source.toolkit.fluxcd.io/v1beta1"
# unable to recognize "./cluster/base/flux-system": no matches for kind "HelmRepository" in version "source.toolkit.fluxcd.io/v1beta1"

📣  Post installation

Verify Flux

kubectl get pods -n flux-system
# NAME                                       READY   STATUS    RESTARTS   AGE
# helm-controller-5bbd94c75-89sb4            1/1     Running   0          1h
# kustomize-controller-7b67b6b77d-nqc67      1/1     Running   0          1h
# notification-controller-7c46575844-k4bvr   1/1     Running   0          1h
# source-controller-7d6875bcb4-zqw9f         1/1     Running   0          1h

VSCode SOPS extension

VSCode SOPS is a neat little plugin for those using VSCode. It will automatically decrypt SOPS secrets when the file is open in the editor and encrypt them when save and exit the file.

👉  Debugging

Manually sync Flux with your Git repository

flux reconcile source git flux-system
# ► annotating GitRepository flux-system in flux-system namespace
# ✔ GitRepository annotated
# ◎ waiting for GitRepository reconciliation
# ✔ GitRepository reconciliation completed
# ✔ fetched revision main/943e4126e74b273ff603aedab89beb7e36be4998

Show the health of you kustomizations

kubectl get kustomization -A
# NAMESPACE     NAME          READY   STATUS                                                             AGE
# flux-system   apps          True    Applied revision: main/943e4126e74b273ff603aedab89beb7e36be4998    3d19h
# flux-system   core          True    Applied revision: main/943e4126e74b273ff603aedab89beb7e36be4998    4d6h
# flux-system   crds          True    Applied revision: main/943e4126e74b273ff603aedab89beb7e36be4998    4d6h
# flux-system   flux-system   True    Applied revision: main/943e4126e74b273ff603aedab89beb7e36be4998    4d6h

Show the health of your main Flux GitRepository

flux get sources git
# NAME           READY	MESSAGE                                                            REVISION                                         SUSPENDED
# flux-system    True 	Fetched revision: main/943e4126e74b273ff603aedab89beb7e36be4998    main/943e4126e74b273ff603aedab89beb7e36be4998    False

Show the health of your HelmReleases

flux get helmrelease -A
# NAMESPACE   	    NAME                  	READY	MESSAGE                         	REVISION	SUSPENDED
# cert-manager	    cert-manager          	True 	Release reconciliation succeeded	v1.5.2  	False
# default        	hajimari                True 	Release reconciliation succeeded	1.1.1   	False
# networking  	    ingress-nginx       	True 	Release reconciliation succeeded	3.30.0  	False

Show the health of your HelmRepositorys

flux get sources helm -A
# NAMESPACE  	NAME                 READY	MESSAGE                                                   	REVISION                                	SUSPENDED
# flux-system	bitnami-charts       True 	Fetched revision: 0ec3a3335ff991c45735866feb1c0830c4ed85cf	0ec3a3335ff991c45735866feb1c0830c4ed85cf	False
# flux-system	hajimari-charts      True 	Fetched revision: 1b24af9c5a1e3da91618d597f58f46a57c70dc13	1b24af9c5a1e3da91618d597f58f46a57c70dc13	False
# flux-system	ingress-nginx-charts True 	Fetched revision: 45669a3117fc93acc09a00e9fb9b4445e8990722	45669a3117fc93acc09a00e9fb9b4445e8990722	False
# flux-system	jetstack-charts      True 	Fetched revision: 7bad937cc82a012c9ee7d7a472d7bd66b48dc471	7bad937cc82a012c9ee7d7a472d7bd66b48dc471	False
# flux-system	k8s-at-home-charts   True 	Fetched revision: 1b24af9c5a1e3da91618d597f58f46a57c70dc13	1b24af9c5a1e3da91618d597f58f46a57c70dc13	False

🤖  Automation

  • Renovate is a very useful tool that create PRs in this Github repository when Docker images, Helm charts or anything else that can be tracked has a newer version. The configuration for renovate is located here.

  • system-upgrade-controller watch for new k3s releases and upgrade your nodes when new releases are found.

A couple of Github workflows included in this repository help automate some processes.

Ajouter une tainte à un noeud

kubectl taint nodes k3s-node-hdw-1 hardware=true:NoSchedule

kubectl label nodes k3s-node-hdw-1 nodetype=hardware-gateway

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

1 star

Watchers

3 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors 4

  •  
  •  
  •  
  •  

Languages