Skip to content

Commit 75ec5cb

Browse files
codeaprendizcodeaprendiz
committed
chore: added task-040 cluster-upgrade
1 parent 1e85451 commit 75ec5cb

File tree

2 files changed

+183
-1
lines changed

2 files changed

+183
-1
lines changed

‎README.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -74,4 +74,5 @@
7474
### Cluster Maintenance
7575

7676
- [task-038-os-upgrades](local-mac/cluster-maintenance/task-038-os-upgrades)
77-
- [task-039-cluster-upgrades-kubeadm](local-mac/cluster-maintenance/task-039-cluster-upgrades-kubeadm)
77+
- [task-039-cluster-upgrades-kubeadm](local-mac/cluster-maintenance/task-039-cluster-upgrades-kubeadm)
78+
- [task-040-cluster-upgrade](local-mac/cluster-maintenance/task-040-cluster-upgrade)

‎local-mac/cluster-maintenance/task-040-cluster-upgrade/ReadMe.md

+181
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
## Cluster Upgrade
2+
3+
### This lab tests your skills on upgrading a kubernetes cluster. We have a production cluster with applications running on it. Let us explore the setup first.
4+
5+
- What is the current version of the cluster?
6+
7+
```bash
8+
controlplane $ kubectl get node
9+
NAME STATUS ROLES AGE VERSION
10+
controlplane Ready master 8m11s v1.18.0
11+
node01 Ready <none> 7m39s v1.18.0
12+
```
13+
14+
15+
### How many nodes can host workloads in this cluster?
16+
17+
- Inspect the applications and taints set on the nodes.
18+
19+
```bash
20+
controlplane $ kubectl describe node controlplane | grep -i taint
21+
Taints: <none>
22+
controlplane $ kubectl describe node node01 | grep -i taint
23+
Taints: <none>
24+
```
25+
26+
### How many applications are hosted on the cluster?
27+
28+
- Count the number of deployments.
29+
30+
```bash
31+
controlplane $ kubectl get deployments
32+
NAME READY UP-TO-DATE AVAILABLE AGE
33+
blue 5/5 5 5 13m
34+
red 2/2 2 2 13m
35+
```
36+
37+
### What nodes are the pods hosted on?
38+
39+
```bash
40+
controlplane $ kubectl get pods -o wide --no-headers=true | awk {'print $7'} | uniq
41+
node01
42+
```
43+
44+
45+
### You are tasked to upgrade the cluster. User's accessing the applications must not be impacted. And you cannot provision new VMs. What strategy would you use to upgrade the cluster?
46+
47+
- Upgrade one node at a time while moving workloads to other.
48+
49+
### What is the latest stable version available for upgrade?
50+
51+
- Use kubeadm tool
52+
53+
```bash
54+
controlplane $ kubeadm upgrade plan
55+
.
56+
COMPONENT CURRENT AVAILABLE
57+
API Server v1.18.0 v1.18.15
58+
Controller Manager v1.18.0 v1.18.15
59+
Scheduler v1.18.0 v1.18.15
60+
Kube Proxy v1.18.0 v1.18.15
61+
CoreDNS 1.6.7 1.6.7
62+
Etcd 3.4.3 3.4.3-0
63+
64+
You can now apply the upgrade by executing the following command:
65+
66+
kubeadm upgrade apply v1.18.15
67+
```
68+
69+
70+
### We will be upgrading the master node first. Drain the master node of workloads and mark it UnSchedulable
71+
72+
```bash
73+
controlplane $ kubectl drain controlplane --ignore-daemonsets
74+
node/controlplane cordoned
75+
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-amd64-4plkl, kube-system/kube-keepalived-vip-vff7m, kube-system/kube-proxy-h9bcg
76+
evicting pod default/blue-8455cd8cd7-tmsvk
77+
evicting pod default/blue-8455cd8cd7-fqsz8evicting pod default/blue-8455cd8cd7-hdvjv
78+
evicting pod default/blue-8455cd8cd7-lcbgc
79+
evicting pod default/red-59d898f784-t5src
80+
evicting pod default/blue-8455cd8cd7-wd8g8
81+
evicting pod default/red-59d898f784-blfx7
82+
evicting pod kube-system/coredns-66bff467f8-87kss
83+
evicting pod kube-system/coredns-66bff467f8-sfv25
84+
evicting pod kube-system/katacoda-cloud-provider-69dc659fc-2z6l2
85+
I0122 15:40:56.209205 8518 request.go:621] Throttling request took 1.171958663s, request: GET:https://172.17.0.46:6443/api/v1/namespaces/default/pods/blue-8455cd8cd7-hdvjv
86+
pod/katacoda-cloud-provider-69dc659fc-2z6l2 evicted
87+
pod/blue-8455cd8cd7-lcbgc evicted
88+
pod/blue-8455cd8cd7-tmsvk evictedpod/red-59d898f784-blfx7 evicted
89+
pod/red-59d898f784-t5src evicted
90+
pod/blue-8455cd8cd7-wd8g8 evicted
91+
pod/blue-8455cd8cd7-fqsz8 evicted
92+
pod/blue-8455cd8cd7-hdvjv evicted
93+
pod/coredns-66bff467f8-87kss evicted
94+
pod/coredns-66bff467f8-sfv25 evicted
95+
node/controlplane evicted
96+
controlplane $
97+
```
98+
99+
### Upgrade the master/controlplane components to exact version v1.19.0
100+
101+
- Upgrade kubeadm tool (if not already), then the master components, and finally the kubelet.
102+
- Practice referring to the kubernetes documentation page.
103+
- Note: While upgrading kubelet, if you hit dependency issue while running the apt-get upgrade kubelet command, use the apt install kubelet=1.19.0-00 command instead
104+
105+
```bash
106+
controlplane $ sudo apt update
107+
controlplane $ apt-get upgrade kubelet
108+
controlplane $ apt install kubelet=1.19.0-00
109+
controlplane $ kubeadm upgrade apply v.1.19.0
110+
controlplane $ apt install kubeadm=1.19.0-00
111+
controlplane $ kubeadm upgrade apply v1.19.0
112+
controlplane $ kubectl version --short
113+
Client Version: v1.20.2
114+
Server Version: v1.19.0
115+
```
116+
117+
### Mark the master/controlplane node as "Schedulable" again
118+
119+
```bash
120+
controlplane $ kubectl uncordon controlplane
121+
node/controlplane already uncordoned
122+
```
123+
124+
125+
### Next is the worker node. Drain the worker node of the workloads and mark it UnSchedulable
126+
127+
- Next is the worker node. Drain the worker node of the workloads and mark it UnSchedulable
128+
129+
```bash
130+
controlplane $ kubectl drain node01 --ignore-daemonsets
131+
node/node01 already cordoned
132+
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-flannel-ds-amd64-6khwh, kube-system/kube-keepalived-vip-b67nx, kube-system/kube-proxy-sk2kr
133+
evicting pod default/blue-8455cd8cd7-x44zx
134+
evicting pod default/red-59d898f784-mm8ws
135+
evicting pod kube-system/katacoda-cloud-provider-ff5bf677c-x254c
136+
evicting pod default/blue-8455cd8cd7-7njxs
137+
evicting pod default/blue-8455cd8cd7-djrp9
138+
evicting pod default/blue-8455cd8cd7-ngj87
139+
evicting pod default/red-59d898f784-whqm2
140+
evicting pod kube-system/coredns-f9fd979d6-6pdqb
141+
evicting pod default/blue-8455cd8cd7-zq44v
142+
I0122 16:30:27.641206 10906 request.go:655] Throttling request took 1.148844687s, request: GET:https://172.17.0.11:6443/api/v1/namespaces/default/pods/blue-8455cd8cd7-djrp9
143+
pod/blue-8455cd8cd7-ngj87 evicted
144+
pod/katacoda-cloud-provider-ff5bf677c-x254c evicted
145+
pod/blue-8455cd8cd7-djrp9 evicted
146+
pod/blue-8455cd8cd7-7njxs evicted
147+
pod/blue-8455cd8cd7-zq44v evicted
148+
pod/red-59d898f784-whqm2 evicted
149+
pod/coredns-f9fd979d6-6pdqb evicted
150+
pod/blue-8455cd8cd7-x44zx evicted
151+
pod/red-59d898f784-mm8ws evicted
152+
node/node01 evicted
153+
```
154+
155+
### Upgrade the worker node to the exact version v1.19.0
156+
157+
```bash
158+
controlplane $ kubectl get nodes -o wide
159+
NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
160+
controlplane Ready master 91m v1.19.0 172.17.0.11 <none> Ubuntu 18.04.5 LTS 4.15.0-122-generic docker://19.3.13
161+
node01 Ready,SchedulingDisabled <none> 90m v1.18.0 172.17.0.12 <none> Ubuntu 18.04.5 LTS 4.15.0-122-generic docker://19.3.13
162+
163+
controlplane $ ssh 172.17.0.12
164+
Warning: Permanently added '172.17.0.12' (ECDSA) to the list of known hosts.
165+
node01 $
166+
167+
node01 sudo apt update
168+
node01 apt-get upgrade kubelet
169+
node01 kubeadm upgrade apply v.1.19.0
170+
node01 apt install kubeadm=1.19.0-00
171+
node01 kubeadm upgrade apply v1.19.0
172+
node01 $ kubeadm upgrade node
173+
node01 $ apt install kubelet=1.19.0-00
174+
```
175+
176+
### Remove the restriction and mark the worker node as schedulable again.
177+
178+
```bash
179+
controlplane $ kubectl uncordon node01
180+
node/node01 uncordoned
181+
```

0 commit comments

Comments
 (0)