forked from kubevirt/kubevirt
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Consume the latest kubevirt-ci fixing the kind lane.
Pin the sriov config daemon version and move all the pfs into the node's namespace. References in kubevirt/kubevirtci#248 Signed-off-by: Federico Paolinelli <[email protected]>
- Loading branch information
Showing
8 changed files
with
170 additions
and
16 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 485ca765a95ec0a7797a1a6128fc0edf9d358a70 | ||
| dfa2be010812c39ce653331db5bd974ae347c652 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,149 @@ | ||
| # kubevirtci K8s provider dev guide. | ||
|
|
||
| The purpose of kubevirtci is to create pre-provisioned K8s clusters as container images, | ||
| allowing people to easily run a K8s cluster. | ||
|
|
||
| The target audience is developers of kubevirtci, who want to create a new provider, or to update an existing one. | ||
|
|
||
| Please refer first to the following documents on how to run k8s-1.17.0:\ | ||
| [k8s-1.17.0 cluster-up](https://github.com/kubevirt/kubevirtci/blob/master/cluster-up/cluster/k8s-1.17.0/README.md) | ||
|
|
||
| In this doc, we will go on what kubevirtci provider image consist of, what its inner architecture, | ||
| flow of start a pre-provisioned cluster, flow of creating a new provider, and how to create a new provider. | ||
|
|
||
| A provider includes all the images (K8s base image, nodes OS image) and the scripts that allows it to start a | ||
| cluster offline, without downloading / installing / compiling new resources. | ||
| Deploying a cluster will create containers, which communicate with each other, in order to act as a K8s cluster. | ||
| It's a bit different from running bare-metal cluster where the nodes are physical machines or when the nodes are virtual machines on the host itself, | ||
| It gives us isolation advantage and state freezing of the needed components, allowing offline deploy, agnostic of the host OS, and installed packages. | ||
|
|
||
| # Project structure | ||
| * cluster-provision folder - creating preprovisioned clusters. | ||
| * cluster-up folder - spinning up preprovisioned clusters. | ||
| * gocli - gocli is a binary that assist in provisioning and spinning up a cluster. sources of gocli are at cluster-provision/gocli. | ||
|
|
||
| # K8s Deployment | ||
| Running `make cluster-up` will deploy a pre-provisioned cluster. | ||
| Upon finishing deployment of a K8s deploy, we will have 3 containers: | ||
| * k8s-1.17.0 vm container - a container that runs a qemu VM, which is the K8s node, in which the pods will run. | ||
| * Registry container - a shared image registry. | ||
| * k8s-1.17.0 dnsmasq container - a container that run dnsmasq, which gives dns and dhcp services. | ||
|
|
||
| The containers are running and looks like this: | ||
| ``` | ||
| [root@modi01 1.17.0]# docker ps | ||
| CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES | ||
| 3589e85efc7d kubevirtci/k8s-1.17.0 "/bin/bash -c '/vm.s…" About an hour ago Up About an hour k8s-1.17.0-node01 | ||
| 4742dc02add2 registry:2.7.1 "/entrypoint.sh /etc…" About an hour ago Up About an hour k8s-1.17.0-registry | ||
| 13787e7d4ac9 kubevirtci/k8s-1.17.0 "/bin/bash -c /dnsma…" About an hour ago Up About an hour 127.0.0.1:8443->8443/tcp, 0.0.0.0:32794->2201/tcp, 0.0.0.0:32793->5000/tcp, 0.0.0.0:32792->5901/tcp, 0.0.0.0:32791->6443/tcp k8s-1.17.0-dnsmasq | ||
| ``` | ||
|
|
||
| Nodes: | ||
| ``` | ||
| [root@modi01 kubevirtci]# oc get nodes | ||
| NAME STATUS ROLES AGE VERSION | ||
| node01 Ready master 83m v1.17.0 | ||
| ``` | ||
|
|
||
| # Inner look of a deployed cluster | ||
| We can connect to the node of the cluster by: | ||
| ``` | ||
| ./cluster-up/ssh.sh node01 | ||
| ``` | ||
|
|
||
| List the pods | ||
| ``` | ||
| [vagrant@node01 ~]$ sudo crictl pods | ||
| POD ID CREATED STATE NAME NAMESPACE ATTEMPT | ||
| 403513878c8b7 10 minutes ago Ready coredns-6955765f44-m6ckl kube-system 4 | ||
| 0c3e25e58b9d0 10 minutes ago Ready local-volume-provisioner-fkzgk default 4 | ||
| e6d96770770f4 10 minutes ago Ready coredns-6955765f44-mhfgg kube-system 4 | ||
| 19ad529c78acc 10 minutes ago Ready kube-flannel-ds-amd64-mq5cx kube-system 0 | ||
| 47acef4276900 10 minutes ago Ready kube-proxy-vtj59 kube-system 0 | ||
| df5863c55a52f 11 minutes ago Ready kube-scheduler-node01 kube-system 0 | ||
| ca0637d5ac82f 11 minutes ago Ready kube-apiserver-node01 kube-system 0 | ||
| f0d90506ce3b8 11 minutes ago Ready kube-controller-manager-node01 kube-system 0 | ||
| f873785341215 11 minutes ago Ready etcd-node01 kube-system 0 | ||
| ``` | ||
|
|
||
| Check kubelet service status | ||
| ``` | ||
| [vagrant@node01 ~]$ systemctl status kubelet | ||
| ● kubelet.service - kubelet: The Kubernetes Node Agent | ||
| Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: disabled) | ||
| Drop-In: /usr/lib/systemd/system/kubelet.service.d | ||
| └─10-kubeadm.conf | ||
| Active: active (running) since Wed 2020-01-15 13:39:54 UTC; 11min ago | ||
| Docs: https://kubernetes.io/docs/ | ||
| Main PID: 4294 (kubelet) | ||
| CGroup: /system.slice/kubelet.service | ||
| ‣ 4294 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/boo... | ||
| ``` | ||
|
|
||
| Connect to the container that runs the vm: | ||
| ``` | ||
| CONTAINER=$(docker ps | grep vm | awk '{print $1}') | ||
| docker exec -it $CONTAINER bash | ||
| ``` | ||
|
|
||
| From within the container we can see there is a process of qemu which runs the node as a virtual machine. | ||
| ``` | ||
| [root@855de8c8310f /]# ps -ef | grep qemu | ||
| root 1 0 36 13:39 ? 00:05:22 qemu-system-x86_64 -enable-kvm -drive format=qcow2,file=/var/run/disk/disk.qcow2,if=virtio,cache=unsafe -device virtio-net-pci,netdev=network0,mac=52:55:00:d1:55:01 -netdev tap,id=network0,ifname=tap01,script=no,downscript=no -device virtio-rng-pci -vnc :01 -cpu host -m 5120M -smp 5 -serial pty | ||
| ``` | ||
|
|
||
| # Flow of K8s provisioning (1.17 for example) | ||
| `cluster-provision/k8s/1.17.0/provision.sh` | ||
| * Runs the common cluster-provision/k8s/provision.sh. | ||
| * Runs cluster-provision/cli/cli (bash script). | ||
| * Creates a container for dnsmasq and runs dnsmasq.sh in it. | ||
| * Create a container, and runs vm.sh in it. | ||
| * Creates a vm using qemu, and checks its ready (according ssh). | ||
| * Runs cluster-provision/k8s/scripts/provision.sh in the container. | ||
| * Update docker trusted registries. | ||
| * Start kubelet service and K8s cluster. | ||
| * Enable ip routing. | ||
| * Apply additional manifests, such as flannel. | ||
| * Wait for pods to become ready. | ||
| * Pull needed images such as Ceph CSI, fluentd logger. | ||
| * Create local volume directiories. | ||
| * Shutdown the vm and commit its container. | ||
|
|
||
| # Flow of K8s cluster-up (1.17 for example) | ||
| Run | ||
| ``` | ||
| export KUBEVIRT_PROVIDER=k8s-1.17.0 | ||
| make cluster-up | ||
| ``` | ||
| * Runs cluster-up/up.sh which sources the following: | ||
| * cluster-up/cluster/k8s-1.17.0/provider.sh (selected according $KUBEVIRT_PROVIDER), which sources: | ||
| * cluster-up/cluster/k8s-provider-common.sh | ||
| * Runs `up` (which appears at cluster-up/cluster/k8s-provider-common.sh). | ||
| It Triggers `gocli run` - (cluster-provision/gocli/cmd/run.go) which create the following containers: | ||
| * Cluster container (that one with the vm from the provisioning, vm.sh is used with parameters here that starts an already created vm). | ||
| * Registry. | ||
| * Container for dnsmasq (provides dns, dhcp services). | ||
|
|
||
| # Creating new K8s provider | ||
| Clone folders of k8s, folder name should be x/y as in the provider name x-y (ie. k8s-1.17.0) and includes: | ||
| * cluster-provision/k8s/1.17.0/provision.sh # used to create a new provider | ||
| * cluster-provision/k8s/1.17.0/publish.sh # used to publish new provider | ||
| * cluster-up/cluster/k8s-1.17.0/provider.sh # used by cluster-up | ||
| * cluster-up/cluster/k8s-1.17.0/README.md | ||
|
|
||
| # Example - Adding a new manifest to K8s 1.17 | ||
| * First add the file at cluster-provision/manifests, this folder would be copied to /tmp in the container, | ||
| by cluster-provision/cli/cli as part of provision. | ||
| * Add this snippet at cluster-provision/k8s/scripts/provision.sh, before "Wait at least for 7 pods" line. | ||
| ``` | ||
| custom_manifest="/tmp/custom_manifest.yaml" | ||
| kubectl --kubeconfig=/etc/kubernetes/admin.conf create -f "$custom_manifest" | ||
| ``` | ||
| * Run ./cluster-provision/k8s/1.17.0/provision.sh, it will create a new provision and test it. | ||
| * Run ./cluster-provision/k8s/1.17.0/publish.sh, it will publish the new created image to docker.io | ||
| * Update k8s-1.17.0 image line at cluster-up/cluster/images.sh, to point on the newly published image. | ||
| * Create a PR with the following files: | ||
| * The new manifest. | ||
| * Updated cluster-provision/k8s/scripts/provision.sh | ||
| * Updated cluster-up/cluster/images.sh. | ||
|
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1 +1 @@ | ||
| 78dcc4f8aaec467bba8ae325646056128268a169 | ||
| f5c7e4db7238ac4130c4e76698807b1c46359b7b |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters