Skip to content

Commit

Permalink
Added OLM and Marketplace manifests
Browse files Browse the repository at this point in the history
For doing that without manifest duplication, the operator RBAC and
deployment manifests are generated now too.

Signed-off-by: Marc Sluiter <[email protected]>
slintes committed Apr 2, 2019
1 parent dad1392 commit ae968d4
Showing 36 changed files with 1,542 additions and 255 deletions.
1 change: 1 addition & 0 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -24,6 +24,7 @@ script:
- make apidocs
- make client-python
- make manifests DOCKER_PREFIX="docker.io/kubevirt" DOCKER_TAG=$TRAVIS_TAG # falls back to latest if not on a tag
- make olm-verify

deploy:
- provider: script
12 changes: 10 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
@@ -90,7 +90,7 @@ verify-build:
hack/verify-build.sh

manifests:
hack/dockerized "DOCKER_PREFIX=${DOCKER_PREFIX} DOCKER_TAG=${DOCKER_TAG} IMAGE_PULL_POLICY=${IMAGE_PULL_POLICY} VERBOSITY=${VERBOSITY} ./hack/build-manifests.sh"
hack/dockerized "CSV_VERSION=${CSV_VERSION} DOCKER_PREFIX=${DOCKER_PREFIX} DOCKER_TAG=${DOCKER_TAG} IMAGE_PULL_POLICY=${IMAGE_PULL_POLICY} VERBOSITY=${VERBOSITY} ./hack/build-manifests.sh"

.release-functest:
make functest > .release-functest 2>&1
@@ -121,6 +121,12 @@ builder-build:
builder-publish:
./hack/builder/publish.sh

olm-verify:
hack/dockerized "./hack/olm.sh verify"

olm-push:
hack/dockerized "CSV_VERSION=${CSV_VERSION} QUAY_USERNAME=${QUAY_USERNAME} QUAY_PASSWORD=${QUAY_PASSWORD} ./hack/olm.sh push"

.PHONY: \
go-build \
go-test \
@@ -145,4 +151,6 @@ builder-publish:
cluster-down \
cluster-clean \
cluster-deploy \
cluster-sync
cluster-sync \
olm-verify \
olm-push
Binary file added assets/kubevirt_logo.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
151 changes: 151 additions & 0 deletions docs/devel/olm-integration.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,151 @@
# KubeVirt's OLM and Operator Marketplace Integration

## Introduction

### Operator Lifecycle Manager (OLM)

https://github.com/operator-framework/operator-lifecycle-manager

OLM is the Operator Lifecycle Manager, which consists of 2 operators:

#### OLM Operator

Installs application operators based in information in ClusterServiceVersions

CRDs:

- ClusterServiceVersion (CSV):
contains application metadata: name, version, icon, required resources, installation, etc...
provided by developer, together with CRD declarations and package description. The latter declares channels and their CSV version.
installed by Catalog Operator

- OperatorGroup:
declares on which namespaces OLM should operate
provided and installed by developer, or in UI

#### Catalog Operator

Prepares installations of operators by installing the application's CRDs and CSVs

CRDs:

- CatalogSource:
declares available packages
provided and installed by Marketplace Operator based on CatalogSourceConfig

- Subscription:
declares which version of an operator to install (which channel from which source)
provided and installed by developer, or in UI

- InstallPlan:
calculated list of resources to be created in order to automatically install/upgrade a CSV
created and insalled by the Catalog Operator, needs manual or automatic approval

### Operator Marketplace

https://github.com/operator-framework/operator-marketplace

The Operator Marketplace has another operator

CRDs:

- OperatorSource:
declares where to find applications bundles (CSV + CRD + package)
provided and installed by developer, and/or already installed pointing to official repositories (community operators)

- CatalogSourceConfig:
declares which packages to enable in the marketplace
created and deployed by marketplace operator

## KubeVirt Manifests

Our OLM / Marketplace manifest templates live in /manifests/release/olm. As for all manifests, you need to run
`make generate && make manifests` for getting their final version in the `_out/` directory.

The bundle subdirectory contains:
- the ClusterServiceVersion manifest
- the CRD manifest
- the Package manifest: this contains the available distribution channels and their corresponding CSV name
These files are pushed to Quay (after they are processed with)

Then we have:
- the OperatorSource manifest: this will be deployed to your cluster.
- a Subscription manifest: only needed when not created using the OKD console.
- a OperatorGroup manifest: can be created in the console, too??

Last but not least there is a preconditions manifest: if want to test the CSV manifest manually, without
OperatorSource and Subscription, you can deploy this manifest in order to satisfy all conditions, which are declared
in the CSV manifest, so that the OLM operator can deploy the KubeVirt operator.

## Test a new version

Note 1: We use a k8s cluster >= v1.11 for this. You might want to use a OKD cluster with OLM and Marketplace already installed.
Note 2: You need a Quay.io account

- create manifests with your repository and version info, e.g.:

TODO: actually use CSV_VERSION!!!

`CSV_VERSION=<csv-version> DOCKER_PREFIX="docker.io/<docker_user>" DOCKER_TAG="<tag>" sh -c 'make generate && make manifests'`
- verify manifests:
`make olm-verify`
- push images:
`DOCKER_PREFIX="index.docker.io/<docker_user>" DOCKER_TAG="<tag>" make bazel-push-images`
- push the operator bundle:
`CSV_VERSION=<csv-version> QUAY_USER=<username> QUAY_PASSWORD=<password> make olm-push`
Note: you need to update the CSV version (and so run `make manifests`) on every push! (or maybe delete an old version before pushing again?)

- install OLM and Marketplace (see below)

- install KubeVirt OperatorSource:
`cd _out/manifests/release/olm`
`kubectl apply -f kubevirt-operatorsource.yaml`
- check that a CatalogSourceConfig and a CatalogSource were created in the marketplace namespace
- WORKAROUND: the OKD console only shows operators from CatalogSources in the `olm` namespace. In order to get it there,
you need to edit the CatalogSourceConfig and change the targetNamespace from `marketplace` to `olm`. The new
CatalogSource should be created automatically.
- create the kubevirt namespace:
`kubectl create ns kubevirt`
- install the OperatorGroup for the new namespace:
`kubectl apply -f operatorgroup.yaml`
- create a Subscription:
`kubectl apply -f kubevirt-subsription.yaml`
- check that a InstallPlan was created
- check that the KubeVirt operator was installed
- install a KubeVirt CR

Bonus: install the OKD Console:

- we need cluster-admin permissions for the kube-system:default account:
`kubectl create clusterrolebinding defaultadmin --clusterrole cluster-admin --serviceaccount kube-system:default`
- in the OLM repository, run `./scripts/run_console_local.sh`
- open `localhost:9000` in a browser

## Release a new version

Travis cares for this on every release.

## Installing OLM on Kubernetes

- clone github.com/operator-framework/operator-lifecycle-manager
- `cd deploy/upstream/quickstart`
- `kubectl apply -f olm.yaml`
- if you get an error, try again, CRDs might have been too slow

## Installing Marketplace on Kubernetes

- clone github.com/operator-framework/operator-marketplace
- `cd deploy/upstream/manifests`
- `kubectl apply -f upstream/`
- if you get an error about rolebinding, repeat with `--validate=false`

## Sources

CSV description: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/Documentation/design/building-your-csv.md
Publish bundles: https://github.com/operator-framework/community-operators/blob/master/docs/testing-operators.md
Install OLM: https://github.com/operator-framework/operator-lifecycle-manager/blob/master/Documentation/install/install.md
Install and use Marketplace: https://github.com/operator-framework/operator-marketplace

## Important

- the Quay repo name needs to match the package name (https://github.com/operator-framework/operator-marketplace/issues/122#issuecomment-470820491)
15 changes: 13 additions & 2 deletions hack/build-manifests.sh
Original file line number Diff line number Diff line change
@@ -23,6 +23,7 @@ source hack/common.sh
source hack/config.sh

manifest_docker_prefix=${manifest_docker_prefix-${docker_prefix}}
kubevirt_logo_path="assets/kubevirt_logo.png"

rm -rf ${MANIFESTS_OUT_DIR}
rm -rf ${MANIFEST_TEMPLATES_OUT_DIR}
@@ -44,12 +45,18 @@ done
# then process variables
args=$(cd ${KUBEVIRT_DIR}/manifests && find . -type f -name "*.yaml.in.tmp")
for arg in $args; do

infile=${KUBEVIRT_DIR}/manifests/${arg}

final_out_dir=$(dirname ${MANIFESTS_OUT_DIR}/${arg})
final_templates_out_dir=$(dirname ${MANIFEST_TEMPLATES_OUT_DIR}/${arg})
mkdir -p ${final_out_dir}

final_templates_out_dir=$(dirname ${MANIFEST_TEMPLATES_OUT_DIR}/${arg})
mkdir -p ${final_templates_out_dir}

manifest=$(basename -s .in.tmp ${arg})
infile=${KUBEVIRT_DIR}/manifests/${arg}
manifest="${manifest/VERSION/${csv_version}}"

outfile=${final_out_dir}/${manifest}
template_outfile=${final_templates_out_dir}/${manifest}.j2

@@ -61,6 +68,8 @@ for arg in $args; do
--container-tag=${docker_tag} \
--image-pull-policy=${image_pull_policy} \
--verbosity=${verbosity} \
--csv-version=${csv_version} \
--kubevirt-logo-path=${kubevirt_logo_path} \
--input-file=${infile} >${outfile}

${KUBEVIRT_DIR}/tools/manifest-templator/manifest-templator \
@@ -71,6 +80,8 @@ for arg in $args; do
--container-tag="{{ docker_tag }}" \
--image-pull-policy="{{ image_pull_policy }}" \
--verbosity=${verbosity} \
--csv-version=${csv_version} \
--kubevirt-logo-path=${kubevirt_logo_path} \
--input-file=${infile} >${template_outfile}
done

6 changes: 4 additions & 2 deletions hack/builder/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
FROM fedora:28

ENV LIBVIRT_VERSION 5.0.0
ENV LIBVIRT_VERSION 5.1.0

# Install packages
RUN dnf install -y dnf-plugins-core && \
@@ -23,7 +23,9 @@ RUN dnf install -y dnf-plugins-core && \
rsync-daemon \
rsync \
qemu-img \
protobuf-compiler && \
protobuf-compiler \
python3-devel \
redhat-rpm-config && \
dnf -y clean all

ENV GIMME_GO_VERSION=1.11.5
4 changes: 3 additions & 1 deletion hack/builder/build.sh
Original file line number Diff line number Diff line change
@@ -5,4 +5,6 @@ SCRIPT_DIR="$(
pwd
)"

docker build -t kubevirt/builder:28-5.0.0 -f ${SCRIPT_DIR}/Dockerfile ${SCRIPT_DIR}
. ${SCRIPT_DIR}/version.sh

docker build -t kubevirt/builder:${VERSION} -f ${SCRIPT_DIR}/Dockerfile ${SCRIPT_DIR}
11 changes: 9 additions & 2 deletions hack/builder/publish.sh
Original file line number Diff line number Diff line change
@@ -1,4 +1,11 @@
#!/usr/bin/env bash

docker tag kubevirt/builder:28-5.0.0 docker.io/kubevirt/builder:28-5.0.0
docker push docker.io/kubevirt/builder:28-5.0.0
SCRIPT_DIR="$(
cd "$(dirname "$BASH_SOURCE[0]")"
pwd
)"

. ${SCRIPT_DIR}/version.sh

docker tag kubevirt/builder:${VERSION} docker.io/kubevirt/builder:${VERSION}
docker push docker.io/kubevirt/builder:${VERSION}
1 change: 1 addition & 0 deletions hack/builder/version.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
VERSION=28-5.1.0
6 changes: 6 additions & 0 deletions hack/config-default.sh
Original file line number Diff line number Diff line change
@@ -9,3 +9,9 @@ namespace=kubevirt
cdi_namespace=cdi
image_pull_policy=${IMAGE_PULL_POLICY:-IfNotPresent}
verbosity=${VERBOSITY:-2}

# try to derive csv_version from docker tag. But it must start with x.y.z, without leading v
default_csv_version="${docker_tag/latest/0.0.0}"
default_csv_version="${default_csv_version/devel/0.0.0}"
[[ $default_csv_version == v* ]] && default_csv_version="${default_csv_version/v/}"
csv_version=${CSV_VERSION:-$default_csv_version}
4 changes: 2 additions & 2 deletions hack/config.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
unset binaries docker_images docker_prefix docker_tag docker_tag_alt manifest_templates \
master_ip network_provider kubeconfig manifest_docker_prefix namespace image_pull_policy verbosity
master_ip network_provider kubeconfig manifest_docker_prefix namespace image_pull_policy verbosity csv_version

KUBEVIRT_PROVIDER=${KUBEVIRT_PROVIDER:-${PROVIDER}}

@@ -13,4 +13,4 @@ test -f "hack/config-provider-${KUBEVIRT_PROVIDER}.sh" && source hack/config-pro
test -f "hack/config-local.sh" && source hack/config-local.sh

export binaries docker_images docker_prefix docker_tag docker_tag_alt manifest_templates \
master_ip network_provider kubeconfig namespace image_pull_policy verbosity
master_ip network_provider kubeconfig namespace image_pull_policy verbosity csv_version
5 changes: 4 additions & 1 deletion hack/generate.sh
Original file line number Diff line number Diff line change
@@ -23,11 +23,14 @@ ${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=vm >${KUBEVIR
${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=vmim >${KUBEVIRT_DIR}/manifests/generated/vmim-resource.yaml
${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=kv >${KUBEVIRT_DIR}/manifests/generated/kv-resource.yaml
${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=kv-cr --namespace={{.Namespace}} --pullPolicy={{.ImagePullPolicy}} >${KUBEVIRT_DIR}/manifests/generated/kubevirt-cr.yaml.in
${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=rbac --namespace={{.Namespace}} >${KUBEVIRT_DIR}/manifests/generated/rbac.authorization.k8s.yaml.in
${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=kubevirt-rbac --namespace={{.Namespace}} >${KUBEVIRT_DIR}/manifests/generated/rbac-kubevirt.authorization.k8s.yaml.in
${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=cluster-rbac --namespace={{.Namespace}} >${KUBEVIRT_DIR}/manifests/generated/rbac-cluster.authorization.k8s.yaml.in
${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=operator-rbac --namespace={{.Namespace}} >${KUBEVIRT_DIR}/manifests/generated/rbac-operator.authorization.k8s.yaml.in
${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=prometheus --namespace={{.Namespace}} >${KUBEVIRT_DIR}/manifests/generated/prometheus.yaml.in
${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=virt-api --namespace={{.Namespace}} --repository={{.DockerPrefix}} --version={{.DockerTag}} --pullPolicy={{.ImagePullPolicy}} --verbosity={{.Verbosity}} >${KUBEVIRT_DIR}/manifests/generated/virt-api.yaml.in
${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=virt-controller --namespace={{.Namespace}} --repository={{.DockerPrefix}} --version={{.DockerTag}} --pullPolicy={{.ImagePullPolicy}} --verbosity={{.Verbosity}} >${KUBEVIRT_DIR}/manifests/generated/virt-controller.yaml.in
${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=virt-handler --namespace={{.Namespace}} --repository={{.DockerPrefix}} --version={{.DockerTag}} --pullPolicy={{.ImagePullPolicy}} --verbosity={{.Verbosity}} >${KUBEVIRT_DIR}/manifests/generated/virt-handler.yaml.in
${KUBEVIRT_DIR}/tools/resource-generator/resource-generator --type=virt-operator --namespace={{.Namespace}} --repository={{.DockerPrefix}} --version={{.DockerTag}} --pullPolicy={{.ImagePullPolicy}} --verbosity={{.Verbosity}} >${KUBEVIRT_DIR}/manifests/generated/virt-operator.yaml.in

(cd ${KUBEVIRT_DIR}/tools/vms-generator/ && go build)
vms_docker_prefix=${DOCKER_PREFIX:-registry:5000/kubevirt}
4 changes: 2 additions & 2 deletions hack/kubevirt-builder/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM kubevirt/builder@sha256:cd662847df816a0c5cf1a99dcb905301d305befe93d50bee8d00fdd5c74beb19
FROM kubevirt/builder:28-5.1.0

ENV GIMME_GO_VERSION=1.11.5
ENV GOPATH="/go" GOBIN="/usr/bin"
@@ -13,7 +13,7 @@ RUN \
go get -u github.com/rmohr/go-swagger-utils/swagger-doc && \
go get -u github.com/onsi/ginkgo/ginkgo

RUN pip install j2cli
RUN pip install j2cli && pip3 install operator-courier

COPY rsyncd.conf /etc/rsyncd.conf

Loading

0 comments on commit ae968d4

Please sign in to comment.