Karmada operator

Overview

The Karmada operator is a method for installing, upgrading, and deleting Karmada instances. It builds upon the basic Karmada resource and controller concepts, provides convenience to centrally manage entire lifecycle of Karmada instances in a global cluster. With the operator, you can extend Karmada with custom resources (CRs) to manage your instances not only in local clusters but also in remote clusters.

This document is an overview of how the operator works from a user perspective.

Developer quick start

This section describes how to install karmada-operator and create a Karmada instance with CR.

Prerequisites

• Kubernetes 1.16+
• Helm v3+

Deploy karmada-operator

Helm install

Go to the root directory of the karmada-io/karmada repo. Before installing the Helm Chart, ensure the karmada-operator is set to the preferred released version. You can check the latest tag on GitHub releases.

To install the Helm Chart with the release name karmada-operator in the namespace karmada-system, run the following command, replacing ${preferred-released-version} with the desired version:

helm install karmada-operator -n karmada-system --create-namespace --dependency-update ./charts/karmada-operator --set operator.image.tag=${preferred-released-version} --debug

Using YAML resource

The karmada-operator workload requires ClusterRole to watch and manage CR resources. In preparation for this, create a ClusterRole (with a ClusterRoleBinding and a ServiceAccount) containing the required privileges for the karmada-operator.

kubectl create namespace karmada-system
kubectl apply -f operator/config/deploy/karmada-operator-clusterrole.yaml
kubectl apply -f operator/config/deploy/karmada-operator-clusterrolebinding.yaml
kubectl apply -f operator/config/deploy/karmada-operator-serviceaccount.yaml

Deploy the karmada-operator workload.

kubectl apply -f operator/config/deploy/karmada-operator-deployment.yaml

The pod of karmada-operator in the karmada-system namespace will be running.

kubectl get po -n karmada-system

NAME                               READY   STATUS    RESTARTS   AGE
karmada-operator-5b7f485c5-g5lj5   1/1     Running   0          26s

Install Karmada operator crds

kubectl apply -f operator/config/crds/

Create a Karmada instance

The Karmada operator provides a Karmada CR that can define most configurations for Karmada components. It includes image messages, replica, the args of binary file, and custom label, annotation, and featuregate. For details, see API.

A Karmada CR represents a Karmada instance, which is a namespace-scoped resource. The example below is to create a simple Karmada CR in the test namespace:

kubectl create namespace test
kubectl apply -f - <<EOF
apiVersion: operator.karmada.io/v1alpha1
kind: Karmada
metadata:
  name: karmada-demo
  namespace: test
EOF

You can also create a Karmada CR directly using the sample provided by the Karmada operator.

kubectl create namespace test
kubectl apply -f operator/config/samples/karmada.yaml

Wait for around 40 seconds, and the pods of the Karmada components will be running in the same namespace as the Karmada CR.

kubectl get po -n test
karmada-demo-aggregated-apiserver-587bc5c697-v27vb      1/1     Running   0          12s
karmada-demo-apiserver-55968d9f8c-mp8hf                 1/1     Running   0          35s
karmada-demo-controller-manager-64455f7fd4-stls6        1/1     Running   0          5s
karmada-demo-etcd-0                                     1/1     Running   0          37s
karmada-demo-kube-controller-manager-584f978bbd-fftwq   1/1     Running   0          5s
karmada-demo-metrics-adapter-57cb5f56b6-4vwk2           1/1     Running   0          5s
karmada-demo-metrics-adapter-57cb5f56b6-zbhjk           1/1     Running   0          5s
karmada-demo-scheduler-6d77b7547-hgz8n                  1/1     Running   0          5s
karmada-demo-webhook-6f5944f5d8-bpkqz                   1/1     Running   0          5s

Generate kubeconfig for karmada

kubectl get secret -n test karmada-demo-admin-config -o jsonpath={.data.kubeconfig} | base64 -d > ~/.kube/karmada-apiserver.config
export KUBECONFIG=~/.kube/karmada-apiserver.config

Tip:

If no spec.hostCluster.secretRef is specified in CR, the Karmada instance will be installed in the cluster where karmada-operator is located.

Upgrade a Karmada instance

Once a Karmada instance is created, the CR resource is automatically filled with default values. To upgrade the Karmada instance, for example, you can upgrade the Karmada version to v1.5.0 or higher:

kubectl patch karmada karmada-demo -n test --type merge -p '
{
  "spec": {
    "components": {
      "karmadaAggregatedAPIServer": {
        "imageTag": "v1.5.0"
      },
      "karmadaControllerManager": {
        "imageTag": "v1.5.0"
      },
      "karmadaScheduler": {
        "imageTag": "v1.5.0"
      },
      "karmadaWebhook": {
        "imageTag": "v1.5.0"
      }
    }
  }
}'

Delete a Karmada instance

Deleting a Karmada CR is a delicate operation that requires careful attention. Once the Karmada CR is deleted, the associated Karmada instance will also be deleted. It is important to proceed with caution when deleting a Karmada CR due to the potential risks involved.

kubectl delete karmada karmada-demo -n test

If you want to delete a Karmada CR without cascading deletion of the associated Karmada instance, you can run the following command before performing the deletion operation.

kubectl label karmada karmada-demo -n test operator.karmada.io/disable-cascading-deletion=true

Custom Karmada CR

This feature allows you to configure the Karmada CR to install Karmada instances flexibly. For details, see karmada.yaml.

Set Karmada component replicas

The replicas of all Karmada components can be modified. For example, you can scale the etcd pod replicas to 3:

apiVersion: operator.karmada.io/v1alpha1
kind: Karmada
metadata:
  name: karmada-demo
  namespace: test
spec:
  components:
    etcd:
      local:
        replicas: 3

Custom label and annotation

All Karmada components allow for custom labels and annotations to be set. These are merged into both pod and workload resources.

apiVersion: operator.karmada.io/v1alpha1
kind: Karmada
metadata:
  name: karmada-demo
  namespace: test
spec:
  components:
    karmadaAPIServer:
      labels: 
        <custom-label-key>: <custom-label-value>
      annotations:
        <custom-annotation-key>: <custom-annotation-value>

Change karmada-apiserver service type

The service type of karmada-apiserver is ClusterIP by default. You can change it to NodePort:

...
karmadaAPIServer:
  imageRepository: registry.k8s.io/kube-apiserver
  imageTag: v1.31.3
  replicas: 1
  serviceType: NodePort
  serviceSubnet: 10.96.0.0/12
...

Add karmada-apiserver SANs

You can add more SANs to karmada-apiserver certificate:

...
karmadaAPIServer:
  imageRepository: registry.k8s.io/kube-apiserver
  imageTag: v1.31.3
  replicas: 1
  serviceSubnet: 10.96.0.0/12
  certSANs:
  - "kubernetes.default.svc"
  - "127.0.0.1"
...

Install karmada addon

By default, the Karmada operator does not install the descheduler and search addons. If you want to use them, you should add definitions to the Karmada CR. Here is an example of the descheduler addon:

apiVersion: operator.karmada.io/v1alpha1
kind: Karmada
metadata:
  name: karmada-demo
  namespace: test
spec:
  components:
    karmadaDescheduler: {}

If you want to install with the defaults, simply define an empty struct for descheduler.

Expose Karmada API Server

By default, the Karmada API Server's Service type is set to ClusterIP, which means it can only be accessed within the Kubernetes cluster. If you wish to access the Karmada API Server from outside the cluster, there are several methods to expose it. The following will introduce these methods and provide the necessary configuration steps.

Using a LoadBalancer Service Type

If your Kubernetes cluster runs on a cloud provider that supports LoadBalancer (such as AWS, GCP, Azure, etc.), you can change the Karmada API Server's Service type to LoadBalancer. This will automatically allocate or use an external IP address for the Karmada API Server, allowing you to access it from outside the cluster.

Using a NodePort Service Type

You also can change the Karmada API Server's Service type to NodePort. This exposes the Karmada API Server on a specific port on each node, allowing you to access it via any node's IP address and that port.

Using an Ingress Controller

If you already have an Ingress controller deployed in your cluster, you can create an Ingress resource to expose the Karmada API Server. The Ingress controller will route external traffic to the Karmada API Server's Service, enabling external access.

For example, you can create a following Ingress resource to route external traffic to the Karmada API Server:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: karmada-apiserver-ingress
  namespace: karmada-system
  annotations:
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  ingressClassName: nginx
  rules:
  - host: karmada.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: karmada-apiserver
            port:
              number: 443

Using Port Forwarding

If you only need temporary access to the Karmada API Server or prefer not to permanently expose it, you can use kubectl port-forward to forward a local port to the Karmada API Server's Pod. This method is ideal for development and debugging but is not recommended for production environments.

Contributing

The karmada/operator repo is part of Karmada from 1.5 onwards. If you're interested in the Karmada operator and want to contribute your code and ideas, welcome to open PRs and issues. See CONTRIBUTING for details on submitting patches and the contribution workflow.