pravega · ibumarskov · Nov 16, 2021 · Nov 23, 2021 · Nov 23, 2021 · Nov 23, 2021
diff --git a/README.md b/README.md
@@ -18,6 +18,7 @@ The project is currently alpha. While no breaking API changes are currently plan
     * [Upgrade the Zookeeper Operator](#upgrade-the-operator)
     * [Uninstall the Operator](#uninstall-the-operator)
     * [The AdminServer](#the-adminserver)
+    * [Automatic backup (tech preview)](#automatic-backup-tech-preview)
  * [Development](#development)
     * [Build the Operator Image](#build-the-operator-image)
     * [Direct Access to Cluster](#direct-access-to-the-cluster)
@@ -369,6 +370,40 @@ The list of available commands are
 /commands/zabstate
 ```
 
+### Automatic backup (tech preview)
+In Zookeeper operator was implemented dedicated controller responsible for automatic backups of zookeeper data. Current 
+implementation provides periodic copying of transaction logs and snapshots (on-disk representation of the data) to 
+dedicated persistence volume with specified storage class within kubernetes cluster. Advantages and disadvantages of such
+approach described in [article](https://www.elastic.co/blog/zookeeper-backup-a-treatise).
+Exmaple CR of zookeeper backup:
+```yaml
+apiVersion: "zookeeper.pravega.io/v1beta1"
+kind: "ZookeeperCluster"
+metadata:
+  name: "example-backup"
+spec:
+    zookeeperCluster: "zookeeper-cluster"
+    schedule: "0 0 */1 * *"
+    backupsToKeep: "7"
+    dataStorageClass: "backup-class"
+    image:
+      repository: "pravega/zkbackup"
+      tag: "0.1"
+```
+
+Parameters:
+- *zookeeperCluster* (required) - name of zookeeper cluster to backup.
+- *schedule* (optional) - the schedule in Cron format.
+- *backupsToKeep* (optional) - number of stored backups.
+- *dataStorageClass* (required) - storage class used for persistence volume for backups. Storage class and related provisioner should be configured separately.
+- *image* (optional) - image for backup procedure.
+
+Backup controller takes following responsibilities:
+- Provide cluster health check and cancel backup operation if required.
+- Detect ZK leader pod and prepare/reconfigure CronJob configuration (Backup pod should land on node where leader is elected).
+- Schedule CronJob with a backup script.
+- Provide mechanism to periodic checks to make sure CronJob configuration is updated and valid (for example in case of new leader election)
+
 ## Development
 
 ### Build the operator image

diff --git a/build_backup/Dockerfile b/build_backup/Dockerfile
@@ -0,0 +1,6 @@
+FROM ubuntu:20.04
+
+WORKDIR /zookeeper
+COPY backup.sh /zookeeper/backup.sh
+RUN chmod +x /zookeeper/backup.sh
+RUN apt-get update && apt-get install -y netcat
diff --git a/build_backup/backup.sh b/build_backup/backup.sh
@@ -0,0 +1,64 @@
+#!/bin/bash
+
+echo "--------------- Check required parameters ---------------"
+_req_envs="
+Required parameters: \n
+BACKUPDIR: $BACKUPDIR \n
+ZOOKEEPERDATADIR: $ZOOKEEPERDATADIR \n
+BACKUPS_TO_KEEP: $BACKUPS_TO_KEEP \n
+"
+echo -e "$_req_envs"
+
+if [ -z "$BACKUPDIR" ] || [ -z "$ZOOKEEPERDATADIR" ] || [ -z "$BACKUPS_TO_KEEP" ]; then
+  echo -e "Some required env variables aren't defined.\n"
+  exit 1
+fi
+
+# Create backup directory if absent.
+# ----------------------------------
+echo "--------------- Check backup/tmp dirs ---------------"
+if [ ! -d "$BACKUPDIR" ] && [ ! -e "$BACKUPDIR" ]; then
+    mkdir -p "$BACKUPDIR"
+else
+    printf "Backup directory $BACKUPDIR is existed.\n"
+fi
+
+# TO DO: provide additional check of zookeeper health
+
+# Backup and create tar archive
+# ------------------------------
+echo "--------------- Backup ---------------"
+TIMESTAMP=$( date +"%Y%m%d%H%M%S" )
+# Include the timestamp in the filename
+FILENAME="$BACKUPDIR/zookeeper-$TIMESTAMP.tar.gz"
+
+tar -zcvf $FILENAME -P $ZOOKEEPERDATADIR > /dev/null 2>&1
+RC=$?
+
+if [ $RC -ne 0 ]; then
+    printf "Error generating tar archive.\n"
+    exit 1
+else
+    printf "Successfully created a backup tar file.\n"
+fi
+
+
+# Cleanup old backups
+# -------------------
+echo "--------------- Cleanup ---------------"
+echo "List of backups:"
+cd $BACKUPDIR
+
+BACKUPS_AMOUNT=`find . -path "*/zookeeper-*.tar.gz" -type f -printf "\n%AD %AT %p" | wc -l`
+TO_DELETE=$(( $BACKUPS_AMOUNT - $BACKUPS_TO_KEEP ))
+if [ $TO_DELETE -gt 0 ] ; then
+    echo "Keeping only $BACKUPS_TO_KEEP full backups"
+    ls -t | tail -n -$TO_DELETE | xargs -d '\n' rm -rf
+else
+    echo "There are less backups than required, nothing to delete."
+fi
+
+# Cleanup old backups
+# -------------------
+echo "--------------- Current backups ---------------"
+ls -lt
diff --git a/deploy/crds/zookeeper.pravega.io_v1beta1_zookeeperbackup_cr.yaml b/deploy/crds/zookeeper.pravega.io_v1beta1_zookeeperbackup_cr.yaml
@@ -0,0 +1,12 @@
+apiVersion: zookeeper.pravega.io/v1beta1
+kind: ZookeeperBackup
+metadata:
+  name: example-zookeeperbackup
+spec:
+  zookeeperCluster: "tf-zookeeper"
+  schedule: "0 0 */1 * *"
+  backupsToKeep: "7"
+  dataStorageClass: "backup-class"
+  image:
+    repository: "pravega/zkbackup"
+    tag: "0.1"
diff --git a/deploy/crds/zookeeper.pravega.io_zookeeperbackups_crd.yaml b/deploy/crds/zookeeper.pravega.io_zookeeperbackups_crd.yaml
@@ -0,0 +1,78 @@
+apiVersion: apiextensions.k8s.io/v1
+kind: CustomResourceDefinition
+metadata:
+  name: zookeeperbackups.zookeeper.pravega.io
+spec:
+  group: zookeeper.pravega.io
+  names:
+    kind: ZookeeperBackup
+    listKind: ZookeeperBackupList
+    plural: zookeeperbackups
+    singular: zookeeperbackup
+  scope: Namespaced
+  versions:
+  - name: v1beta1
+    schema:
+      openAPIV3Schema:
+        description: ZookeeperBackup is the Schema for the zookeeperbackups API
+        properties:
+          apiVersion:
+            description: 'APIVersion defines the versioned schema of this representation
+              of an object. Servers should convert recognized schemas to the latest
+              internal value, and may reject unrecognized values. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#resources'
+            type: string
+          kind:
+            description: 'Kind is a string value representing the REST resource this
+              object represents. Servers may infer this from the endpoint the client
+              submits requests to. Cannot be updated. In CamelCase. More info: https://git.k8s.io/community/contributors/devel/sig-architecture/api-conventions.md#types-kinds'
+            type: string
+          metadata:
+            type: object
+          spec:
+            description: ZookeeperBackupSpec defines the desired state of ZookeeperBackup
+            properties:
+              backupsToKeep:
+                default: "7"
+                description: Number of backups to store
+                type: string
+              dataCapacity:
+                default: 1Gi
+                description: Data Storage Capacity
+                type: string
+              dataStorageClass:
+                description: Data Storage Class name
+                type: string
+              image:
+                description: Image for backup procedure
+                properties:
+                  pullPolicy:
+                    description: PullPolicy describes a policy for if/when to pull
+                      a container image
+                    enum:
+                    - Always
+                    - Never
+                    - IfNotPresent
+                    type: string
+                  repository:
+                    type: string
+                  tag:
+                    type: string
+                type: object
+              schedule:
+                default: 0 0 */1 * *
+                description: Schedule in Cron format
+                type: string
+              zookeeperCluster:
+                description: Name of the ZookeeperCluster to backup
+                type: string
+            required:
+            - zookeeperCluster
+            type: object
+          status:
+            description: ZookeeperBackupStatus defines the observed state of ZookeeperBackup
+            type: object
+        type: object
+    served: true
+    storage: true
+    subresources:
+      status: {}
diff --git a/deploy/role.yaml b/deploy/role.yaml
@@ -0,0 +1,40 @@
+apiVersion: rbac.authorization.k8s.io/v1
+kind: Role
+metadata:
+  creationTimestamp: null
+  name: zookeeper-operator.fullname
+rules:
+  - apiGroups:
+      - zookeeper.pravega.io
+    resources:
+      - '*'
+      - zookeeperbackups
+    verbs:
+      - '*'
+  - apiGroups:
+      - ""
+    resources:
+      - pods
+      - services
+      - endpoints
+      - persistentvolumeclaims
+      - events
+      - configmaps
+      - secrets
+    verbs:
+      - '*'
+  - apiGroups:
+      - apps
+    resources:
+      - deployments
+      - daemonsets
+      - replicasets
+      - statefulsets
+    verbs:
+      - '*'
+  - apiGroups:
+      - policy
+    resources:
+      - poddisruptionbudgets
+    verbs:
+      - '*'
diff --git a/go.mod b/go.mod
@@ -19,6 +19,7 @@ require (
 	github.com/gorilla/mux v1.8.0 // indirect
 	github.com/imdario/mergo v0.3.12 // indirect
 	github.com/json-iterator/go v1.1.12 // indirect
+	github.com/mitchellh/hashstructure/v2 v2.0.2
 	github.com/onsi/ginkgo v1.16.4
 	github.com/onsi/gomega v1.14.0
 	github.com/operator-framework/api v0.10.7 // indirect

diff --git a/go.sum b/go.sum
@@ -283,6 +283,7 @@ github.com/docker/go-units v0.4.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDD
 github.com/docker/libtrust v0.0.0-20150114040149-fa567046d9b1/go.mod h1:cyGadeNEkKy96OOhEzfZl+yxihPEzKnqJwvfuSUqbZE=
 github.com/docker/libtrust v0.0.0-20160708172513-aabc10ec26b7 h1:UhxFibDNY/bfvqU5CAUmr9zpesgbU6SWc8/B4mflAE4=
 github.com/docker/libtrust v0.0.0-20160708172513-aabc10ec26b7/go.mod h1:cyGadeNEkKy96OOhEzfZl+yxihPEzKnqJwvfuSUqbZE=
+github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96 h1:cenwrSVm+Z7QLSV/BsnenAOcDXdX4cMv4wP0B/5QbPg=
 github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96/go.mod h1:Qh8CwZgvJUkLughtfhJv5dyTYa91l1fOUCrgjqmcifM=
 github.com/docopt/docopt-go v0.0.0-20180111231733-ee0de3bc6815/go.mod h1:WwZ+bS3ebgob9U8Nd0kOddGdZWjyMGR8Wziv+TBNwSE=
 github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4/go.mod h1:HtrtbFcZ19U5GC7JDqmcUSB87Iq5E25KnS6fMYU6eOk=
@@ -296,6 +297,7 @@ github.com/elastic/go-sysinfo v1.0.1/go.mod h1:O/D5m1VpYLwGjCYzEt63g3Z1uO3jXfwyz
 github.com/elastic/go-sysinfo v1.1.1/go.mod h1:i1ZYdU10oLNfRzq4vq62BEwD2fH8KaWh6eh0ikPT9F0=
 github.com/elastic/go-windows v1.0.0/go.mod h1:TsU0Nrp7/y3+VwE82FoZF8gC/XFg/Elz6CcloAxnPgU=
 github.com/elastic/go-windows v1.0.1/go.mod h1:FoVvqWSun28vaDQPbj2Elfc0JahhPB7WQEGa3c814Ss=
+github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153 h1:yUdfgN0XgIJw7foRItutHYUIhlcKzcSf5vDpdhQAKTc=
 github.com/elazarl/goproxy v0.0.0-20180725130230-947c36da3153/go.mod h1:/Zj4wYkgs4iZTTu3o/KG3Itv/qCCa8VVMlb3i9OVuzc=
 github.com/emicklei/go-restful v0.0.0-20170410110728-ff4f55a20633/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
 github.com/emicklei/go-restful v2.9.5+incompatible/go.mod h1:otzb+WCGbkyDHkqmQmT5YD2WR4BBwUdeQoFo8l/7tVs=
@@ -737,7 +739,10 @@ github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrk
 github.com/mitchellh/go-testing-interface v1.0.0/go.mod h1:kRemZodwjscx+RGhAo8eIhFbs2+BFgRtFPeD/KE+zxI=
 github.com/mitchellh/go-wordwrap v1.0.0/go.mod h1:ZXFpozHsX6DPmq2I0TCekCxypsnAUbP2oI0UX1GXzOo=
 github.com/mitchellh/gox v0.4.0/go.mod h1:Sd9lOJ0+aimLBi73mGofS1ycjY8lL3uZM3JPS42BGNg=
+github.com/mitchellh/hashstructure v0.0.0-20170609045927-2bca23e0e452 h1:hOY53G+kBFhbYFpRVxHl5eS7laP6B1+Cq+Z9Dry1iMU=
 github.com/mitchellh/hashstructure v0.0.0-20170609045927-2bca23e0e452/go.mod h1:QjSHrPWS+BGUVBYkbTZWEnOh3G1DutKwClXU/ABz6AQ=
+github.com/mitchellh/hashstructure/v2 v2.0.2 h1:vGKWl0YJqUNxE8d+h8f6NJLcCJrgbhC4NcD46KavDd4=
+github.com/mitchellh/hashstructure/v2 v2.0.2/go.mod h1:MG3aRVU/N29oo/V/IhBX8GR/zz4kQkprJgF2EVszyDE=
 github.com/mitchellh/iochan v1.0.0/go.mod h1:JwYml1nuB7xOzsp52dPpHFffvOCDupsG0QubkSMEySY=
 github.com/mitchellh/mapstructure v0.0.0-20160808181253-ca63d7c062ee/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
 github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=

diff --git a/pkg/apis/zookeeper/group.go b/pkg/apis/zookeeper/group.go
@@ -0,0 +1,16 @@
+/**
+ * Copyright (c) 2018 Dell Inc., or its subsidiaries. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ */
+
+// Package zookeeper contains zookeeper API versions.
+//
+// This file ensures Go source parsers acknowledge the zookeeper package
+// and any child packages. It can be removed if any other Go source files are
+// added to this package.
+package zookeeper