helm: pod: ceph-csi-rbd-provisioner: CrashLoopBackOff; container: csi-snapshotter: flag provided but not defined: -enable-volume-group-snapshots

[tw-yshuang]

Describe the bug

ceph-csi-rbd-provisioner encounters a CrashLoopBackOff issue when using the custom values.yaml. The values.yaml only modifies the parameter: csiConfig, while all other settings remain at their default values.

Environment details

• Image/version of Ceph CSI driver : quay.io/cephcsi/cephcsi:canary
• Helm chart version : v3.16.2
• Kernel version : Linux 6.8.0-49-generic
• Mounter used for mounting PVC (for cephFS its fuse or kernel. for rbd its
  krbd or rbd-nbd) : krbd
• Kubernetes cluster version : v.1.32.0
• Ceph cluster version : 18.2.2

Steps to reproduce

Steps to reproduce the behavior:

1. Setup details:
   i. Using containerized or host installation method to set a Ceph-Cluster, e.g. mon, osd, etc., related services
   ii. Build a K8s Cluster
   iii. Download the ceph-csi repo, and edit the csiConfig in charts/ceph-csi-rbd/values.yaml:

   csiConfig:
      - clusterID: "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx"
        monitors:
          - "<ceph-node1>:6789"
          - "<ceph-node2>:6789"
          - "<ceph-node3>:6789"
          - ...

2. Deployment to trigger the issue
   execute: helm install --namespace "default" "ceph-csi-rbd" ceph-csi/ceph-csi-rbd -f values.yaml

3. See error
   i. Check the pod:

   kubectl get pod
   # ========================== output ==========================
   NAME                                        READY   STATUS              RESTARTS     AGE
   ceph-csi-rbd-nodeplugin-8nwl7               3/3     Running             0            5s
   ceph-csi-rbd-nodeplugin-dwkz8               3/3     Running             0            5s
   ceph-csi-rbd-nodeplugin-hz5vp               3/3     Running             0            5s
   ceph-csi-rbd-nodeplugin-pj2v2               3/3     Running             0            5s
   ceph-csi-rbd-provisioner-85996b84cb-f24cx   6/7     CrashLoopBackOff    1 (3s ago)   5s
   ceph-csi-rbd-provisioner-85996b84cb-xqldq   6/7     CrashLoopBackOff    1 (3s ago)   5s
   ceph-csi-rbd-provisioner-85996b84cb-zsxxr   6/7     CrashLoopBackOff    1 (3s ago)   5s

   ii. The describe of the ceph-csi-rbd-provisioner:

   kubectl describe pod ceph-csi-rbd-provisioner-xxxxxxxx
   # ========================== output ==========================
    Type     Reason     Age                    From               Message
     ----     ------     ----                   ----               -------
   ...
   Warning  BackOff    5s (x21 over 4m17s)    kubelet            Back-off restarting failed container csi-snapshotter in pod ceph-csi-rbd-provisioner-85996b84cb-f24cx_default(0e24ca96-68cc-400e-ae17-8f7e94c45f7e)

Actual results

The ceph-csi-rbd-provisioner pods will keep restarting and become the CrashLoopBackOff status.
The Static-PVC method is not affected. It can normally work.
Only the snapshot function can not enable.

Expected behavior

The Snapshotter related parameters are using the default setting, so I expect that the snapshotter can successfully created and work.

Logs

If the issue is in snapshot creation and deletion please attach complete logs
of below containers.

• csi-snapshotter and csi-rbdplugin/csi-cephfsplugin container logs from the
  provisioner pod.

  kubectl logs pods/ceph-csi-rbd-provisioner-xxxxxxxx csi-snapshotter
  # ========================== output ==========================
  flag provided but not defined: -enable-volume-group-snapshots
  Usage of /csi-snapshotter:
    -add_dir_header
  ...

Additional context

I notice in the charts/ceph-csi-rbd/values.yaml has a csi-snapshotter related default setting: enableVolumeGroupSnapshots: false.
I am not sure what the relationship is between enable-volume-group-snapshots and enableVolumeGroupSnapshots: false, hope this information can help.

[Madhu-1]

@tw-yshuang what version of helm chart are you using? and also what is the csi-snapshotter version

[tw-yshuang]

@tw-yshuang what version of helm chart are you using?

 - helm version
   version.BuildInfo{Version:"v3.16.2", GitCommit:"13654a52f7c70a143b1dd51416d633e1071faffb", GitTreeState:"clean", GoVersion:"go1.22.7"}

 - csi-snapshotter version
   registry.k8s.io/sig-storage/csi-snapshotter:v8.2.0

[Madhu-1]

sorry i looking for cephcsi helm chart version you are using

[tw-yshuang]

sorry i looking for cephcsi helm chart version you are using

I try the same method on the current devel, release-v3.13 branchs and tag v3.13.0.

[Madhu-1]

sorry i looking for cephcsi helm chart version you are using

I try the same method on the current devel, release-v3.13 branchs and tag v3.13.0.

you should not have problem with devel or release 3.13 branch because we dont set -enable-volume-group-snapshots anymore, can you please check on the chart version you are using

[tw-yshuang]

sorry i looking for cephcsi helm chart version you are using

I try the same method on the current devel, release-v3.13 branchs and tag v3.13.0.

you should not have problem with devel or release 3.13 branch because we dont set -enable-volume-group-snapshots anymore, can you please check on the chart version you are using

In the latest testing environment, the chart version is ceph-csi-rbd-3.13.0, and I am using the default charts/ceph-csi-rbd/values.yaml, but I only changed the csiConfig parameter.