CSI Driver installation failed because of ro /usr

[rainerleber]

I want to use the CSI Driver on Gardener Clusters. Unfortunately the /usr filesystem is set to read only (https://github.com/gardenlinux/gardenlinux/blob/f27744b247fb05b858fbfcfa272196a871094168/features/gardener/README.md)

Previous v1.24 everything is working

+ INSTALL_BLOBFUSE_PROXY=true
+ DISABLE_UPDATEDB=true
+ SET_MAX_OPEN_FILE_NUM=true
+ SET_READ_AHEAD_SIZE=true
+ READ_AHEAD_KB=15380
+ KUBELET_PATH=/var/lib/kubelet
+ [ /var/lib/kubelet != /var/lib/kubelet ]
+ HOST_CMD=nsenter --mount=/proc/1/ns/mnt
+ nsenter --mount=/proc/1/ns/mnt cat /etc/os-release
+ grep ^ID=
+ cut -d= -f2
+ tr -d "
+ DISTRIBUTION=gardenlinux
+ nsenter --mount=/proc/1/ns/mnt uname -m
+ ARCH=x86_64
+ echo Linux distribution: gardenlinux, Arch: x86_64
+ . ./blobfuse-proxy/install-proxy.sh
+ set -xe
+ [ gardenlinux != ubuntu ]
+ echo skip install blobfuse/blobfuse2 for gardenlinux....
+ updateBlobfuseProxy=true
+ [ -f /host/usr/bin/blobfuse-proxy ]
+ [ true = true ]
+ echo copy blobfuse-proxy....
+ rm -rf /host//var/lib/kubelet/plugins/blob.csi.azure.com/blobfuse-proxy.sock
Linux distribution: gardenlinux, Arch: x86_64
skip install blobfuse/blobfuse2 for gardenlinux....
copy blobfuse-proxy....
+ cp /blobfuse-proxy/blobfuse-proxy /host/usr/bin/blobfuse-proxy --force
cp: cannot create regular file '/host/usr/bin/blobfuse-proxy': Read-only file system

What happened:
CSI Driver will not start

What you expected to happen:
CSI Driver is started

How to reproduce it:
Set /usr to ro

Anything else we need to know?:

Environment:

• CSI Driver version: 1.25.0
• Kubernetes version (use kubectl version): 1.29.2
• OS (e.g. from /etc/os-release): Gardener
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[andyzhangx]

hi @rainerleber /usr dir is set as readonly by gardener, right? in the CSI driver setting, we don't set the /usr dir as readonly during volume mount.

[rainerleber]

Hi @andyzhangx yes it was a bit misleading what i want to say is even /usr/local is read only which is mounted in the chart for fuseproxy. I see in the code that from Version 1.24 onwards fuseproxy is enabled by default which try to copy the fuseproxy binary to /usr/local/ (this is not possible in gardener environments because of the ro). Disabeling fuse proxy in the chart is not possible because of the logic in the bash script - wich always try to copy the binary. So my idea to use fuseproxy is: to copy it to a different place and start it from there.

[andyzhangx]

could you set --node.enableBlobfuseProxy=false in helm chart install or set INSTALL_BLOBFUSE_PROXY env as false in driver daemonset on the node? and what's the possible install path in gardener ?

[rainerleber]

The path could be something under var. I try disabeling the blobfuseproxy. The problem why the script always try to install is here i think:

blob-csi-driver/pkg/blobfuse-proxy/install-proxy.sh

Line 77 in 2804061

# install blobfuse-proxy

=> Even in disabled state the script tries to upgrade and copy :-)

At the moment we use 1.23 with disabled fuseproxy but we want to use the fuseproxy for more stability in the future.

[andyzhangx]

@rainerleber is /usr/local/bin path writable in gardener node?

[rainerleber]

@andyzhangx

unfortunately not

[andyzhangx]

@rainerleber then what path is the writable? the dir should be in $PATH list

[rainerleber]

@andyzhangx Thanks for the quick fix.

Is it possible to include an option to change the default directory from /usr/local/bin to e.g. /var/bin? Since the $PATH is defined in the container, the Containerfile could be extended to accommodate this.

[andyzhangx]

@andyzhangx Thanks for the quick fix.

Is it possible to include an option to change the default directory from /usr/local/bin to e.g. /var/bin? Since the $PATH is defined in the container, the Containerfile could be extended to accommodate this.

@rainerleber I think yes, is /var/bin writable on Gardener node?

[rainerleber]

@andyzhangx yes everything/most in tree from /var/.... is writable