Reload modules on a multi-arch cluster

[pfermi]

When the head node and the compute node have different microarchitectures, modules loaded in the head node must be reloaded in the compute node to get the right binaries. The command to force this reloading is module update, but if only done once, already loaded modules appear as inactive.

EESSI/2023.06 loaded successfully

Inactive Modules:
  1) FFTW.MPI/3.3.10-gompi-2023b
  2) FFTW/3.3.10-GCC-13.2.0
  3) FlexiBLAS/3.3.1-GCC-13.2.0
  4) GCC/13.2.0
  5) GCCcore/13.2.0
  6) GROMACS
  7) OpenBLAS/0.3.24-GCC-13.2.0
  8) OpenMPI/4.1.6-GCC-13.2.0
  9) OpenSSL/1.1
 10) PMIx/4.2.6-GCCcore-13.2.0
 11) Python-bundle-PyPI/2023.10-GCCcore-13.2.0
 12) Python/3.11.5-GCCcore-13.2.0
 13) SQLite/3.43.1-GCCcore-13.2.0
 14) ScaLAPACK/2.2.0-gompi-2023b-fb
 15) SciPy-bundle/2023.11-gfbf-2023b
 16) Tcl/8.6.13-GCCcore-13.2.0
 17) UCC/1.2.0-GCCcore-13.2.0
 18) UCX/1.15.0-GCCcore-13.2.0
 19) cffi/1.15.1-GCCcore-13.2.0
 20) cryptography/41.0.5-GCCcore-13.2.0
 21) foss/2023b
 22) gfbf/2023b
 23) gompi/2023b
 24) hwloc/2.9.2-GCCcore-13.2.0
 25) libevent/2.1.12-GCCcore-13.2.0
 26) libfabric/1.19.0-GCCcore-13.2.0
 27) libffi/3.4.4-GCCcore-13.2.0
 28) libpciaccess/0.17-GCCcore-13.2.0
 29) libxml2/2.11.5-GCCcore-13.2.0
 30) mpi4py/3.1.5-gompi-2023b
 31) networkx/3.2.1-gfbf-2023b
 32) numactl/2.0.16-GCCcore-13.2.0
 33) pybind11/2.11.1-GCCcore-13.2.0
 34) virtualenv/20.24.6-GCCcore-13.2.0

Instead we have to apply module update twice, getting:

EESSI/2023.06 loaded successfully

Activating Modules:
  1) FFTW.MPI/3.3.10-gompi-2023b
  2) FFTW/3.3.10-GCC-13.2.0
  3) FlexiBLAS/3.3.1-GCC-13.2.0
  4) GCC/13.2.0
  5) GCCcore/13.2.0
  6) GROMACS/2024.3-foss-2023b
  7) OpenBLAS/0.3.24-GCC-13.2.0
  8) OpenMPI/4.1.6-GCC-13.2.0
  9) OpenSSL/1.1
 10) PMIx/4.2.6-GCCcore-13.2.0
 11) Python-bundle-PyPI/2023.10-GCCcore-13.2.0
 12) Python/3.11.5-GCCcore-13.2.0
 13) SQLite/3.43.1-GCCcore-13.2.0
 14) ScaLAPACK/2.2.0-gompi-2023b-fb
 15) SciPy-bundle/2023.11-gfbf-2023b
 16) Tcl/8.6.13-GCCcore-13.2.0
 17) UCC/1.2.0-GCCcore-13.2.0
 18) UCX/1.15.0-GCCcore-13.2.0
 19) cffi/1.15.1-GCCcore-13.2.0
 20) cryptography/41.0.5-GCCcore-13.2.0
 21) foss/2023b
 22) gfbf/2023b
 23) gompi/2023b
 24) hwloc/2.9.2-GCCcore-13.2.0
 25) libevent/2.1.12-GCCcore-13.2.0
 26) libfabric/1.19.0-GCCcore-13.2.0
 27) libffi/3.4.4-GCCcore-13.2.0
 28) libpciaccess/0.17-GCCcore-13.2.0
 29) libxml2/2.11.5-GCCcore-13.2.0
 30) mpi4py/3.1.5-gompi-2023b
 31) networkx/3.2.1-gfbf-2023b
 32) numactl/2.0.16-GCCcore-13.2.0
 33) pybind11/2.11.1-GCCcore-13.2.0
 34) virtualenv/20.24.6-GCCcore-13.2.0

Finally, in order to have this reload done automatically when a job is submitted to Slurm, write the following code to a file and point to it with the BASH_ENV variable. It is important to note that reloading the EESSI module causes an infinite loop if the BASH_ENV variable is set.

#!/bin/bash
#
# This script is sourced by Slurm when launching a job with SBATCH or SRUN.

# BASH_ENV must be unset when loading/reloading the EESSI module to avoid infinite loop
unset BASH_ENV
original_MODULEPATH="${MODULEPATH}"
module -q update
if [[ "${MODULEPATH}" != "${original_MODULEPATH}" ]]; then
  echo "Reloading for architecture ${EESSI_SOFTWARE_SUBDIR}"
  module update
fi
unset original_MODULEPATH