Deployment and running of Spark and Dask.

[bkmgit]

Big data software such as Spark and Dask uses ip addresses and fixed ports that are usually not provided by cluster managers such as SLURM. Deployment of these should be made possible on EESSI, and documentation on their use provided. While it is helpful to have these as modules, self installation in user home directory should also be enabled.

[ocaisa]

This is not related to EESSI (or the software stack in general) but the underlying system and the (network) policies used there. This is not something that EESSI can influence, aside from maybe documenting what tweaking might be required.

[bkmgit]

NERSC does seem to support this, https://docs.nersc.gov/analytics/dask/ not sure what their setup is though.

[bkmgit]

A suggested network policy would be helpful to have. Would initially suggest an internal zone for the compute nodes that has many open ports (if using TCP/IP), but which is separated from communication to the outside world.

[ocaisa]

NERSC does seem to support this, https://docs.nersc.gov/analytics/dask/ not sure what their setup is though.

Sure, many sites support Dask, I'm just saying that it is dependent on network policies (and these can even differ between login and compute nodes).