Skip to content

Commit

Permalink
spack-stack 1.7.0 updates for readthedocs (#730)
Browse files Browse the repository at this point in the history
* initial pass for 1.5.1
* add mapl warning
* r2d2 error note for skylab
  • Loading branch information
ashley314 authored Apr 11, 2024
1 parent 1005424 commit 7c92fa2
Show file tree
Hide file tree
Showing 7 changed files with 33 additions and 17 deletions.
2 changes: 1 addition & 1 deletion docs/FAQ/FAQ.rst
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,7 @@ This is often accompanied by failure of the python tests in ``ioda``. A likely

Conda installs its own packages like ``hdf5``, ``NetCDF``, and ``openssl`` that can conflict with libraries installed via the `spack-stack <https://github.com/jcsda/spack-stack.git>`_. This applies in particular to the IODA Python API, which is now enabled by default in ``ioda``.

These conflicts are not easily addressed since the dependencies are built into ``conda`` through `rpaths <https://en.wikipedia.org/wiki/Rpath>`_. At this time we recommend that you avoid using conda if possible when building and running JEDI applications, and use alternative methods described in the `spack-stack documentation <https://spack-stack.readthedocs.io/en/1.5.1/MaintainersSection.html#testing-adding-packages-outside-of-spack>`_ instead.
These conflicts are not easily addressed since the dependencies are built into ``conda`` through `rpaths <https://en.wikipedia.org/wiki/Rpath>`_. At this time we recommend that you avoid using conda if possible when building and running JEDI applications, and use alternative methods described in the `spack-stack documentation <https://spack-stack.readthedocs.io/en/latest/MaintainersSection.html#testing-adding-packages-outside-of-spack>`_ instead.

Git LFS Smudge error when running ``ecbuild``
---------------------------------------------
Expand Down
4 changes: 2 additions & 2 deletions docs/inside/developer_tools/cmake.rst
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ Installing CMake and CTest
^^^^^^^^^^^^^^^^^^^^^^^^^^

This step is only necessary if you are working outside preconfigured JEDI environments or containers, and when not following the
`spack-stack <https://spack-stack.readthedocs.io/en/1.5.1/>`_ instructions to set up your environment.
`spack-stack <https://spack-stack.readthedocs.io/en/latest/>`_ instructions to set up your environment.

For the Mac, use `homebrew <https://brew.sh/>`_ to install CMake.

Expand Down Expand Up @@ -199,7 +199,7 @@ source directories.
Installing ecbuild
^^^^^^^^^^^^^^^^^^

As before, the steps shown in this section are only necessary if you are working outside preconfigured JEDI environments or containers, and when not following the recommendation to use `spack-stack <https://spack-stack.readthedocs.io/en/1.5.1/>`_ to set up your environment.
As before, the steps shown in this section are only necessary if you are working outside preconfigured JEDI environments or containers, and when not following the recommendation to use `spack-stack <https://spack-stack.readthedocs.io/en/1.7.0/>`_ to set up your environment.

For all systems, you need to have CMake, eigen3 installed before installing ecbuild.
To install these on the Mac:
Expand Down
2 changes: 1 addition & 1 deletion docs/using/jedi_environment/cloud/singlenode.rst
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ After launching the instance through the AWS console, select the instance and cl
[jcsda-usaf-aws-us-east-2]
region = us-east-2
**For AWS Red Hat 8:** After logging in, follow the instructions in https://spack-stack.readthedocs.io/en/1.5.1/PreConfiguredSites.html#amazon-web-services-red-hat-8 to load the basic spack-stack modules for GNU. Proceed with loading the appropriate modules for your application, for example for the ``{skylab_version}`` release:
**For AWS Red Hat 8:** After logging in, follow the instructions in https://spack-stack.readthedocs.io/en/1.7.0/PreConfiguredSites.html#amazon-web-services-red-hat-8 to load the basic spack-stack modules for GNU. Proceed with loading the appropriate modules for your application, for example for the ``{skylab_version}`` release:

.. code-block:: bash
Expand Down
2 changes: 1 addition & 1 deletion docs/using/jedi_environment/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ But JEDI does not exist in a vacuum. Like any modern, sophisticated software pa

In order to help JEDI users and developers quickly create a productive and consistent computing environment, the JEDI team provides a number of portability tools. These include:

* A complete software stack called `spack-stack <https://github.com/jcsda/spack-stack>`_ for compiled and Python dependencies based on the open-source `spack <https://github.com/spack/spack>`_ package manager, originally developed by the `Lawrence Livermore National Laboratory (LLNL) <https://computing.llnl.gov/projects/spack-hpc-package-manager>`_, with `instructions <https://spack-stack.readthedocs.io/en/1.5.1/>`_ for building and using spack-stack on HPC, the cloud, and generic macOS and Linux systems.
* A complete software stack called `spack-stack <https://github.com/jcsda/spack-stack>`_ for compiled and Python dependencies based on the open-source `spack <https://github.com/spack/spack>`_ package manager, originally developed by the `Lawrence Livermore National Laboratory (LLNL) <https://computing.llnl.gov/projects/spack-hpc-package-manager>`_, with `instructions <https://spack-stack.readthedocs.io/en/latest/>`_ for building and using spack-stack on HPC, the cloud, and generic macOS and Linux systems.
* Machine images for cloud computing (e.g. AMIs for `Amazon Web Services <https://aws.amazon.com>`_)
* :doc:`Environment modules <modules>` for selected HPC systems
* Docker and Singularity software :doc:`containers <containers/container_overview>`.
Expand Down
4 changes: 2 additions & 2 deletions docs/using/jedi_environment/modules.rst
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Using spack-stack modules to build and run JEDI
===============================================

The instructions in this section are specific to the use of spack-stack environment modules (``lmod/lua`` or ``tcl/tk``) for building and running JEDI applications. For general information on using spack-stack to build and run software, see the `spack-stack documentation <https://spack-stack.readthedocs.io/en/1.5.1>`_.
The instructions in this section are specific to the use of spack-stack environment modules (``lmod/lua`` or ``tcl/tk``) for building and running JEDI applications. For general information on using spack-stack to build and run software, see the `spack-stack documentation <https://spack-stack.readthedocs.io/en/1.7.0>`_.

One of the big advantages of spack-stack is that it automatically generates modules for all compiled packages and Python packages and works in exactly the same way on HPCs, on the cloud, and on a personal computer. Environment modules are available on basically all HPC systems and any modern macOS or Linux distribution, and are an easy and effective way to manage software libraries. There are two main flavors, the older ``tcl/tk`` modules and the newer ``lmod/lua`` modules, with the latter being superior and therefore preferred, if available. The two implementations share similar commands, such as:

Expand Down Expand Up @@ -36,7 +36,7 @@ General Instructions

This section outlines the general steps to set up spack-stack modules for the desired HPC. Please see :ref:`hpc_users_guide` for more information on how to run jobs on the different HPCs.

1. Load HPC specific modules from `spack-stack pre-configured sites documentation <https://spack-stack.readthedocs.io/en/1.5.1/PreConfiguredSites.html>`_.
1. Load HPC specific modules from `spack-stack pre-configured sites documentation <https://spack-stack.readthedocs.io/en/1.7.0/PreConfiguredSites.html>`_.

2. Load appropriate modules for JEDI and the ``{skylab_v}`` release.

Expand Down
14 changes: 7 additions & 7 deletions docs/using/jedi_environment/spackbuild.rst
Original file line number Diff line number Diff line change
Expand Up @@ -5,38 +5,38 @@ Building spack-stack modules for JEDI

The instructions in this section are specific to building spack-stack environments to support the JEDI applications
({skylab_v} release). For general information on using spack-stack to build and run software, see the
`spack-stack documentation <https://spack-stack.readthedocs.io/en/1.5.1/>`_. Make sure you have read through
`spack-stack documentation <https://spack-stack.readthedocs.io/en/1.7.0/>`_. Make sure you have read through
the documentation before attempting to build your own stack. The commands below only highlight the differences
to the general documentation, steps in between that are not written down here are identical.

Supported in this release are macOS with ``clang`` (``ewok`` graphics dependencies don’t build with ``gcc``), Red Hat with
``gcc``, Ubuntu with ``gcc``, Ubuntu with ``intel``.

Before building the spack-stack modules on macOS, you must first install homebrew and update the local path to your
homebrew installation. See https://brew.sh/ and https://spack-stack.readthedocs.io/en/1.5.1/NewSiteConfigs.html#homebrew-notes
homebrew installation. See https://brew.sh/ and https://spack-stack.readthedocs.io/en/1.7.0/NewSiteConfigs.html#homebrew-notes
for more specific information on this task. It is recommended to start with a fresh homebrew installation before setting up
the spack-stack prerequisites. See https://docs.brew.sh/FAQ#how-do-i-uninstall-homebrew for information on how to uninstall
homebrew from your current environment.

Install the prerequisites:

For macOS, follow the instructions `in the spack-stack docs <https://spack-stack.readthedocs.io/en/1.5.1/NewSiteConfigs.html>`_.
For macOS, follow the instructions `in the spack-stack docs <https://spack-stack.readthedocs.io/en/1.7.0/NewSiteConfigs.html>`_.
Be sure to start at the top of the page and read the notes about the macOS and Intel Arm platform particularly if your machine
has a newer M1 or M2 chip.

Later in the same document you can find prerequisite install instructions for `Red Hat <https://spack-stack.readthedocs.io/en/1.5.1/NewSiteConfigs.html##prerequisites-red-hat-centos-8-one-off>`_.
Later in the same document you can find prerequisite install instructions for `Red Hat <https://spack-stack.readthedocs.io/en/1.7.0/NewSiteConfigs.html##prerequisites-red-hat-centos-8-one-off>`_.

Check out the code:

.. code-block:: bash
git clone -b 1.5.1 --recursive https://github.com/JCSDA/spack-stack spack-stack-1.5.1
git clone -b 1.7.0 --recursive https://github.com/JCSDA/spack-stack spack-stack-1.7.0
Go into the ``spack-stack-1.5.1`` directory and source the spack-stack ``setup.sh`` script:
Go into the ``spack-stack-1.7.0`` directory and source the spack-stack ``setup.sh`` script:

.. code-block:: bash
cd spack-stack-1.5.1
cd spack-stack-1.7.0
source setup.sh
Use the following command to create the spack-stack environment for ``{skylab_version}``:
Expand Down
22 changes: 19 additions & 3 deletions docs/using/running_skylab/running_skylab.rst
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,11 @@ List of spack, software, and AMIs

Versions used:

- spack-stack-1.5.1 from November 2023
- spack-stack-1.7.0 from April 2024

* https://github.com/JCSDA/spack-stack/tree/1.5.1
* https://github.com/JCSDA/spack-stack/tree/1.7.0

* https://spack-stack.readthedocs.io/en/1.5.1
* https://spack-stack.readthedocs.io/en/1.7.0

- AMI available in us-east-1 region (N. Virginia)

Expand Down Expand Up @@ -104,6 +104,10 @@ The commands for loading the modules to compile and run SkyLab are provided in s
Make sure you are building CRTMV3 within the jedi-bundle using the `ecbuild_bundle command <https://github.com/JCSDA-internal/jedi-bundle/blob/5.0.0/CMakeLists.txt#L38>`_.

.. warning::

If you are using ``spack-stack 1.7.0``, different versions of ``mapl`` are used with different variants, depending on the version of the compiler and whether the system is used for UFS or GEOS.
Please reference `spack-stack 1.7.0 documentation <https://spack-stack.readthedocs.io/en/1.7.0/PreConfiguredSites.html>`_ in a note and table under "3.1. Officially supported spack-stack installations" for more information.

.. _build-jedi-bundle:

Expand Down Expand Up @@ -240,6 +244,18 @@ You can then proceed with
cd $JEDI_SRC/simobs
python3 -m pip install -e .
.. note::

If you are using ``spack-stack 1.7.0``, when installing ``r2d2`` you might recieve the following error:

.. code-block::
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
cylc-flow 8.2.3 requires protobuf<4.22.0,>=4.21.2, but you have protobuf 3.20.1 which is incompatible.
Successfully installed protobuf-3.20.1 r2d2-2.3.0
You can ignore this for now and note that is says ``Successfully installed protobuf-3.20.1 r2d2-2.3.0``

Note: You need to run :code:`source venv/bin/activate` every time you start a
new session on your machine.

Expand Down

0 comments on commit 7c92fa2

Please sign in to comment.