NOTE: The interfaces of flux-core are being actively developed and are not yet stable. The github issue tracker is the primary way to communicate with the developers.
See also our Online Documentation.
flux-core implements the communication layer and lowest level services and interfaces for the Flux resource manager framework. It consists of a distributed message broker, broker plug-in modules that implement various distributed services, and an API and set of utilities to utilize these services.
flux-core is intended to be the first building block used in the construction of a site-composed Flux resource manager. Other building blocks are also in development under the flux-framework github organization, including a fully functional workload scheduler.
Framework projects use the C4 development model pioneered in the ZeroMQ project and forked as Flux RFC 1. Flux licensing and collaboration plans are described in Flux RFC 2. Protocols and API's used in Flux will be documented as Flux RFC's.
Click to expand and see our full dependency table
flux-core requires the following packages to build:
| redhat | ubuntu | version | note |
|---|---|---|---|
| autoconf | autoconf | ||
| automake | automake | ||
| libtool | libtool | ||
| make | make | ||
| pkgconfig | pkg-config | ||
| glibc-devel | libc6-dev | ||
| zeromq4-devel | libzmq3-dev | >= 4.0.4 | |
| czmq-devel | libczmq-dev | >= 3.0.1 | |
| jansson-devel | libjansson-dev | >= 2.6 | |
| libuuid-devel | uuid-dev | ||
| lz4-devel | liblz4-dev | ||
| libarchive-devel | libarchive-dev | ||
| ncurses-devel | libncurses-dev | ||
| hwloc-devel | libhwloc-dev | >= v1.11.1 | |
| sqlite-devel | libsqlite3-dev | >= 3.0.0 | |
| lua | lua5.1 | >= 5.1, < 5.5 | |
| lua-devel | liblua5.1-dev | >= 5.1, < 5.5 | |
| lua-posix | lua-posix | ||
| python36-devel | python3-dev | >= 3.6 | |
| python36-cffi | python3-cffi | >= 1.1 | |
| python36-yaml | python3-yaml | >= 3.10.0 | |
| python36-jsonschema | python3-jsonschema | >= 2.3.0, < 4.0 | |
| phthon3-sphinx | python3-sphinx | 1 | |
| phthon3-docutils | python3-docutils | 1 |
*Note 1 - only needed if optional man pages are to be created.
The following optional dependencies enable additional testing:
| redhat | ubuntu | version |
|---|---|---|
| aspell | aspell | |
| aspell-en | aspell-en | |
| valgrind-devel | valgrind | |
| mpich-devel | libmpich-dev | |
| jq | jq |
yum install autoconf automake libtool make pkgconfig glibc-devel zeromq4-devel czmq-devel libuuid-devel jansson-devel lz4-devel libarchive-devel hwloc-devel sqlite-devel lua lua-devel lua-posix python36-devel python36-cffi python36-yaml python36-jsonschema python3-sphinx aspell aspell-en valgrind-devel mpich-devel jq
apt install autoconf automake libtool make pkg-config libc6-dev libzmq3-dev libczmq-dev uuid-dev libjansson-dev liblz4-dev libarchive-dev libhwloc-dev libsqlite3-dev lua5.1 liblua5.1-dev lua-posix python3-dev python3-cffi python3-yaml python3-jsonschema python3-sphinx aspell aspell-en valgrind libmpich-dev jq
./autogen.sh # skip if building from a release tarball
./configure
make
make check
If you use VSCode we have a Dev Container provided via the assets in .devcontainer.
Click to expand for more information
You can follow the [tutorial](https://code.visualstudio.com/docs/remote/containers-tutorial) where you'll basically need to:- Install Docker, or compatible engine
- Install the Development Containers extension
Then you can go to the command palette (View -> Command Palette) and select Dev Containers: Open Workspace in Container.
and select your cloned Flux repository root. This will build a development environment from fluxrm/testenv
that are built from src/test/docker (the focal tag) with a few tweaks to add linting and dev tools.
In addition to the usual flux dev requirements, you get:
- bear
- fd
- gdb
- GitHub CLI
- ripgrep
- and several useful vscode extensions in the vscode server instance, pre-configured for lua, c and python in flux-core
You are free to change the base image and rebuild if you need to test on another operating system!
When your container is built, when you open Terminal -> New Terminal, surprise! You're
in the container! The dependencies for building Flux are installed. Try building Flux - it will work without a hitch!
./autogen.sh
./configure --prefix=/usr/local
make
# This will install in the container!
make install
# This will test in the container!
make check
# If you want a compilation database
make clean
./scripts/generate_compile_commands # this runs `bear make check` by default to generate for all tests as wellAnd try starting flux
flux start --test-size=4Note that the above assumes installing flux to /usr/local. If you install elsewhere, you'll need to adjust your
LD_LIBRARY_PATH or similar. IPython is provided in the container for Python development, along with other linting tools.
If you ever need to rebuild, you can either restart VSCode and open in the same way (and it will give you the option)
or you can do on demand in the command palette with Dev Containers: Rebuild Container (with or without cache).
Important it's recommended that you commit (or otherwise write to the .git folder) from the outside of the container. The reason is two-fold - first, if you need to sign your commits, the container doesn't have access and won't be able to. Second, if you write to .git it will change permissions of the directory. If you accidentally do this and need to fix, you can run this from your terminal outside of VSCode:
$ sudo chown -R $USER .git/
# and then commitHopefully we will find a workaround for this so everything works from inside of a VSCode terminal. For the time being, make sure you fix permissions and commit from outside the container on your local machine!
A Flux instance is composed of a set of flux-broker processes
that bootstrap via PMI (e.g. under another resource manager), or locally
via the flux start command.
No administrator privilege is required to start a Flux instance as described below.
To start a Flux instance (size = 8) on the local node for testing:
src/cmd/flux start --size 8
A shell is spawned that has its environment set up so that Flux commands can find the message broker socket. When the shell exits, the session exits.
To start a Flux instance (size = 64) on a cluster using SLURM:
srun --pty --mpi=none -N64 src/cmd/flux start
The srun --pty option is used to connect to the rank 0 shell. When you exit this shell, the session terminates.
Within a session, the path to the flux command associated with the
session broker will be prepended to PATH, so use of a relative or
absolute path is no longer necessary.
To see a list of commonly used commands run flux with no arguments,
flux help, or flux --help
$ flux help
Usage: flux [OPTIONS] COMMAND ARGS
-h, --help Display this message.
-v, --verbose Be verbose about environment and command search
-V, --version Display command and component versions
-p, --parent Set environment of parent instead of current instance
For general Flux documentation, please visit
https://flux-framework.readthedocs.io
Common commands from flux-core:
config Manage/query Flux configuration
cron Schedule tasks on timers and events
dmesg manipulate broker log ring buffer
dump Write KVS snapshot to portable archive
env Print the flux environment or execute a command inside it
exec Execute processes across flux ranks
filemap Map files into a Flux instance
{get,set,ls}attr Access, modify, and list broker attributes
jobs list jobs submitted to Flux
pstree display job hierarchies
uptime Tell how long Flux has been up and running
shutdown Shut down a Flux instance
resource list/manipulate Flux resource status
restore Read KVS snapshot from portable archive
kvs Flux key-value store utility
mini Minimal Job Submission Tool
job Job Housekeeping Tool
overlay Show flux overlay network status
ping measure round-trip latency to Flux services
proxy Create proxy environment for Flux instance
queue Manipulate flux queues
start bootstrap a local Flux instance
startlog Show Flux instance start and stop times
top Display Running Flux Jobs
version Display flux version information
Most of these have UNIX manual pages as flux-<sub-command>(1),
which can also be accessed using ./flux help <sub-command>.
During launch, Flux brokers use a PMI server provided by the launcher to exchange network endpoints and other information. Flux also provides a PMI server embedded in the Flux job shell to support launching parallel applications such as MPI or subordinate Flux instances. In both cases, the PMI version 1 wire protocol is Flux's preferred PMI interface, but other options are available.
See the Flux FAQ for more information on debugging and configuring Flux to interoperate with foreign launchers and different MPI versions. Open a flux-core issue if you encounter a situation that is not covered in the FAQ.
SPDX-License-Identifier: LGPL-3.0
LLNL-CODE-764420