NOTE: The interfaces of flux-core are being actively developed and are not yet stable. The github issue tracker is the primary way to communicate with the developers.
flux-core implements the communication layer and lowest level services and interfaces for the Flux resource manager framework. It consists of a distributed message broker and plug-in comms modules that implement various distributed services.
A set of message broker instances are launched as a comms session. Each instance has a rank numbered 0 to (size - 1). Instances are interconnected with three overlay networks: a k-ary tree rooted at rank 0 that is used for request/response messages and data reductions, an event overlay that is used for session-wide broadcasts, and a ring network that is used for debugging requests. Overlay networks are implemented using ZeroMQ.
The message broker natively supports the following services: logging, which aggregates syslog-like log messages at rank 0; heartbeat, which broadcasts a periodic event to synchronize housekeeping tasks; module loader, which loads and unloads comms modules; and reparent, which allows overlay networks to be rewired on the fly.
flux-core also includes the following services implemented as comms modules: kvs, a distributed key-value store; live, a service that monitors overlay network health and can rewire around failed broker instances; modctl, a distributed module control service; barrier, a MPI-like barrier implementation; api, a routing service for clients connecting to a broker instance via a UNIX domain socket; and wreck a remote execution service.
A number of utilities are provided for accessing these services,
accessible via the flux command front-end (see below),
Experimental programming abstractions are provided for various recurring needs such as data reduction, multicast RPC, streaming I/O, and others. A PMI implementation built on the Flux KVS facilitates scalable MPI launch. A set of Lua bindings permits rapid development of Flux utilities and test drivers.
flux-core is intended to be the first building block used in the construction of a site-composed Flux resource manager. Other building blocks are being worked on and will appear in the flux-framework github organization as they get going.
Framework projects use the C4 development model pioneered in the ZeroMQ project and forked as Flux RFC 1. Flux licensing and collaboration plans are described in Flux RFC 2. Protocols and API's used in Flux will be documented as Flux RFC's.
flux-core requires the following packages to build:
zeromq4-devel >= 4.0.4 # built --with-libsodium
czmq-devel >= 3.0.1
munge-devel
json-c-devel
lua-5.1-devel
luaposix
libhwloc-devel >= v1.11.0
# for python bindings
python-devel >= 2.7
python-cffi >= 1.1
# for man pages
asciidoc
Spec files for building zeromq4 and czmq packages on a RHEL 6 based system are provided for your convenience in foreign/rpm.
If you want to build the MPI-based test programs, make sure that
mpicc is in your PATH before you run configure. These programs are
not built if configure does not find MPI.
./autogen.sh # skip if building from a release tarball
./configure
make
make check
A Flux comms session is composed of a set of flux-broker processes
that boostrap via PMI (e.g. under another resource manager), or locally
via the flux start command.
No administrator privilege is required to start a Flux comms session as described below.
To start a Flux comms session (size = 8) on the local node:
src/cmd/flux start --size 8
A shell is spawned that has its environment set up so that Flux commands can find the message broker socket. When the shell exits, the session exits.
To start a Flux comms session (size = 64) on a cluster using SLURM, first ensure that MUNGE is set up on your cluster, then:
srun --pty --mpi=none -N64 src/cmd/flux-broker
The srun --pty option is used to connect to the rank 0 shell. When you exit this shell, the session terminates.
Within a session, the path to the flux command associated with the
session broker will be prepended to PATH, so use of a relative or
absolute path is no longer necessary.
To see a list of commonly used commands run flux with no arguments,
flux help, or flux --help
$ flux --help
Usage: flux [OPTIONS] COMMAND ARGS
...
The flux-core commands are:
help Display manual for a sub-command
keygen Generate CURVE keypairs for session security
start Bootstrap a comms session interactively
kvs Access the Flux the key-value store
...
Most of these have UNIX manual pages as flux-<sub-command>(1),
which can also be accessed using ./flux help <sub-command>.