README.no-local-dir

Magpie No Local Directory
-------------------------

Magpie requires local scratch space on each node to be available for
creation of configuration files, log files, and other miscellaneous
files.  This local scratch space is typically /tmp on most systems.

This local scratch space need not be a local disk.  It could
hypothetically be memory based tmpfs.

However, some environments may not have such local scratch space or
the scratch space is extremely small and cannot handle these
additional files.

Begining with Magpie 1.60, an option called MAGPIE_NO_LOCAL_DIR is
available that would allow users to specify network mounted locations
for all the various LOCAL_DIR configuration variables (such as
HADOOP_LOCAL_DIR, HBASE_LOCAL_DIR, etc.).  Magpie will manipulate all
paths to be unique by adding the node name into the path and
configuring all projects to use this path.

However, this requires a patch to several projects, which are as
follows:

Hadoop - Support for no local dir begins with Hadoop 2.2.0.
       - Patches in /patches/hadoop with 'no-local-dir.patch' filename
         suffix.
       - Requires 'alternate-ssh.patch' patch to be applied first.

Spark - Support for no local dir begins with Spark 1.1.1.
      - Patches in /patches/spark with 'no-local-dir.patch' filename
        suffix.
      - Requires 'alternate.patch' patch to be applied first.

Hbase - Support for no local dir begins with 0.98.0.
      - Patches in /patches/hbase with 'no-local-dir.patch' filename
        suffix.
      - Requires 'alternate-ssh.patch' patch to be applied first.

Kafka - Support for no local dir begins with 2.11-0.9.0.0.
      - Patches in /patches/kafka with 'no-local-dir.patch' filename
        suffix.

It is believed no local dir works in remaining projects without
patches.

The submission scripts do not include MAGPIE_NO_LOCAL_DIR
configuration fields by default.  To include them:

cd submission-scripts/script-templates
edit Makefile and set MAGPIE_NO_LOCAL_DIR to 'y'
run 'make'

You should also set LOCAL_DIR_PREFIX to something else other than
/tmp, so the default LOCAL_DIR is the desired network location.

After doing so you can enable MAGPIE_NO_LOCAL_DIR in your submission
scripts.