ubuntu 9.10

StarCluster AMI Cookbook Ubuntu 9.10

Below are pseudo-instructions for creating the latest StarCluster AMI (Ubuntu 9.10 Karmic) from scratch:

Launch AMI and Install Packages

Launch an alestic ubuntu AMI (x86: ami-bb709dd2, amd64: ami-55739e3c)
Login via ssh as the ubuntu user and switch to root:
```
sudo -i
```
Edit /etc/apt/sources.list and add multiverse to the end of all lines except deb-src lines
Update the package list and upgrade all packages:
```
apt-get update
apt-get upgrade
```

Install the necessary build tools:

apt-get --reinstall install build-essential python-dev swig gfortran -y
apt-get remove python-nose g77 python-setuptools -y

Install and configure blas/lapack:

rm -rvf /usr/lib{,64}/atlas/
apt-get remove liblapack"*" -y
apt-get remove libblas"*" -y
apt-get -y --reinstall install libatlas-base-dev libatlas3gf-base

Install and configure gfortran:

apt-get --reinstall install gfortran-4.2 -y
ln -sf /usr/bin/gfortran{-4.2,}

Remove any existing numpy/scipy installations:

rm -rvf /usr/local/lib/python2.6/*-packages/numpy*
rm -rvf /usr/local/lib/python2.6/*-packages/scipy*

Install and configure the base dependencies:

apt-get install python-dev git-core vim mercurial subversion cvs encfs
openmpi-bin libopenmpi-dev nfs-kernel-server python-django
sun-java6-jdk keychain screen tmux zsh ksh csh tcsh python-pip
python-setuptools python-virtualenv python-imaging python-boto unzip
rar unace build-essential gfortran ec2-api-tools ec2-ami-tools
mysql-server mysql-client apache2 libapache2-mod-wsgi sysv-rc-conf pssh
emacs cython irssi python-distutils-extra python-profiler htop
vim-scripts python-multiprocessing python-multiprocessing-dbg
python-ctypes

Install latest versions of ipython and nose:
```
pip install ipython
pip install nose
```

Configure Root Login ==

The alestic AMI's have been configured to disable root logins. Follow the instructions below to undo this behavior:

Edit /etc/ec2-init/ec2-config.cfg and set disable_root=0
Edit /root/.ssh/authorized_keys and remove prefix commands from pubkey entry (anything before the ssh-rsa...)

Configure MOTD and Bash Completion

Customize motd in /etc/update-motd.d/* and /etc/motd.tail
Uncomment bash completion section in /etc/bash.bashrc

Configure NFS

StarCluster relies on /etc/init.d/nfs existing. Run the command below to ensure this exists:

ln -s /etc/init.d/nfs-kernel-server /etc/init.d/nfs

Configure Sun Grid Engine

Launch one of the 32bit or 64bit StarCluster AMI's and copy /opt/sge6-fresh to /opt/sge6-fresh on your new image host.
Unpack python-drmaa (0.2) to /opt, set SGE6_ROOT='/opt/sge6-fresh' in setup.py and run:
```
python setup.py install
```
Unpack python-drmaa (0.4b3) to /opt and run the following command inside the source folder:
```
python setup.py install
```

Link the DRMAA library to /usr/lib64/libdrmaa.so:

ln -s /opt/sge6-fresh/lib/lx24-{x86|amd64}/libdrmaa.so.1.0 /usr/lib{64}/libdrmaa.so.1.0
ln -s /usr/lib{64}/libdrmaa.so.1.0 /usr/lib{64}/libdrmaa.so

Test that the following commands complete without error:
```
python -c "import DRMAA"
python -c "import drmaa"
```

Rebuild Ubuntu's OpenMPI packages with Sun Grid Engine Support

Install the build deps for libopenmpi-dev:
```
apt-get build-dep libopenmpi-dev
```
Get the source for the libopenmpi-dev debian package:
```
cd /usr/local/src
apt-get source libopenmpi-dev
```
Change into the libopenmpi-dev package's debian folder:
```
cd openmpi-1.3.2/debian
```
Modify the 'rules' file and add --with-sge to the configure arguments:
```
vim rules
(add --with-sge to configure args)
```
Rebuild the libopenmpi-dev package:
```
cd ..
dpkg-buildpackage -rfakeroot -b
```
Install the newly rebuild package:
```
cd ..
dpkg -i *.deb
```

Verify Sun Grid Engine support:

ompi_info | grep -i grid
    MCA ras: gridengine (MCA v2.0, API v2.0, Component v1.3.2)

Configure Optimized Atlas/UMFPACK/AMD/FFTW/Scipy/Numpy

The next sections describe installing an optimized scipy/numpy using Atlas/UMFPACK/AMD/FFTW libraries from source. A lot of these instructions were merged from http://scipy.org/Installing_SciPy/Linux

Install lapack/atlas from source

Checkout npinto-toolbox:

cd /usr/local/src
git clone git://github.com/jtriley/npinto-toolbox
cd npinto-toolbox

Run the ATLAS installer:

./install_atlas3.8.3_Ubuntu9.10-{x86,x86_64}-root.bash

Remove /usr/lib/liblapack.so:
```
rm /usr/lib/liblapack.so
```

Update /usr/lib/gfortran.so symlink:

ln -s /usr/lib/libgfortran.so.3 /usr/lib/gfortran.so

Install UMFPACK/AMD/UFconfig from source

Create a directory to unpack the source into:
```
mkdir /usr/local/src/umfpack
```
Get the latest versions of AMD, UFconfig and UMFPACK and untar them into /usr/local/src/umfpack

Modify /usr/local/src/umfpack/UFconfig/UFconfig.mk to look like (change -m64 to -m32 in the flags below for a 32bit system):

CC = gcc
CFLAGS = -O3 -fexceptions -m64 -fPIC -DNCHOLMOD
F77 = gfortran
F77FLAGS = -O -m64 -fPIC

BLAS = -lptf77blas -lptcblas -latlas -lgfortran
LAPACK = -llapack -lptf77blas -lptcblas -latlas -lgfortran

Run 'make' in the /usr/local/src/umfpack/UMFPACK directory:
```
cd /usr/local/src/umfpack/UMFPACK
make
```

Copy resulting libraries and include files:

cp /usr/local/src/umfpack/AMD/Lib/libamd.a /usr/lib
cp /usr/local/src/umfpack/UMFPACK/Lib/libumfpack.a /usr/lib
cp /usr/local/src/umfpack/AMD/Include/amd.h /usr/include
cp /usr/local/src/umfpack/UFconfig/UFconfig.h /usr/include
cp /usr/local/src/umfpack/UMFPACK/Include/*.h /usr/include

Create /usr/lib/libumfpack.so symlink:

ln -s /usr/lib/libumfpack.a /usr/lib/libumfpack.so

Install FFTW from source

Download fftw-3.2.2 from http://www.fftw.org into /usr/local/src and unpack
Change into the fftw source folder:
```
cd /usr/local/src/fftw-3.2.2
```

Configure fftw (change -m64 to -m32 for 32bit):

export CFLAGS="-O3 -fomit-frame-pointer -fstrict-aliasing -ffast-math -pthread -fPIC -m64"
export FFLAGS="-g -O2 -fPIC -m64"
export CXXFLAGS="-g -O2 -fPIC -m64"
./configure --enable-sse2 --enable-threads --with-combined-threads --prefix=/usr

Build fftw:
```
make install
```

Install Scipy/Numpy from source

Download numpy-1.5.0b1/scipy-0.8.0 into /usr/local/src and untar them

Copy numpy_scipy_site.cfg from npinto-toolbox:

cp /usr/local/src/npinto-toolbox/install_scripts/numpy_scipy_site.cfg /usr/local/src/numpy-1.5.0b1/site.cfg
cp /usr/local/src/npinto-toolbox/install_scripts/numpy_scipy_site.cfg /usr/local/src/scipy-0.8.0/site.cfg

Run the configure check for numpy:

cd /usr/local/src/numpy-1.5.0b1
python setup.py config

Copy the SMCFLAGS from the ATLAS VERSION output and export CFLAGS to be the same as SMCFLAGS
Copy the F77FLAGS from the ATLAS VERSION output and export F77FLAGS to be the same
Install numpy:
```
python setup.py install
```

Run the following command:

python -c "import numpy; numpy.show_config()"

The output of the above command should look something like:

In [3]: import numpy

In [4]: numpy.show_config()
atlas_threads_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = f77

blas_opt_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    language = c

atlas_blas_threads_info:
    libraries = ['ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    language = c

lapack_opt_info:
    libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
    library_dirs = ['/usr/lib']
    define_macros = [('ATLAS_INFO', '"\\"3.8.3\\""')]
    language = f77

lapack_mkl_info:
  NOT AVAILABLE

blas_mkl_info:
  NOT AVAILABLE

mkl_info:
  NOT AVAILABLE

Install scipy:

cd /usr/local/src/scipy-0.8.0
python setup.py install