Using the High Performance Computing (HPC) systems at Sheffield

Presentation

The presentation given during the session is at https://mikecroucher.github.io/Intro_to_HPC/#/

Description of Sheffield's HPC Systems

The University of Sheffield has two HPC systems:-

• Sharc Sheffield's newest system. It contains about 2000 CPU cores all of which are latest generation.
• Iceberg Iceberg is Sheffield's old system. It contains 3440 CPU cores but many of them are very old and slow.

The two systems are broadly similar but have small differences in the way you load applications using the module system.

We recommend that you use Sharc as much as possible.

Connecting to the HPC systems from Windows

To use the HPC from a Windows machine, you need a way to connect - we recommend you install mobaXterm. This is available from http://mobaxterm.mobatek.net. On a University machine, you need to install the portable version (highlighted in the image below):

You can install the full version if you wish to install mobaXterm on your own machine. (Note that you can connect to iceberg from any location, but will need to be on the University network or use VPN to connect to sharc.)

mobaXterm also contains mobaTextEditor which you can use to write your programs.

Using iceberg

You can connect to iceberg and sharc via mobaXterm:

To run a scala program on a Linux machine, it will need to be compiled using the Scala build tool. This requires a very strict directory structure and a .sbt file specifying dependencies. We illustrate this on the helloWorld example.

Create the directory structure

1. Start by creating the project directory at the top (~) level. We call this hello in this case.
2. Then create the directory structure required by sbt

You can control directory creation via the mobaXterm session:

or you can do everything via the command line by typing the following:

Either way, you need to create the above 4 embedded directories.

Create .sbt file and .scala file on the Windows machine

The .sbt file contains the dependencies required by the program. Take a look at the .sbt file included here for the helloWorld program. The .scala program is also available.

Copy the .sbt file over to the HPC

The .sbt file needs to be placed at the top level of the project.

Copy the .scala file over to the HPC

The .scala file needs to be placed in the scala directory.

Compile the project

Import spark and run program