Advanced Parallel Computing for Scientic Applications

Autumn Term 2010 Prof. I. F. Sbalzarini ETH Zentrum, CAB G34 CH-8092 Z urich

Getting started on Brutus

This little reference is just meant to help you getting started on Brutus. It only covers the very basics that are necessary to start working on the exercises. I am sure there will be more questions coming up once you are working. The rst address when looking for answers is the Brutus Wiki, its easy to understand and continuously updated:

http://brutuswiki.ethz.ch (then follow the link to Brutus) Login to Brutus: Please visit brutuswiki.ethz.ch/brutus/Accessing the cluster. Open a terminal (Linux or UNIX) and enter the following command: ssh <yourusername>@brutus.ethz.ch After entering your password, you should now be in your home directory via one of the Brutus login nodes. By using the option -Y when logging in, you set up an X11 session such that opening windows are displayed on your screen. If you are working under Microsoft Windows, you have to install an SSH client, e.g. PuTTY, where you can start an interactive login session. Use brutus.ethz.ch as the Host Name and choose SSH as the Connection Type. As an alternative to using PuTTY, you can install Cygwin, a Linux-like environment for Windows. Copying a le to/from Brutus: Please visit brutuswiki.ethz.ch/brutus/File transfer. When using a Linux or Unix terminal (also via Mac): Type in your local terminal: scp <pathorig>/<origfile> <username>@brutus.ethz.ch:<pathdest>/<destfile> or scp <username>@brutus.ethz.ch:<pathorig>/<origfile> <pathdest>/<destfile> to copy to or from Brutus from or to your local workstation. Or, type in the Brutus terminal: scp <username>@<localhost>:<pathorig>/<origfile> <pathdest>/<destfile> or scp <pathorig>/<origfile> <username>@<localhost>:<pathdest>/<destfile> to copy to or from Brutus from or to your local workstation. The option -c blowfish will speed up le transfer. When working with Microsoft Windows: Clients like FileZilla (also available for Linux or Mac) support secure le transport using a GUI. On Macs only: The application Macfusion allows you to mount your Brutus home in your Finder such that you can open/modify les on Brutus with your local applications. Requires OS X 10.5 Leopard.

Editing a text le on Brutus: You can choose between the text editors vi, nedit, and emacs. NEdit and Emacs will have a GUI if you established an X11 session. NEdit does not work without GUI. Compiling and running your program on Brutus: Please visit brutuswiki.ethz.ch/brutus/Compiling an application and brutuswiki.ethz.ch/brutus/Using the batch system Serial code (no usage of MPI): Go to the directory your source code lies in. Compile with gcc <srcfile> -o <myprog>. Submit the job to the queuing system with bsub ./<myprog>. You can check the status (pending or running) of your job with the command bjobs. Please, DO NOT RUN YOUR APPLICATION ON THE LOGIN NODE! Login nodes are only the gateway to the cluster. You are allowed to run short programs (CPU time < 1 min) interactively (use the -I option) on the login nodes for testing and debugging purposes, or for quick pre- or post-processing. Anything else is prohibited. Parallel code using OpenMP: Todays C and Fortran compiler support OpenMP. To compile an OpenMP source le you can add the -fopenmp ag to the compiler. Before submitting, type export OMP NUM THREADS=<N> where N is the number of threads you like to use. To submit the parallel job to Brutus batch systems, the command looks like follows for N threads: bsub -n N ./<my openmp prog> Parallel code using MPI: Before you compile your code, you have to load the MPI module: module load open mpi (Modules aect only the shell in which the module command is executed and are unloaded when you logout. To see which modules you have loaded, use the command module list.) Now you compile your code like the serial code, you just have to replace the command gcc by mpicc. Submitting the job is a bit dierent because you have to specify that you want to run a parallel program and how many processes you want. The command is bsub -n N mpirun ./<myprog> where N is the number of processes. When you submit a job, the batch system analyzes its requirements (run-time, number of processors, memory, communication network) and dispatches it to the right queue. This process is fully automatic. By default there is a run-time limit of one hour. If you want to execute a longer job, use the option -W HH:MM. Also, use the -W ag if you have a short job that you like to test on Brutus since short jobs are scheduled more often. Submit a shell script: Please submit your shellscript as follows: bsub < script. Dont use: bsub /path/to/script Checking on your jobs: To see all of your current jobs, their status (pending or running), their job-ID: bjobs or bjobs -l for additional details. Impressive: the option -u all shows you all users jobs in the batch system... To kill one of your jobs: bkill <jobID> To kill all your jobs: bkill 0 Once a job is nished you will nd a le called lsf.o<jobID> in your working directory. It contains information about the completed job (when, from where submitted, how long did it run etc.), and the programs output and error messages that have been sent to standard output.

Logout from Brutus: When you want to stop your connection to Brutus, use the command logout or exit. The logout does not work if you have still applications running on the login node, however, your jobs waiting in the queuing system or running on the compute nodes are not aected by and do not inhibit your logout.