LSF: Running MPI Jobs on Odyssey

 

Details about MPI software available on Odyssey is here. It also has important information about ensuring your environment is set up correctly to launch MPI jobs.

MPI and LSF are loosely coupled on Odyssey. In order to have your processes launched on the nodes reserved via LSF, you must use the mpirun.lsf command. Here is an example script:

#!/bin/csh
#BSUB -n 16
#BSUB -e err
#BSUB -o out
#BSUB -a openmpi
#BSUB -R "span[ptile=8]"
#BSUB -q short_parallel
mpirun.lsf ./a.out


Here is an explanation of the important BSUB parameters specific to MPI jobs:

  • -n 16
    The number of processing cores to reserve in LSF. Should equal number of MPI ranks unless also using threads.
  • -a openmpi
    This indicates you are using the OpenMPI library. Possible options are: openmpi or mvapich.
  • -R “span[ptile=8]”
    Use this if you want to specify the number of MPI processes per node. So for this example the job would run on two nodes using 8 cores per node. Parameter is optional. Values can range from 1 to 8.


To debug your MPI code you should use Totalview. Look at our FAQ on debugging parallel code for more information.

Site last updated April 18, 2014