LSF: Running MPI Jobs on Odyssey

Details about MPI software available on Odyssey can be found here. It also has important information about ensuring your environment is set up correctly to launch MPI jobs.

MPI and LSF are loosely coupled on Odyssey. In order to have your processes launched on the nodes reserved via LSF, you must use the mpirun.lsf command. Here is an example script:

#BSUB -n 16
#BSUB -e err
#BSUB -o out
#BSUB -a openmpi
#BSUB -R "span[ptile=8]"
#BSUB -q short_parallel
mpirun.lsf ./a.out

Here is an explanation of the important BSUB parameters specific to MPI jobs:

  • -n 16
    The number of processing cores to reserve in LSF. Should equal number of MPI ranks unless also using threads.
  • -a openmpi
    This indicates you are using the OpenMPI library. Possible options are: openmpi or mvapich.
  • -R "span[ptile=8]"
    Use this if you want to specify the number of MPI processes per node. So for this example the job would run on two nodes using 8 cores per node. Parameter is optional. Values can range from 1 to 8.

To debug your MPI code you should use Totalview.

CC BY-NC 4.0
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. Permissions beyond the scope of this license may be available at Attribution.