TOTALVIEW: Debugging Parallel Applications

Totalview from Rogue Wave Software is a debugging tool particularly suitable for debugging of parallel applications. It provides both X Window-based Graphical User Interface (GUI) and command line interface (CLI) environments for debugging. This page is intended to get you started with Totalview on Odyssey.

To use Totalview on Odyssey, first you need to load the Totalview module-file to set the correct environment settings. This is done most conveniently by placing the below command in your .bashrc startup file:

module load devel/totalview-8.8.0-1

In order to debug MPI parallel applications, you also need to load one of the currently available MPI modules by, e.g.,

module load hpc/openmpi-1.4.2_intel-11.1.072

(or place it in your startup file, e.g., .bashrc for the bash shell).

In order to use Totalview, your code must be compiled with the -g option. It is also recommended to suppress any level of optimization by compiling your application with the -O0 option.

Fortran 77: [username@iliadaccess03 ~]$ mpif77 -g -O0 -o tv_test.x tv_test.f

Fortran 90: [username@iliadaccess03 ~]$ mpif90 -g -O0 -o tv_test.x tv_test.f90

C:          [username@iliadaccess03 ~]$ mpicc -g -O0 -o tv_test.x tv_test.c

C++:        [username@iliadaccess03 ~]$ mpicxx -g -O0 -o tv_test.x tv_test.cpp

Note: The instrumented executable should be used for debugging only, not in production runs. After your code is up and running, for actual production runs you need to recompile your application with the desired level of optimization.

To use Totalview, you need to log in with an X window forwarding enabled. If you access Odyssey from a Unix-like system, you have to use the -X or -Y option to ssh. The -Y option often works better for Mac OS X. For instructions on how to enable X11 forwarding when accessing Odyssey from Windows click here.

ssh -l username odyssey.fas.harvard.edu -X

After loading the Totalview module and compiling with the -g option, request an interactive session, e.g.:

[username@iliadaccess02 ~]$ bsub -Is -q interact -n 4 bash

Job <5229809> is submitted to queue <interact>.

<<Waiting for dispatch ...>>

<<Starting on heroint4>>

Loading module devel/intel-11.1.072.

Loading module hpc/openmpi-1.4.2_intel-11.1.072.

Loading module devel/totalview-8.8.0-1.

[username@heroint4 ~]$

This will start an interactive (bash) shell and load the module-files included in your startup .bashrc file. Then launch the debugger with one of the following commands:

totalview mpirun -a -np 4 ./tv_test.x

or

mpirun -np 4 -tv ./tv_test.x

or

mpirun -np 4 -debug ./tv_test.x

The Totalview startup GUI will pop up and display debugging startup parameters, as illustrated below. After reviewing them, click OK.

Go to the process window, and click the “Go” button.

Note: At this stage Totalview displays the source code of the mpirun function, NOT the source code of your application.

After you click “GO” in the process window, a small window will pop up, asking whether the mpirun process should be stopped. Click “Yes”.

Then, in the “Stack Trace” section of the process window you should see the name of the main program of your application. You can now display the source code by clicking on it. To start debugging, create a break point by clicking on a line number in the source pane, and click “Go”. After that, you can use other buttons (“Next”, “Step”, “Out”, etc).

This page was created by Plamen G. Krastev

Last modified on August 9, 2011 by plamenkrastev@fas.harvard.edu

TOTALVIEW: Debugging Parallel Applications

Contents: