Debugging HPC programs

This page is dedicated to debugging methods for HPC codes. New HPC developers should know these basic options to save time in their work. All methods here provide a way to trace the related bug, which means finding the exact code line that is generating the bug.

If you are new in HPC programming or in debugging, a small tutorial on how to use the following flags is available. See help debug. There are also examples for FPE and Uninitiliazed values debugging. All methods are then based on the same philosophy.

For reference :
Compilers used :

• gcc and gfortran 4.8.2 (from ubuntu 14.04 x86_64)
• icc and ifort 14.0.3 (from Intel Parallel Studio 2013 SP1 x86_64)

Tools used :

• valgrind 3.10.0
• gdb 7.7

Files used to simulate most of common bugs : deb_f.f90 , deb_c.c.

When developing HPC programs, bugs encountered are often the sames. Here is a list of most common bugs :

• Floating point exceptions called fpe (Invalid, Overflow, Zero)
• Uninitialized values reading
• Allocation/deallocation issues
• Array out of bound reading/writing
• IO issues
• Memory leak
• Stack overflow
• Buffer overflow

There are many other types of bugs, but these are the most common and the most easy to solve when using the appropriate tools.

First of all is to identify the presence of a bug :

• Program returns an error message
• Program returns an error exit code (other than 0)
• Program finishes with NaN or +Inf values
• Program ends unexpectedly
• Other cases, many scenario are possible

~$ gfortran myokprog.f90
~$ ./a.out
Hello world !
~$ echo $?
0

~$ gfortran mybugprog.f90
~$ ./a.out
Program received signal SIGSEGV: Segmentation
fault - invalid memory reference.
Backtrace for this error:
#0 0x7FFC993C87D7
#1 0x7FFC993C8DDE
#2 0x7FFC9901FC2F
Segmentation fault (core dumped)
$ echo $?
139

Here is the list of debug flags/tools to use to trace bugs discussed above. First part is generic (Quick debug strategy), while the second part is specific for each bug.

Most of the time, these compilation options will find your bug (except for gcc which has only few debug options) :

If C code, try FPE strategy (see below).

If not enough, compile with :

And launch the program with valgrind :

~$ valgrind myprog.exe

Most of the time it will get the error.

There are three types of FPE :

• Zero : when you divide by zero, very common in HPC. For example : A/0.0=+∞
• Invalid : when the operation is mathematically impossible. For example : acos(10.0) = NaN
• Overflow/Underflow : when you reach maximum/minimum number that system can hold. For example : exp(10E15) = A huge number = +Inf

Behavior : FPE will not generate an error at runtime or at compilation time (GCC/INTEL).

When you try to read a non initialized variable. The program may not stop, and all following calculations will be based on a random value. This is common with MPI programs (Ghosts, etc).
Three main types of initialized variables :

• Static variable : variable uninitialized is static
• Dynamic variable : variable uninitialized is dynamic
• Not allocated variable : try to use a non allocated dynamic variable

Behavior :

• Static variable : no error at runtime
• Dynamic variable : no error at runtime
• Not allocated variable : segmentation fault at runtime

IO errors are often very explicit. No need to use a debugging tool. However, Valgrind and fpe options can detect some related errors (bad reading = bad initialized value or = fpe, etc.)

Do not forget to set -g -fbacktrace (gfortran) or -g -traceback (icc/ifort) to get useful error information.

Simply be careful by securing all read/write (get output code and check it).