1 debugging with the totalview source code debugger ed hinkel sales engineer totalview technologies...
TRANSCRIPT
![Page 1: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/1.jpg)
Debugging
with the
TotalView
Source Code Debugger
Ed Hinkel
Sales Engineer
TotalView Technologies
MIT
March 6, 2008
![Page 2: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/2.jpg)
2
Agenda
• TotalView Technologies Intro• Source Code Debugging
- Setup
- Navigation
- Data View and Analysis • Memory Debugging • Parallel Debugging • Debugging Large Apps• Questions / Comments
![Page 3: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/3.jpg)
3
TotalView Technologies Corporate Overview
• The Most Experienced Technologists in Parallel Debugging• Technology originally developed at BBN in late 80’s• Developed from scratch specifically for debugging parallel
applications
• TotalView is recognized worldwide as the gold standard for debugging in multi-core, data intensive, high-performance, distributed, and clustered computing environments
• The debugging leader in the HPC, EDU, and Commercial sectors• Founded as Etnus, Inc. in 1999, Renamed TotalView Technologies
in 2007• 50 employees (heavily engineering influenced)• Over 1,400 customers in 55 countries• Over 10K developers with over 2 million cores under license
• Award winning product line (Supercomputing Online's Product of the Year)
![Page 4: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/4.jpg)
4
What Is TotalView?
![Page 5: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/5.jpg)
5
What is TotalView?
• A comprehensive debugging solution for demanding multi-core applications
• C, C++, Fortran 77 & 90, UPC
• Wide compiler & platform support
• Multi-threaded Debugging
• Parallel Debugging• MPI, PVM, Others
• Remote Debugging
• Memory Debugging Capabilities• Integrated into the Debugger
• Powerful and Easy GUI• Visualization
• CLI for Scripting
![Page 6: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/6.jpg)
6
Supported Compilers and Architectures
• Platform Support• Linux x86, x86-64, ia64, Power• Mac Power and Intel• Solaris Sparc and AMD64• AIX, Tru64, IRIX, HP-UX ia64• Cray X1, XT3, XT4, IBM BGL, BGP, SiCortex
• Languages / Compilers• C/C++, Fortran, UPC, Assembly• Many Commercial & Open Source Compilers
• Parallel Environments• MPI (MPICH1 & 2, LAM, Open MPI, poe, MPT, Quadrics,
MVAPICH, & many others )• UPC
![Page 7: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/7.jpg)
7
Architecture for Cluster Debugging
• Single Front End (TotalView)• GUI• debug engine
• Debugger Agents (tvdsvr)• Low overhead, 1 per node• Traces multiple rank processes
• TotalView communicates directly with tvdsvrs• Not using MPI• Protocol optimization
Compute Nodes
Provides Robust, Scalable and efficient operation with Minimal Program Impact
![Page 8: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/8.jpg)
TotalView Basics
_________________
Startup, Process Control
& Navigation
![Page 9: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/9.jpg)
9
Starting TotalView
Normaltotalview [ tv_args ] prog_name [–a
prog_args ]
Attach to running program totalview [ tv_args ] prog_name –pid PID#
[–a prog_args ]
Attach to remote processtotalview [ tv_args ] prog_name –remote
name [–a prog_args ]
Attach to a core filetotalview [ tv_args ] prog_name
corefile_name [ –a prog_args ]
Command Line GUI
![Page 10: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/10.jpg)
10
Interface Concepts
• Root Window• State of all processes being
debugged
• Process Window• Detailed state of a single
process• Thread within a process
• Point of control• Control the process and
possibly other related processes
![Page 11: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/11.jpg)
11
TotalView Root Window
Host name
Action Point
ID number
Expand - Collapse
ToggleProcess
Status
TotalView
Thread ID #
Rank #
(if MPI program)
Hierarchical/
Linear Toggle
![Page 12: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/12.jpg)
12
Process Window Overview
Stack Trace Pane Stack Frame Pane
Source Pane
Tabbed Area
Toolbar
![Page 13: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/13.jpg)
13
Stack Trace and Stack Frame Panes
Language Name Function Pointer
![Page 14: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/14.jpg)
14
Source Code Pane
![Page 15: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/15.jpg)
15
Viewing Source Code
• TV always tries to display source code• If it cannot you will see assembly
• -g puts ‘symbol table’ and ‘source code + line number’ info into your application• These are references, usually by relative path from the object file to
source file
• TV takes the basename and the path
• TotalView will first try to use this info to find the source file
• Then it will search a TV search path for the basename• Paths can be set via $tree function
• CLI variables provides for setting source search paths - see documentation for details
![Page 16: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/16.jpg)
16
Debugging Assembly Code
Display/Debug Source, Assembly or Both
![Page 17: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/17.jpg)
17
Process Status
![Page 18: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/18.jpg)
18
Stepping Commands
Based on
PC location
![Page 19: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/19.jpg)
19
Basic Process Control
•Control Group–All the processes created or attached together
•Share Group–All the processes that share the same image
•Workers Group–All the processes & threads that are not recognized as manager or service processes or threads
•Lockstep Group–All threads at the same PC
Automatic Grouping
![Page 20: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/20.jpg)
20
Finding Functions, Variables, and Source Files
Menu:
View > Lookup
----------
Accelerator Keys:
f, v
----------
“Closest Match”
Search Results
![Page 21: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/21.jpg)
21
Action Points
Breakpoints
----------
Barrier Points
----------
Conditional Breakpoints
----------
Evaluation Points
----------
Watchpoints
![Page 22: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/22.jpg)
22
Setting Breakpoints
Setting action points
Single left-click outlined source code line numbers
Action Points Tab
Lists all action points
Dive on an action point to focus it in source pane
Action point properties
Context menu when right-clicking the action point
Deleting action points
Left-click in Source Pane
Context menu in Source Pane / Action Points Tab
Disabling action points
Context menu
Left-click in Action Points Tab
Saving all action points
Action Point > Save All
![Page 23: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/23.jpg)
23
Setting Breakpoints
![Page 24: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/24.jpg)
24
Conditional Breakpoint
![Page 25: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/25.jpg)
25
Evaluation Points
• Generalization of Conditional Breakpoints
• C/C++ or Fortran
• Call functions
• Set variables
• Test conditions
• Test small source code patches
• Help set up program circumstances
![Page 26: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/26.jpg)
26
Test Fixes on the Fly
![Page 27: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/27.jpg)
27
Watchpoints
Use Tools > Watchpoint from a
Variable Window.
Watchpoints are set on a fixed
memory region.
When the contents of watched
memory change, the watch-
point is triggered and
TotalView stops the
program. Watchpoints are
not set on a variable. You
you need to be aware of
the variable scope.
Watchpoints can be conditional
or unconditional
Use intrinsic variables $newval
and $oldval in the conditional
expression
![Page 28: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/28.jpg)
28
Using Set PC to Replay Code
![Page 29: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/29.jpg)
29
Help System
• Context sensitive buttons on many dialog windows
• Help menu in the main windows
• Launches an html browser
• Navigate or search the full content
• Also available in pdf and hard copy
• Check out the tip of the week archive
![Page 30: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/30.jpg)
30
TotalView Documentation
![Page 31: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/31.jpg)
TotalView Basics
_________________
Viewing and Editing Data
![Page 32: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/32.jpg)
32
Diving on Variables
You can use Diving to:
… get more information
… to open a variable in a Variable Window.
… to chase pointers in complex data structures.
You can Dive on:
… variable names to open a variable window
… function names to open the source in the Process
Window.
… processes and threads in the Root Window.
How do I Dive?• Double-click the left mouse button on selection• Single-click the middle mouse button on selection.• Select Dive from context menu opened with the right
mouse button
![Page 33: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/33.jpg)
33
Diving on Variables
![Page 34: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/34.jpg)
34
Undiving
In a Process Window: retrace the path that has been explored with
multiple dives.
In a Variable Window: replace contents with the previous contents.
You can also remove changes in the variable window with Edit > Reset Default.
![Page 35: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/35.jpg)
35
Dive in All
Dive in All displays an
element in an array of
structures as if it were a
simple array.
![Page 36: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/36.jpg)
36
The Variable Window
Editing Variables• Click once on the value• Cursor switches into edit more• Esc key cancels editing• Enter key commits a change• Editing values changes the memory of the
program
• Window contents are updated
automatically
• Changed values are highlighted
• “Last Value” column is available
![Page 37: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/37.jpg)
37
Expression List Window
Add to the expression list using contextual menu with right-click on a variable,
or by typing an expression directly in the window
![Page 38: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/38.jpg)
38
Expression List Window
• Reorder, delete, add• Sort the expressions• Edit expressions in place• Dive to get more info
• Updated automatically• Expression-based• Simple values/expressions• View just the values you want to monitor
![Page 39: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/39.jpg)
39
Four Ways to Look at Variables
• Glance
• Stack frame
• Hover
• Source pane
• Dive to data window
• Source, Stack or Variable Window
• Arrays, structures, explore
• Monitor via expression list
• Source, Stack or Variable window
• Keep an eye on scalars and expressions
![Page 40: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/40.jpg)
40
Viewing Arrays
![Page 41: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/41.jpg)
41
Slicing Arrays
Slice notation is [start:end:stride]
![Page 42: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/42.jpg)
42
Filtering Arrays
![Page 43: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/43.jpg)
43
Visualizing Arrays
•Visualize array data using Tools > Visualize
from the Variable Window•Large arrays can be sliced down to a
reasonable size first•Visualize is a standalone program•Data can be piped out to other visualization
tools •Visualize allows to
spin, zoom, etc.•Data is not updated
with Variable Window;
You must re-visualize•$visualize() is a
directive in the
expression system,
and can be used in
evaluation point
expressions.
![Page 44: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/44.jpg)
44
Typecasting Variables
• Edit the type of a variable
• Changes the way TotalView interprets the data in your program
• Does not change the data in your program
• Often used with pointers
• Type cast to a void or code type to snoop around in memory
Give TotalView a starting memory address and
TotalView will interpret and display your memory
from that location.
![Page 45: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/45.jpg)
45
Type Casts Read from Right to Left
Examples:
• int[10]* Pointer to an array of 10 int
• int*[10] Array of 10 pointers to int
• int*[10]* Pointer to an array of 10 pointers to int
• int[5]*[10] Array of 10 pointers to arrays of 5 int
![Page 46: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/46.jpg)
46
Typecasting Examples
• Cast float * to float [100]* to see a dynamic array’s
values
• Cast to built-in types like $string to view a variable as a
null-terminated string (automatic cast for char *)
• Cast to $void for no type interpretation or for displaying
regions of memory
• Cast to $code[100] to see 100 instructions of
disassembly
• Cast to your own structs, objects, Fortran user defined
types, common block definitions, etc.
![Page 47: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/47.jpg)
47
STLView
STLView transforms templates into readable and
understandable information
–STLView supports std::vector, std::list, std::map, std::string
–See doc for which STL implementations are supported
![Page 48: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/48.jpg)
TotalView understands your C++ templates and gives you a
choice ...
Boxes with solid lines around line numbers indicate locations with
replicated code
C++ Templates
![Page 49: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/49.jpg)
49
Managing SignalsFile > Signals
Error Stop the process and flag as errorStop Stop the processResend Pass the signal to the target and do nothing: use with signal handlersIgnore Discard the signal
![Page 50: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/50.jpg)
TotalView Basics
_________________
Memory Debugging
![Page 51: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/51.jpg)
51
What is a Memory Bug?
• A Memory Bug is a mistake in the
management of heap memory
• Failure to check for error conditions
• Leaking: Failure to free memory
• Dangling references: Failure to clear pointers
• Memory Corruption
• Writing to memory not allocated
• Over running array bounds
![Page 52: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/52.jpg)
52
The Agent and Interposition
Heap Interposition
Agent (HIA)
Malloc API
User Code and Libraries
Allocation
TableDeallocati
on
Table
Process
TotalView
![Page 53: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/53.jpg)
53
TotalView HIA Technology
• Advantages of TotalView HIA Technology• Use it with your existing builds
• No Source Code or Binary Instrumentation
• Programs run nearly full speed• Low performance overhead • Efficient memory usage
• Low memory overhead• Support wide range of platforms and compilers
![Page 54: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/54.jpg)
54
Memory Debugger Features
• Automatically detect allocation problems
• View the heap
• Leak detection
• Block painting
• Dangling pointers
• Deallocation/reallocation notification
• Guard Blocks
• Memory Comparisons between processes
• Collaboration features
![Page 55: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/55.jpg)
55
Heap Graphical View
![Page 56: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/56.jpg)
56
Heap Graphical View
![Page 57: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/57.jpg)
57
Leak Detection
• Leak Detection
• Based on Conservative Garbage Collection
• Can be performed at any point in runtime
• Helps localize leaks in time
• Multiple Reports
• Backtrace Report
• Source Code Structure
• Graphically Memory Location
![Page 58: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/58.jpg)
58
Leak Detection
![Page 59: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/59.jpg)
59
Guard Blocks & Memory Corruption
![Page 60: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/60.jpg)
60
Guard Blocks & Memory Corruption
![Page 61: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/61.jpg)
61
Memory Comparisons
• “Diff” live processes
• Compare processes across
cluster
• Compare with baseline
• See changes between point A
and point B
• Compare with saved
session
• Provides memory usage
change from last run
![Page 62: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/62.jpg)
62
Memory Status
![Page 63: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/63.jpg)
63
• What is MemoryScape?• Streamlined• Lightweight• Intuitive• Collaborative• Memory Debugging
• Features• Shows
• Memory Errors
• Memory Status
• Memory Leaks
• Bounds Violations
• MPI Memory Debugging
• Remote Memory Debugging
• Tech• Low Overhead• No Instrumentation
• Interface• Inductive• Collaboration• Multi-process
MemoryScape
![Page 64: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/64.jpg)
64
Script Mode
• Automation Support• MemoryScape lets users run tests and check programs for
memory leaks without having to be in front of the program
• Simple command line program called memscript• Doesn’t start up the GUI
• Can be run from within a script or test harness
• The user defines• What configuration options are active
• What things MemoryScape is looking for
• What actions MemoryScape should take for each type of event
that may occur
![Page 65: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/65.jpg)
TotalView Basics
_________________
Parallel Application Debugging
![Page 66: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/66.jpg)
66
Challenges of Debugging in a Multi-Core Age
• Concurrency• Stochastic errors are often many times harder to solve than others
• Achieving reproducibility of race conditions, deadlocks, live-locks, and other concurrent bugs is the key
• Precise thread-level control of all the processes in the distributed application• If you can’t control the threads then you are simply hoping that the
problem happens
• If you can’t reproduce the problem you can’t easily pose questions about why
• Constructs to enable problem reproduction• Scripting
• Expression Evaluation
• Visibility into all the relevant data• Thread specific stacks and thread-private variables
• Easy ways to view complex data
![Page 67: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/67.jpg)
67
Debugging Multithreaded Programs
When debugging multithreaded programs, you want to:• Know where to look to get thread status.
• Be able to switch the focus from one thread to another quickly
and easily.• Understand how asynchronous thread control commands (step,
go, halt) and breakpoints are used.
A parallel program has a lot more states than ‘Running or
stopped’ There are more degrees of freedom for program
control. TotalView gives you a full set of features to
manage this complexity. It is important to understand how
the different classes of commands and features work so as
to avoid confusion.
![Page 68: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/68.jpg)
68
TotalView Provides
• MPI-Aware Easy launch mechanisms
• Seamless Parallel and Remote Debugging
• Powerful Process Control Features
• MPI Message Queue Display
• High Degree of Scalability
• Scriptability for unattended operation
![Page 69: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/69.jpg)
69
Preparing for debugging
• Compiling Your Application
• Provide TotalView with debug symbols
• Add '-g' to your compile line
• Turn off optimizations
• Remove any optimization flags ‘-o’
![Page 70: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/70.jpg)
70
Starting an MPI Job Within TotalViewStarting an MPI Job Within TotalView
QuickTime™ and aBMP decompressor
are needed to see this picture.
QuickTime™ and aBMP decompressor
are needed to see this picture.
QuickTime™ and aBMP decompressor
are needed to see this picture.
Indirect launch
•Choose MPI implementation•Set parameters
•Enable Memory Debugging•Indicate your MPI
Start from command line
•mpirun -tv -np 4 my_program (mpich)•totalview poe -a -np 4 my_program
![Page 71: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/71.jpg)
71
Running TotalView with SiCortex Applications
• The TotalView Debugger runs as a cross-debugger
within the SiCortex-MIPS Linux environment.
• The SiCortex version is a 64-bit application runs on
a x86- 64 system running a 64-bit kernel.
• Debugging on SiCortex uses the remote features of
TotalView
![Page 72: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/72.jpg)
72
Running TotalView with SiCortex Applications
• TVD needs to execute a command on the target system
from the development host.
• By default, this version uses the ssh -x command.
• It is suggested to use ssh set so that allows password-less
commands.
• The program’s executable file must be visible from both the
development host and the target system.
• Place the executables in a directory that is visible on both machines
through the same path.
• Having the executable visible in separate directories that are
accessed through the same path on both machines will also work.
![Page 73: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/73.jpg)
73
Running TotalView with SiCortex Applications
The SiCortex version of TotalView uses a different
set of naming conventions, using an ”sc” prefix
• sctv8 instead of tv8
• sctototalview instead of totalview
• sctv8cli vs. tv8cli for the Command Line I/F
![Page 74: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/74.jpg)
74
Running TotalView with SiCortex Applications
TVD must debug the MIPS version of srun, not the x86-64 version of srun.
TotalView can be Invoked as follows:
sctv8 -r SiCortex_node ./srun -a srun_arguments
Via the GUI:
• Use the File > New Program dialog box.
• Within the Parallel tab, select SiCortex from the pull- down
list.
(This is the preferred way to start MPI programs from within TVD.)
![Page 75: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/75.jpg)
75
Root Window with MPI
• Status Info
• T = stopped
• B = Breakpoint
• E = Error
• W = Watchpoint
• R = Running
• M = Mixed
• Navigation
• Dive to refocus
• Dive anew to get a second process window
![Page 76: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/76.jpg)
76
Call Graph
• Quick view of
program state
• Each call stack is a path
• Functions are nodes
• Calls are edges
• Labeld with the MPI
rank
• Construct process
groups
• Look for outliers
![Page 77: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/77.jpg)
77
Process Control Concepts
• Each process window is always focused on a specific process.
• Process focus can be easily switched
• Processes can be ‘held’ - they will not run till unheld.
• Breakpoints can be set to stop the process or the group
• Breakpoint and command scope can be simply controlled
![Page 78: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/78.jpg)
Switching Processes
• You can switch the focus of the process window by using the P+ and P- buttons on the toolbar. – P+ takes you to the next process.
– P- takes you to the previous process.
• You can also navigate directly to processes by diving on process entries in the root window.– The next slide describes using the root window in
more detail.
![Page 79: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/79.jpg)
• Process control commands have a 'scope’. For MPI debugging the following scopes are interesting:• Control Group Scope: the entire MPI job, including
starter (if there is a separate starter)
• Share Group Scope: all the rank processes (if you
are focused on a rank process)
• Process Scope: the process that the Process
Window is focused on. • Arbitrary sets of processes can be controlled
by defining a process group with the command line interface. • See later slides and Chapter 11 of the users manual
• Nothing MPI specific about process control
Process Control with MPI
![Page 80: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/80.jpg)
Looking at Variables across Processes
• TotalView allows you to look at
the value of a variable in all MPI
processes
• Right Click on the variable
• Select the View > View Across• TotalView creates an array
indexed by process • You can filter and visualize
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
![Page 81: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/81.jpg)
81
View MPI Message Queues
• Information visible whenever MPI rank processes are halted
• Provides information from the MPI layer• Unexpected messages
• Pending Sends
• Pending Receives
• Use this info to debug• Deadlock situations
• Load balancing
![Page 82: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/82.jpg)
82
Message Queue Graph
• Hangs &Deadlocks
• Pending Messages• Receives
• Sends
• Unexpected
• Inspect• Individual
entries
• Patterns
![Page 83: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/83.jpg)
83
Message Queue Debugging
• Filtering • Tags• MPI Communicators
• Cycle detection • Find deadlocks
![Page 84: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/84.jpg)
84
Large Jobs:Subset Attach
• You can be attached to different subsets at different times through the run
• You can attach to a subset, run till you see trouble and then 'fan out' to look at more processes if necessary.
• This greatly reduces overhead
TotalView does not need to
be attached to the entire job
![Page 85: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/85.jpg)
85
Large Jobs:Strategies
• Reduce N
• Problem: Each process added requires overhead
• Strategy: Reduce the number of processes TotalView is attached to
• Simply reducing N is best, however data or algorithm may require large N
• Technique: subset attach mechanism
• Focus Effort
• Problem: Some debugger operations are much more intensive than others, when multiplied by N this is a big deal
• Strategy: Reduce the interaction between the debugger and the processes
• Technique: Use TotalView's process control features to
• Avoid single stepping
• Focus on one or a small set of processes
![Page 86: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/86.jpg)
86
Large Jobs:Focus on One Process
• If you want to single step through a section of code • Perhaps do it on one process only…
• Set a process-width breakpoint at the beginning
• Once everything you want/expect is lined up, holdone process
• Do a group-width ‘go’ to get all the other processes running
• Unhold that one process and change the width of the commands in the tool bar to process
• Next and step that one process • The other processes are running so they will participate in
communication with your process of interest
![Page 87: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/87.jpg)
87
Thanks!
QUESTIONS?
![Page 88: 1 Debugging with the TotalView Source Code Debugger Ed Hinkel Sales Engineer TotalView Technologies MIT March 6, 2008](https://reader036.vdocuments.us/reader036/viewer/2022062500/56649e885503460f94b8c13b/html5/thumbnails/88.jpg)
88