net debugging techniques
DESCRIPTION
Advanced .Net Debugging of native (unmanaged) and Managed CLR code using NTSD, SOS, SosEx, KD.TRANSCRIPT
.NET Debugging Techniques
Bala Subra
Software Bugs are Expensive
Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.- Brian Kernighan
Why Debugging?
We need Reliable Software Users Choose Reliability over Price “Software bugs or errors are so prevalent and
so detrimental that they cost the US economy an estimated $59 billion annually, or about 0.6 percent of the GDP.” www.nist.gov/public_affairs/releases/n02-10.htm
Most Commonly Found Defects
http://scan.coverity.com/report/Coverity_White_Paper-Scan_Open_Source_Report_2009.pdf
Technical Requirements
Understand unmanaged (native) vs. Managed debugging.
Explore production-environment debugging. Deep-dive into CLR fundamentals. Understand .NET memory management. Debug hangs, crashes, and memory leaks.
Agenda
Importance of debugging Available Tools Basic Tasks and Walkthroughs Postmortem Debugging
Common Traits for good debuggers Willingness to venture outside of your "own"
code Curiosity Patience Treat dependent code as just that - code Ability to see patterns
Importance of debugging
Perfect code is an Illusion Legacy Code Deeper Understanding Helps you learn & write better code in the
future
Debugging Basics What are you trying to find
and fix? Two main types of code
errors Syntax: Compiler catches
most if not all of these for you.
Semantic or logical: Syntactically correct yet program may “crash and burn” at run-time!
Autos Locals Watch Call Stack Command Window QuickWatch Dialog Breakpoints window Threads Modules Processes Memory Disassembly Registers
Execution Control: Breakpoints Stepping through your
code Starting / Stopping Breaking Stepping through your
application (F10, F11 or Toolbar
buttons) Run to a specific location
Run To Cursor (right-click menu)
Situations under which breakpoints are not feasible Timing sensitive issues Breakpoint triggers too
often Live Debug not possible Debugging production
systems
Tools available for Debugging .NET Visual Studio 2008 (& soon 2010)
2010: Historical Debugging Visualizations for locks, threads
CorDBG / MDbg Debugging Tools for Windows
Focus for Today Used by everybody for everybody
Visual Studio 2008: Debugger Tips and Tricks Execution Control Breakpoints Symbols Multi-Threaded debugging Post Mortem Debugging Behind the Debugger Magic
VS Debugger Overview
Debugs many different code Environments Native Windows
X86 X64 IA64
Managed Code Windows (32 & 64 bit) SQLCLR
Script T-SQL Native Device Programs
VS Debugger Architecture
VSDebugPackageVSDebugPackage
SDMSDM
CPDE(Managed)
CPDE(Managed)
NatDbgDE(Native)
NatDbgDE(Native)
Your EngineHere
Your EngineHere
Your EE HereYour EE Here
http://msdn.microsoft.com/en-us/library/bb161718.aspx
Execution Control
Step Filtering for Managed Code Adds support for automatically stepping over
simple properties Right click to “Step into Specific”: pick Step into
Target Switch off: How?
Breakpoints Tracepoints
Print a variety of program state types without stopping Program Location, including Stack Expressions (including @clk in native for quick timing) Thread Info <Your Message Here>
Run a Macro Useful on stop on certain condition that is difficult to
express as a ‘Watch window’ expression. For eg. “Stop if this method is in Call Stack”
Visual Studio 2010
Tagging Filtering Import from others & Export Breakpoints
Symbols
Ensure Symbols are switched on in Final/Retail/Optimized Builds
Archive Symbols using Symbol Server VS 2010
Team Build support for Symbol Server Add Symbol & Source indexing into the Build’s
Workflow
Symbols: Reference Source Support
Symbols: Visual Studio Options
Symbol Loading Internals VS Never loads Mismatched symbols Path Plan:
Where the EXE think it is On the path we create from the “Symbols” Dialog On the path at HKLM/HKCU
Software\Microsoft\VisualStudio\MSPDB\SymbolSearchPath
On the path @ Any of these: _NT_ALT_Symbol_Path _NT_Symbol_Path SystemRoot
Threads Thread Categories Flagging Threads in the List (for tracking) Using Breakpoint Filters Freezing & Thawing Stack Tips Naming Threads
Managed: Thread.Name Native: Use the SetThreadName execution
wrapper http://blogs.msdn.com/stevejs/archive/2005/12/19/
505815.aspx
Post Mortem Analysis
Windows Error Reporting http://winqual.microsoft.com http://msdn.microsoft.com/en-us/library/aa939342.a
spx
Debugging Open the Disassembly Window Open the Autos window to see pertinent Registers http://blogs.msdn.com/greggm/archive/2004/12/15/315673.
aspx
Trust Statics/Globals Trust Stacks when you have Symbols
VS2010: Managed/Interop Support Support for reading & writing Minidumps from
Processes with Managed Code Support for Mixed Mode debugging on x64
How does the Debugger do Minidumps Use dbgHelp.dll Method:
MiniDumpReadDumpStream to read streams from MiniDump File
Read the following Streams SystemInfoStream ThreadListStream ModuleListStream MemoryListStream
Create Container Objects in the debugger that wrap the instances from the MiniDump
Wrap memory as needed by StackWalking or Data Inspection
Benefits of Debugging Tools for Windows Small Footprint
XCopy Enabled Ideal for debugging problems on machines that are
locked down Frequent releases
Updated for new versions of Windows Which debuggers does it include?
User Mode Debuggers: windbg/ntsd/cdb Kernel Mode Debugger: kd
Powerful Extensions & Instrumentation Extensible by us
Debugging: Package Content
Symbol Indexing Tools Source Indexing Tools Stand-alone Tools
AgeStore AdPlus BreakIN DbgSrv GFlags TList Remote
Installing Debugging tools for Windows Download Point:
www.microsoft.com/whds/devtools/debugging/default.msp
Default options sufficient By Default installs into
C:\Program Files\debugging Tools for Windows Directory Listing
Debugger Interaction: 1st Steps Command Mode or GUI? User mode prompt can get us a Head Start Get the Exception Code Understand the Environment Set the Correct Symbols Start from the Current Execution Context Check the Loaded Module
Basic Tasks : Running the debugger Attaching to Process
By Process ID: -p <process id> By Process Name: -pn <process name> TList Command
Running Under the Debugger NTSD.EXE <Command Line> NTSD.EXE C:\Windows\notepad.exe Caveats: Various Components may go into Debug
Mode
Demos
Working with the Target Last Event Registers Memory Variables Stack Unassemble Process Information Thread Information Address Information
Basic Tasks : Symbols
Additional Metadata about the Code Managed Types far more self-descriptive Private vs Public Symbols Microsoft Public Symbol Server How to tell the debugger the Location
Pointing to MS Public Server: .symfix Pointing to additional Paths: .sympath+ Reloading Symbols: .reload
Custom Symbol Servers
Walkthrough for Symbols
Symbol Server Large Store of Symbol & Binary Files Files are organized based on properties:
Name Type Time stamp Size of the Image RSDS Signature
Binary files can be stored in different location Files can be compressed
Building a Symbol Server Tools:
PdbCopy.exe BinPlace.Exe (WDK) SymStore.exe Extending the build process (Batch Files)
Basic Tasks : SOS
Powerful managed code debugger extension Introspect on the internal state of the CLR Son of Strike
Loading SOS .NET 2.0 : .loadby sos mscorwks .NET 4.0 : .loadby sos clr
Help Command !help displays all commands !help <command> displays help for specific command
SOSEX is another useful debugger extension http://www.stevestechspot.com
Debugger Extension walkthroughs
Basic Tasks : Thread
Basic Tasks : Thread Basic Unit of code Execution Before you launch a new thread, think twice Sync-blocks
Plethora of information about objects SOS Thread commands
!Threads: List all managed Thread !ClrStack: Displays Managed Callstack for currently active
Thread ~<ThreadNum>s: Switches currently active Thread ~*e!ClrStack: Shows Callstack for all managed threads !syncBlk
Deadlock problem walkthrough
Basic Tasks: Managed Heap & Garbage collection Automatic Memory Management
Sits on top of the Windows Memory Manager Currently consists of 3 generations (0, 1, 2) Caveat: Native resources must be explicitly cleaned up
SOS Commands DumpHeap DumpObj (do) GCRoot
Visualizing Runtime Object Graphs http://www.lovettsoftware.com/blogengine.net/post/
2010/01/15/Visualizing-Runtime-Object-Graphs.aspx
Walkthrough for managed heap commands
Resource Leaks What is a Resource?
Handles File Object Process Object Thread Object Isolation layer between User Mode code & Kernel
Synchronization Primitives Heap Memory Allocator Virtual Memory Allocator COM Allocator
Tools for Heap Memory Tracking UMDH
Tracks Heap based Memory Requires OS Instrumentation to be Enabled (gflags)
LeakDiag Uses Microsoft Detours Library Tracks different types of Memory Allocators
Heap Allocator Virtual Memory Allocator COM Allocator C Runtime Allocator
Debugger Command: !heap Static Source Code Analysis Tools: Prefast (WDK)
Memory Leaks: C Run-Time Functions _CrtDumpMemoryLeaks()
Performs leak checking where called. You want to place this call at all possible exits of your app.
_CrtSetDbgFlag () Sets debugging flags for the C run-time library. _CRTDBG_REPORT_FLAG Gets current flag(s) _CRTDBG_LEAK_CHECK_DF Perform
automatic leak checking at program exit through a call to _CrtDumpMemoryLeaks
Memory Leaks: Visual Studio _CRTDBG_MAP_ALLOC_
Memory allocation number (inside curly braces) Block type (normal, client or CRT) Memory location in hex Size of block in bytes Contents of the first 16 bytes in hex File name Line number
Heap Corruptions Violate the Integrity of Memory allocated on
the Heap Stray Pointers Overruns Underruns Over-Deletion Reuse after Deletion
One of the toughest problem to Debug
Windows Memory Architecture
ApplicationApplication
Virtual Memory ManagerVirtual Memory Manager
Heap ManagerHeap Manager
Default Process
Heap
Default Process
Heap
C RuntimeHeap
C RuntimeHeap Other HeapsOther Heaps
Heap Block Structure
Current Size
Current Size
Previous Size
Previous Size
SegIndexSeg
Index FlagsFlags UnusedUnused Tag IndexTag
Index
Pre-allocation MetadataPre-allocation Metadata Post-allocation MetadataPost-allocation Metadata
User accessible partUser accessible part
Pre-allocation Metadata
Suffix BytesSuffix Bytes
Fill Area(debug)Fill Area(debug)
Heap ExtraHeap Extra
Post-allocation Metadata
User accessible partUser accessible part
User accessible partUser accessible part
Tools for Debugging Heap Corruptions
Goal is to Break when the corruption occurs AND not after
PageHeap helps with that goal Annotates heap blocks to trigger fault at the time
of write Light PageHeap uses Fill Patterns Full PageHeap uses Fill Patterns and Guard
Pages Very Memory Intensive
Postmortem Debugging
Scenarios Live debugging not feasible Reproducing the problem is difficult
Static Snapshot of a Live Process Use the same debugger to debug offline Limitations
It is a snapshot; so you can't control execution Depending on type of dumpfile, some SOS
commands may not work.
Postmortem Debugging: How to generate dumpfiles Using the debuggers
.dump /mf c:\CoreDump.dmp Automatic
ADPlus Windows Error Reporting
https://winqual.microsoft.com Available to everyone
Windows Error Reporting (WER) Architecture
Error SentError SentDr.
WatsonDr.
WatsonProcessCrash
ProcessCrash
Crash data over HTTPSCrash data over HTTPS
Fault response over HTTPS
Windows Error Reporting Service
ISVISV
Query Fault Data
Postmortem Debugging: How to debug Dumpfiles? Slightly different than Native code debugging
The data Access Layer (DAC) Implemented in mscordacwks.dll Different for each version of the CLR
Debugging Dump files use the -z switch with path to the dump file
Walkthroughs with ADPlus & Postmortem debugging
When Not to use Native Debugging During Code Development Tracing the Code 100% managed code Need frequent variable inspection Need frequent references to the source files Debugging Partial Dumps Kernel Mode Debugging
Some pages are paged out
Summary
Importance of debugging Be aware of Magic Tools available for Debugging .NET Basic debugging Tasks
Running the debuggers
Questions?
Books Advanced .NET Debugging: Mario Hewardt Windows Internals: Mark E. Russinovich, David A.
Solomon with Alex Ionescu Windows via C/C++: Jeffrey M. Richter,
Christophe Nasarre Blogs
http://blogs.msdn.com/ms_joc/ http://www.wintellect.com/cs/blogs/jrobbins/default
.aspx