net debugging workshop
DESCRIPTION
Workshop from DevConnections 2014 covering .NET debugging in production environments.TRANSCRIPT
![Page 1: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/1.jpg)
.NET Debugging Workshop
#devconnections
![Page 2: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/2.jpg)
SESSION TITLE
#devconnections
Sasha Goldshtein
CTO, Sela GroupMicrosoft C# MVP, Azure MRS@goldshtn blog.sashag.net
#devconnections
![Page 3: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/3.jpg)
.NET DEBUGGING WORKSHOP
In This Workshop…• Debugging issues in production
environments• Automating triage and analysis
processes• Analyzing system and application
performance#devconnections
![Page 4: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/4.jpg)
.NET DEBUGGING WORKSHOP
Production Debugging• Requirements– Obtain actionable
information about crashes and errors
– Obtain accurate performance information
• Limitations– Can’t install Visual
Studio– Can’t suspend
production servers– Can’t run intrusive
tools
#devconnections
![Page 5: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/5.jpg)
.NET DEBUGGING WORKSHOP
In the DevOps Process…• Automatic build (CI)• Automatic deployment (CD)• Automatic monitoring• Automatic error triage and
analysis• Automatic remediation#devconnections
![Page 6: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/6.jpg)
.NET DEBUGGING WORKSHOP
The Tools• Sysinternals Procdump• DebugDiag• Windows SDK– Debugging Tools for Windows–Windows Performance Toolkit
• PerfView#devconnections
![Page 7: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/7.jpg)
Dump Files
#devconnections
![Page 8: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/8.jpg)
.NET DEBUGGING WORKSHOP
Dump Files• A user dump is a snapshot of a running
process• A kernel dump is a snapshot of the entire
system• Dump files are useful for post-mortem
diagnostics and for production debugging– Anytime you can’t attach and start live
debugging, a dump might help
#devconnections
![Page 9: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/9.jpg)
.NET DEBUGGING WORKSHOP
Limitations of Dump Files• A dump file is a static snapshot– You can’t debug a dump, just analyze it– Sometimes a repro is required (or more
than one repro)
• Sometimes several dumps must be compared
#devconnections
![Page 10: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/10.jpg)
.NET DEBUGGING WORKSHOP
Taxonomy of Dumps• Crash dumps are dumps generated
when an application crashes• Hang dumps are dumps generated
on-demand at a specific moment• These are just names; the contents
of the dump files are the same!#devconnections
![Page 11: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/11.jpg)
.NET DEBUGGING WORKSHOP
Generating a Hang Dump• Task Manager,
right-click and choose “Create Dump File”– Creates a dump in
%LOCALAPPDATA%\Temp
#devconnections
![Page 12: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/12.jpg)
.NET DEBUGGING WORKSHOP
Procdump• Sysinternals utility for creating dumps• Examples:
Procdump -ma app.exe app.dmpProcdump -ma -h app.exe hang.dmpProcdump -ma -e app.exe crash.dmpProcdump -ma -c 90 app.exe cpu.dmpProcdump -m 1000 -n 5 -s 600 -ma app.exe
#devconnections
![Page 13: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/13.jpg)
.NET DEBUGGING WORKSHOP
Windows Error Reporting• WER can create dumps automatically– HKLM\Software\Microsoft\Windows\
Windows Error Reporting\LocalDumps
– http://tinyurl.com/localdumps
• Can be application-specific, not system-wide
#devconnections
![Page 14: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/14.jpg)
.NET DEBUGGING WORKSHOP
DebugDiag• Microsoft tool for
monitoring and dump generation– Very suitable for
ASP.NET– Dump analysis
component included
#devconnections
![Page 15: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/15.jpg)
.NET DEBUGGING WORKSHOP
Debugging Symbols• Debugging symbols link runtime
memory addresses to function names, source file names and line numbers– PDB files– Required for proper debugging and
dump analysis#devconnections
![Page 16: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/16.jpg)
.NET DEBUGGING WORKSHOP
Symbols for Microsoft Binaries
• Microsoft has a public symbol server with PDB files for Microsoft binaries
• Configure _NT_SYMBOL_PATH environment variable
setx _NT_SYMBOL_PATH srv*C:\symbols*http://msdl.microsoft.com/download/symbols
#devconnections
![Page 17: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/17.jpg)
.NET DEBUGGING WORKSHOP
Opening Dump Files• Visual Studio can
open dump files– For .NET, CLR 4.0+
and VS2010+ required
– VS2013 recommended
#devconnections
![Page 18: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/18.jpg)
.NET DEBUGGING WORKSHOP
Opening Dump Files• WinDbg is a free
lightweight debugger
• No intrinsic .NET support, but has SOS debugging extension
!analyze -v (CLR 4.0+).loadby sos clr!printexception!clrstack
#devconnections
![Page 19: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/19.jpg)
.NET DEBUGGING WORKSHOP
TRY IT OUT
Configuring LocalDumpsObtaining and opening a dump file
#devconnections
![Page 20: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/20.jpg)
Automatic Dump Analysis
#devconnections
![Page 21: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/21.jpg)
.NET DEBUGGING WORKSHOP
Basic Automation• Run WinDbg automatically on a
bunch of files and log its output:@echo offfor %%f in (.\*.dmp) do (
echo Launching analysis of file %%f...start "Analyzing %%f" "C:\Program Files (x86)\
Windows Kits\8.1\Debuggers\x86\cdb.exe" -z %%f -c ".logopen %%f.log; !analyze -v; .logclose; qd")
#devconnections
![Page 22: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/22.jpg)
.NET DEBUGGING WORKSHOP
Basic Automation
• Parse the results for interesting tokens:for %%f in (.\*.dmp.log) do ( echo In file %%f: findstr "EXCEPTION_MESSAGE MANAGED_OBJECT_NAME" %%f)
#devconnections
![Page 23: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/23.jpg)
.NET DEBUGGING WORKSHOP
ClrMD• Text-based analysis of debugger
command output is very fragile and limited
• ClrMD is a .NET library for analyzing dump files (and running processes)– A managed API for interacting with
the .NET debugging runtime (“SOS API”)– Distributed through NuGet (search “ClrMD”)
#devconnections
![Page 24: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/24.jpg)
.NET DEBUGGING WORKSHOP
ClrMD Basic Classes
#devconnections
DataTargetDataTarget
ClrRuntimeClrRuntime ClrRuntimeClrRuntime
ClrHeapClrHeap ClrThreadClrThread
ClrTypeClrType ClrTypeClrType ClrThreadClrThread
![Page 25: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/25.jpg)
.NET DEBUGGING WORKSHOP
mscordacwks.dll• Managed dump analysis requires
mscordacwks.dll matching the CLR version
• It can be automatically downloaded from the Microsoft symbol server in most cases
#devconnections
![Page 26: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/26.jpg)
.NET DEBUGGING WORKSHOP
Connecting to a Target
#devconnections
![Page 27: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/27.jpg)
.NET DEBUGGING WORKSHOP
Basic Exception Triage
#devconnections
![Page 28: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/28.jpg)
.NET DEBUGGING WORKSHOP
TRY IT OUTGetting stacks from a live process
#devconnections
![Page 29: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/29.jpg)
.NET DEBUGGING WORKSHOP
Inspecting the Heap• Enumerate all heap
objects and statistics• Find specific objects• Inspect GC
information (roots, finalization queues, etc.)
#devconnections
ClrHeapEnumerateObjectsGetObjectTypeEnumerateRoots
ClrTypeGetSizeEnumerateRefsOfObjectGetFieldValue
![Page 30: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/30.jpg)
.NET DEBUGGING WORKSHOP
Wait Information• Threads have a list
of blocking objects, which have owner threads
• Wait analysis and deadlock detection is made possible
ClrThreadBlockingObjects
BlockingObjectReasonObjectHasSingleOwnerOwner/OwnersWaiters
#devconnections
![Page 31: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/31.jpg)
ETW and PerfView
#devconnections
![Page 32: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/32.jpg)
.NET DEBUGGING WORKSHOP
Event Tracing for Windows• High-performance facility for emitting
100K+ log events per second with rich payloads and stack trace support
• Used widely across Windows, .NET, drivers, services, third party components
#devconnections
![Page 33: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/33.jpg)
.NET DEBUGGING WORKSHOP
ETW Participants
• A provider generates ETW events• A controller starts and stops ETW
collection• A consumer logs, analyzes, or
processes ETW events
#devconnections
![Page 34: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/34.jpg)
.NET DEBUGGING WORKSHOP
ETW Scenarios• Profile an app in sampling mode• Perform wait-time analysis• Log disk accesses including stacks• Log GC and JIT events• Log memory allocation statistics (C++)• Custom application event log
#devconnections
![Page 35: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/35.jpg)
.NET DEBUGGING WORKSHOP
Custom ETW Events
#devconnections
![Page 36: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/36.jpg)
.NET DEBUGGING WORKSHOP
ETW Tools• xperf.exe: Command-line tool for ETW
capturing and processing• wpr.exe: Command-line and GUI for
end users• wpa.exe: Visual trace analysis tool• PerfView.exe: Visual tool for capturing
and recording ETW events from managed providers and the CLR
#devconnections
![Page 37: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/37.jpg)
.NET DEBUGGING WORKSHOP
Capturing a Trace• Xperf
xperf -on DiagEasy...xperf -d diag.etl
• WPR
#devconnections
![Page 38: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/38.jpg)
.NET DEBUGGING WORKSHOP
What’s In A Trace?• A trace is a huge list
of events• Events have multiple
columns (payload)• Useless without
additional processing
#devconnections
![Page 39: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/39.jpg)
.NET DEBUGGING WORKSHOP
Trace Processing with Xperf• I/O summary report
per file
xperf -i fileio.etl-o fileio.csv -a diskio -summary
• Interactive profiling report (for a specific process)xperf -i cpu.etl-o cpu.html -symbols-a stacks -process app.exe -butterfly
#devconnections
![Page 40: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/40.jpg)
.NET DEBUGGING WORKSHOP
Managed Stacks• To display managed stack traces
correctly, additional CLR data is required• WPR & PerfView take care of this
automatically• If using Xperf, see:
http://msdn.microsoft.com/en-us/library/windows/desktop/hh448186.aspx
#devconnections
![Page 41: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/41.jpg)
.NET DEBUGGING WORKSHOP
TRY IT OUTCollecting file I/O information
#devconnections
![Page 42: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/42.jpg)
.NET DEBUGGING WORKSHOP
Trace Analysis with WPA
#devconnections
List of graphsList of graphs
Graph displayGraph display
Ungrouped columns
Ungrouped columnsGrouped
columnsGrouped columns Grouping
barGrouping
bar
![Page 43: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/43.jpg)
.NET DEBUGGING WORKSHOP
Stack Summaries• Learn how to read
stack summaries– Group by Stack
column– Expand “hot path”,
like in profiler• Stack resolution
requires symbols (slow)
#devconnections
![Page 44: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/44.jpg)
.NET DEBUGGING WORKSHOP
PerfView• ETW collection and analysis tool
tailored for .NET applications (but not only)
• Can be used as a sampling profiler• Can be used as an allocation profiler• Can be used for heap snapshot analysis
#devconnections
![Page 45: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/45.jpg)
.NET DEBUGGING WORKSHOP
Collecting Data w/ PerfView• CLI
PerfView run app.exe
• GUI
#devconnections
![Page 46: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/46.jpg)
.NET DEBUGGING WORKSHOP
PerfView Collection Options
#devconnections
Profiling wall-clock time
Profiling wall-clock time
Allocation profiling
Allocation profiling
File/registry accesses
File/registry accesses
CPU sampling profiling
CPU sampling profiling
![Page 47: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/47.jpg)
.NET DEBUGGING WORKSHOP
PerfView Tables
#devconnections
Grouping options
Grouping options Filtering
optionsFiltering options
Call stack treeCall stack tree
In-trace activity highlighter
In-trace activity highlighter
![Page 48: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/48.jpg)
.NET DEBUGGING WORKSHOP
Memory Leak Analysis• PerfView can
generate heap snapshots (smaller than a dump), analyze, and compare them
• Can also import dumps directly
#devconnections
![Page 49: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/49.jpg)
.NET DEBUGGING WORKSHOP
TRY IT OUTLeak analysis with PerfView
#devconnections
![Page 50: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/50.jpg)
.NET DEBUGGING WORKSHOP
Automatic ETW Analysis• The TraceEvent
library provides an API for ETW analysis– Understands kernel
and CLR events– Supports call stacks
(incl. managed)
#devconnections
![Page 51: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/51.jpg)
.NET DEBUGGING WORKSHOP
Example Analysis Scenarios• Monitor the system
for CLR exceptions w/ stacksExceptionTraceData
• Get a profiling trace and look for regressionsTraceLogSampledProfileTraceDataTraceCallStack
#devconnections
![Page 52: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/52.jpg)
.NET DEBUGGING WORKSHOP
Summary• Production debugging and
performance investigation is here, and entirely possible thanks to dumps and ETW traces
• Integrate automatic error analysis and triage into your devops process
#devconnections
![Page 53: NET Debugging Workshop](https://reader038.vdocuments.us/reader038/viewer/2022102823/54680829af7959925f8b5651/html5/thumbnails/53.jpg)
SESSION TITLE
#devconnections
Rate This Session Now!Rate with Mobile App:1. Select the session from the
Agenda or Speakers menus
2. Select the Actions tab
3. Click Rate Session
Rate Using Our Website:1. Register at www.devconnections.com/logintoratesession
2. Go to www.devconnections.com/ratesession
3. Select this session from the list and rate it
Tell Us What
You Thought
of This Session
Be Entered to
WIN
Prizes!