windows debugging and troubleshooting
DESCRIPTION
More info on http://www.techdays.be.TRANSCRIPT
Windows Debuggingand Troubleshooting
Daniel PearsonDavid Solomon Expert Seminars
AgendaIntroduction to the Debugging Tools for WindowsUnderstanding Windows and x86/x64 ArchitecturesUnderstanding Application CrashesIntroducing Application VerifierAdvanced Debugging Techniques
Daniel Pearson7 years working at MicrosoftSenior Escalation Lead in the Windows base operating system teamLead in the Mobile Internet sustained engineering team
3 years at Digital Equipment CorporationSupporting Intel and Alpha systems running Windows NT
Instructor with David SolomonDavid, co–author of the Windows Internals series
Introduction to the Debugging Tools for Windows
Types of Windows DebuggersThe Debugging Tools install four debuggersOf those, cdb, ntsd, kd and WinDbg, only three can effectively be used to debug applicationsAll the debuggers share a common debugging engine, dbgeng.dll
Support for all architectures supported by WindowsMatch the architecture of the installed system, e.g. a 64–bit debugger for an x64–based processorThe debuggers can support cross–architecture debugging in certain scenarios
WinDbg is a Windows–based debugging toolAble to perform both application and system level debugging
Starting the DebuggerSeveral ways to select a debugging targetCan be configured to attach to a running process, spawn a new process or open a crash dumpThe target application ends when the debugging session ends
Must know the name or the identifier of the targetTools such as Task Manager, tasklist or tlist can be used to display the target identifier
Support for noninvasive debuggingNoninvasive debugging minimizes the debugger's interference with the target applicationUseful for situations where the application or debugging interface is not responding
Configuring the Windows DebuggersWinDbg supports the use of workspacesAllows customization of the debugger including the look and feel and handling of eventsAbility to create named workspaces that can be reused or shared with other users
Support included for a command line interfacePossible to configure the handling of the debugger and any initial commands that are sentUseful in automating the debugger for hard to track down or low–rate occurrence issues
Access to symbols to perform debuggingMost third party application vendors don’t make symbols for their applications available
Understanding SymbolsA collection of symbols contained within a single fileSymbols are the named units of code or data within a module, e.g. function names or local variablesThe debugger can interpret code and data using memory locations or by resolving symbol names
#include <stdio.h>
int main(int argc, char *argv[]){ char *s = "Hello";
printf("%s, %s\n", s, argv[1]); return 0;}
0:000> uf hello!main00321180 55 push ebp00321181 8bec mov ebp,esp00321183 8b450c mov eax,dword ptr [ebp+0c]00321186 ff7004 push dword ptr [eax+4]00321189 68c4103200 push offset hello!`string' (003210c4)0032118e 68bc103200 push offset hello!`string' (003210bc)00321193 ff1570103200 call dword ptr [hello!_imp__printf]00321199 83c40c add esp,0c0032119c 33c0 xor eax,eax0032119e 5d pop ebp0032119f c3 ret
Configuring SymbolsCan be challenging to locate the required symbolsSymbols need to match the version of the target system including any service packs or hotfixesUsing a symbol server can simplify the configuration process
Set the system wide environment variable_NT_SYMBOL_PATH=srv*C:\SYMBOLS*http://msdl.microsoft.com/download/symbols
Troubleshoot symbol loading errors with !sym noisyTurns on additional tracing information inside of symsrv.dll, the symbol server DLL
Cache
Getting AssistanceThe most useful information is the Help fileThe Debugging Tools documentation contains information for supported debugging commands Install the latest Debugging Tools for Windows for the most recent version of the documentation
Use the .hh command from within the debuggerAny text will be searched for in the index of the Debugging Tools for Windows documentation
Discovering commands with auto–completePressing the Tab key in the command window cycles through the debugger commands
Demo
Understanding Windows and 32–bit and 64–bit Architectures
CPUs and RegistersRegisters, small areas of extremely fast storageUsually measured by the number of bits they holde.g. a “32–bit” register or a “64–bit” register
x86 architecture provides 16 basic program registersEAX, EBX, ECX, EDX, EDI, ESI, EBP, ESP CS, DS, SS, ES, FS, GS EFLAGS EIP
x64 adds an additional 8 general–purpose registersRAX, RBX, RCX, RDX, RDI, RSI, RBP, RSP, R8–R15
General–purpose Segment
Displaying RegistersAccessible using the r debugger command0:001> rrax=000007fffffdb000 rbx=0000000000000000 rcx=000007fffffdf000rdx=0000000077347ec0 rsi=0000000000000000 rdi=0000000000000000rip=00000000772a0530 rsp=000000000230fa38 rbp=0000000000000000 r8=0000000000000000 r9=0000000077347ec0 r10=0000000000000000r11=0000000000000000 r12=0000000000000000 r13=0000000000000000r14=0000000000000000 r15=0000000000000000iopl=0 nv up ei pl zr na po nccs=0033 ss=002b ds=002b es=002b fs=0053 gs=002b efl=00000246ntdll!DbgBreakPoint:00000000`772a0530 cc int 3
Virtual MemoryWindows provides support for a flat addressed virtual environmentThe processor, with support from the operating system, manages virtual memory via a mechanism called paging
Linear address space is divided into fixed–size pagesx86 and x64 CPUs support a minimum page size of 4 KB
2 GB systemspace
2 GB userspace
x86 virtual address space layout
Virtual MemoryWindows provides support for a flat addressed virtual environmentThe processor, with support from the operating system, manages virtual memory via a mechanism called paging
Linear address space is divided into fixed–size pagesx86 and x64 CPUs support a minimum page size of 4 KB
8 TB systemspace
8 TB userspace
x64 virtual address space layout
Displaying Virtual MemoryAccessible using the d debugger commands0:000> db 00000000`ff18000000000000`ff180000 4d 5a 90 00 03 00 00 00-04 00 00 00 ff ff 00 00 MZ..............00000000`ff180010 b8 00 00 00 00 00 00 00-40 00 00 00 00 00 00 00 [email protected]`ff180020 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................00000000`ff180030 00 00 00 00 00 00 00 00-00 00 00 00 e8 00 00 00 ................00000000`ff180040 0e 1f ba 0e 00 b4 09 cd-21 b8 01 4c cd 21 54 68 ........!..L.!Th00000000`ff180050 69 73 20 70 72 6f 67 72-61 6d 20 63 61 6e 6e 6f is program canno00000000`ff180060 74 20 62 65 20 72 75 6e-20 69 6e 20 44 4f 53 20 t be run in DOS 00000000`ff180070 6d 6f 64 65 2e 0d 0d 0a-24 00 00 00 00 00 00 00 mode....$.......
Threads and ProcessesProcess, an instance of a programA container that includes a private virtual address space, executable code and dataContains at least one unit of execution, a thread
Thread, a unit of execution within the systemIncludes the contents of a volatile set of registers that represent the state of the processorScheduled by the Windows kernel for execution
A unique identifier is assigned to bothAllocated from a shared table within system address space
Displaying Threads and ProcessesUsing the !teb debugger commandEach thread within a process contains a Thread Environment Block, linked the process blockViewable using the !teb debugger command
Using the !peb debugger commandEach process contains a single Process Environment Block, viewable using the !peb command
Using the inbuilt ~ commandThe ~ command is used to identify threads, ~* represents all the threads within a processThe ~s command can be used to switch between threads in a debugger
Thread StacksA storage location used by threadsUsed to store information such as parameters, local variables and return addressesThe amount of storage per thread is configurable by the application developer
Useful to identify the flow of code in an applicationUnderstanding the flow of code can assist in troubleshooting why an application crashed or is hungUsing the stack pointer register as a base is useful when viewing a stack trace is not successful
A unique stack is allocated to each threadTwo stacks are assigned to application threads, the other in system address space
Displaying Thread StacksAccessible using the k debugger commands0:000> kChild-SP RetAddr Call Site00000000`0021fa58 00000000`77169e9e USER32!NtUserGetMessage+0xa00000000`0021fa60 00000000`ff181064 USER32!GetMessageW+0x3400000000`0021fa90 00000000`ff18133c notepad!WinMain+0x18200000000`0021fb10 00000000`76ce652d notepad!DisplayNonGenuineDlgWorker+0x2da00000000`0021fbd0 00000000`7727c521 kernel32!BaseThreadInitThunk+0xd00000000`0021fc00 00000000`00000000 ntdll!RtlUserThreadStart+0x1d
Demo
Understanding Application Crashes
Why Windows Applications CrashThe result of an unhandled exceptionAn event that occurs that requires the execution of code outside the normal flow of controlCan be initiated by either software or hardware during execution
Windows uses structured exception handlingRaising an exception causes the exception dispatcher to search for an exception handlerAllows the application to be given control when an exception occurs
Unhandled exceptions are passed to a system filterThe kernel filter UnhandledExceptionFilter, attempts to report the fault to the system
What Happened to the Doctor?Dr Watson replaced with WerFault in Windows VistaWindows Error Reporting enables users to notify Microsoft of application and kernel faultsAllows Microsoft to provide end users with troubleshooting information, solutions or updates
A central location is now provided for usersThe Action Center in Windows 7 or Problem Reports and Solutions in Windows Vista allows users to check for new solutions, manage reporting history and view details of reports
Additional support for non–critical eventsIncluding unresponsive applications, performance issues and application specific events
Configuring WerFaultDefault configuration is to not take a full dumpConfigurable using HKLM\SOFTWARE\Microsoft\Windows\Windows Error Reporting\LocalDumpsPossible to change the default location, maximum storage count and type of dump created
Ability to exclude reports on a per application basisPrevents Windows from sending problem reports for those applications that are listedUseful for internal applications, those that sensitive by nature or are under development
Doesn’t affect applications with their own supportApplication such as Microsoft SQL Server use their own external utilities to write crash dumps
Attaching to a Crashed ApplicationApplication not terminated until the filter returnsIn most cases there’s a window in which a debugger can be attached to the process
Must know the name or the PID of the applicationWerFault.exe is passed a reference to the PID of the faulting application as a parameterC:\Windows\System32\WerFault.exe –u –p 4668 –s 180
Allows a user to create a dump of the applicationUseful when the system isn’t configured by default to save crashes or the default crash options don’t contain enough information to diagnose the issue you’re attempting to troubleshoot
PID
Demo
Introducing Application Verifier
Application VerifierA runtime verification tool for native codeUseful for identifying errors that can be difficult to diagnose under normal conditionsWorks by monitoring the application’s interaction with the operating system
Available as a separate download from MicrosoftSearch for Application Verifier from http://microsoft.com/downloads
Injects verification DLLs into the applicationBase support built into the operating system with additional support from verification DLLsThe number of DLLs loaded depends on the verification tests selected by the user
Configuring Application VerifierConfigurable using the Application Verifier toolEnables the user to choose which tests are run against the selected application and to view logsConfiguration information is written to the HKEY_LOCAL_MACHINE registry hive
Certain verification layers require a debuggerAll of the basic tests require that the application be configured to run under a debuggerThe user is reminded when selecting any of the tests from the Basics category
Support for using a command line interfaceUseful for pushing changes to multiple applications or those that require further configuration
Demo
Advanced Debugging Techniques
Taking a Dump of an ApplicationPossible to force dump creation of an applicationTaking a dump is useful as it allows you to restart the application while you perform further analysis
Using the built in Windows Task ManagerSelect the Processes tab, right–click on the application and select Create Dump FileThe resulting dump file is written to the directory defined by the user’s TEMP variable
Using the Debugging Tools for WindowsAfter attaching to the process, create a dump using one of the .dump commandsAllows for more control over what information is included, e.g. .dump /mA notepad.dmp
Attaching and Using a Kernel DebuggerSupport for redirection using a kernel debuggerSupported transports include a serial, USB or IEEE 1394 cable, named pipes or a network connectionNetwork support is to be included with Windows 8 and requires a supported network adapter
The system must be started in debugging modeConfigurable by modifying the boot configuration database, e.g. bcdedit /debug on
Useful in several advanced scenariosControl required of the Windows kernel or access to kernel debugging featuresDebugging service initialization when no user is logged into the system
Demo
Further InformationWindows Internals, 5th EditionRussinovich, Mark, and David Solomon. 2009. Microsoft Press
Advanced Windows DebuggingHewardt, Mario, and Daniel Pravat. 2007. Addison–Wesley Press
Windows via C/C++, 5th EditionRichter, Jeffrey, and Christophe Nasarre. 2007. Microsoft Press
Further InformationMemory Dump, Software Trace, Debugging, Malware and Intelligence Analysis Portalhttp://dumpanalysis.org
Advanced Windows Debugging and Troubleshootinghttp://blogs.msdn.com/ntdebugging
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.