Download - Kdump - Linux Scalability Effort Homepage
![Page 1: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/1.jpg)
1
KdumpA Kexec Based Kernel Crash Dumping Mechanism
Vivek Goyal ([email protected])Eric W. Biederman ([email protected])
Hariprasad Nellitheertha ([email protected])
![Page 2: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/2.jpg)
2
Background● Dump capture from crashing kernel's context
✔ Resource lockup✔ Corrupt data structures
● Dedicated dump drivers✔ Limited number of target devices✔ Maintenance was a big issue✔ Dependency on crashing kernel reduced and not
eliminated completely
![Page 3: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/3.jpg)
3
Background contd...● Stand alone dumpers
✔ Need to maintain low level hardware specific code✔ Filtering is not possible
● Kernel reboot based dumper✔ Memory constraint might prevent capturing full dump✔ Significant amount of code being run in crashing
kernel context✔ Core kernel invasive code
![Page 4: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/4.jpg)
4
Design Goals of The New Solution● Simple and minimally invasive into the kernel code● Highly Reliable● Available on most architectures● Easy to Maintain● Flexibility in terms of dump contents and targets
✔ Full dump or kernelpages only dump✔ Dump to disk or across the network
● Ease of Use
![Page 5: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/5.jpg)
5
Kdump – Overview● A new kernel, often called capture kernel, is booted
after the crash● Previous kernel's memory is preserved● Dump is captured from the context of capture kernel● Kerneltokernel boot loader enables booting a new
kernel after a crash● Kexec is underlying kernel to kernel bootloader
![Page 6: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/6.jpg)
6
KerneltoKernel Boot Loader● Running kernel acts as a loader for the new kernel● System directly jumps from one kernel to another
✔ Skips BIOS or Firmware stage● Reboots are extremely fast (33% reduction in time)● Memory can be preserved across reboots
✔ Since BIOS is skipped, it is left to the OS to retain or erase memory
● Finds application in crash dumping tools
![Page 7: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/7.jpg)
7
Kexec● Allows a Linux kernel to boot another kernel● Currently available on i386, x86_64 and ppc64 platforms● Two components
✔ User space tool – kexectools✔ Kernel System Call (sys_kexec_load)
● Load a new kernel
kexec l <kernelimage> append=<options>● Exec new kernel
kexec e
![Page 8: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/8.jpg)
8
Kexec On Panic● An extension of Kexec functionality● Enables booting a new kernel after system crash● Devices are not shutdown● New kernel runs from a reserved memory
location✔ Protection against ongoing DMA at the time of crash
![Page 9: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/9.jpg)
9
Kexec On Panic Contd..● Loading capture kernel
kexec p <kernelimage> append=<options>
● Execution of capture kernel✔ panic()✔ AltSysrqc
![Page 10: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/10.jpg)
10
Kdump Overview
First Kernel Capture Kernel
System Crashpanic(), AltSysrqc
Kexec Enabled Boot
LocalDisk
cp, dd
RemoteDisk
scpftp
/proc/vmcore
gdb
StorageMedia
EarlyUserSpace(initrd)
![Page 11: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/11.jpg)
11
Kexec on Panic Preloading
ReservedRegion
Capture Kernel Image
Purgatory
Parameter Segment
0K
16M
48M
4G
● Reserve memory for capture kernel (crashkernel=X@Y)
● Preload the capture kernel
● Capture kernel runs from reserved memory location
![Page 12: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/12.jpg)
12
Kexec on Panic Purgatory
ReservedRegion
Capture Kernel Image
Purgatory
Parameter Segment
0K
16M
48M
4G
● Purgatory is an ELF relocatable object and it contains setup code and sha256 hash
● Sha256 hash ensures integrity of the new kernel's preloaded data
![Page 13: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/13.jpg)
13
Kexec on Panic – Post Crash (x86)
Save CPU States
Restore APICS toBIOS State
Purgatory(Sha256 hash+Setup)
Execute new kernel
Crash● CPU states are saved and
other CPUs are halted using NMI
● LAPIC/IOAPIC are disabled and put back into PIC or virtual wire mode
● Purgatory is run and control is transferred to new kernel
![Page 14: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/14.jpg)
14
Kexec on Panic – Saving CPU States
CPU[1]
Null Note
CPU[NR_CPUS]
NT_PRSTATUS
CPU[0]
Filled with Zeros
1K 1K 1K
● CPU register states are saved in ELF note format● 1K of memory is reserved statically per CPU
![Page 15: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/15.jpg)
15
Kdump – ELF Header Generation● Dump information
across the kernel is exchanged in an ELF format Core file
● Kexectools prepare ELF headers and preload them in the reserved region
ReservedRegion
Capture Kernel Image
Purgatory
Parameter Segment
0K
16M
48M
4G
ELF Core Headers
![Page 16: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/16.jpg)
16
Kdump – ELF Header Generation● One PT_LOAD type ELF program header is
created for every contiguous memory chunk● Kexectools use /proc/iomem to retrieve System
RAM information on i386 platform● Address of the start of ELF header is passed to
the capture kernel using command line option “elfcorehdr=”
![Page 17: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/17.jpg)
17
Kdump – Backup Region
Capture Kernel Image
Purgatory
Parameter Segment
0K
16M
48M
4G
ELF Core Headers
Backup Region
640K
Copy Operation
● Kernel uses some fixed memory locations to boot
● First 640K of memory is required for booting SMP capture kernel on i386
● Contents of first 640K of memory are backed up in Backup Region
![Page 18: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/18.jpg)
18
Kdump – Backup Region● Kexectools reserve the memory for backup
region while loading capture kernel.● Purgatory contains the code for backing up first
640K of memory after crash.● Other architectures can define their own backup
region (If need be).
![Page 19: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/19.jpg)
19
Kdump – Booting into Capture Kernel
● Capture kernel uses limited amount of memory to boot
● Command line option “memmap=exactmap” is used to limit the memory regions capture kernel uses
● Kexec tools append memmap= command line options automatically
![Page 20: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/20.jpg)
20
Kdump – Capturing the Dump● Accessing dump image in ELF Core format
✔ /proc/vmcore● Accessing dump image in linear raw format
✔ /dev/oldmem
![Page 21: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/21.jpg)
21
Kdump – ELF Format Core File (/proc/vmcore)
ELFHeader
Program Header
PT_NOTE
Program Header
PT_LOAD
Per CPURegisterStates
Dump Image
● ELF32/ELF64 format headers● Physical addresses are filled for all the regions● Virtual addresses are filled only for linearly
mapped memory region
![Page 22: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/22.jpg)
22
Kdump – Analysis Tools● gdb
✔ Virtual view of memory✔ Can debug linearly mapped region of memory✔ User space utility to regenerate ELF headers to create
the ELF headers for vmalloc regions ● crash
✔ Physical view of memory
![Page 23: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/23.jpg)
23
Advantages● Increased reliability
✔ Dump is captured from a newly booted kernel● Enhanced flexibility
✔ Dump image can be saved to virtually any storage media supported by kernel
✔ Filtering mechanism can be plugged in
![Page 24: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/24.jpg)
24
Advantages Contd..● Ease of use
✔ Standard utilities can be used to save the dump image either locally or remotely
✔ Standard analysis tools like gdb can be directly used for limited debugging
![Page 25: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/25.jpg)
25
Limitations● Devices are not shutdown/reset after a crash
which might result in a driver initialization failure in capture kernel
● Nondisruptive dumping is not possible
![Page 26: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/26.jpg)
26
Current Status● Initial i386 implementation is mainline now
(2.6.13rc1)● Driver initialization issues are being addressed
✔ Shared Interrupts➢ irqpoll commandline option, Disabling PCI interrupts etc.
✔ Driver hardening➢ Reset the device if it is not reset already.
![Page 27: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/27.jpg)
27
ToDos● Port kdump to other platforms like x86_64 and
ppc64● Modify “crash” tool to be able to analyze kdump
generated dump images● Implement kernel pages only filtering mechanism● Relocatable Kernel for binary image unification● Initialize APICs before timer initialization
![Page 28: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/28.jpg)
28
Downloads● Kdump patches for kexectools and test reports
are available at:http://lse.sourceforge.net/kdump/
![Page 29: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/29.jpg)
29
Questions?
![Page 30: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/30.jpg)
30
Legal Statement● Copyright ©2005 IBM.
● This work represents the view of the author and does not necessarily represent the view of IBM.
● IBM, and the IBM logo, are trademarks or registered trademarks of International Business Machines Corporation in the United States, other countries, or both.
● UNIX is a registered trademark of The Open Group in the United States and other countries.
● Linux is a registered trademark of Linux Torvalds in the United States, other countries, or both.
![Page 31: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/31.jpg)
31
Legal Statement Contd...● Other company, product, and service names may be trademarks or
service marks of others.
● References in this publication to IBM products or services do not imply that IBM intends to make them available in all countries in which IBM operates.
● This document is provided "AS IS" with no express or implied warranties. Use the information in this document at your own risk.
![Page 32: Kdump - Linux Scalability Effort Homepage](https://reader035.vdocuments.us/reader035/viewer/2022071601/613d30e2736caf36b75a64fc/html5/thumbnails/32.jpg)
32
Thank You