Kdump

lkong

2014-11-10

Contents

1 Kdump 3

2 Agenda 3

3 Background 43.1 Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

4 Kexec - Overview 54.1 Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

5 Kdump - Overview 6

6 Kdump - Overview 6

7 Kdump - Overview 77.1 Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77.2 Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

8 Kdump - Overview 108.1 Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

9 Install and configure kdump 119.1 Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

10 Install and configure kdump 1310.1 Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

11 Kdump on Xen guest 15

1

12 Kdump on Xen guest 16

13 Kdump on Xen guest 17

14 Xen dump 18

15 Using crash tool to check core file 18

16 Related bugs 19

17 Reference 1917.1 Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

18 Q & A 20

19 Q & A 2119.1 Note . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2

1 Kdump slide

2 Agenda slide

• Background

• Kexec - Overview

• Kdump - Overview

• Install and configure kdump

3

• Kdump on Xen guest

• Xen dump

• Using crash tool to check core file

• Related bugs

• Reference

• Q & A

3 Background slide

• Linux kernel is a rather robust entity, nevertheless kernel panic stilloccurs

• Dump tools like: LKCD, netdump, diskdump have there limitations

• Guest may hit problems in its whole life cycle, such as crash, reboot,hang, shutdown, etc

• People may want to know current state of the guest OS for troubleshooting

• kexec - directly boot into a new kernel

3.1 Note notes

• Unable to save memory dumps to local RAID (md) devices, outsidenetwork

4

• Unstable

• Not mul version support – So kdump borns

• Kdump is a much more flexible tool

4 Kexec - Overview slide

• Kexec is a fastboot mechanism that allows booting a Linux kernel fromthe context of an already running kernel without going through BIOS

• kexec performs the function of the boot loader from within the kernel

• Include two components

– User space tool - kexec-tools

– Kernel System Call (kexec_load())

• Using kexec consists of

– loading the kernel to be rebooted to into memory

kexec -l kernel-image --initrd=ini-trd-image --append=command-line-optiocat /sys/kernel/kexec_loaded

– actually rebooting to the pre-loaded kernel

kexec -e

4.1 Note notes

yum install kexec-tools kexec -lkexec -l vmlinuz-3.10.0-118.el7.x86_64 --initrd=initramfs-3.10.0-118.el7.x86_64.img --append="BOOT_IMAGE=/vmlinuz-3.10.0-118.el7.x86_64 root=UUID=e2a47529-701a-4780-bb30-af886fdc899d ro xen_emul_unplug=unnecessary console=tty0 vconsole.keymap=us rd.lvm.lv=rhel_dhcp-9-37/swap rd.lvm.lv=rhel_dhcp-9-37/root console=ttyS0,115200n8 crashkernel=auto vconsole.font=latarcyrheb-sun16 rhgb quiet LANG=en_US.UTF-8 --args-linux"

5

5 Kdump - Overview slide

• Kdump uses kexec to quickly boot to dump-capture kernel

• Previous kernel’s memory is preserved before crash

• Dump information across the kernel is exchanged in and ELF formatCore file

• Can use common commands, such as cp and scp, to copy the memoryimage to a dump file on the local disk, or across the network to a re-mote system.

• Kdump and kexec are currently supported on the x86, x86_64, ppc64,ia64, and s390x architectures. And kdump is install in RHEL5,RHEL6, RHEL7 as default.

• Accessing dump image in ELF Core format

– /proc/vmcore

• Accessing dump image in linear raw format

– /dev/oldmem

6 Kdump - Overview slide

• Write Out the Dump File

cp /proc/vmcore <dump-file>

mknod /dev/oldmem c 1 12dd if=/dev/oldmem of=oldmem.001

6

• Based on the architecture and type of image (relocatable or not), onecan choose to load the uncompressed vmlinux or compressed bzIm-age/vmlinuz of dump-capture kernel

For i386 and x86_64:- Use vmlinux if kernel is not relocatable.- Use bzImage/vmlinuz if kernel is relocatable.For ppc64:- Use vmlinuxFor ia64:- Use vmlinux or vmlinuz.gzFor s390x:- Use image or bzImage

7 Kdump - Overview slide

• Arch specific command line options to be used while loading dump-capture kernel

For i386, x86_64 and ia64:"1 irqpoll maxcpus=1 reset_devices"

For ppc64:"1 maxcpus=1 noirqdistrib reset_devices"

• If you use a uncompressed image remember to add ’–args-linux’ to ker-nel command line (no need for ia64)

7.1 Note notes

• The "irqpoll" boot parameter reduces driver initialization failures dueto shared interrupts in the dump-capture kernel

• Boot parameter "1" boots the dump-capture kernel into single-usermode without networking. If you want networking, use "3"

7

7.2 Note notes

• Production/the first/standard kernel====crash, capture, the secondkernel

kexec -p <kernel-image> --append=<options>

• Execution of capture kernel

– panic()

– Alt-Sysrq-c

8

9

8 Kdump - Overview slide

10

8.1 Note notes

9 Install and configure kdump slide

• Install kexec-tools

11

yum install kexec-tools

If you wish to configure kdump using a graphical user interface instead of the command line, install the system-config-kdump utility:yum install system-config-kdump

• Configure kdump (Remember to set crashkernel on the kernel com-mand line)

crashkernel=128Mcrashkernel=128M@16Mcrashkernel=512M-2G:64M,2G-:128Mcrashkernel=512M-2G:64M,2G-:128M@16Mgrub2-mkconfig -o /boot/grub2/grub.cfg

9.1 Note notes

• A limitation in the current implementation of the Intel IOMMU drivercan occasionally prevent the kdump service from capturing the coredump image. To use kdump on Intel architectures reliably, it is ad-vised that the IOMMU support is disabled.

• GRUB_CMDLINE_LINUX

• chkconfig kdump on;service kdump status;service kdump start

• influence crashkernel size: 1) arch 2) total amount of installed systemmemory, 128 MB + 4 bits for every 4KB page

• <4G, not recommand auto. Y must larger than 16M. Usually 128M+Ym, Y is calculate.

• On many systems, kdump can reserve memory automatically. This be-havior is enabled by default. However, automatic memory reservationonly works on systems which have more than a certain amount of totalavailable memory

12

10 Install and configure kdump slide

• Configure your kernel for kdump

– System kernel config options

∗ CONFIG_KEXEC=y

∗ CONFIG_SYSFS=y

∗ CONFIG_DEBUG_INFO=Y

– Dump-capture kernel config options

∗ CONFIG_CRASH_DUMP=y

∗ CONFIG_PROC_VMCORE=y

∗ CONFIG_HIGHMEM64G=y or CONFIG_HIGHMEM4G=y(only for i386)

∗ CONFIG_RELOCATABLE=y

∗ CONFIG_PHYSICAL_START=0x100000

• Configure file: /etc/kdump.conf

• Configure file: /etc/sysconfig/kdump.conf

10.1 Note notes

• (If CONFIG_SMP=y, then specify maxcpus=1 on the kernel com-mand linewhen loading the dump-capture kernel, see section "Load the Dump-capture

13

Kernel".

• Allows the kernel to be placed somewhere else in the memory.

• If kernel is not relocatable CONFIG_RELOCATABLE=n then bzIm-age will decompress itself to the above physical address

and run from there. Otherwise bzImage will run from the address where ithas been loaded by the boot loader

• – Enabling the kdump Service

#systemctl enable kdump.service#systemctl start kdump.service

– Testing the dump Configration

#systemctl is-active kdump#echo 1 > /proc/sys/kernel/sysrq or#echo c > /proc/sysrq-trigger

• core_collector makedumpfile -c –message-level 1 -d 31 ,-c makedump-file ,-

-message-level 1 (1 )-d 31( zero page, cache page,cache private, user data, free page )

• "kexec system call" in "Processor type and features." CONFIG_KEXEC

• "Filesystem" -> "Pseudo filesystems -> "sysfs file system support"CONFIG_SYSFS=y

14

• "Compile the kernel with debug info" in "Kernel hacking." CON-FIG_DEBUG_INFO=Y

• "kernel crash dumps" support under "Processor type and features"CONFIG_CRASH_DUMP=y

• "/proc/vmcore support" under "Filesystems" -> "Pseudo filesys-tems".CONFIG_PROC_VMCORE=y

• On i386, enable high memory support under "Processor type and fea-tures" CONFIG_HIGHMEM64G=y

• On i386 and x86_64, disable symmetric multi-processing support un-der "Processor type and features" CONFIG_SMP=n

• "Build a relocatable kernel" support under "Processor type and fea-tures" CONFIG_RELOCATABLE=y

• "Physical address where the kernel is loaded" (under "Processor typeand features"). CONFIG_PHYSICAL_START=0x1000000

•

11 Kdump on Xen guest slide

• kdump in rhel6 PV guest is not supported

• Delete or comment out (using #) the line with the Kdump_not_supported_on_Xen_domU_guestmarker, if it’s present

• Booting without paravirt drivers (Method 1)

15

– Preparing the kdump initrd (dumprd), if needed

dracut -f /boot/initramfs-$(uname -r).img $(uname -r)

– Add the kernel command line parameter xen_emul_unplug=nevercrashkernel=128M to the kernel’s command line and reboot.

– Start the kdump service and trigger a kernel panic in guest (Needdisable selinux first)

service kdump restartecho c > /proc/sysrq-trigger

– Login the guest and check the vmcore, vmcore can be analyzed

12 Kdump on Xen guest slide

• Booting without paravirt drivers (Method 2)

– Change kernel command line parameter xen_emul_unplug=unnecessary

– Black list the xen pv modules

cat /etc/modprobe.d/blacklist.conf[...]blacklist xen_blkfrontblacklist xen_netfront

– Remove the paravirt driver if needed

modprobe -r xen_netfrontmodprobe -r xen_blkfront

– Append "xen_emul_unplug=unnecessary" to kernel commandline in grub.cfg

16

– Reboot the guest, and login, you will see there is a new kdumpramdisk generated under /boot

– Start the kdump service and trigger a kernel panic in guest

service kdump restartecho c > /proc/sysrq-trigger

– Login the guest and check the vmcore, vmcore can be analyzed

13 Kdump on Xen guest slide

• Booting with paravirt drivers, and without unplug

– Preparing the kdump initrd (dumprd), if needed

dracut -f /boot/initramfs-$(uname -r).img $(uname -r)

– Edit /etc/modprobe.d/blacklist.conf by adding the three linesshown below to blacklist the drivers used for the emulated de-vices.

blacklist ata_piixblacklist 8139tooblacklist 8139cp

– Change xen_emul_unplug=never to xen_emul_unplug=unnecessaryin kernel command line and reboot

– Start the kdump service and trigger a kernel panic in gues

service kdump restartecho c > /proc/sysrq-trigger

– Login the guest and check the vmcore, vmcore can be analyzed

17

14 Xen dump slide

• Manual dump core

– By default, xen would first pause the guest before dump core for it

– Manual dump core is applicable to both PV and HVM, no xenpvdriver is needed for HVM.

• Automatic dump core

– ’enable-dump’ must be set as ’yes’ in /etc/xen/xend-config.sxp

– Automatic dump core is applicable to both PV and HVM, as longas xenpv driver is installed for HVM guest.

15 Using crash tool to check core file slide

• Install crash tool

yum install crash

• Install debuginfo packages

yum install kernel-debuginfo-$(uname -r)yum install kernel-debuginfo-common-$(uname -r)

• Using crash tool to check

crash /usr/lib/debug/lib/modules/$(uname -r)/vmlinux $core_file

• Commonly used commands

18

helphlogbtforeach btpsvmfilesexit or q

16 Related bugs slide

• Bug 771712 - kexec and kdump for Xen PVonHVM guests

• Bug 1007328 - RHEL7 guest hangs when triggering crash with kdumpenabled on a Xen4.3 host

• Bug 657506 - Xen 32bit HVM guest will have large time-drift aftersave/restore

17 Reference slide

• Section B.1, “Memory Requirements for kdump" [Kernel Crash DumpGuide]

• Section B.2, “Minimum Threshold for Automatic Memory Reserva-tion”[Kernel Crash Dump Guide]

• Guest dump core

• Enable and test kdump in RHEL7.0 guest

• Guest kdump

19

17.1 Note notes

• Minimum Threshold is for crashkernel=auto

18 Q & A slide

1. compressed linux kernel image is not equal to a relocatable ker-nel image. You can get a relocatable kernel image by set CON-FIG_RELOCATABLE=y when you bulid a kernel image

2. When startup kdump service and trigger a kernel panic, kdump willfirst save the dump core file then reboot your host automatically. Whenkdump failed to save the core file it will perform as the value of defaultin /etc/kdump.conf:

• default=reboot -> just reboot the host

• default=halt -> Bring the system to a halt, requiring manual re-set poweroff

• default=poweroff -> Bring the system to a poweroff, requiringmanual poweron

• default=shell -> Drop to an shell session inside the initramfs fromwhere you can manually perform additional recovery actions. Ex-iting this shell reboots the system

3. Option –args-linuxWhen you use kexec tool to load a second kernel, some times you needto add option ’–args-linux’ to kernel command line, I try to find thereason on google but can not find any useful information. If you try toload a new kernel but failed, it is safe to add/remove –args-linux fromthe kernel command line and try again.

20

19 Q & A slide

1. What’s the exact definition of relocatable kernel?

I can not found any official definition of *relocatable kernel*. But thefollowing lines can explain what a relocatable kernel is:

A non-relocatable kernel was loaded at a fixed memory address in orderto work(usually 1M of the memory space), loading it in a different placewouldn’t work. When we use kdump, we need to load the second kernel in adifferent place(The fixed memory address is reserved by the firstkernel), so when we build a kernel as a second kernel we need to makesure the kernel can be loaded at a different memory address, we have twoways to implement this:

1. Build relocatable kernel*Relocatable kernel can be loaded at different 4K-aligned addresses*,but always below 1 GB.If we want to build a relocatable kernel we need to configure the kerneloption:CONFIG_RELOCATABLE=y

Because kernel is relocatable, it can be run from any physical addresshence kexec boot loader will load it in memory region reserved for it(we can specify the memory address by add crashkerne=X@Y orcrashkernel=auto to first kernel’s command line)

Also when we set CONFIG_RELOCATABLE=y we can choose bzImage/vmlinuzkernel format

2. Hard code the load memory address and tell the start of the memoryregion to the first kernel by crasnkernel=X@YIn this way we can build the second kernel by set kernel option:CONFIG_PHYSICAL_START=0x100000and add that address on the first kernel’s command line:crashkernel=X@Y

When we set CONFIG_PHYSICAL_START=0x100000 the Y shoud be 16M, of courseyou can set a different value for CONFIG_PHYSICAL_START, but 0x100000 is

21

recommand for x86 arch.

Also when we set CONFIG_RELOCATABLE=n or not set CONFIG_RELOCATABLE, wecan only choose vmlinux kernel format.

If we want to learn more about relocatable kernel, we may learn moreabout kernel, of cource this would be another big topic.#+BEGIN_EXAMPLE

19.1 Note notes

• when start kdump service need disable selinux first??

22