trace kernel code tips
TRANSCRIPT
![Page 2: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/2.jpg)
12/14/14 2/38
Attribution – ShareAlike 3.0
You are free
● to copy, distribute, display, and perform the work
● to make derivative works
● to make commercial use of the work
Under the following conditions
– Attribution. You must give the original author credit.
– Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one.
For any reuse or distribution, you must make clear to others the license terms of this work.
● Any of these conditions can be waived if you get permission from the copyright holder.
Your fair use and other rights are in no way affected by the above.
License text: http://creativecommons.org/licenses/by-sa/3.0/legalcode
Corrections, suggestions, contributions and translations are welcome!
Rights to Copycopyright © 2014 Viller Hsiao
![Page 3: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/3.jpg)
12/14/14 3/38
Who am I ?
Viller Hsiao
source url: http://goo.gl/n7NAfS
Embeded Linux / RTOS engineer
2007 ~ Present Realtek Corp. 2003 ~ 2007 BenQ/Qisda
![Page 4: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/4.jpg)
12/14/14 4/38
An Embedded Programmer
source url: http://ext.pimg.tw/aabb52043/1367377582-3526768649_n.png
makefile menuconfigmakefile
gcc
armcc
ld objdump
memorymanagement Linker
script
HardwareSpec
uclibc
newlib
glibc
musl
uboot
kbuild
![Page 5: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/5.jpg)
12/14/14 5/38
Topic Today
How to trace C sources environment
We have cross reference tools.ctags, cscope, LXR, source insight ... and so on
BUT
![Page 6: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/6.jpg)
12/14/14 6/38
Difficulties of Tracing C/C++ Sources
● Preprocessor related● Too much #if/#elif/#endif ● Too much level of macros● Configurable included files
![Page 7: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/7.jpg)
12/14/14 7/38
Difficulties of Tracing C/C++ Sources (Cont.)
● Runtime related● Some debugging and error handling stuff● Multi-task environment● Hard to realize HW state change
Different task context
![Page 8: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/8.jpg)
12/14/14 8/38
Topic Today
How to trace C sources environment
Dynamically
Trace code from runtime execution result(Mainly in linux kernel source)
![Page 9: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/9.jpg)
12/14/14 9/38
Tracing from real execution log
Provides clues to trace code
http://www.clker.com/cliparts/c/c/7/p/7/4/earth-hi.pnghttp://goo.gl/4PnwFe
![Page 10: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/10.jpg)
12/14/14 10/38
Agenda Today
● Build environment● Config system● Makefile / kbuild● Toolchain tips
● Runtime trace● Qemu log● Linux ftrace
![Page 11: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/11.jpg)
12/14/14 11/38
menuconfig
● Too much words in menuconfig● Hard to map definition to code world (CONFIG_XXX)
![Page 12: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/12.jpg)
12/14/14 12/38
menuconfig Tips – Search
● Press '/' to search specific configurations
![Page 13: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/13.jpg)
12/14/14 13/38
menuconfig Tips – Search (Cont.)
Dependency
Path in menu
● Search result
Kconfig path
![Page 14: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/14.jpg)
12/14/14 14/38
menuconfig Tips – Result output
● Linux/.config## Automatically generated file; DO NOT EDIT.# Linux/arm 3.14.25 Kernel Configuration#CONFIG_ARM=yCONFIG_SYS_SUPPORTS_APM_EMULATION=y
## General setup#CONFIG_INIT_ENV_ARG_LIMIT=32CONFIG_CROSS_COMPILE=""# CONFIG_COMPILE_TEST is not setCONFIG_LOCALVERSION=""# CONFIG_LOCALVERSION_AUTO is not setCONFIG_HAVE_KERNEL_GZIP=yCONFIG_HAVE_KERNEL_LZ4=yCONFIG_KERNEL_GZIP=y# CONFIG_KERNEL_LZMA is not set# CONFIG_KERNEL_XZ is not set
## Automatically generated file; DO NOT EDIT.# Linux/arm 3.14.25 Kernel Configuration#CONFIG_ARM=yCONFIG_SYS_SUPPORTS_APM_EMULATION=y
## General setup#CONFIG_INIT_ENV_ARG_LIMIT=32CONFIG_CROSS_COMPILE=""# CONFIG_COMPILE_TEST is not setCONFIG_LOCALVERSION=""# CONFIG_LOCALVERSION_AUTO is not setCONFIG_HAVE_KERNEL_GZIP=yCONFIG_HAVE_KERNEL_LZ4=yCONFIG_KERNEL_GZIP=y# CONFIG_KERNEL_LZMA is not set# CONFIG_KERNEL_XZ is not set
![Page 15: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/15.jpg)
12/14/14 15/38
Tips of Tracing makefile
● make V=1● make --dry-run
![Page 16: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/16.jpg)
12/14/14 16/38
Tips of Tracing makefile (Cont.)
● make --debug
![Page 17: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/17.jpg)
12/14/14 17/38
Tips of Tracing Shell Script
● #!/bin/sh “-x”
● Result
#!/bin/sh -x
FOO=something_produced_from_command
if [ x${FOO} != x'' ]; then echo "get foo"else echo "no foo"fi
#!/bin/sh -x
FOO=something_produced_from_command
if [ x${FOO} != x'' ]; then echo "get foo"else echo "no foo"fi
+ FOO=something_produced_from_command+ '[' xsomething_produced_from_command '!=' x ']'+ echo 'get foo'get foo
+ FOO=something_produced_from_command+ '[' xsomething_produced_from_command '!=' x ']'+ echo 'get foo'get foo
![Page 18: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/18.jpg)
12/14/14 18/38
Tips of Tracing Macros/Definitions
● gcc options: “-save-temps” or “-E”
● Use astyle to beauty result
$ astyle -A8 main.i$ astyle -A8 main.i
$ gcc -save-temps -o main main.c
$ lsmain main.c main.i main.o main.s Makefile
$ gcc -save-temps -o main main.c
$ lsmain main.c main.i main.o main.s Makefile
![Page 19: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/19.jpg)
12/14/14 19/38
Tips of Tracing Macros/Definitions (Cont.)
![Page 20: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/20.jpg)
12/14/14 20/38
Tips of Structure Definitions
● Point to Real definition in the headers
# 5 "include/linux/linkage.h" 2# 1 "include/linux/stringify.h" 1# 6 "include/linux/linkage.h" 2# 1 "include/linux/export.h" 1# 26 "include/linux/export.h"
struct kernel_symbol{ unsigned long value; const char *name;};
# 5 "include/linux/linkage.h" 2# 1 "include/linux/stringify.h" 1# 6 "include/linux/linkage.h" 2# 1 "include/linux/export.h" 1# 26 "include/linux/export.h"
struct kernel_symbol{ unsigned long value; const char *name;};
![Page 21: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/21.jpg)
12/14/14 21/38
Kbuild Tips
● Add compile flags for some object
● Remove compile flags for some object
● All subdirectory
CFLAGS_foo.o = -O0 -g -save-tempCFLAGS_foo.o = -O0 -g -save-temp
CFLAGS_REMOVE_foo.o = -OsCFLAGS_REMOVE_foo.o = -Os
subdir-ccflags-y = -save-tempssubdir-ccflags-y = -save-temps
![Page 22: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/22.jpg)
12/14/14 22/38
Qemu Tips
Setup the search rule
● Qemu log (“-d op -D /path/to/qemu.log”)
![Page 23: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/23.jpg)
12/14/14 23/38
Qemu Tips (Cont.)
Example: get first user space entry point
Setup the search rule
● Qemu log + regular expression
![Page 24: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/24.jpg)
12/14/14 24/38
Qemu Tips (Cont.)
● Other tips● Qemu log + disassembly (objdump)
● Qemu + gdb ( + qemu log)– Breakpoint / stack trace– Qemu options
● -s or -gdb tcp::2345: enable gdb port● -S: freeze CPU at startup
![Page 25: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/25.jpg)
12/14/14 25/38
Ftrace – Overview
● Kernel function tracer ( > 2.6.28 ?)● Record execution log of kernel function● Implemented by “-pg”
– Use dynamic ftrace to reduce overhead– Add “-g -O0” to export more information
● Accessing API – debugfs– /path/to/debug/tracing/
mount -t debugfs none /sys/kernel/debugmount -t debugfs none /sys/kernel/debug
![Page 26: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/26.jpg)
12/14/14 26/38
Ftrace – Configs
![Page 27: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/27.jpg)
12/14/14 27/38
Ftrace – Steps
[tracing]$ echo “*mtd*” > set_ftrace_filter
[tracing]$ echo “*blk*” >> set_ftrace_filter
[tracing]$ echo “*:mod:ext3” >> set_ftrace_filter
[tracing]$ echo function > current_tracer
[tracing]$ echo 1 > tracing_on; do_test; echo 0 > tracing_on;
[tracing]$ cat trace
[tracing]$ echo “*mtd*” > set_ftrace_filter
[tracing]$ echo “*blk*” >> set_ftrace_filter
[tracing]$ echo “*:mod:ext3” >> set_ftrace_filter
[tracing]$ echo function > current_tracer
[tracing]$ echo 1 > tracing_on; do_test; echo 0 > tracing_on;
[tracing]$ cat trace
![Page 28: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/28.jpg)
12/14/14 28/38
Ftrace – Available Tracer
[tracing] $ cat available_tracers
function
function_graph nop
[tracing] $ cat available_tracers
function
function_graph nop
![Page 29: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/29.jpg)
12/14/14 29/38
ftrace
● function tracer
![Page 30: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/30.jpg)
12/14/14 30/38
ftrace
● function tracer
![Page 31: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/31.jpg)
12/14/14 31/38
ftrace
● function_graph tracer
![Page 32: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/32.jpg)
12/14/14 32/38
ftrace
● function_graph tracer
Taskswitch
![Page 33: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/33.jpg)
12/14/14 33/38
ftrace
● Better front-end● trace-cmd● kernelshark
![Page 34: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/34.jpg)
12/14/14 34/38
Ftrace for Code Tracing
● Ftrace provides clues of● Per-module view● Top-down view● Procedure excution flow● Tasks/Threads model relationship
![Page 35: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/35.jpg)
12/14/14 35/38
Ftrace – Other tracer
● Trace event● ftrace + tracepoint
● Probe event● ftrace + kprobe
● Other tracer● Stack tracer● Irqoff latency tracer● Syscall tracer
● And so forth … but not the topic today
![Page 36: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/36.jpg)
12/14/14 36/38
Conclusion
● Trace code● Cross reference tool helps● Drill-down trace
– Start from interface functions● Filesystem hooks: open/read/write/ioctl ...● probe/exit● ISR entry● Architecture interface
– Global functions: EXPORT_SYMBOLS● Execution log provide some hints, too.
– Log from build flow– Qemu instruction trace log– Kernel ftrace
![Page 37: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/37.jpg)
12/14/14 37/38
Reference
[1] Building embedded linux demo case
i. Sherif Mousa (Mar 26, 2014), “Building Embedded Linux”
ii. Linaro wiki, “cross-NG”
[2] Linux-3.14 documentation,
i. “kbuild/makefile.txt”
ii. “trace/ftrace.txt”
[3] 池田宗廣,大岩尚宏,島本裕志,竹步晶雄,平松雅巳 (Jan. 2014), “Linux kernel hacks”
[4] man of
i. astyle
ii. qemu
iii. Gnu make
![Page 38: Trace kernel code tips](https://reader033.vdocuments.us/reader033/viewer/2022042715/55a203501a28ab42268b47d5/html5/thumbnails/38.jpg)
12/14/14 Viller Hsiao
THE END