week 7 system calls, kernel threads, kernel debugging

Week 7System Calls, Kernel

Threads, Kernel Debugging

Sarah Diesburg

Florida State University

First…

Any questions on Part 1 – 5 system calls Part 2 – xtime proc module

Story of Kernel Development

Some context…

In the old days…

There were no modules or virtual machines The kernel is a program

Has code, can compile, re-compile, make executable

When changes needed to be made, developers make changes in source and re-compile

How is the kernel different from a regular program? Mostly in how it is executed

Boot loader loads the kernel image/executable during boot time

Sets kernel mode Jumps to the entry point in the image/executable

Remember the generic booting sequence?

Quick Question

How would you make changes to the kernel and run those changes?1. Make changes to the source

2. Re-complie the kernel source

3. Re-install the kernel source

4. Make sure the bootloader sees the new kernel image (grub)

5. Reboot and profit!

Getting more modern..

Modules were created as bits of code that can be loaded and unloaded by the kernel in kernel mode

Made development easier Instead of re-compiling, re-installing, and

rebooting into the new kernel, one could just re-compile and load a module

Quick Question

How would you make changes to a module and run those changes?1. Make changes to module source code

2. Re-compile the module

3. Load the new module

Present Day

Reboots into new kernels and loading new modules often freezes machines

Enter virtual machine software Process that emulates the hardware necessary to

run an OS in user-space Guest OS is executed inside the virtual machine

process!

New System Calls

Fun but tricky!

Implementing System Calls

Need to implement the functions above. But how?

int start_elevator(void);

int issue_request(int #1, int #2,

int #3);

int stop_elevator(void);

Adding a System Call to Kernel Files to add:

LINUX_DIR/PROJECT_NAME/Makefile LINUX_DIR/PROJECT_NAME/PROJECT_NAME.c LINUX_DIR/PROJECT_NAME/NEW_SYSCALLS.c

Files to modify: LINUX_DIR/arch/x86/kernel/syscall_table_32.S LINUX_DIR/include/asm-generic/unistd.h LINUX_DIR/include/linux/syscalls.h LINUX_DIR/Makefile

Sample System Call

Let’s add a sample module to the kernel that defines a sample system call

test_newsyscall(int test_int); Takes an int test_int and issues a printk on it Returns test_int

Problem – If our module isn’t loaded, what happens if we call our sample system call?

Project 2 System Call Model

Elevator module Core kernel

User program

User issues system call, core kernel looks up system call in system call table

User program

Elevator module performs system call action

User program

Elevator module returns result of system call

User program

Core kernel forwards result of system call to user program

What happens if elevator module is not loaded?

Core kernel

User program

Core kernel

User program

User issues system call, core kernel looks up system call in system call table

Core kernel

User program

Elevator module is not loaded to perform the action…. OOPS!

Module System Calls

We must create a wrapper system call! Wrapper will call module function if module

loaded, else returns an error Must be created in a separate, built-in kernel file

in the project folder

Function Pointers

We will implement our system call wrapper with a function pointer Pointer to a function

Function pointer can point to any function that you implement that Takes the same input variable types Returns the same return type

Function Pointers

long (*STUB_test_newsyscall)(int test_int) = NULL;

Function pointer that Returns a long Name is STUB_test_newsyscall Takes parameter int test_int Function pointer set to NULL

Can set function pointer to a local function you implement

Elevator Project

Create a file in your elevator project that just contains the system call information KERNEL_DIR/PROJECT_DIR/newsyscalls.c

KERNEL_DIR/PROJECT_DIR/newsyscalls.c#include <linux/linkage.h>

#include <linux/kernel.h>

#include <linux/module.h>

/* System call stub. We initialize the stub function to be NULL. */

EXPORT_SYMBOL(STUB_test_newsyscall);

/* System call wrapper. If the stub is not NULL, it will be run, otherwise returns -ENOSYS */

asmlinkage long sys_test_newsyscall(int test_int)

if (STUB_test_newsyscall)

return STUB_test_newsyscall(test_int)

return -ENOSYS;

Function pointer

return -ENOSYS;

Export the pointer so we can access

it later

return -ENOSYS;

System call wrapper

Elevator Project

Next create a separate file that Holds your module code Registers the system call pointer Actually implements the system call behavior

Inside KERNEL_DIR/PROJECT_DIR/PROJECT_NAME.C/* Extern system call stub declarations */extern long (*STUB_test_newsyscall)(int test_int);

long my_test_newsyscall(int test){ printk("%s: Your int is %i\n", __FUNCTION__, test); return test;}

my_module_init() { STUB_test_newsyscall=&(my_test_newsyscall);

return 0;}

my_module_exit() {STUB_test_newsyscall=NULL;

return 0;}

Gain access to stub function pointer.

return 0;}

Local function that implements syscall

return 0;}

Set stub function pointer to local function in init

return 0;}

Reset stub function pointer to NULL on

module unload

KERNEL_DIR/PROJECT_DIR/Makefileobj-m := my_module.o

obj-y := newsyscalls.o

KDIR := /lib/modules/2.6.32/build

PWD := $(shell pwd)

default:

$(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules

PWD := $(shell pwd)

default:

Compile as a module

PWD := $(shell pwd)

default:

Compile as kernel built-in

Core Kernel Additions

Add the new system call to the core kernel system call table Modify three files

Add the project directory to the main Makefile Modify one file

Modifying syscall_table_32.S….long sys_preadv.long sys_pwritev.long sys_rt_tgsigqueueinfo /* 335 */.long sys_perf_event_open

Modifying syscall_table_32.S….long sys_preadv.long sys_pwritev.long sys_rt_tgsigqueueinfo /* 335 */.long sys_perf_event_open.long sys_test_newsyscall /* 337 */

Add new system call to the end of the file. Remember the number – you’ll need it in userspace!

Modifying unistd.h

/* midfile */#define __NR_perf_event_open 241__SYSCALL(__NR_perf_event_open, sys_perf_event_open)

#undef __NR_syscalls#define __NR_syscalls 242/* midfile */

Can be found around line 623…

Modifying unistd.h

#define __NR_test_newsyscall 242__SYSCALL(__NR_test_newsyscall, sys_test_new_syscall)

Modifying unistd.h

#define __NR_test_newsyscall 242__SYSCALL(__NR_test_newsyscall, sys_test_new_syscall)

Modifying syscalls.h

asmlinkage long sys_perf_event_open(

struct perf_event_attr __user *attr_uptr,

pid_t pid, int cpu, int group_fd, unsigned

long flags);

#endif

/* EOF */

Modifying syscalls.h

asmlinkage long sys_perf_event_open(

struct perf_event_attr __user *attr_uptr,

pid_t pid, int cpu, int group_fd, unsigned

long flags);

asmlinkage long sys_test_newsyscall(int test_int);

#endif

/* EOF */

Modifying KERNEL_DIR/Makefile# Objects we will link into vmlinux / subdirs we need to visit

init-y := init/

drivers-y := drivers/ sound/ firmware/

net-y := net/

libs-y := lib/

core-y := usr/

endif # KBUILD_EXTMOD

Modifying KERNEL_DIR/Makefile# Objects we will link into vmlinux / subdirs we need to visit

init-y := init/

drivers-y := drivers/ sound/ firmware/

net-y := net/

libs-y := lib/

core-y := usr/ my_module/

endif # KBUILD_EXTMOD

Found around line 475… Can replace “my_module” with the name of your PROJECT_DIR

Getting it all to work

1. Re-compile the kernel

2. Install modules, install kernel

3. Make new initramfs image

4. Reboot

5. Test with a user-space program…

Sample User-space Program#include <stdio.h>#include <stdlib.h>#include <sys/syscall.h>#include <linux/unistd.h>

#define __SYS_TEST_ELEVATOR 337

int main(){ int test=5; long ret;

ret=syscall(__SYS_TEST_ELEVATOR, test); if(ret<0) perror("system call error"); else printf("Function successful, returned %i\n", ret);

return 0;}

syscall()

int syscall(int number, ...);

Performs the system call based on the system call’s number Number can be found in the syscall_table_32.S file (our example

was 337)

User-space Program Output Output when my_module not loaded

system call error: Function not implemented

Output when my_module loaded

Function successful, returned 5

Kthreads

Run the main logic of your module in a kthread!

Refresher: hello.c

#include <linux/init.h>#include <linux/module.h>MODULE_LICENSE(“Dual BSD/GPL”);

static int hello_init(void){ printk(KERN_ALERT “Hello, world!\n”); return 0;}

static void hello_exit(void){ printk(KERN_ALERT “Goodbye, sleepy world.\n”);}

module_init(hello_init);module_exit(hello_exit);

Kernel Modules

Remember, kernel modules are very event-based

We need a way to start an independent thread of execution in response to an event e.g. start_elevator() for project 2…

kthread_run

kthread_run(threadfn, data, namefmt, ...)

Creates a new thread and tells it to run threadfn – the name of the function the thread should run data – data pointer for threadfn (can be NULL if the function

does not take any args) namefmt – name of the thread (displayed during “ps” command)

Returns a task_struct

kthread_run example

struct task_struct *t;

t = kthread_run(run, NULL, “my_elevator");

if (IS_ERR(t)){

ret=PTR_ERR(t);

kthread_stop

int kthread_stop(struct task_struct * k);

Sets kthread_should_stop for k to return true, wakes the thread, and waits for the thread to exit

Returns the result of the thread function

kthread_stop_example

ret=kthread_stop(t);

if(ret != -EINTR)

printk("Main logic tread stopped.\n“);

Thread Function Example

static int run(void *arg){ /* Lock here */ while(!kthread_should_stop()) {

/* Do stuff */

/* Unlock here */ schedule(); /* Lock here */ }

/* Unlock here */ printk("%s: kernel thread exits.\n", __FUNCTION__); return 0;}

Thread Function Example

static int run(void *arg){ /* Lock here */ while(!kthread_should_stop()) {

/* Do stuff */

/* Unlock here */ schedule(); /* Lock here */ }

/* Unlock here */ printk("%s: kernel thread exits.\n", __FUNCTION__); return 0;}

schedule() is very important here. Why?

Inefficient Solution

Thread will continue to run even though it has nothing to do Eats up resources

Investigate the kthread interface to find ways to Put thread to sleep Wake up thread

There is more than one way to do this…

Debugging

Kernel Debugging Configurations Timing info on printks __depreciated logic Detection of hung tasks SLUB debugging Kernel memory leak detector Mutex/lock debugging Kmemcheck Check for stack overflow Linked list debugging

Select Kernel Hacking

Enable Debugging Options

Debugging through procfs

Necessary for elevator project! General process

Identify data to monitor in your module Create a proc entry to monitor this data Run your module Query /proc/<entry> for that information at any

Kernel Oops and Other Errors Kernel errors often only appear on first tty

(terminal interface) Why?

How can I see my first tty? On regular system – CTRL+ALT+F1

CTRL+ALT+F7 to go back to X screen On VMware – CTRL+ALT+SPACE+F1

CTRL+ALT+SPACE+F7 to go back to X screen

Reason for failure

Current drivers

Call Trace

Failed command

Defensive Programming

• Infinite loops and deadlocks at the kernel level hang your machine– Ctrl-Alt-Del has NO effect– Ctrl-C does not matter– Ctrl-D does not matter– You may only reboot

• How do you protect yourself?– Use schedule() strategically– Use preemptable versions of functions

Debugging Tools not Covered LTT – Linux Tracing Framework gdb – Invoking gbd on the kernel image kgdb – A remote debugger for the kernel Magic SysRq printk – Rate limiting, turning on/off

Next Time

Locks Linked lists Elevator algorithms

week 7 system calls, kernel threads, kernel debugging

sample system

system callspart

kernel sourcere

new system callsfun

kernel threads

kernel imageexecutable

kernel different

result of system callproject

Documents

suresh kernel debugging techniques

from printk to qemu: xen/linux kernel debugging

section 4 processes, kernel threads, user threads, locks ·...

kernel debugging tutorial

generation of debugging interfaces for linux kernel services

ladebug kernel debugging tutorial

intro to kernel debugging 1 _ matt's blog

reverse debugging of kernel failures in deployed systems

debugging kernel oops

debugging linux kernel problems

configuring kernel debugging on windows 7 with · pdf...

security for freebsd and kernel debugging

introduction to debugging the freebsd kernel · 2020. 1....

debugging and profiling in microsoft visualstudio with...

yvr18-308: kernel bugs and regressions debugging best

the os kernel: implementing processes and threads kernel...

windows kernel internals ii processes, threads,...

kernel debugging and tracing -...

debugging linux kernel with gdb - ku...

week 7 system calls, kernel threads, kernel...