Running Applications on theNetBSD Rump Kernel

Justin Cormack @justincormack

Rump kernels

Slides at http://eurobsdcon.myriabit.eu/

3

What is a rump kernel?It is the (NetBSD) kernel without support for

• executing binaries

• scheduling threads

• managing hardware privilege levels

• most memory management eg virtual memory

4

Just drivers• Drivers are the bit of the kernel that turns raw hardware into nice

abstractions like files and sockets

• Get something else (host) to do the other parts (host environment)

• Or do without it, just compile in single purpose application.

5

What use is that?• You get the missing capabilities from a hypercall layer.

• Like a very simple virtual machine

• man 3 rumpuser

• Provides memory allocation, threads, mutexes, console output, random

numbers, clock

• This can run in userspace for example, but can be implemented in any

environment fairy easily

6

What use is that?• Original use cases around testing drivers.

• You can test a kernel driver without booting the OS.

• You can test much of the OS without booting into it.

• Debugging is easy as it is just a userpsace program

• Also for running kernel drivers in userspace eg for file systems

7

How was it used?• Typically either written for rump like the NetBSD tests

rump_init(); rump_sys_mkdir(path, 0777);

• or using LD_PRELOAD to change some calls using rumphijack

library

• The interface was essentially the syscall interface (minus parts not

supported by rump)

8

New developments

Xen port• In August last year we got a Xen port of the rump kernel working. In

addition to porting the hypercall layer, we also got NetBSD libc to

compile against the rump kernel.

• We replaced the syscalls in libc with calls to the rump kernel syscall

implementations.

• This enabled running real applications, in particular we had LuaJIT

running, and a simple web server.

• Not just syscalls, more of a POSIX API.

10

Increased portabilityThe POSIX implementation of the rump kernel now runs on

• NetBSD

• FreeBSD, OpenBSD, DragonflyBSD

• Solaris

• Linux, including Android

• on architectures including arm, x86, amd64, powerpc, sparc64 and mips

11

Increased testingThere is continuous integration using TravisCI and buildbot

• Tests all commits to the buildrump.sh script on multiple operating

systems and architectures

• Tests NetBSD -current hourly on multiple operating systems and

architectures

• Good test of NetBSD portability issues

• Uses ljsyscall to test functionality - good syscall coverage, fast.

12

Continuous integration

13

Runningapplications

rumprun-posix• Late last year I started work on rumprun, which allowed running

unmodified NetBSD binaries with the rump kernel.

• The main issue with this is that you have two libcs involved, so there are

symbol conflicts. However with judicious symbol renaming it turned out

to be possible to fix this.

• In particular you can compile userspace binaries for the core NetBSD

commands like ifconfig, etc.

• Can test userspace without booting into OS

15

How does it work?• Compile NetBSD object code and libraries on host machine

• Link code and libraries to new object with -nostdinc

• Rename read symbols to rump___sysimpl_read . . .

• Fix up main and other things which will be called from a stub

• Localise symbols to avoid conflicts

• Link with host libc and rump libraries

• See script for more details, it is messy...

16

# ./rumpdyn/bin/rump_allserver unix://sock

# export RUMP_SERVER=unix://sock

# ./bin/ifconfig

lo0: flags=8049 mtu 33648

inet6 ::1 prefixlen 128

inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1

inet 127.0.0.1 netmask 0xff000000

# ./bin/ping -o 127.0.0.1

PING 127.0.0.1 (127.0.0.1): 64 data bytes

64 bytes from 127.0.0.1: icmp_seq=0 ttl=255 time=0.000000 ms

----127.0.0.1 PING Statistics----

1 packets transmitted, 1 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.000000/0.000000/0.000000/0.000000 ms

17

Supported commandsarp cp dumpfs fsck_msdos ls mount_ext2fs ndp pax

reboot sysctl wpa_supplicant cat dd ed halt mkdir

mount_ffs newfs pcictl rm umount cgdconfig df fsck

ifconfig mknod mount_msdos newfs_ext2fs ping rmdir

vnconfig chmod disklabel fsck_ext2fs ktrace modstat

mount_tmpfs newfs_msdos ping6 rndctl wlanctl chown

dump fsck_ffs ln mount mv npfctl raidctl route

wpa_passphrase

18

What runs?• Almost anything in the NetBSD core can probably be compiled

• There is now a cross compiler for rumprun, that should work with most

code

• There is experimental pthreads support, not yet well tested

• Programs that fork or use other things not supported by rump may be a

problem

• Some of the missing rump features are emulated or partially emulated,

eg anon mmap

19

rumprun - TODOs• Clean up and upstream libc build modifications

• Fix other architectures, currently only working on x86, amd64

• Build more of userspace tools and libraries

• Use for NetBSD tests - ability to run tests on a kernel without booting it

very useful when developing.

• Improve build process

• Continuous integration and testing on NetBSD current.

20

More newdevelopments

rump on green threads• A few months back I added a "green threads" userspace implementation

to the userspace rump implementation.

• The default implementation uses pthreads.

• Uniprocessor, single thread

• Useful for embedded implementations

• Also useful if you want to make a deterministic implementation

22

PCI support• Some PCI support has been added

• Linux userspace support.

• Linux has uio and vfio frameworks for userspace pci drivers.

• uio supported now, only works for some cards

• vfio may be later; requires an iommu.

• Can be used for developing NetBSD drivers even so - used for wireless

development

• Really want to add BSD support

23

rump on bare metal• New project to run a rump kernel on (currently x86) bare metal

• https://github.com/rumpkernel/rumpuser-baremetal

• Includes PCI support, with virtio drivers for KVM, bhyve

• Only a few hundred lines of code to deal with interrupts etc, written in a

week

• Planning ARM port to mbed

24

rump on microkernels• Genode, a microkernel OS framework, has started using the rump kernel

to support NetBSD file system drivers on microkernels.

• Less than 3,000 lines of (untrusted) glue code.

• See genode.org for more

• Minix3 is another potential user, it already uses a lot of NetBSD code.

25

Logo

26

Architecture

Four different environments• hosted, e.g. userspace

• paravirtualized, e.g. Xen

• "bare metal", e.g. hardware or hypervisor with virtio

• microkernels (as servers)

28

Architecture

29

Documentation• Documentation is much improved.

• All at http://wiki.rumpkernel.org/

• Best places to start are Getting Started and Kernel development

tutorials.

• A longer introduction is the rump kernel book

30

Use cases1. Driver development

2. Tests

3. Drivers for other environments

4. Applications with userspace drivers, eg networking

5. Running code securely eg file system code

6. Very lightweight "containers" with their own OS library

7. .. .

31

What needs doing?• Upstream and improve rumprun

• Improve documentation

• Portability: native Windows, OSX

• Userspace IP stacks: need good performance on 10Gb

• dogfooding

• .. .

32

Get involved• http://rumpkernel.org/

• Freenode #rumpkernel

• Mailing list rumpkernel-users

• twitter @rumpkernel

• 25 November operatingsystems.io conference in London

• 26 November hackday in London

• Fosdem 2015

33