freebsd and drivers
TRANSCRIPT
FreeBSD and DriversGili Yankovitch, Nyx Software Security Solutions
Key Points● What is FreeBSD?● FreeBSD Drivers: How to create, compile and run a driver.● Char devices● Network Hooking
○ L3○ L2
● Interaction with the network stack
What is FreeBSD?● “FreeBSD is a free UNIX-like operating system descended from Research
Unix via the Berkeley Software Distribution (BSD).” - Wikipedia● The BSD Project was founded in 1976 by Bill Joy.● Contained code written by AT&T (Who later sued people related to BSD)● In 1993 the first FreeBSD distribution was released.
○ Two years AFTER the Linux Kernel was founded.
What is FreeBSD?● https://www.freebsd.org/● Unlike Linux, it comes with a lot of user mode tools● Doesn’t come in many flavours (distributions)● Supported architectures: amd64, i386, ia64, powerpc, powerpc64, sparc64,
mips, armv6, aarch64.● Unfortunately, lacks a lot of features implemented in Linux.
○ Namespaces, Good L2 hooking (yay -_-) and more…
● It is very unfortunate but there is very little documentation onFreeBSD on the internet. :(
○ This means that if you are stuck, you need to deal with it on your own.■ True story.
How to get FreeBSD● Download
○ https://www.freebsd.org/where.html
● Git:○ https://github.com/freebsd
■ Yes they use GitHub.
● On a FreeBSD system, the sources are usually at:○ /usr/src/sys/
Folder Structure● In contrast to Linux, FreeBSD has a lot of folders in its root directory.
○ kern/ - Core kernel implementation.○ libkern/ - Core kernel libraries (printf, uprintf, strcpy etc…).○ fs/ - File systems implementation.○ net/ netinet/ - Core net and Inet implementation.○ sys/ - Include directory. Contains a lot of *.h files.○ amd64/ arm/ mips/ … - Architecture specific sources.○ modules/ dev/ - Drivers.○ ...
First And Foremost - Prints!● There are two types of prints from the kernel:
○ printf○ uprintf
● Both appear in dmesg● uprintf prints to your current console● printf prints to tty0
Char Devices
Compiling Our First Driverstatic int nethook_loader (struct module *m, int what, void *arg){ int err = 0; switch (what) { case MOD_LOAD: /* kldload */ uprintf ("Nethook KLD loaded. \n"); break; case MOD_UNLOAD : uprintf ("Nethook KLD unloaded. \n"); break; } return err;}static moduledata_t nethook_mod ={ "nethook", nethook_loader , NULL};DECLARE_MODULE (nethook, nethook_mod , SI_SUB_KLD , SI_ORDER_ANY );
nethook.c
Don’t return anything different than 0!Different value will prevent you from unloading the module!!
Compiling Our First Driver
● Yep. That simple.
SRCS=nethook.cKMOD=nethook
.include <bsd.kmod.mk>
Makefile
Running the Driver● Just like Linux, we need to inject it to the Kernel:
$ kldload ./nethook.ko
$ kldunload ./nethook.ko
$ kldstat
● Removing from Kernel:
● Modules list:
Creating A Char Device● This actually has a very good tutorial:
○ https://www.freebsd.org/doc/en/books/arch-handbook/driverbasics-char.html
● But here’s the snippets anyhow:○ Create a struct with function pointers to read, write, open, close.
■ ioctl seems to fail with this method...
/* Character device entry points */static struct cdevsw echo_cdevsw = {
.d_version = D_VERSION ,
.d_open = echo_open ,
.d_close = echo_close ,
.d_read = echo_read ,
.d_write = echo_write ,
.d_name = "echo",};
● Just like Linux...
Creating A Char Device● Then all you need to do is register:
static struct cdev *echo_dev;
static int nethook_loader (struct module *m, int what, void *arg){...error = make_dev_p (MAKEDEV_CHECKNAME | MAKEDEV_WAITOK ,
&echo_dev, &echo_cdevsw , 0, UID_ROOT , GID_WHEEL , 0600, "echo");
…}
Char Device Kernel ObjectPointer to File Operations
Owner UID of File System NodeOwner GID of File System NodeFile System Node PermissionsName of File System Node
Creating A Char Device● Read operation● Note the struct uio:
○ uio_resid - Space left in buffer sent from user mode (read length usually)○ uio_offset - Current write offset
● uio knows whether it’s a read or a write depending on current action.○ Here uiomove() writes from kernel buffer to user mode buffer.
static int echo_read (struct cdev *dev __unused , struct uio *uio, int ioflag __unused ){
size_t amt;int error;
amt = MIN(uio->uio_resid, uio->uio_offset >= echomsg->len + 1 ? 0 : echomsg ->len + 1 - uio->uio_offset );
if ((error = uiomove(echomsg->msg, amt, uio)) != 0)uprintf("uiomove failed! \n");
return (error);}
Creating A Char Device● Corresponding write().
static int echo_write (struct cdev *dev __unused , struct uio *uio, int ioflag __unused ){
size_t amt;int error;
if (uio->uio_offset != 0 && (uio->uio_offset != echomsg->len))return (EINVAL);
/* Copy the string in from user memory to kernel memory */amt = MIN(uio->uio_resid, (BUFFERSIZE - echomsg->len));
error = uiomove(echomsg->msg + uio->uio_offset , amt, uio);
if (error != 0)uprintf("Write failed: bad address! \n");
return (error);}
Networking● DISCLAIMER:
○ Before you begin to build your own network driver, be absolutely sure you understand the below.
● OK lets continue...
Network Stack
Networking● Just like Linux has its skb structure, FreeBSD has a basic buffer system● It’s called: mbuf● mbufs are buffer chains of size 256
○ Larger buffers are possible in an mbuf cluster but unfortunately usually it’s not the case.
● When you get a packet larger than 256 bytes, you get an mbuf chain● Mellanox created a module called OFED to help port drivers from Linux to
Freebsd.○ It’s a great place to start learning about networking in FreeBSD.○ Unfortunately it lacks a HELL LOT of functionality sometimes needed.
Meet struct mbuf
● Yeah I know it’s weird and complicated.○ Our interest is in m_hdr and in m_dat.M_databuf (Which means a normal packet)
struct mbuf { struct m_hdr m_hdr ; union { struct { struct pkthdr MH_pkthdr ; /* M_PKTHDR set */ union { struct m_ext MH_ext ; /* M_EXT set */ char MH_databuf [MHLEN]; } MH_dat; } MH; char M_databuf [MLEN]; /* !M_PKTHDR, !M_EXT */ } M_dat;};
/sys/mbuf.h
Meet struct m_hdr
● mh_next - Already mentioned this is an mbuf chain● mh_nextpkt - mbufs provide us with a linked-list of packets storage place.● mh_data - Pointer to beginning of data within the data buffer● mh_len - Length of data in this mbuf
struct m_hdr { struct mbuf *mh_next; /* next buffer in chain */ struct mbuf *mh_nextpkt ; /* next chain in queue/record */ caddr_t mh_data ; /* location of data */ int32_t mh_len ; /* amount of data in this mbuf */ uint32_t mh_type :8, /* type of data in this mbuf */ mh_flags:24; /* flags; see below */#if !defined(__LP64__) uint32_t mh_pad ; /* pad for 64bit alignment */#endif};
/sys/mbuf.h
nbuf Structure
mbuf
mh_data
mbuf aaaaa mh_next
mh_len
mbuf aaaaa
mbuf aaaaa
mh_
next
pkt
Whatever you do, do NOT access these directly!● Seriously. For everything you need there’s a function.● When in doubt, see man mbuf (9).● mbuf function names are non indicative, so I’ll explain a few here:
Allocating and freeing buffers● m_get(int how, int type) - Allocates a new mbuf and sets its type.● m_free(struct mbuf *m) - Frees a single mbuf.● m_freem(struct mbuf *m) - Frees an entire mbuf chain.● m_dup(struct mbuf *m, int how) - Duplicates an entire mbuf.● m_copym(struct mbuf *mbuf, int offset, int len, int how) - Copy only a portion
of the mbuf to a new mbuf chain.● m_copydata(const struct mbuf *mbuf, int offset, int len, caddr_t buf) - Copy
the mbuf data to a different buffer.● m_length(struct mbuf *m, struct mbuf ** last) - Returns the entire mbuf chain
length (in bytes).
Shorten or Lengthen the Buffer● m_adj(struct mbuf *m, int len) - Shorten the buffer from the beginning.
mbuf aaaaa mbuf
mh_data
mh_nextmbuf
mh_len mh_lenmh_len
mbuf
mh_data
void shorten_my_mbuf (struct mbuf *m){ m_adj(m);}
m is still pointing to the first mbuf!!
Shorten or Lengthen the Buffer● m_prepend(struct mbuf *m, int len, int how) - Prepend len bytes in te
beginning.
mbuf aaaaa mbuf
mh_data
mh_next
mh_len mh_len
mbuf aaaaa mh_next
mh_data mh_data
mh_len mh_len
mbuf
Accessing data● Because mbufs are divided to 256-bytes parts, header might fall between two
mbufs.● Accessing the header linearly might cause an unexpected behaviour.
mbuf aaaaa mbufhea derWrite.. OVERFLOW...
● NEVER access directly, or before using this:● m_pulldown(struct mbuf *mbuf, int offset, int len, int *offsetp)
mbuf aaaaa mbufhea dermbuf a mbufheader
Might allocate a new mbuf
Interfaces● Interfaces in FreeBSD are represented by struct ifnet
struct ifnet { struct vnet *if_vnet; /* pointer to network stack instance */ TAILQ_ENTRY (ifnet) if_link; /* all struct ifnets are chained */... char if_xname [IFNAMSIZ]; /* external name (name + unit) */... struct ifaddrhead if_addrhead ; /* linked list of addresses per if */... u_short if_index ; /* numeric abbreviation for this if */
int (*if_output) /* output routine (enqueue) */ (struct ifnet *, struct mbuf *, const struct sockaddr *, struct route *); void (*if_input) /* input routine (from h/w driver) */ (struct ifnet *, struct mbuf *);… void (*if_transmit ) /* initiate output routine */ (struct ifnet *, struct mbuf *); u_int if_fib ; /* interface FIB */...};
/net/if_var.h
L3 Hooking● Just like Linux has netfilter, FreeBSD has a framework called pfil● It enables to create a list of filters for both IN and OUT packets.● Unlike Linux, pfil allows hooking in only one place for incoming and outgoing
packets.
L3 Hooking● Hooking is easy. Use:
struct pfil_head *pfh_inet;
/* Initializing L3 Hooking */ if (!(pfh_inet = pfil_head_get (PFIL_TYPE_AF , AF_INET))) { uprintf ("Failed getting packet filter head \n");
return ESRCH; }
pfil_add_hook(in_filter, NULL, PFIL_IN | PFIL_WAITOK, pfh_inet);
static int in_filter(void *arg, struct mbuf **m, struct ifnet *ifp, int dir, struct inpcb *inp)
● Then, register the callback:
● Hook signature:
The L2-L3 Input StackDriver
ifp->if_input()
ether_input_internal
BPF
LAGG
ng_ether
BridgingHooks
ether_demuxVLAN Handlingvlan_input_p()
IP: ip_input() IPv6 ARP: arpintr() ATALK AARP
pfil_run_hooks()
PFil
The L2-L3 Input Stack
Driver
ip_output
pfil_run_hooks()
if_output()
ether_output
Bridge
ng_ether
PFilHooks
ifp->if_transmit()
L2 Hooking● Apparently, it’s not as trivial hooking to the network stack in L2● For example, in order to make Libpcap work, NIC drivers need to explicitly call
Libpcap kernel hooks to redirect L2 flow to it.● Suggested implementation in user mode:
○ BPF - Explained in previous lectures○ Libpcap - Explained above
○ Nethook - Memory-mapping based network handling. Exists in both Linux, Windows and FreeBSD.
● Despite what is said above, you can use netgraph to attach to ng_ether.○ There is a way to use it more easily. Source code will be uploaded later.
● DDB is the static kernel debugger. You can read about it here:○ https://www.freebsd.org/cgi/man.cgi?ddb(4)
● Compile kernel with:○ Options DDB
● Compiling the kernel:○ Configs are in:
■ amd64/conf/GENERIC■ Always copy GENERIC to a new file and edit it.
Other useful tips
$ cd /usr/src/$ make buildkernel KERNCONF=GENERIC.MYCONF && make installkernel KERNCONF=GENERIC.MYCONF && shutdown -r now
● If kernel hangs, useful VirtualBox command (Opens DDB):$ VBoxManage debugvm <VM Name> injectnmi
Questions? :)