a look at the netsniff-ng toolkit - a suite of high - jon schipp

48
A Look at the Netsni-NG Toolkit A Suite of High-Performance Networking Tools Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net Derbycon 2013 Louisville, KY Sept. 29, 2013 Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net A Look at the Netsni-NG Toolkit

Upload: others

Post on 18-Feb-2022

1 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

A Look at the Netsni↵-NG ToolkitA Suite of High-Performance Networking Tools

Jon Schipp

E-mail:

[email protected]

Web:

jonschipp.com, sickbits.net

Derbycon 2013

Louisville, KY

Sept. 29, 2013

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 2: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

About Me

Security Engineer1 National Center for Supercomputing Applications

Founder1 Dubois County Linux User Group2 Southern Indiana Computer Klub

Contributer to Netsni↵-NG1 Documentation, review, testing, etc.2 Not an expert, still learning

Network security monitoring junkie

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 3: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Background

Started in 2009 for fun; GNU GPL, version 2.0

1 Packages available on Debian, Fedora, Red Hat, Gentoo, ArchLinux, Slackware etc.

2 Included in Security Onion, NST, Xplico’s Network ForensicToolkit, and GRML Linux

Analysis of network problems

Debugging tool for network (protocol-)development

Tra�c monitoring, security auditing, stress testing, and more

Maintainers: Daniel Borkmann (Red Hat) and Tobias Klauser(University of Zurich)

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 4: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Toolkit Overview

netsni↵-ng - a high-performance zero-copy analyzer, pcapcapturing and replaying tool

trafgen - a high-performance zero-copy network tra�cgenerator

flowtop - a top-like netfilter connection tracking tool

mausezahn - a packet generator and analyzer for HW/SWappliances with a Cisco-CLI

bpfc - a Berkeley Packet Filter (BPF) compiler with Linuxextensions

ifpps - a top-like kernel networking and system statistics tool

curvetun - a lightweight multiuser IP tunnel based on ellipticcurve cryptography

astraceroute - an autonomous system (AS) trace route utility

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 5: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Big Picture

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 6: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Gain

In sum, about as fast you can get from user space{netsni↵-ng, trafgen}

3rd party drivers may be faster (e.g. Myricom Sni↵er10G)

Use of PF PACKET’s zero-copy RING’s (TX and RX)

1 Fewer context switches and interrupts2 Less calls to sendto() and recvfrom()3 Bypass the normal packet processing path through the

networking stack

Control over entire packet including data-link layer

Optimized for Linux

Small code footprint

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 7: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Active Development

Netsni↵-NG is at good point in its life cycle1 A lot of new additions and features over the last year

Netsni↵-NG’s core developers are contributors to the Linuxkernel (netdev)

1 Result: Integration of latest developments into Netsni↵-NG

Reviewing the source code to see use of latest additions

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 8: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Latest Developments: TPACKET V3

Addition to the kernel that allows for moree�cient capturing 1

1 Available in Linux 3.2+

Improvements1 15-20% reduction in cpu-usage2 20% increase in packet capture rate

Implementation1 Netsni↵-NG: Yes2 Libpcap: No, still using TPACKET V2

libpcap (and thus programs that use) use TPACKET V2

Reduces cache misses in the translation lookaside bu↵er (TLB)

Writes packets linearly rather than in slots

1

https://lkml.org/lkml/2011/6/21/463

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 9: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Latest Developments: ANC PAY OFFSET

New addition to the kernel’s BPF code 2

1 Available in Linux kernel 3.10+

Instruction provides a filter for packet headers (no payload)

Implementation1 Bpfc: Yes (Instruction: po↵)2 Libpcap: No

$ cat header.bpfc

ld poff

ret a

$ bpfc header.bpfc > header.bpf

$ netsniff-ng --in eth0 --filter header.bpf

2

https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit

/?id=3e5289d5e3f98b7b5b8cac32e9e5a7004c067436

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 10: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Latest Developments: Virtual Netlink Monitoring Device

New addition allows for creation of a virtual netlink device 3

Monitor netlink messages between kernel and user space bymaking them available to PF PACKET sockets

1 Available in Linux kernel 3.11 4

Implementation1 Packet analyzer neutral

modprobe nlmon

ip link add type nlmon

ip link set nlmon0 up

netsniff-ng --in nlmon0 --out dump.pcap --silent

3

http://comments.gmane.org/gmane.linux.network/273556

4

http://seclists.org/tcpdump/2013/q3/21

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 11: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Netsni↵-NG

Fast network analyzer, pcap recorder, pcap replayerSpeeds matching PF RING transparent mode=0 (default)

Uses PF PACKET sockets with mmap(2) RX and TX RING 5

PCAP recording backend for Security Onion, Xplico, NST etc.

Supports di↵erent pcap types and I/O methods,i.e. scatter-gather, mmap(2), and read(2)/write(2)

Enables the BPF JIT compiler 6

Performs basic OS tuning for improved reception 7

5

https://github.com/torvalds/linux/blob/master/Documentation/networking/packet mmap.txt

6

https://lwn.net/Articles/437981/

7

https://github.com/borkmann/netsni↵-ng/blob/master/sock.c

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 12: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Netsni↵-NG, Packet Types

Netsni↵-NG also supports Linux packet(7) types 8 9

PACKET HOST for a packet addressed to the local host$ netsniff-ng --in eth0 --host

PACKET BROADCAST for a physical layer broadcast packet$ netsniff-ng --in eth0 --broadcast

PACKET MULTICAST for a packet sent to a physical layermulticast address$ netsniff-ng --in eth0 --multicast

PACKET OTHERHOST for a packet to some other host thathas been caught by a device driver in promiscuous mode$ netsniff-ng --in eth0 --others

PACKET OUTGOING for a packet originated from the localhost that is looped back to a packet socket$ netsniff-ng --in eth0 --outgoing

8

https://github.com/torvalds/linux/blob/master/include/uapi/linux/if packet.h

9

http://man7.org/linux/man-pages/man7/packet.7.html

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 13: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Netsni↵-NG, Examples

Full content, daily packet capture. Excels at this.

$ netsniff-ng --in eth1 --out /pcaps/ --silent--ring-size 2GiB --prio-high --interval 24hrs

Capture multiple raw 802.11 tra�c pcap files, each 1GiB

$ netsniff-ng -i wlan0 -o raw.pcap --rfraw -s--interval 1GiB --bind-cpu 0

Replay interface to interface

$ netsniff-ng --in eth0 --out eth1 --type host--kernel-pull 100 --mmap -s -b 3

Apply BPF filter

$ netsniff-ng -i eth0 -f "tcp and port 22”$ netsniff-ng -i eth0 -f filter.bpf$ netsniff-ng -i eth0 icmp

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 14: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Netsni↵-NG, Example Output (ICMP Echo)

Verbose (--verbose)

Default RX ring bu↵er size, number of frames that can fit inbu↵er, max frame size (increase w/ --jumbo-support)

Packet

MAC OUI lookup, Geo IP, protocol dissector, ASCII, HexStatistics

Incoming, passed filter, dropped, drop rate, capture duration

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 15: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Trafgen

Fast multithreaded low-level network tra�c generator

Uses AF PACKET sockets with mmap(2)ed TX RING

Powerful packet configuration syntax, more flexible thanpktgen

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 16: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Trafgen vs. Pktgen

Pktgen is faster on smaller packet sizes, nearly the sameotherwise

e.g. 64 byte: Trafgen: 520,000 PPS; Pktgen: 620,000 PPS

Trafgen is more customizableTrafgen: Entire packet; Pktgen: Not payload, few options, etc.

Packet ConfigurationPktgen is configured by sending commands to procfsTrafgen is configured by reading a file with trafgen expressions

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 17: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Trafgen vs. Pktgen Cont.

Pktgen example: 64 byte frameecho "clone skb 0" > /proc/net/pktgen/kpktgend 0

echo "pkt size 60" > /proc/net/pktgen/kpktgend 0

echo "count 15000000" > /proc/net/pktgen/kpktgend 0

echo "srcmac 00:1e:c9:b2:d5:41" > /proc/net/pktgen/kpktgend 0

echo "dstmac 00:1e:c9:b2:d7:51" > /proc/net/pktgen/kpktgend 0

echo "dst 10.1.1.1" > /proc/net/pktgen/kpktgend 0

echo "udp src min 514" > /proc/net/pktgen/kpktgend 0

echo "udp src max 514" > /proc/net/pktgen/kpktgend 0

echo "udp dst min 514" > /proc/net/pktgen/kpktgend 0

echo "udp dst max 514" > /proc/net/pktgen/kpktgend 0

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 18: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Trafgen vs. Pktgen Cont.

Trafgen example: 64 byte frame{

0x00, 0x1e, 0xc9, 0xb2, 0xd7, 0x4f, 0x00, 0x1e, 0xc9, 0xb2,0xd5, 0x41, 0x08, 0x00, 0x45, 0x00, 0x00, 0x2e, 0x00, 0x00,0x00, 0x00, 0x20, 0x11, 0x8f, 0xbe, 0x00, 0x00, 0x00, 0x00,0x0a, 0x01, 0x01, 0x01, 0x00, 0x09, 0x00, 0x09, 0x00, 0x1a,0x00, 0x00, 0xbe, 0x9b, 0xe9, 0x55, 0x00, 0x00, 0x00, 0x01,0x52, 0x2f, 0xc0, 0xaf, 0x00, 0x06, 0x62, 0x42, 0x00, 0x00,

}

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 19: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Trafgen, Performance

1 Gb linerate reported on modern machines (unaware of 10Gb)Dell PowerEdge 1950 (approx. 6-7 years old)e.g. $ trafgen --in $cfg --out $int --cpus 1 -t 01Gb line-rate attempts (Broadcom NetXtreme II BCM5708 PCI-X)

1 64 - 24%

2 128 - 48%

3 256 - 92%

4 512 - 96%

5 1024 - 97%

6 1280 - 98%

7 1518 - 98%

10Gb line-rate attempts (Myricom 10G-PCIE-8B)1 64 - 3%

2 128 - 6%

3 256 - 12%

4 512 - 24%

5 768 - 34%

6 1024 - 44%

7 1280 - 52%

8 1518 - 61%

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 20: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Trafgen, Examples

Send 15mio packets using all cores

$ trafgen --in configs/64.cfg --out eth2 --num15000000Generate packets and perform mortality test (ICMP Echo)

$ trafgen --dev eth0 --conf fuzzing.cfg--smoke-test 10.0.0.1

Generate raw 802.11 frames

$ trafgen --dev wlan0 --rfraw --confbeacon-test.cfg --verbose --cpus 2

Pass built-in configuration through stdin

$ trafgen -e |trafgen --in - --out eth0 --num 1 --cpp

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 21: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Trafgen Expression Language - Features

Macros1 fill(), rnd(), drnd(), const8()...const64(), seqinc(), ddec() etc.2 fill() - fill(x, n) - fill(0x↵, 10) - Fill (repeat) data x for n bytes3 rnd() - rnd(n) - rnd(582) - Generate n bytes of random data

Data Elements & Types1 Hexidecimal, Decimal, Octal, Binary, Characters, Strings2 0x↵, 100, 05607, 0b11111111, ’A’, ”Sweet!”3 ...and shellcode, ”\xca \xfe \xba \xbe”

C Preprocessor1 Ability to pass configuration through CPP ( --cpp )

2 C #define’s can be used via #include <stddef.h>

3 e.g. #define ETH P IP 0x0800

/* Internet Protocol packet */

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 22: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Trafgen Packet Configuration Examples

Single packet definition: { ..packet data.. }

Create a 64 byte packet of all 0x↵’s: { fill {0xff, 64} }

Packet with 512 bytes of random data: { drnd(512) }

$ python -c "print ’{\n’ + ’ \" MyPacket\",\n’ * 8 + ’}\n’"

{"MyPacket",..."MyPacket",

}$ netsniff-ng --in eth0

eth0 56 1366850783s.90970749ns

[ Eth MAC (74:79:50:61:63:6b => 79:50:61:63:6b:65), Proto (0x6574) ]

[ Vendor (Unknown => Unknown) ]

[ Chr etMyPacketMyPacketMyPacketMyPacketMyPacketMyPacket ]

[ Hex 65 74 4d 79 50 61 63 6b 65..]

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 23: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Trafgen Packet Configuration Examples Cont. 1

$ cat udp.cfg

/* Note: dynamic elements make trafgen slower! *//* Use CPU 1-4 */

cpu(0:3): {

/* Ethernet */drnd(12),

/* IPv4 Protocol */c16(0x0800),

/* IP Header */drnd(20),

/* UDP Header */drnd(8),

/* Data */rnd(1472)

}

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 24: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Trafgen Packet Configuration Examples, Cont. 2

$ trafgen -e/* Note: dynamic elements make trafgen slower! */#include <stddef.h>

{/* MAC Destination */fill(0xff, ETH ALEN),

/* MAC Source */0x00, 0x02, 0xb3, drnd(3),

/* IPv4 Protocol */c16(ETH P IP),

/* IPv4 Version, IHL, TOS */0b01000101, 0,

/* IPv4 Total Len */c16(59),

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 25: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

/* IPv4 Ident */drnd(2),

/* IPv4 Flags, Frag Off */0b01000000, 0,

/* IPv4 TTL */64,

/* Proto TCP */0x06,

/* IPv4 Checksum (IP header from, to) */csumip(14, 33),

/* Source IP */drnd(4),

/* Dest IP */drnd(4),

/* TCP Source Port */drnd(2),

/* TCP Dest Port */c16(80),

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 26: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

/* TCP Sequence Number */drnd(4),

/* TCP Ackn. Number */c32(0),

/* TCP Header length + TCP SYN/ECN Flag */c16((0x8 << 12) | TCP FLAG SYN | TCP FLAG ECE)

/* Window Size */c16(16),

/* TCP Checksum (offset IP, offset TCP) */csumtcp(14, 34),

/* TCP Options */0x00, 0x00, 0x01, 0x01, 0x08, 0x0a, 0x06,0x91, 0x68, 0x7d, 0x06, 0x91, 0x68, 0x6f,

/* Data blob */"gotcha!",

}

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 27: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

In Conjunction: Netsni↵-NG & Trafgen

PCAP-2-Trafgen

$ netsniff-ng --in pkt.pcap --out - --num 1 -f tcp

{0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x00, 0x02, 0xb3, 0xba,

0x34, 0x93, 0x08, 0x00, 0x45, 0x00, 0x00, 0x3b, 0x97, 0xea,

0x40, 0x00, 0x40, 0x06, 0xc1, 0x1d, 0x6a, 0x11, 0x43, 0x2e,

0xa0, 0x1f, 0x94, 0x56, 0x96, 0xa0, 0x00, 0x50, 0xae, 0xd6,

0x7e, 0x01, 0x00, 0x00, 0x00, 0x00, 0x80, 0x42, 0x00, 0x10,

0x8d, 0xb3, 0x00, 0x00, 0x01, 0x01, 0x08, 0x0a, 0x06, 0x91,

0x68, 0x7d, 0x06, 0x91, 0x68, 0x6f, 0x67, 0x6f, 0x74, 0x63,

0x68, 0x61, 0x21,

}

Together now!

$ netsniff-ng --in pkt.pcap --out - --num 1 -f tcp |

trafgen --in - --out eth0 --num 1000000

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 28: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Gencfg: Packet Configuration Generator

Gencfg - Shell script for testing and generating packetconfigurations for trafgen 10

Configurable Ethernet, IP, and port addresses

Currently generates configurations for:1 UDP Syslog Messages2 RFC2544 Ethernet size frame set3 802.11 beacon frames with configurable 8 character SSID

Examples:$ gencfg -G rfc2544 -s 192.168.1.10 -d 192.168.1.20 -P 123

$ gencfg -G syslog -d 10.1.1.2 -M 00:0c:29:8d:4d:a2

$ gencfg -G beacon -T "FreeWifi" | trafgen --in -

--out wlan0 --rfraw --num 1000

10

https://github.com/jonschipp/gencfg

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 29: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Case Study: Performance Metrics

Packet capture performance test

Create and generate packet distributions with Trafgen 11

1 RFC 2544 ”Benchmarking Methodology for NetworkInterconnect Devices”, Section 9.1

2 i.e. 64, 128, 256, 512, 1024, 1280, 15183 Generate with scripting language e.g. Python, ”0x↵” * 64

Verify tra�c on receiving machine with Ifpps

Script a timer and SIGINT for packet capturing tools

Calculate average packet loss e.g. Awk plus capinfos

11

https://github.com/jonschipp/gencfg

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 30: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Case Study: Full Content Daily PCAPs

Daily PCAP capture and rotation with Netsni↵-NG

1 Record packets from 3 NIC’s attached to 3 separate networks2 Bind to specific CPU and run process as high priority3 Run as non-root user and redirect stats to file

Resulting filenames, $timestamp.pcap e.g 1367028821.pcap

$ netsniff-ng --bind-cpu 0 --in eth1 --out /pcap/dmz/

--prefix "" --ring-size 4GiB --silent --prio-high --verbose

--interval 24hrs --user 1000 --group 1000 >>dmz.stats 2>&1

$ netsniff-ng --bind-cpu 1 --in eth2 --out /pcap/wlan/

--prefix "" --ring-size 512MiB -Q --silent --prio-high

--verbose --interval 24hrs --user 1000 --group 1000

>>wlan.stats 2>&1

$ netsniff-ng --bind-cpu 2 --in eth3 --out /pcap/lan/

--prefix "" --ring-size 4GiB -Q --silent --prio-high

--verbose--interval 24hrs --user 1000 --group 1000

>>lan.stats 2>&1

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 31: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Performance Considerations: Hardware

High-speed capture/generation requires careful system tuning,end of story.

1 NAPI, RSS (for multiqueue NICs), RPS, XPS, Queue Length,O✏oading methods, Interrupt Throttling and Coalescence

2 Multi-core CPU, hardware locality, I/OATDMA and DirectCache Access

3 Plenty of RAM for large mmap(2)’d bu↵ers, fast diskconfiguration, fast bus to transmit data e.g. PCIe

Every system is di↵erent and should be treated as such

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 32: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Performance Considerations: Software

Latest mainline Linux kernel for the latest fixes and features

Find optimal bu↵er sizes for sockets and driver queues$ sysctl -w net.core.rmem max=8388608

$ ifconfig eth0 txqueuelen 10000

Setting SMP and IRQ a�nity, exploit cache locality$ echo "0" > /proc/irq/48/smp affinity list

$ taskset -c 0 trafgen --cpus 1 ...

BPF JIT compiler and Load Balancing (Hardware or Software)$ echo "1" > /proc/sys/net/core/bpf jit enable

Write PCAP’s to RAM based filesystem e.g. tmpfs$ mount -t tmpfs -o size=24g tmpfs /mnt/ramdisk

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 33: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Bpfc

Is a Berkely Packet Filter compiler

Supports internal Linux extensions

Write/compile seccomp-bpf 12and xt bpf 13 (netfilter) filters$ iptables -A INPUT -m bpf --bytecode-file my.bpf

Filter opcodes can be passed to netsni↵-ng:$ bpfc arp > arp.bpf && netsniff-ng -f arp.bpf

Useful for:1 Complex filters that cannot be expressed with

the high-level syntax2 Low-level kernel BPF machine/JIT debugging3 Learn how to read, write, and debug BPF (pedagogical)

12

https://en.wikipedia.org/wiki/Seccomp

13

http://www.spinics.net/lists/netfilter-devel/msg25306.html

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 34: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Linux Kernel Extensions

Bpfc supports BPF extensions 14

#pktlen Length of packet

#proto Ethernet type field

#type Packet type

#poff Detected payload start offset

#ifidx Interface index

#nla Netlink attribute of type X with offset A

#nlan Nested Netlink attribute of type X with offset A

#mark Packet mark

#queue NIC queue index

#hatype NIC hardware type

#rxhash Receive hash

#cpu Current CPU

#vlant VLAN TCI value

#vlanp VLAN present

14

https://github.com/torvalds/linux/blob/master/include/linux/filter.h

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 35: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Bpfc, Examples

High-level BPF (Mnemonics)libpcap equiv ’ether src aa:bb:cc:dd:ee:↵’

$ cat ethernet.bpfc

ld [8]

jne #0xccddeeff, Drop

ldh [6]

jne #0xaabb, Drop

ret #1514

Drop: ret #0

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 36: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Bpfc, Examples Cont.

Low-level BPF (Op Codes)Compile source file to BPF machine language:

$ bpfc ethernet.bpfc

{ 0x20, 0, 0, 0x00000008 },{ 0x15, 0, 3, 0xccddeeff },{ 0x28, 0, 0, 0x00000006 },{ 0x15, 0, 1, 0x0000aabb },{ 0x6, 0, 0, 0x000005ea },{ 0x6, 0, 0, 0x00000000 },

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 37: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Bpfc, Examples Cont.

Use of linux kernel extension, filter by CPU

$ cat cpu.bpfcld #cpujeq #0,L1,L2L1: ret #-1L2: ret #0

Compile filter

$ bpfc cpu.bpfc > cpu.bpf{ 0x20, 0, 0, 0xfffff024 },{ 0x15, 0, 1, 0x00000000 },{ 0x6, 0, 0, 0xffffffff },{ 0x6, 0, 0, 0x00000000 },Run with netsni↵-ng

$ netsniff-ng --in eth0 -f cpu.bpf

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 38: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Notable Examples

Used trafgen to create a UDP fragmentation DoS attack1 by Jesper Dangaard Brouer, Linux kernel network developer2 [net-next PATCH V2 0/6 net]: frag performance tuning

cachelines for NUMA/SMP systemshttp://lists.openwall.net/netdev/2013/01/29/44

Used bpfc to prove/exploit a Linux BPF x86 JIT compiler bug1 by Markus Kotter2 net: bpf jit: fix an o↵-one bug in x86 64 cond jump target

http://carnivore.it/2011/12/27/linux_3.0_bpf_jit_

x86_64_exploit

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 39: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Ifpps

Reads kernel stats from procfs, e.g.$ watch -n 1 "cat /proc/net/dev | column -t"

Does not use user space monitoring libraries which lead toinaccurate statistics under high load.

1 What some people do: iptraf (libpcap): 246,000 pps2 What the system actually sees: ifpps: 1,378,000 pps

Replace use of vmstat and ifstat (in conjunction) with a singleifpps window. PPS, BPS, CPU, IRQ, I/O Wait etc.

Display percentage of linerate ( -P )$ ifpps eth0

$ ifpps --promisc --dev eth0

$ ifpps --loop -p --csv -d wlan0 > gnuplot.dat

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 40: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Ifpps, Cont. 1

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 41: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Ifpps, Cont. 2

Pull data out of the CSV format (not technically CSV, uses spaces,not commas)

$ awk ’! /^#/ { print "Packets Per Second: " $3 }’ifpps.csv | head -3

Packets Per Second: 225810Packets Per Second: 256062Packets Per Second: 240789

Convert timestamp to a human readable format with gawk

$ gawk ’$4 > 0 || $5 > 0 { print strftime("%x-%X",$1)" | Drops: "$4 " Errors: "$5 }’01/20/2013-07:53:45 AM | Drops: 900 Errors: 001/20/2013-07:54:02 AM | Drops: 100 Errors: 001/20/2013-07:54:58 AM | Drops: 0 Errors: 3

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 42: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Ifpps, gnuplot 1set auto xset yrange [0:*]plot ’run.csv’ using 4 title "RX Drops" with lines,’run.csv’ using 3 title "RX Packets" with linesset output ’run.png’set terminal png truecolor size 600,900

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 43: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Ifpps, gnuplot 2

set auto xset yrange [0:*]plot ’drops.csv’ using 4 title "RX Drops" with lines,set output ’rx-drop.png’set terminal png truecolor size 600,900

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 44: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Flowtop

Top-like ncurses connection trackerBuilt on libnetfilter conntrack library (requires iptables)GeoIP, process name, PID, port/service, reverse DNS

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 45: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Installation

Always recommend cloning from the main repository1 Actively developed2 Latest features3 Latest fixes

Debian/Ubuntu installation example

$ apt-get install git build-essential flex bison

ccache libnl-3-dev libnl-genl-3-dev libgeoip-dev

libnetfilter-conntrack-dev libncurses5-dev

liburcu-dev libnet1-dev libpcap-dev zlib1g-dev

$ git clone https://github.com/borkmann/netsniff-ng

$ cd netsniff-ng

$ make && make install

Simple!

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 46: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

What Could Be Next?

astraceroute

DNS traceroute to detect malicious DNS injections on transittra�c (SIGCOMM 2012 paper)

mausezahn

Improve imported code and integrate into the main reponetsni↵-ng, mausezahn

New proto dissectors/generators like SCTP, DCCP, BGP, etc.netsni↵-ng

Compressed on-the-fly bitmap indexing for large PCAP filesTry to find sane way to utilize multicore with packet fanout

netsni↵-ng, trafgen, mausezahn

Optimize performance (AF PACKET plumbing)Performance benchmark on 10Gbit/s

flowtop

Byte and packet countersToolkit integration into RHEL!

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 47: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Notes

With permission reused parts of Daniel’s Red Hat talkhttp://pub.netsniff-ng.org/paper/devconf_2013.pdf

Trafgen Expression Language articlehttps://sickbits.net/trafgen-expression-language/

Please read manuals & documentation for more informationgithub.com/borkmann/netsniff-ng/tree/master/

<tool>.8

http://pub.netsniff-ng.org/docs/

Linux Distribution specific documentationhttps://help.ubuntu.com/community/Netsniff-NG

https://fedoraproject.org/wiki/Help:Netsniff-NG

Find configurations via Googleext:txf | ext:cfg | ext:bpf netsniff

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit

Page 48: A Look at the Netsniff-NG Toolkit - A Suite of High - Jon Schipp

Conclusion

Thanks!!!

[email protected]

github.com/borkmann/netsniff-ng

Jon Schipp E-mail: [email protected] Web: jonschipp.com, sickbits.net

A Look at the Netsni↵-NG Toolkit