kernel recipes 2015: kernel packet capture technologies

Kernel packet capture technologies

Éric Leblond

Stamus Networks

October 1, 2015

Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 1 / 54

1 Introduction

2 Why capture

3 Libcap and raw socket

4 AF_PACKET

5 PF_RING

6 AF_PACKET goes multi*

7 Netmap

8 Latest AF_PACKET evolution

9 ++zero copy

10 Conclusion


Éric Leblond

Co-founder of Stamus NetworksCompany providing network probe based on SuricataFocusing on bringing you the best of Suricata IDS technology

Open source hackerSuricata core developerNetfilter core team member


Raw socket: definition

A raw socket is an internet socket that allows direct sending and receiving of InternetProtocol packets without any protocol-specific transport layer formatting.

Wikipedia


"The End of the Internet"

[raw socket ...] spells catastrophe for the integrity of the Internet.

Steve Gibson in 2001


"The End of the Internet"

Talking about introduction of raw socket in MS WindowsAllow users to write any packetsCould be used to abuse protocol and [poorly implemented] OS

More info at http://www.informit.com/articles/article.aspx?p=27289


http://www.informit.com/articles/article.aspx?p=27289

Raw socket: usage

Send and receiveSend low level message: icmp, igmpImplement new protocol in userspace

SniffingCapture trafficPromiscuous modeUse by network monitoring tools

Debugging tools: tcpdump, wiresharkMonitoring tools: iptraf, ntop, NSAIntrusion detection systems: snort, bro, suricata


Network Intrusion Detection System: definition

An intrusion detection system (IDS) is a device or software application that monitorsnetwork or system activities for malicious activities or policy violations and producesreports to a management station.

Wikipedia


Network Intrusion Detection System: challenge

IDS detection rule

Some dataComplexity of rule

Work on recontructed streamProtocol field analysisPattern recognition on ungzipped content (http_server_body)

Got around 15000 rules in standard rulesetNeed to inspect 10Gbps of trafic or more


Suricata: Open source & multi threaded IDS

IDS and IPS engineGet it here: http://www.suricata-ids.orgProject started in 2008Open Source (GPLv2)Funded by consortium members (and originaly USgovernment)Run by Open Information Security Foundation (OISF)More information about OISF athttp://www.oisf.net/


http://www.suricata-ids.org

http://www.oisf.net/

Suricata Features

High performance, scalable through multi threadingProtocol identificationFile identification, extraction, on the fly MD5 calculationTLS handshake analysis, detect/prevent things like DiginotarHardware acceleration support:Useful logging like HTTP request log, TLS certificate log, DNS loggingLua scripting for detection


libpcap

Multi OS abstraction for packet captureAll *nix, WindowsMulti layer: Network, USB, . . .


Raw socket: the initial implementation

A dedicated socket type

#include <sys / socket . h>#include < n e t i n e t / i n . h>raw_socket = socket ( AF_INET , SOCK_RAW, i n t p ro toco l ) ;

Straight socket modeGet packet per packet via recvmsgOptional ioctl

Get timestamp


Memories of another time

"640 K ought to be enough for anybody." Memory contraint designNo preallocationOn demand only


Disclaimer


IDS design

Monoprocess

No Performance for you, go home now.

Marty Roesch about multithread and network data processing, 2010

Suricata architecture


NAPI (2001-200?)

Reducing interrupts usageInterrupts tempest at high packet rateAll CPU time is sued to handle the interruptsNIC driver needs to be updated

No direct change for packet captureChange internal to device driverDirect performance impact on packet capture


NAPI performance

Table extracted from luca.ntop.org/Ring.pdf


luca.ntop.org/Ring.pdf

Problem of the socket mode

Internal pathData in card bufferData copied to skbData copied to socketData read and copied by userspace


Memory map approach

Sharing is the solutionKernel expose some memoryUserspace access memory directlySpare a message sending for every packets

mmap internal pathData in card bufferData copied to skbData copied to ring bufferUserspace access data via pointer in ring buffer


TPACKET_V2

setupsocket(): creation of the capture socketsetsockopt(): allocation of the circular buffer (ring) via PACKET_RX_RING optionmmap(): mapping of the allocated buffer to the user process

capturepoll(): to wait for incoming packets

shutdownclose(): destruction of the capture socket and deallocation of all associated resources.


Performance

Graph extracted from luca.ntop.org/Ring.pdf



Suricata architecture

MMAP optionSupport of TPACKET_V2Zero copy mode

Implied changesAccess data via pointer to ring buffer cellRelease data callback


PF_RING original design (2004)

Architecturering designmmapcapture only interface

skip kernel pathput in ring buffer and discard

user access the ring buffer

ProjectProject started by Luca DeriAvailable as separate sources


PF_RING performance

Show real improvement on small size packetsPre optimisation resultBetter result in following version due to a better poll handling

Table extracted from luca.ntop.org/Ring.pdf



PF_RING going multicore (around 2008?)

Sharing the loadEach core has a finite bandwidth capability

Multicore CPU were introduced in 2006Sharing load become common

Previously separate hardware was used to split the network load

Straight forward solutionAllow multiple sockets to be attached to one interfaceLoad balance over the attached sockets


Suricata autofp multi reader


PF_RING code

Build system and sourcesCustom build systemNo autotools or cmakeInclude patched drivers

SVN stats

g i t log −−format=format : "%s " | s o r t | uniq −c | s o r t −n | t a i l −n1015 Minor change20 f i x20 minor changes22 l i b re f resh30 L i b r a r y re f resh43 minor change67 minor f i x


David Miller in da place


AF_PACKET load balancing (2011)

Multiple sockets on same interfaceKernel does load balancingMultiple algorithms

LB algorithmRound-robinFlow: all packets of a given flow are send to the same socketCPU: all packets treated in kernel by a CPU are send to the same socket


AF_PACKET CPU Load balancing

RSS queuesMultiqueue NIC have multiple TX RXData can be split in multiple queues

Programmed by userFlow load balanced

RSS queues load balancingNIC does load balancing using hash functionCPU affinity is set to ensure we keep the cache line


Suricata workers mode


tpacket_v3 (2011)

The problemCell are fixed sizeSize is the one of biggest packet (MTU)Small packets use same memory as big one

Variable size cellsRing bufferUpdate memory mapping to enable variable sizesUse a get pointer to next cell approach


Netmap (2012)

Similar approach than PF_RINGskip kernel pathput in ring buffer and discard

User access the ring bufferPaired with network card ring

More info http://queue.acm.org/detail.cfm?id=2103536


http://queue.acm.org/detail.cfm?id=2103536

Performances

Table by Luigi Rizzo


AF_PACKET rollover option (2013)

Single intensive flowLoad balancing is flow basedOne intensive flow saturate core capacityLoad needs to be shared

PrincipleMove to next ring when ring is fullAs a load balancing modeAs a fallback method


Rollover and suricata (1/2)

Graph by Victor Julien


Rollover and suricata (2/2)

A TCP streaming issueRollover activation lead to out of order packetsFool TCP stream reconstruction by suricataResult in invalid streams

Possible solutionEvolve autofop multicaptureDecode and dispatch packets


DPDK (2012-)

Data Plane Development Kitset of libraries and driverdesign for fast packet processingimpact on software architecture

Architecturemulticore frameworkhuge page memoryring bufferspoll-mode drivers


Suricata workers mode limit

Packet treatment can be really longInvolve I/O on disk or networkHuge computation like regular expression

Ring buffers are limited in sizeA slow packet can block a whole bufferSuricata need to dequeue faster


Need to evolve Suricata architecture

Switch to asynchronousRelease ring buffer elements as fast as possibleBuffer in userspace

An enhanced autofp approach?Fast decodeCopy data to packet pool of detect threadWith a fast decisionRelease data


Conclusion (1/2)

A small subject and a huge evolutionHas follow evolution of hardware architectureAlways need to deal with more speed

10Gbps is common100Gbps is in sight

Multiple technologiesVanilla kernel propose some solutionsPatching may be required to do more


Conclusion (2/2)

Do you have questions ?

Contact meMail: [email protected]: @Regiteric

More informationSuricata: http://www.suricata-ids.orgPF_RING: http://www.ntop.org/products/packet-capture/pf_ring/netmap: http://info.iet.unipi.it/~luigi/netmap/dpdk: http://dpdk.org/


[email protected]

http://www.suricata-ids.org

http://www.ntop.org/products/packet-capture/pf_ring/

http://info.iet.unipi.it/~luigi/netmap/

http://dpdk.org/

kernel recipes 2015: kernel packet capture technologies

Software