kernel packet capture technologies€¦ · some data complexity of rule work on recontructed stream...
Post on 24-May-2020
4 Views
Preview:
TRANSCRIPT
Kernel packet capture technologies
Éric Leblond
Stamus Networks
October 1, 2015
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 1 / 54
1 Introduction
2 Why capture
3 Libcap and raw socket
4 AF_PACKET
5 PF_RING
6 AF_PACKET goes multi*
7 Netmap
8 Latest AF_PACKET evolution
9 ++zero copy
10 Conclusion
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 2 / 54
Éric Leblond
Co-founder of Stamus NetworksCompany providing network probe based on SuricataFocusing on bringing you the best of Suricata IDS technology
Open source hackerSuricata core developerNetfilter core team member
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 4 / 54
Raw socket: definition
A raw socket is an internet socket that allows direct sending and receiving of InternetProtocol packets without any protocol-specific transport layer formatting.
Wikipedia
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 6 / 54
"The End of the Internet"
[raw socket ...] spells catastrophe for the integrity of the Internet.
Steve Gibson in 2001
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 7 / 54
"The End of the Internet"
Talking about introduction of raw socket in MS WindowsAllow users to write any packetsCould be used to abuse protocol and [poorly implemented] OS
More info at http://www.informit.com/articles/article.aspx?p=27289
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 8 / 54
Raw socket: usage
Send and receiveSend low level message: icmp, igmpImplement new protocol in userspace
SniffingCapture trafficPromiscuous modeUse by network monitoring tools
Debugging tools: tcpdump, wiresharkMonitoring tools: iptraf, ntop, NSAIntrusion detection systems: snort, bro, suricata
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 9 / 54
Network Intrusion Detection System: definition
An intrusion detection system (IDS) is a device or software application that monitorsnetwork or system activities for malicious activities or policy violations and producesreports to a management station.
Wikipedia
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 10 / 54
Network Intrusion Detection System: challenge
IDS detection rule
Some dataComplexity of rule
Work on recontructed streamProtocol field analysisPattern recognition on ungzipped content (http_server_body)
Got around 15000 rules in standard rulesetNeed to inspect 10Gbps of trafic or more
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 11 / 54
Suricata: Open source & multi threaded IDS
IDS and IPS engineGet it here: http://www.suricata-ids.orgProject started in 2008Open Source (GPLv2)Funded by consortium members (and originaly USgovernment)Run by Open Information Security Foundation (OISF)More information about OISF athttp://www.oisf.net/
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 12 / 54
Suricata Features
High performance, scalable through multi threadingProtocol identificationFile identification, extraction, on the fly MD5 calculationTLS handshake analysis, detect/prevent things like DiginotarHardware acceleration support:Useful logging like HTTP request log, TLS certificate log, DNS loggingLua scripting for detection
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 13 / 54
libpcap
Multi OS abstraction for packet captureAll *nix, WindowsMulti layer: Network, USB, . . .
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 15 / 54
Raw socket: the initial implementation
A dedicated socket type
#include <sys / socket . h>#include < n e t i n e t / i n . h>raw_socket = socket ( AF_INET , SOCK_RAW, i n t p ro toco l ) ;
Straight socket modeGet packet per packet via recvmsgOptional ioctl
Get timestamp
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 16 / 54
Memories of another time
"640 K ought to be enough for anybody." Memory contraint designNo preallocationOn demand only
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 17 / 54
Disclaimer
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 18 / 54
IDS design
Monoprocess
No Performance for you, go home now.
Marty Roesch about multithread and network data processing, 2010
Suricata architecture
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 19 / 54
NAPI (2001-200?)
Reducing interrupts usageInterrupts tempest at high packet rateAll CPU time is sued to handle the interruptsNIC driver needs to be updated
No direct change for packet captureChange internal to device driverDirect performance impact on packet capture
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 21 / 54
NAPI performance
Table extracted from luca.ntop.org/Ring.pdf
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 22 / 54
Problem of the socket mode
Internal pathData in card bufferData copied to skbData copied to socketData read and copied by userspace
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 23 / 54
Memory map approach
Sharing is the solutionKernel expose some memoryUserspace access memory directlySpare a message sending for every packets
mmap internal pathData in card bufferData copied to skbData copied to ring bufferUserspace access data via pointer in ring buffer
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 24 / 54
TPACKET_V2
setupsocket(): creation of the capture socketsetsockopt(): allocation of the circular buffer (ring) via PACKET_RX_RING optionmmap(): mapping of the allocated buffer to the user process
capturepoll(): to wait for incoming packets
shutdownclose(): destruction of the capture socket and deallocation of all associated resources.
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 25 / 54
Memory organization
Ascii artblock #1 block #2
+---------+---------+ +---------+---------+| frame 1 | frame 2 | | frame 3 | frame 4 |+---------+---------+ +---------+---------+
block #3 block #4+---------+---------+ +---------+---------+| frame 5 | frame 6 | | frame 7 | frame 8 |+---------+---------+ +---------+---------+
ComponentsFrame contains a datagram dataBlocks are physically contiguous region of memory
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 26 / 54
Performance
Graph extracted from luca.ntop.org/Ring.pdf
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 27 / 54
Suricata architecture
MMAP optionSupport of TPACKET_V2Zero copy mode
Implied changesAccess data via pointer to ring buffer cellRelease data callback
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 28 / 54
PF_RING original design (2004)
Architecturering designmmapcapture only interface
skip kernel pathput in ring buffer and discard
user access the ring buffer
ProjectProject started by Luca DeriAvailable as separate sources
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 30 / 54
PF_RING performance
Show real improvement on small size packetsPre optimisation resultBetter result in following version due to a better poll handling
Table extracted from luca.ntop.org/Ring.pdf
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 31 / 54
PF_RING going multicore (around 2008?)
Sharing the loadEach core has a finite bandwidth capability
Multicore CPU were introduced in 2006Sharing load become common
Previously separate hardware was used to split the network load
Straight forward solutionAllow multiple sockets to be attached to one interfaceLoad balance over the attached sockets
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 32 / 54
Suricata autofp multi reader
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 33 / 54
PF_RING code
Build system and sourcesCustom build systemNo autotools or cmakeInclude patched drivers
SVN stats
g i t log −−format=format : "%s " | s o r t | uniq −c | s o r t −n | t a i l −n1015 Minor change20 f i x20 minor changes22 l i b re f resh30 L i b r a r y re f resh43 minor change67 minor f i x
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 34 / 54
David Miller in da place
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 36 / 54
AF_PACKET load balancing (2011)
Multiple sockets on same interfaceKernel does load balancingMultiple algorithms
LB algorithmRound-robinFlow: all packets of a given flow are send to the same socketCPU: all packets treated in kernel by a CPU are send to the same socket
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 37 / 54
AF_PACKET CPU Load balancing
RSS queuesMultiqueue NIC have multiple TX RXData can be split in multiple queues
Programmed by userFlow load balanced
RSS queues load balancingNIC does load balancing using hash functionCPU affinity is set to ensure we keep the cache line
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 38 / 54
Suricata workers mode
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 39 / 54
tpacket_v3 (2011)
The problemCell are fixed sizeSize is the one of biggest packet (MTU)Small packets use same memory as big one
Variable size cellsRing bufferUpdate memory mapping to enable variable sizesUse a get pointer to next cell approach
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 40 / 54
Netmap (2012)
Similar approach than PF_RINGskip kernel pathput in ring buffer and discard
User access the ring bufferPaired with network card ring
More info http://queue.acm.org/detail.cfm?id=2103536
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 42 / 54
Performances
Table by Luigi Rizzo
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 43 / 54
AF_PACKET rollover option (2013)
Single intensive flowLoad balancing is flow basedOne intensive flow saturate core capacityLoad needs to be shared
PrincipleMove to next ring when ring is fullAs a load balancing modeAs a fallback method
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 45 / 54
Rollover and suricata (1/2)
Graph by Victor Julien
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 46 / 54
Rollover and suricata (2/2)
A TCP streaming issueRollover activation lead to out of order packetsFool TCP stream reconstruction by suricataResult in invalid streams
Possible solutionEvolve autofop multicaptureDecode and dispatch packets
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 47 / 54
DPDK (2012-)
Data Plane Development Kitset of libraries and driverdesign for fast packet processingimpact on software architecture
Architecturemulticore frameworkhuge page memoryring bufferspoll-mode drivers
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 48 / 54
Suricata workers mode limit
Packet treatment can be really longInvolve I/O on disk or networkHuge computation like regular expression
Ring buffers are limited in sizeA slow packet can block a whole bufferSuricata need to dequeue faster
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 50 / 54
Need to evolve Suricata architecture
Switch to asynchronousRelease ring buffer elements as fast as possibleBuffer in userspace
An enhanced autofp approach?Fast decodeCopy data to packet pool of detect threadWith a fast decisionRelease data
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 51 / 54
Conclusion (1/2)
A small subject and a huge evolutionHas follow evolution of hardware architectureAlways need to deal with more speed
10Gbps is common100Gbps is in sight
Multiple technologiesVanilla kernel propose some solutionsPatching may be required to do more
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 53 / 54
Conclusion (2/2)
Do you have questions ?
Contact meMail: eleblond@stamus-networks.comTwitter: @Regiteric
More informationSuricata: http://www.suricata-ids.orgPF_RING: http://www.ntop.org/products/packet-capture/pf_ring/netmap: http://info.iet.unipi.it/~luigi/netmap/dpdk: http://dpdk.org/
Éric Leblond (Stamus Networks) Kernel packet capture technologies October 1, 2015 54 / 54
top related