Download - The adventures of a Suricate in eBPF land
The adventures of a Suricate in eBPF land
É. Leblond
Stamus Networks
Nov. 10, 2016
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 1 / 34
1 eBPF technology
2 Suricata meets eBPF
3 AF_PACKET bypass via eBPF
4 Conclusion
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 1 / 34
Extended Berkeley Packet Filter
Berkeley Packet FilterVirtual machine inside kernelArithmetic operations and tests on the packet dataFilters are injected by userspace in kernel via syscall
Extended BPFExtended virtual machine: more operators, data and function accessVarious attachment points
SocketSyscallTraffic control
Kernel and userspace shared structuresHash tablesArrays
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 2 / 34
LLVM backend
From C file to eBPF codeWrite C codeUse eBPF LLVM backend (since LLVM 3.7)Get ELF fileExtract and load section in kernel
BCC: BPF Compiler collectionInject eBPF into kernel from high level scripting languageTrace syscalls and kernel functionshttps://github.com/iovisor/bcc
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 3 / 34
1 eBPF technology
2 Suricata meets eBPF
3 AF_PACKET bypass via eBPF
4 Conclusion
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 3 / 34
AF_PACKET
Linux raw socketRaw packet capture methodSocket based or mmap based
Fanout modeLoad balancing over multiple socketsMultiple load balancing functions
Flow basedCPU basedRSS based
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 4 / 34
AF_PACKET
Linux raw socketRaw packet capture methodSocket based or mmap based
Fanout modeLoad balancing over multiple socketsMultiple load balancing functions
Flow basedCPU basedRSS based
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 4 / 34
Suricata workers mode
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 5 / 34
Load balancing and hash symmetry
Stream reconstructionUsing packets sniffed from networkto reconstruct TCP stream as seenby remote application
Non symmetrical hash breakOut of order packets
Effect of non symmetrical hash
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 6 / 34
Broken symmetry
HistoryT. Herbert introduce asymmetrical hash function in flow
Kernel 4.2
Users did start to complainAnd our quest did beginFixed in 4.6 and pushed to stable by David S. Miller
Intel NIC RSS hashXL510 hash is not symmetricalXL710 could be symmetrical
Hardware is capableDriver does not allow itPatch proposed by Victor Julien
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 7 / 34
Broken symmetry
HistoryT. Herbert introduce asymmetrical hash function in flow
Kernel 4.2
Users did start to complainAnd our quest did beginFixed in 4.6 and pushed to stable by David S. Miller
Intel NIC RSS hashXL510 hash is not symmetricalXL710 could be symmetrical
Hardware is capableDriver does not allow itPatch proposed by Victor Julien
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 7 / 34
eBPF cluster
Userspace to the rescueProgram your own hash function in userspaceAvailable since Linux 4.3Developed by Willem de BruijnUsing eBPF infrastructure by Alexei Storovoitov
eBPF cluster: ippairIP pair load balancingPerfect for xbitebpf-lb-file variable in af-packet iface configuration
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 8 / 34
eBPF code for ippair
s t a t i c __a lways_ in l ine i n t ipv4_hash ( s t r u c t __sk_buf f ∗skb ){
u i n t32_ t nho f f ;u i n t 32_ t src , ds t ;nho f f = skb−>cb [ 0 ] ;s rc = load_word ( skb , nho f f + o f f s e t o f ( s t r u c t iphdr , saddr ) ) ;ds t = load_word ( skb , nho f f + o f f s e t o f ( s t r u c t iphdr , daddr ) ) ;r e t u r n src + dst ;
}
i n t __sect ion ( " loadbalancer " ) l b ( s t r u c t __sk_buf f ∗skb ) {__u32 nho f f = BPF_LL_OFF + ETH_HLEN;skb−>cb [ 0 ] = nho f f ;sw i tch ( skb−>p ro toco l ) {
case __constant_htons ( ETH_P_IP ) :r e t u r n ipv4_hash ( skb ) ;
case __constant_htons (ETH_P_IPV6 ) :r e t u r n ipv6_hash ( skb ) ;
d e f a u l t :break ;
}r e t u r n skb−>p ro toco l ; /∗ hash on pro to by d e f a u l t ∗ /
} É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 9 / 34
eBPF cluster: prospective
Custom tunnelled trafficTunneling protocol not known by kernel and card
L2TPGTP: 4G protocol
Shared by different flowsResult in poor load balancing
eBPF solutionStrip tunnel headersLoad balance on inner packetsGet fair balancing
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 10 / 34
1 eBPF technology
2 Suricata meets eBPF
3 AF_PACKET bypass via eBPF
4 Conclusion
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 10 / 34
The big flow problem: load balancing
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 11 / 34
The big flow problem: load balancing
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 11 / 34
The big flow problem: load balancing
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 11 / 34
The big flow problem: unfair balancing
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 12 / 34
The big flow problem: unfair balancing
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 12 / 34
The big flow problem: elephant flow
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 13 / 34
The big flow problem: elephant flow
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 13 / 34
The big flow problem: elephant flow
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 13 / 34
The big flow problem
Ring buffer overrunLimited sized ring bufferOverrun cause packets lossthat cause streaming malfunction
Ring size increaseWork aroundUse memoryFail for non burst
Dequeue at NQueue at speed N+M
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 14 / 34
Introducing bypass
Stop packet handling as soon as possibleTag flow as bypassedMaintain table of bypassed flowsDiscard packet if part of a bypassed flow
Bypass methodLocal bypass: Suricata discard packet after decodingCapture bypass: capture method maintain flow table and discard packets ofbypassed flows
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 15 / 34
Bypassing big flow: local bypass
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 16 / 34
Bypassing big flow: capture bypass
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 17 / 34
Bypassing big flow: capture bypass
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 17 / 34
Stream depth bypass
Attacks characteristicIn most cases attack is done at start of TCP sessionGeneration of requests prior to attack is not commonMultiple requests are often not even possible on same TCP session
Stream reassembly depthReassembly is done till stream.reassembly.depth bytes.Stream is not analyzed once limit is reached
Activating stream depth bypassSet stream.bypass to yes in YAML
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 18 / 34
Selective bypass
Ignore some trafficIgnore intensive traffic like NetflixCan be done independently of stream depthCan be done using generic or custom signatures
The bypass keywordA new bypass signature keywordTrigger bypass when signature matchExample of signature
pass h t t p any any −> any any ( content : " s u r i c a t a . i o " ; \ \h t tp_hos t ; bypass ; s id :6666; rev : 1 ; )
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 19 / 34
Selective bypass
Ignore some trafficIgnore intensive traffic like NetflixCan be done independently of stream depthCan be done using generic or custom signatures
The bypass keywordA new bypass signature keywordTrigger bypass when signature matchExample of signature
pass h t t p any any −> any any ( content : " s u r i c a t a . i o " ; \ \h t tp_hos t ; bypass ; s id :6666; rev : 1 ; )
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 19 / 34
Implementation
Suricata updateAdd callback functionCapture method register itself and provide a callbackSuricata calls callback when it wants to offload
NFQ bypass in Suricata 3.2Update capture register functionWritten callback function
Set a mark with respect to a mask on packetMark is set on packet when issuing the verdict
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 20 / 34
Implementation
Suricata updateAdd callback functionCapture method register itself and provide a callbackSuricata calls callback when it wants to offload
NFQ bypass in Suricata 3.2Update capture register functionWritten callback function
Set a mark with respect to a mask on packetMark is set on packet when issuing the verdict
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 20 / 34
And now AF_PACKET
What’s neededSuricata to tell kernel to ignore flowsKernel system able to
Maintain a list of flow entriesDiscard packets belonging to flows in the listUpdate from userspace
nftables is too late even in ingress
eBPF filter using mapseBPF introduce mapsDifferent data structures
Hash, array, . . .Update and fetch from userspace
Looks good!
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 21 / 34
And now AF_PACKET
What’s neededSuricata to tell kernel to ignore flowsKernel system able to
Maintain a list of flow entriesDiscard packets belonging to flows in the listUpdate from userspace
nftables is too late even in ingress
eBPF filter using mapseBPF introduce mapsDifferent data structures
Hash, array, . . .Update and fetch from userspace
Looks good!
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 21 / 34
Using libbpf
Library Linux source in tools/lib/bpf directoryProvide high level function to load eBPF elf fileCreate maps for userDo the relocation
Sample usage
s t r u c t bp f_ob jec t ∗bpfob j = bpf_object__open ( path ) ;bpf_ob jec t__ load ( bp fob j ) ;pfd = bpf_program__fd ( bpfprog ) ;/∗ s to re the map i n our ar ray ∗ /bpf_map__for_each (map, bp fob j ) {
map_array [ l a s t ] . fd = bpf_map__fd (map ) ;map_array [ l a s t ] . name = st rdup ( bpf_map__name (map ) ) ;l a s t ++;
}
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 22 / 34
Kernel code and exchange structure
s t r u c t p a i r {u i n t 64_ t t ime ;u i n t64_ t packets ;u i n t 64_ t bytes ;
} ;
s t r u c t bpf_map_def SEC( "maps" ) f low_tab le_v4 = {. type = BPF_MAP_TYPE_HASH,. key_size = s i z e o f ( s t r u c t f lowv4_keys ) ,. va lue_s ize = s i z e o f ( s t r u c t p a i r ) ,. max_entr ies = 32768 ,
} ;
value = bpf_map_lookup_elem(& f low_tab le_v4 , &tup l e ) ;i f ( value ) {
__sync_fetch_and_add (& value−>packets , 1 ) ;__sync_fetch_and_add (& value−>bytes , skb−>len ) ;value−>t ime = bpf_kt ime_get_ns ( ) ;r e t u r n 0 ;
}r e t u r n −1;
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 23 / 34
Sharing data
Data is updated with statsGetting last flow activity time allow Suricata to handle timeout
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 24 / 34
Userspace code
s t r u c t f lowv4_keys {__be32 src ;__be32 dst ;union {
__be32 por t s ;__be16 por t16 [ 2 ] ;
} ;__u32 ip_p ro to ;
} ;
wh i le ( bpf_map__get_next_key ( mapfd , &key , &next_key ) == 0) {bpf_map__lookup_elem ( mapfd , &key , &value ) ;c lock_get t ime (CLOCK_MONOTONIC, &cur t ime ) ;i f ( cur t ime−>tv_sec ∗ 1000000000 − value . t ime > BYPASSED_FLOW_TIMEOUT) {
f l ows ta t s −>count ++;f l ows ta t s −>packets += value . packets ;f l ows ta t s −>bytes += value . bytes ;bpf_map__delete_elem ( fd , key ) ;
}key = next_key ;
}
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 25 / 34
Test methodology
Test setupIntel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHzIntel Corporation 82599ES 10-Gigabit SFI/SFP+Live traffic:
Around 1Gbps to 2GbpsReal users so not reproducible
TestsOne hour long runDifferent stream depth valuesCollected Suricata statistics counters (JSON export)Graphs done via Timelion(https://www.elastic.co/blog/timelion-timeline)
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 26 / 34
Results: stream bypass at 1mb
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 27 / 34
Results: stream bypass at 512kb
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 28 / 34
A few words on graphics
Tests at 1mbMark show some really high ratebypassPotentialy a big high speed flow
Tests at 512kbWe have on big flow that kill thebandwidthCapture get almost nullEven number of closed bypassedflows is low
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 29 / 34
AF_PACKET bypass on interlaced runs
Limit system to 4 coresSimilar configuration60 sec runs
One with bypassOne without bypass
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 30 / 34
Results
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 31 / 34
AF_PACKET bypass and your CPU is peaceful
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 32 / 34
1 eBPF technology
2 Suricata meets eBPF
3 AF_PACKET bypass via eBPF
4 Conclusion
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 32 / 34
Conclusion
Suricata and eBPFA fresh but interesting methodFeedback welcome
More informationStamus Networks: https://www.stamus-networks.com/Suricata eBPF code:https://github.com/regit/suricata/tree/ebpf-3.12
Libbpf update:https://github.com/regit/linux/tree/libbpf-network-v6
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 33 / 34
Questions ?
Thanks toAlexei StorovoitovDaniel Borkmann
Contact meMail: [email protected]: @regiteric
More informationSuricata eBPF code:https://github.com/regit/suricata/tree/ebpf-3.12
É. Leblond (Stamus Networks) The adventures of a Suricate in eBPF land Nov. 10, 2016 34 / 34