FloSIS: A Highly Scalable Network Flow Capture Systemfor Fast Retrieval and Storage Efficiency

Jihyung Lee, Sungryoul Lee*, Junghee Lee*, Yung Yi, KyoungSoo Park

Department of Electrical Engineering, KAISTThe Attached Institute of ETRI*

AbstractNetwork packet capture performs essential functions

in network management such as attack analysis, networktroubleshooting, and performance debugging. As the net-work edge bandwidth exceeds 10 Gbps, the demand forscalable packet capture and retrieval is rapidly increasing.However, existing software-based packet capture systemsneither provide high performance nor support flow-levelindexing for fast query response. This would either pre-vent important packets from being stored or make it tooslow to retrieve relevant flows.

In this paper, we present FloSIS, a highly scalable,software-based flow storing and indexing system. Flo-SIS is characterized as the following three aspects. First,it exercises full parallelism in multiple CPU cores anddisks at all stages of packet processing. Second, it con-structs two-stage flow-level indexes, which helps mini-mize expensive disk access for user queries. It also storesthe packets in the same flow at a contiguous disk loca-tion, which maximizes disk read throughput. Third, weoptimize storage usage by flow-level content deduplica-tion at real time. Our evaluation shows that FloSIS ona dual octa-core CPU machine with 24 HDDs achieves30 Gbps of zero-drop performance with real traffic, con-suming only 0.25% of the space for indexing.

1 IntroductionNetwork traffic capture plays a critical role in attackforensics and troubleshooting of abnormal network be-haviors. Network administrators often resort to packetcapture tools such as tcpdump [6], wireshark [8], netsniff-ng [3] to dump all incoming packets and to analyze thebehaviors from multiple dimensions. These tools canhelp pinpoint a problem in network configuration and per-formance and even reconstruct the entire trace of an in-trusion attempt by malicious hackers.

As the edge network bandwidth upgrades to beyond10 Gbps, existing packet capture systems have exposed a

few fundamental limitations. First, they exhibit poor per-formance in packet capture and dumping, sometimes un-able to catch up with packet transmission rates in a high-speed network. This is mainly because these tools do notproperly exploit the parallelism with multiple CPU coresand disks, and the existing kernel is not optimized forhigh packet rates. Second, most existing tools do not sup-port indexing of stored packets. Lack of indexing wouldincur a long delay to retrieve the content of interest sincethe search performance would be limited by sequentialdisk access throughput, often translating to hours of de-lay in a multi-TB disk. Third, the huge traffic volume ata high-speed link would significantly limit the monitor-ing period. For example, it takes less than 17 hours tofill up 24 disks of 3TB at the rate of 10 Gbps. For ex-tended monitoring period, one must enhance the storageefficiency by compressing the stored content.

Recent development of high-performance packet I/Olibraries [1, 22, 26, 28] has in part alleviated some of theproblems. For instance, n2disk10g [17] and tcpdump-netmap [7] exploit a scalable packet I/O library to achievea packet capture performance of multi-10 Gbps. More-over, n2disk10g allows parallel packet dumping to diskand packet-level indexing for fast access. However, theprimary problem with these solutions is that they stilldeal with packets instead of network flows. Workingwith packets presents a few fundamental performance is-sues. First, query processing would be inefficient. Mostcontent-level queries would inspect the packets in thesame TCP flow rather than those that belong to com-pletely unrelated flows. Time-ordered packet dumpingwould scatter the packets in the same flow across a disk.Even with indexing, gathering all relevant packets for aquery could either increase disk seeks or waste disk band-width from reading unrelated packets nearby the targets.Second, per-packet indexing would be more expensivethan per-flow indexing. Per-packet indexing would notonly use more metadata disk space but also significantlyincrease the search time.

In this paper, we present FloSIS (Flow-aware Stor-age and Indexing System), a highly scalable software-based network traffic capture system that supports effi-cient flow-level indexing for fast query response. Flo-SIS is characterized by three design choices. First, itachieves high performance packet capture and disk writ-ing by exercising full parallelism in computing resourcessuch as network cards, CPU cores, memory, and harddisks. It adopts the PacketShader I/O Engine (PSIO) [22]for scalable packet capture, and performs parallel diskwrite for high-throughput flow dumping. Towards highzero-drop performance, it strives to minimize the fluctu-ation of packet processing latency. Second, FloSIS gen-erates two-stage flow-level indexes in real time to reducethe query response time. Our indexing utilizes Bloom fil-ters [13] and sorted arrays to quickly reduce the searchspace of a query. Also, it is designed to consume smallamount of memory while it allows flexible queries withwildcards, ranges of connection tuples, and flow arrivaltimes. Third, FloSIS supports flow-level content dedupli-cation in real time for storage savings. Even with dedu-plication, the system preserves the packet arrival time andpacket-level headers to provide exact timing and size in-formation. For an HTTP connection, FloSIS parses theHTTP response header and body to maximize the hit rateof deduplication for HTTP objects.

We find that our design choice brings enormous perfor-mance benefits. On a server machine with dual octa-coreCPUs, four 10 Gbps network interfaces, and 24 SATAdisks, FloSIS achieves up to 30 Gbps for packet captureand disk writing without packet drop. Its indexes take uponly 0.25% of the stored content while avoiding slow lin-ear disk search and redundant disk access. On a machinewith 24 hard disks of 3 TB, this translates into 180 GB for72 TB total disk space, which could be managed entirelyin memory or stored into solid state disks for fast randomaccess. Finally, FloSIS deduplicates 34.5% of the storagespace for 67 GB of a real traffic trace only with 256 MBof extra memory consumption for a deduplication table.In terms of performance, it achieves about 15 Gbps zero-drop throughput with real-time flow deduplication.

We believe that our key techniques in producing highscalability and high zero-drop performance are applicableto other high-performance flow processing systems likeL7 protocol analyzers, intrusion detection systems, andfirewalls. We show how one should allocate various com-puting resources for high performance scalability. Also,our efforts for maintaining minimal processing varianceshould be valuable in high-speed network environmentswhere packet transmission rates vary over time.

2 Design Goals and Overall Architecture

In this section, we highlight our system design goals anddescribe overall system architecture of FloSIS.

2.1 Design Goals(1) High scalability: A high-speed network traffic cap-ture system should acquire tens of millions of packets persecond from multiple 10G network interfaces, and writethem to many hard disks without creating a hotspot. Toachieve high performance, the system requires a highlyscalable architecture that enhances the overall perfor-mance by utilizing multiple CPU cores and hard disksin parallel. Moreover, it should be easily configurable.

(2) High zero-drop performance: The system shouldensure high zero-drop performance. A peak zero-dropperformance refers to the maximum throughput that thesystem can achieve without any packet drop. It is an im-portant performance metric of a network traffic capturesystem since we need to minimize packet drop for accu-rate network monitoring and attack forensics. It typicallydiffers from the peak throughput that allows packet drop.Even if the input traffic rate is much lower than the peakthroughput, a temporary spike of delay caused by block-ing I/O calls or scheduling can incur packet drop regard-less of the amount of available computation resources.

(3) Flow-level packet processing: Indexing plays a crit-ical role in fast query response. Instead of packet-levelindexing [17, 19], we make an index per each flow. Flow-level indexing significantly reduces the index space over-head and improves disk access efficiency. However, itrequires managing the packets by their flows so that thepackets in the same flow are written to the same disk lo-cation. While this incurs extra CPU and memory over-heads, it also provides an opportunity to deduplicate theflow content, making more efficient use of the storage.

(4) Flow-aware load balancing: For efficient flow pro-cessing, the packets in the same flow need to be trans-ferred to the same CPU core from the NICs while theper-core workload should be balanced on a multicore sys-tem. Otherwise, packets should be moved across the CPUcores for flow management, which might cause severelock contention and degrade CPU cache efficiency.

2.2 Overall ArchitectureThe basic operation of FloSIS is simple. It captures thenetwork packets mirrored by one or a few switch ports,manages them by their flows 1, and writes them to disk.It also generates per-flow metadata and its associated in-dex entries, and responds to user queries that find a setof matching flows on disk. In case a flow content is re-dundant, the system deduplicates the flow by having itson-disk metadata point to a version that exists in a disk.

For scalable operation, FloSIS adopts a parallelizedpipelined architecture that effectively exploits the paral-lelism in modern computing resources, and minimizes

1We mainly focus on TCP flows in this paper, and non-TCP packetsare written to disk as they arrive without flow-level indexing.

2

Symmetric RSS

10 Gbps NIC

Symmetric RSS

10 Gbps NIC

CPU Core

Engine Thread

Packet Acquisition

Packet Classification

Flow Metadata & Index Creation

Flow Data Buffering

HDD HDD

Writing Thread Writing Thread

Rx queue

SSD

Indexing Thread Indexing Thread

Packet Processing Procedure

1. Packet acquisition

Read a batch of packets using PSIO

2. Packet classification

Decode packets & update flow buffers

3. Flow metadata & index creation

Create flow metadata & index if a flow is terminated

Indexing threads sort the indexes and write to SSD

4. Flow data buffering

Copy data in flow buffers to w-buffer

Writing threads write fully loaded w-buffers to HDD× 2

Figure 1: The half of overall architecture and thread arrangement of FloSIS on a machine with 16 CPU cores, four 10G NIC ports,24 HDDs, and 2 SSDs

the contention among them. It is multi-threaded withthree distinct thread types: engine, writing, and index-ing threads. The engine thread is responsible for packetacquisition from NICs, flow management, and index gen-eration. It also coordinates flow and index writing to diskwith writing and indexing threads. The writing threadshares the buffers for writing with an engine thread, andperiodically writes them onto a disk. All packets in thesame flow are written consecutively to disk unless theyare deduplicated. The indexing thread shares the flowmetadata and indexes with an engine thread, and sorts theindexes when it gathers enough entries. Also, it writes themetadata and indexes to disk for durability and helps withresolving the queries by searching through the indexes.We use hard disk drives (HDDs) for storing the flowdata and solid state drives (SSDs) for metadata and in-dex information. Our system benefits from cost-effectivepacket data dumping and prompt query response fromfast retrieval of flow metadata and indexes.

Figure 1 shows the mapping of FloSIS threads into aserver machine with 16 CPU cores, four 10G NIC ports,and 24 HDDs and 2 SSDs, which we use as a referenceplatform in this paper. The guiding principle in resourcemapping is to parallelize each thread type as much aspossible to maximize the throughput while minimizingpacket drop from resource contention. Every thread isaffinitized to a CPU core to avoid undesirable interfer-ence from inter-core thread migration. We co-locate mul-tiple writing threads with an engine thread on the sameCPU core since writing threads rarely consume CPU cy-cles and mostly perform blocking disk operations. Sincea writing thread shares the buffers that its engine threadfills in, it would benefit from shared CPU cache if it isco-located with an engine thread. In contrast, an index-ing thread runs on a different CPU core since it period-

ically executes CPU-intensive operations, which mightinterfere with high-speed packet acquisition with enginethreads. Since indexing thread operations are not time-critical, we can place multiple indexing threads on thesame CPU core. Each writing thread has its own diskto benefit from sequential disk I/O without concurrentaccess by other threads. However, SSDs are shared bymultiple indexing threads since they do not suffer fromperformance degradation by concurrent random access.

3 High-speed Flow DumpingIn this section, we describe scalable packet capture andflow dumping of FloSIS. FloSIS uses a fast user-levelpacket capture library that exploits multiple CPU cores,and performs direct I/O to bypass redundant memorybuffering at disk writing while minimizing CPU andmemory consumption for disk I/O.

3.1 High-speed Packet AcquisitionThe first task of FloSIS is to read packets from NICs. Itis of significant importance to allocate enough resourceon the packet acquisition, because its performance servesas an upper-bound of the achievable system throughput.Thus, we allocate so many CPU cores as to read the pack-ets from NICs at line rate. For scalable packet I/O, Flo-SIS employs the PSIO library [22], which allows parallelpacket acquisition by flow-aware distribution of incom-ing packets across available CPU cores. PSIO is knownto achieve a line rate regardless of packet size with batchprocessing and efficient memory management [22]. Also,we use symmetric receive-side scaling (S-RSS) [31] thatmaps both upstream and downstream packets in the sameTCP connection to the same CPU core. S-RSS hashes the4-tuple of a TCP packet to place the packets in the sameconnection in the same RX queue inside a NIC. Each RX

3

Flow Buffer

Packet header buffer

Upstream content buffer

Downstream content buffer

(a) Typical-HTTP Flow (b) HTTP Flow (c) HTTP Flow w/ Redundancy

HTTP response header

HTTP response body

HTTP requestUa

Da

Ub

Db-H

Uc

Db-B Dc-B

Dc-H

Disk Flow content deduplication

Ua Da Ub Db-H Db-B Uc Dc-H… … … …

Flow data map

Flow data of flow (a) Flow data of flow (c)Flow data of flow (b)

Figure 2: Flow data in the flow buffers and the disk (U: Upstream, D: Downstream, H: Head, B: Body). (a) a typical TCP flow,(b) an HTTP flow with a request in an upstream buffer, and a response in a downstream buffer, and (c) another HTTP flow with thesame response body as (b)’s.

queue in a NIC is dedicated to one of the engine threads,and all packets in the queue are processed exclusivelyby the thread. This architecture eliminates the need ofinter-thread synchronization and achieves high scalabil-ity without sharing any state across engine threads.

One may have concern that S-RSS may not distributethe packet load evenly across multiple CPU cores. How-ever, a recent study reveals that S-RSS effectively dis-tributes real traffic at a 10 Gbps link almost evenly acrossmultiple CPU cores, by curbing the maximum load dif-ference at 0.2% for a 16-core machine [31].

3.2 Flow-aware Packet BufferingWhen packets are read from NIC RX queues, the enginethread manages them by their flows. The engine threadclassifies the received packets into individual flows, andeach flow maintains its current TCP state as well as itspacket content. FloSIS keeps all packets in the sameTCP connection at a single conceptual buffer called flowbuffer. A flow buffer consists of three data buffers: packetheader, upstream and downstream buffers as shown inFigure 2. The packet header buffer collects all packetheaders (e.g., Ethernet and TCP/IP headers) and theirtimestamps in the order of their arrival. While FloSISfocuses on flow-level indexing and querying, it also sup-ports packet-level operations to provide information suchas packet retransmission, out-of-order packet delivery,inter-packet delay, etc, where packet header buffers areneeded to reconstruct such information. The upstreamand downstream buffers maintain reassembled content ofTCP segments so that the user can check the entire mes-sage at one disk seek. In case of an HTTP flow, the down-stream buffer is further divided into response header andbody buffers. This header-body division adds efficiencyin deduplication, since the response header tends to differfor the same object (see Section 5 for more details).

One challenge in maintaining the flow buffer is mem-ory management of three data buffers. Since a flowsize is variable, one might be tempted to dynamically(re)allocate the buffers. However, we find that dynamicmemory allocation often induces unpredictable delays,

which increases random packet drops at engine threads.Instead, FloSIS uses user-level memory managementwith pre-allocated memory pools of fixed-sized chunks.For the flow buffer, one chunk is allocated initially foreach data buffer when a flow starts. If more memory isneeded, a new chunk is allocated but the new chunk islinked to the previous one with doubly-linked pointers.The flow management module has to follow the links towrite the data into right memory chunks. While manag-ing non-contiguous memory chunks is more difficult, thebenefit of fixed-cost operations ensures a predictable per-formance at high-speed packet processing.

When a TCP flow terminates, the engine thread gener-ates a flow metadata consisting of the following informa-tion: (i) start and end timestamps, (ii) source and desti-nation 2 IP addresses and port numbers, and (iii) the lo-cation and length of the flow data on disk. Then the datain the flow buffer is moved to a large contiguous buffercalled w-buffer, and the flow buffer is recycled for otherflows. The flow data is also moved to w-buffer if its sizegrows larger than a single w-buffer. The w-buffer servestwo purposes. First, it enables to have all flow data at acontiguous location before disk writing, which ensures toread all flow data with one disk seek. Second, it buffersthe data from multiple flows to maximize the sequentialwrite throughput of an HDD. It is the unit of disk writing,where a larger size would lead to a higher disk through-put while reducing the CPU usage. When the w-bufferis filled up, the engine thread gives the ownership to itswriting thread, which writes to disk and recycles it.

For each flow, a flow data map is written prior to itsflow data in w-buffer. The flow data map contains anarray of disk locations and lengths of flow data buffers.In most cases, three (or four) flow data buffers follow theflow data map, but for a deduplicated buffer, the flow datamap points to an earlier version. If a flow consists ofmultiple w-buffers, the flow data map is responsible forkeeping track of all data buffers in the same flow, allow-ing flows of arbitrary size.

2Interpreted from the client side

4

0

10

20

30

40

0

10

20

30

40

1 2 4 8 16 32 64 128 256 512 1024

CP

U U

tili

zati

on

(%

)

Thro

ug

hp

ut

(Gbp

s)

Buffer Size (KBytes)

Dwrite Throughput Fwrite Throughput Dwrite CPU%

Figure 3: Disk writing throughput and CPU utilization

We comment that FloSIS avoids using buffered I/O,because the file system cache would be of little use forflow dumping and random disk access for user queries.Worse yet, the file system cache behavior becomes unpre-dictable when the cache buffer is full, which often leadsto a catastrophic result such as kernel thrashing. In con-trast, direct I/O in FloSIS performs I/O operations by di-rect DMA-mapping of user buffer to disk, and minimizesthe effect of file system cache.

To figure out what size of w-buffer is appropriate as agood design choice in our reference platform, we run abenchmark test with varying buffer sizes. Figure 3 showsdisk writing throughputs and CPU utilization of direct I/Ousing the O DIRECT flag (Dwrite) and buffered I/O us-ing fwrite() (Fwrite), where we compare the perfor-mance of buffered I/O just to understand the maximumdisk performance. In these microbenchmarks, we run 24threads on our platform, each of which occupies its ownHDD and calls write() (fwrite() in Fwrite) contin-uously, and measure the aggregate throughputs and CPUutilization. Not surprisingly, Fwrite shows good perfor-mance regardless of the buffer size due to its data batch-ing in the file system buffer. Since memory copying toa file system write buffer is much faster than actual diskI/O, the kernel always has enough data to feed to disk.This allows Fwrite to achieve the maximum disk through-put, whereas Dwrite achieves a similar performance onlyat 128 KB or larger buffer sizes due to the absence ofkernel-level buffering. As expected, the CPU usage ofDwrite decreases as the buffer size increases. In FloSIS,we choose 512 KB as the size of w-buffer, which achievesalmost peak performance only with 10% CPU usage.

4 Flow-level Indexing & Query ProcessingIn this section, we explain the two-stage flow-level index-ing of FloSIS and real-time index generation.

4.1 Two-stage Flow-level IndexingFloSIS writes flow data to a file (called a dump file) oneach disk. When the current file size reaches a given

threshold (e.g., 1GB), FloSIS closes it and moves on tothe next file for flow data dumping. The reason for fileI/O instead of raw disk I/O is to allow other tools to ac-cess the dumped content. For sequential disk I/O, we pre-allocate the disk blocks contiguously in the files (e.g., us-ing Linux’s fallocate()).

FloSIS handles flow data stored on each disk by two-stage flow-level indexes as shown in Figure 4. It uses thefirst-stage indexes to determine the files that contain thequeried flows (file indexing). If a dump file has relevantflows, it filters the exact flows using the second-stage in-dexes (flow indexing). The details are as follows.

File indexing: The first-stage indexing is file indexes.Each dump file has in-memory file indexes that consistof two timestamps and four Bloom filters. These indexesare used to determine whether queried flows do not ex-ist in the dump file, and the file can be passed. The twotimestamps record the arrival times of the first and thelast packet stored in the file. Each Bloom filter holds theinformation about one of the 4-tuple of a flow. For ex-ample, the source IP Bloom filter records all source IPsof the flows stored in the dump file. Using the filters,we can quickly figure out whether the dump file does nothave any flow with a queried IP (or a port number).

Even if a Bloom filter confirms a hit with a queriedIP address or a port number, there is still a small prob-ability that it is a false positive. To achieve a tolerablefalse positive rate, we need to adjust the size of a Bloomfilter and the number of hash functions, considering thenumber of elements. Assuming that an average flow sizeis 32 KB [31], we expect 32K flows in a dump file of1 GB. A Bloom filter with 7 hash functions and 128 KB(or 220 bits) of size, would reduce the false positive rate to0.0011%, which should be small enough. This means thatthe memory requirement for file indexes is about 1.5 GBper 3 TB HDD or 36 GB for 24 HDDs.

Flow indexing: The second-stage is flow indexes. Flowindexes of a dump file consist of all flow metadata ofthose stored in the file and four sorted arrays. These areused to retrieve the flow metadata entries that match auser query. Using these entries, FloSIS reads the matchedflow data from the dump file.

Each sorted array contains one of the flow tuple val-ues in increasing order. An element in the array consistsof a tuple value and a pointer to the flow metadata withthat value. For example, a sorted array for destination IPhas the entries with (a flow’s destination IP, a pointer tothe flow metadata) for all flows in the dump file, sortedby the destination IPs. When a query determines a targetdump file in the first stage, sorted arrays are used to con-firm if there are such flows that match the query. Using abinary search, one can locate the flow metadata fast witha specific tuple value in a query.

5

…Packets Packets Packets Packets Packets PacketsFlow Data HDD

w n x 3 z y…

…Flow 2Flow 1 Flow 3 Flow n-2 Flow n-1 Flow nFlow Metadata

Sorted Arrays z 3 w n y x… x y 3 w z n… 3 y n z w x…

Source IP Destination IP Source Port Destination Port

SSD

The first flow: 2014-01-01 14:23:37.224 The last flow: 2014-01-01 14:23:46.735 Timestamps

0100100111…10101101110 1110010110…00011011101 1000001110…01010111011 1011011100…10101100100

Source IP Destination IP Source Port Destination Port

Bloom Filters

Memory

Figure 4: A hierarchical indexing architecture for each file in FloSISIn terms of the space overhead, assuming a dump file

contains 32K flows, four sorted arrays would take up1 MB, and the array of 32-byte flow metadata would con-sume another 1 MB. The total space overhead for flowindexes for 24 HDDs of 3 TB would be 140 GB. Sincethis might be too large to fit in memory, we store theseindexes on SSDs, and keep a fraction of them in memoryif they are necessary for query processing.

4.2 Real-time Index GenerationTo create the two-stage flow-level indexes in real time,FloSIS needs to produce the flow metadata, and updatesthe Bloom filters and the sorted arrays, for each com-pleted flow with a budget of a few CPU cycles. However,sorting is a memory-intensive operation that consumesmany CPU cycles. To avoid the interference caused bysorting, FloSIS separates the index creation in two steps.

In the first step, an engine thread creates the flow meta-data with the 4-tuple values, start time, duration, and disklocation of the flow when the flow terminates. Then,the engine thread computes Bloom filter hash values ofsource and destination IP addresses and port numbers,and updates the Bloom filters. Also, it adds an elementat the end of each sorted array and leaves the array un-sorted. The new element has one of the 4-tuple valuesand a pointer to the newly-created flow metadata. Sortingof the arrays is postponed until the engine thread fills upthe dump file with flow data. Since sorting is skipped, theengine thread creates an index just in a few CPU cycles.

When a dump file is filled up, the second step begins.The engine thread sends an event to an indexing threadto sort the arrays for the 4-tuple values. The indexingthread sorts the arrays, and writes the flow metadata andthe sorted arrays into the SSD. While the indexing threadperforms CPU-intensive sorting, other threads workingon the same core could suffer from a lack of computationcycles, which could affect the overall performance. Toprevent this contention, FloSIS runs the indexing threadson dedicated CPU cores. We note that array sorting andindex writing happen once every few seconds since itwould take at least 5 seconds to fill a dump file of 1 GB,assuming the disk write throughput is 200 MBps. Dedi-cating a CPU core for each indexing thread would waste

CPU cycles, so we co-locate several indexing threads onthe same CPU core as shown in Figure 1.

4.3 Flow Query ProcessingA FloSIS query consists of all or a part of six fields; aperiod of time (a time range), source and destination IPaddresses and port numbers, and a packet-level conditionin the BPF syntax. Each field can be a singleton value ora range of values using an IP prefix, a hyphen (-), or awildcard (‘*’). FloSIS searches the flows that satisfy ev-ery condition in the query, i.e., the query fields are inter-preted as conjunctive. Every query field is optional and itis assumed to be any (’*’) in case it is missing in a query.

An example query looks like following.

./flowget -st 2015-1-1:0:0:0 -et2015-1-2:0:0:0 -sip 10.1.2.3 -dip128.142.33.1/24 -sp 20000-30000 -dp80 -bpf tcp[tcpflags]&(tcp-syn) != 0

flowget is a query client program that accepts a userquery and sends it to the query front-end server of FloSIS.The query searches the flows that overlap with the timerange of one day from January 1st in 2015, and the sourceIP is 10.1.2.3 with ports between 20000 and 30000 withany destination IPs in 128.142.33.1/24 with port 80. Itwants to see only the SYN packets of the matching flows.

When the query front-end server receives the userquery, it distributes the query to all indexing threads andcollects the results. Each indexing thread processes thequery independently using the two-staged indexes of thedump files on their disks. The first step is ‘dump-file fil-tering’ using the file indexes. In this step, the indexingthread identifies which dump files might have matchingflows. The indexing thread first finds a set of dump fileswhose timestamps overlap with the queried time range,and then checks the Bloom filters to see if the dump filemight contain the flows with the requested fields. If aquery field is a range, the indexing thread assumes thatthe corresponding Bloom filter returns a hit since check-ing all possible values would be too costly.

The second step is ’flow filtering’ using flow indexes.With the target dump files to look up, the indexing threadperforms a binary search on each sorted array to find the

6

entries with the requested field. If the field is a range, itfirst checks whether the array has any overlap with therange, and a lookup for the closest entry to one end ofthe range would find all matching entries. Assuming thenumber of entries in an array is 32K, each binary searchwould require at most 15 entry lookups. That is, 60 entrylookups would be enough to find all matching flows in adump file in the worst case, which takes a few microsec-onds on our platform. For each dump file, the indexingthread takes an intersection of the results from the foursorted arrays. After that, the indexing thread retrieves thematching flow metadata entries, and reads the flow datausing the disk location in each flow metadata entry.

The last step in query processing is ‘packet filtering’.The indexing thread runs BPF for detailed query condi-tion on each packet header in the matching flows. A BPFprovides a user-friendly filtering syntax, and is widelyused for packet capture and filtering. While this step re-quires sequential processing, it is applied to a much re-duced set of packets compared to brute-force filtering. Ifa BFP field is missing, then the last step is omitted andall data is passed to the query front-end, which relays itto the query client on a per-flow basis.

5 Flow-level Content DeduplicationIt is well-known that the Internet traffic exhibits a sig-nificant portion of redundancy in the transferred con-tents [11, 31]. For example, a recent study reveals thatone can save up to 59% of the cellular traffic bandwidthwith simple network deduplication with 4KB fixed-sizedchunking [31]. Thus, if we apply flow content dedupli-cation to FloSIS, we may expect a non-trivial saving onstorage space, which would help to extend the monitoringperiod at a high-speed network.

In FloSIS, we choose to adopt “whole-flow dedupli-cation,” which uses the entire flow data as a unit of re-dundancy (i.e., a chunk). If the size of a flow exceedsthat of the w-buffer, we make each w-buffer for the flowas a chunk. Whole-flow deduplication is a reasonabledesign choice, given that 80 to 90% of the total redun-dancy comes from serving the same or aliased HTTP ob-jects [31], and that it consumes a smaller fraction of CPUcycles compared to other alternatives with fine-grainedchunking [11]. Note that CPU cycles are important re-sources in FloSIS that regularly runs CPU-intensive taskssuch as packet acquisition and index sorting.

For flow-level deduplication, an engine thread calcu-lates the SHA-1 hash of the flow-assembled content ineach direction (or only the response body in case of anHTTP transaction). CPU cycles for hash computationare amortized over the packets in a flow as the hash ispartially updated on each packet arrival. This minimizesthe fluctuation of CPU usage, which produces more pre-dictable latency in hash computation. Out-of-order pack-

ets are buffered until missing packets arrive, and the hashis updated over the partially reassembled data. When aflow finishes (or when the flow size exceeds the w-buffersize), the SHA-1 hash is finalized. The hash is used as thecontent name to look up in a global deduplication tablefor cache hit. The deduplication table is a hash table thatstores previous content hashes and their disk locations. Itis shared by all engine threads to maximize the dedupli-cation rate. When a lookup is a cache hit, the flow datamap would be made point to the location of the earlierversion when the flow is written to disk. If it is a cachemiss, the SHA-1 hash and its disk location is inserted tothe table.

We implement the deduplication table so as to min-imize CPU usage and maximize the deduplication per-formance. We use a per-entry access lock to minimizethe lock contention due to frequent accesses by multipleengine threads. We also pre-allocate a fixed number oftable entries at initialization and use FIFO as the replace-ment policy (known to perform as well as LRU in net-work deduplication [31]). Each entry in the table is de-signed to be as small as possible to maximize the numberof in-memory entries: 44 bytes per entry, 20 bytes SHA-1hash, 8 bytes for disk location, and 16 bytes for doubly-linked pointers. The size of the deduplication table (orsimply cache size) is a design parameter that trades offmemory usage and deduplication performance. However,it is expected that the cache size does not have to be verybig, because it is known that even a small cache signif-icantly helps and the performance improvement dimin-ishes as the cache size increases (i.e., diminishing return)[31, 11]. A recent study reveals that one can achieve 30to 35% of deduplication rates with only 512 GB of con-tent cache for a 10 Gbps cellular traffic link [31]. Thistranslates to 16 million entries (or 1 GB) in a dedupli-cation table assuming 32KB of average flow size. Weunderstand that the actual numbers would vary in differ-ent environments, but we believe that a few GB of tableentries should suffice in most cases.

6 ImplementationWe implement FloSIS with 7,271 lines of C code thatinclude engine, writing, and indexing threads as well as aquery front-end server and a client. We mainly use Linuxkernel version 2.6.32-47 for development and testing, butthe code does not depend on the kernel version except forthe PSIO library that requires Linux kernel 2.6.3x due toits driver code.

The number of engine, writing and indexing threadsand their CPU core mapping are configurable depend-ing on the hardware configuration. Each thread is affini-tized to a CPU core and we replicate the thread mappingper each non-uniform memory access (NUMA) domain.FloSIS ensures that the network cards deliver the packets

7

only to the CPU cores in the same NUMA domain sincecrossing NUMA domains is expensive [22].

Since high zero-drop performance is one of our de-sign goals, we pay special attention to the implementationthat avoids unpredictable processing delay, whose keytechniques are summarized in what follows: First, Flo-SIS limits maximum processing latency per each batchof packets from a NIC. Normally, an engine thread readsa batch of packets, processes them all, and moves on toread the next batch. However, when the number of pack-ets in a batch is too large, it handles only a fraction ofthem at a time while leaving the rest in the buffer to min-imize packet drop in a NIC. This technique is appliedto other implementations such as SHA-1 hash calcula-tion and flow management. Second, FloSIS pins eachthread to a CPU core to avoid random scheduling de-lay and to improve CPU cache utilization. Also, it pre-allocates performance-critical memory chunks such asdata, flow, and write buffers, and core data structures likeflow metadata, indexes, flow and deduplication table en-tries at initialization to avoid the run-time overhead of dy-namic memory allocation. FloSIS efficiently recycles thememory space, and dynamically allocates memory onlywhen there is no more available pre-allocated memoryspace. The amount of pre-allocated memory is config-urable, which in our prototype is set based on the recenttraffic measurement [31]. Third, FloSIS minimizes un-predictable disk I/O delay as much as possible. We usedirect I/O with enough buffer size to saturate the disk I/Ocapacity, and pre-allocate disk blocks in each file so thatthey are accessed sequentially. This significantly reducesthe variance in disk I/O latency.

For evaluating FloSIS’s performance, we extend a net-work workload generator initially developed for [23]. Weimplement the workload generator to (i) produce syn-thetic TCP packets with a random payload at a specifiedrate up to 40 Gbps regardless of packet size, and (ii) re-play a real traffic packet trace (in the pcap file format) at aconfigurable replay speed. We ensure that the packets inthe same flow are forwarded to the same destination NICport in the order of their original record.

7 Evaluation

The goals of our evaluation are three folds. First, we eval-uate if FloSIS provides a good performance 3 of packet-and flow-level capture and disk dumping with syntheticand real traffic. Second, we show how much reduction inquery response time FloSIS’s two-stage flow-level index-ing brings over the existing state-of-the-art packet-levelindexing. Third, we evaluate the effectiveness of flow

3Unless specified otherwise, the throughput numbers include Ether-net frame overhead (24 bytes) to reflect the actual line rate.

0

10

20

30

40

64 128 256 512 1024 1514

Thro

ug

hp

ut

(Gbp

s)

Packet Size (Bytes)

tcpdump tcpdump_netmap n2disk10g FloSIS

Figure 5: Synthetic TCP packet dumping throughputs of Flo-SIS, n2disk10g, and tcpdump at 40 Gbps input rate with variouspacket sizes

content deduplication with a real traffic trace and mea-sure the overhead.

7.1 Experiment SetupWe run FloSIS and other packet capture systems on ourreference server machine with dual Intel E5-2690 CPUs(2.90 GHz, 16 cores in total), two dual-port Intel 10 GbpsNICs with 82599 chipsets, 128 GB of RAM, 24 3TBSATA HDDs of 7200 RPM, and 2 SSDs of 512 GB Sam-sung 840 Pro. For workload generation, we run ourpacket generator on a similar machine with the samenumber of CPUs and NICs as the server. For syntheticworkload, the packet generator sends the packets of thesame size at a specified rate. For real traffic tests, thepacket generator replays 67.7 GBs of a packet trace ob-tained from a large cellular ISP in South Korea [31]. Thistrace represents a few minutes of real traffic at one of10 Gbps cellular backhaul links, and has 89 million TCPpackets with 760 bytes of average packet size. The clientand server machines are directly connected by four 10Gcables since the traffic is either synthetically generated orreplayed. However, in practice, the live traffic should beport-mirrored to the server via a switch.

7.2 Packet Capture & DumpingWe measure the performance of traffic capture and diskdumping of FloSIS for synthetic and real traffic work-loads and compare them with those of existing solutions.

7.2.1 Synthetic Packet WorkloadWe first evaluate whether FloSIS achieves a good perfor-mance with packet capture and disk dumping regardlessof incoming packet size. We have the packet generatortransmit the packets of the same size at 40 Gbps, andmeasure the disk writing performance of captured pack-ets. To compare the performance of packet-level captureand disk dumping with other tools, we disable other fea-tures of FloSIS such as flow management, index genera-tion and writing, and deduplication.

8

0

10

20

30

40

FloSIS n2disk10g_I n2disk10g

Thro

ug

hp

ut

(Gbp

s)Peak Zero-drop

Figure 6: Peak and zero-drop throughput of FloSIS andn2disk10g with real traffic trace replay

We compare the performances with tcpdump [6],tcpdump-netmap [7], and n2disk10g [17]. tcpdump isa popular packet capture system based on the libpcaplibrary, but the libpcap library allows only one pro-cess/thread for each network interface. Since our ma-chine has four 10G interfaces, we run 4 tcpdump pro-cesses and have each process write the captured pack-ets to its dedicated HDD. On the other hand, tcpdump-netmap uses the netmap packet I/O framework [28] to ac-celerate the libpcap library performance. n2disk10g [17]uses PF RING DNA [26] as scalable packet acquisitionlibrary, and can be configured to write to multiple disksin parallel. We select the parameters suggested by themanual of n2disk10g [2] for the best behavior. For faircomparison, we disable the indexing module for this test.

Figure 5 shows the throughputs 4 of the systems. Flo-SIS achieves 35.8 to 36.5 Gbps regardless of packet size,which is close to the peak aggregate disk writing perfor-mance as shown in Figure 3. This implies that FloSIS’spacket buffering works well to achieve a sequential diskwrite performance. We observe that n2disk10g performsas well (34.9 to 35.6 Gbps), which is not surprising sinceits packet capture and parallel disk I/O is similar to thatof FloSIS. Unfortunately, tcpdump and tcpdump-netmapperform poorly, achieving only 0.4 to 3.2 Gbps and 0.9to 4.4 Gbps, respectively. While the netmap support im-proves the performance of tcpdump, the inefficiency inthe libpcap library itself seems to limit the improvement.

7.2.2 Real Traffic WorkloadWe now measure the performance of handling the realtraffic by replaying the LTE packet trace at a high speed.For this test, we enable flow management and indexing inFloSIS but disable flow deduplication to focus on the per-formance of flow processing. For comparison, we showthe performances of n2disk10g with and without packet-level indexing. We also measure the peak zero-drop per-

4We do not include the Ethernet frame overhead in throughput cal-culation in this test to focus on the performance of disk writing ratherthan packet capturing.

-

5

10

15

20

25

0

5

10

15

20

25

1 2 3 4 5 6 7 8 9 10

1,0

00

Flo

ws

Qu

ery P

roce

ssin

g T

ime

(Sec

)

Query ID

FloSIS n2disk10g The number of flows

Figure 7: Query processing times over various number ofmatching flows

formances of both systems to estimate the practical per-formance for accurate monitoring.

As depicted in Figure 6, the peak performances ofFloSIS and n2disk10g with/without indexing are simi-larly high as 36.4 Gbps, implying that disk writing band-width is the primary bottleneck, as in the synthetic work-load case. This, in turn, implies that the extra overheadfor flow management and two-stage flow index genera-tion is small enough not to degrade the overall perfor-mance. However, FloSIS achieves better than n2disk10gin terms of zero-drop performance, 30 Gbps by FloSISand 20 Gbps by n2disk10g regardless of indexing. Webelieve that our design choices such as minimizing thecontention by separating the resources between interfer-ing tasks, and aggressive amortization of resource con-sumption help produce more predictable processing de-lays, despite extra tasks like flow management and index-ing. We do not know the root cause for lower zero-dropperformance with n2disk10g since we do not have accessto the source code, but it seems to pay less attention to thelatency burstiness in packet capturing and disk dumping.

7.3 Query ProcessingWe evaluate the effectiveness of flow-level indexing ofFloSIS. For the experiments, we replay the LTE traffictrace, and issue 10 queries to retrieve all flows betweentwo IP addresses randomly chosen by us. The numberof matching flows ranges from 119 to 21,300, and theyare scattered around the disks. We compare the query re-sponse times with those of n2disk10g. The indexing ofn2disk10g similar to that of FloSIS in that it uses tuple-based Bloom filters to skip the dump files, but it performslinear search on per-packet indexes called packet digests.Per-packet indexing not only incurs more space overheadbut also requires more disk seeks to read the packets scat-tered around the disks.

Figure 7 compares the response times of 10 queries.We find that FloSIS outperforms n2disk10g by a factorof 2.2 to 4.9. As the number of flows increases, the per-

9

Queries Q1 Q2 Q3 Q4Bloom filter 100% 0.05% - -Sorted array - 10.83% 100% 99.43%Flow read - 89.12% - 0.57%Latency 3.3ms 330.2ms 80.5s 82.2s

Response 0B 2.7KB 0B 2.7KB

Table 1: Query processing times of singleton and range querieswith/without disk access to read indexes and flow data

formance gap widens since n2disk10g suffers from moredisk activity to gather all data. Unfortunately, we couldnot fill more data to examine the performance differencefurther since n2disk10g that we have is an evaluation ver-sion that runs for only a few minutes.

To investigate the FloSIS behavior with more flowdata, we disable deduplication in FloSIS and fill the LTEtrace repeatedly until we have 10 TBs of stored data.During the data filling phase, we feed in a few randomflows that we retrieve in our queries. We use four typesof queries to evaluate the effectiveness of our indexingstructure. We use two singleton (Q1, Q2) and two rangequeries (Q3, Q4) for the entire time period. One of thetwo queries in each query type (Q1, Q3) looks for a flowthat does not exist on disk, while the other queries (Q2,Q4) would return a small flow (2.7 KB) as a response.

Table 1 shows the response times for all queries. Q1takes only 3.3 milliseconds (ms) since it would requirechecking only the Bloom filters in memory. Q2 takesmore time (330.2 ms) since it has to read in the sorted ar-rays and flow metadata for the matching dump file froman SSD after cache hits with the Bloom filters. It alsoneeds to read in the flow data from an HDD. Q3 and Q4take much longer, 80.5 and 82.2 seconds, respectively,since FloSIS skips the Bloom filters for range queries,and reads in sorted arrays and flow metadata for all dumpfiles. Since there are about 10,000 dump files for 10 TBsof data, the total size of all sorted arrays and flow meta-data would be 20 GB. Actually, we could optimize thebehavior further by reading the sorted arrays first, andread the flow metadata only if the query is a hit with thesorted arrays. Without indexing, it would require readingall data from HDDs which could take at least a few hoursto resolve the queries.

7.4 DeduplicationWe evaluate flow content deduplication with the real traf-fic trace. Deduplication is the most CPU-intensive taskin FloSIS due to the overhead of real-time content hash-ing. We see almost full CPU utilization when we enablededuplication at a high traffic rate. While we expect toimprove the performance in a cost-effective manner withSHA-1 computation offloading to off-chip GPUs [24], wefocus on the CPU-only performance here. We also mea-sure the level of storage compression by deduplication aswe use more deduplication table entries.

0

10

20

30

40

0 10 20 30 40

Ded

up

lica

tio

n R

ate

(%)

Running Time (Sec)

64K 128K 256K 512K 1M 2M 4M

Figure 8: Deduplication rates over various numbers of dedu-plication table entries

FloSIS achieves about 15 Gbps of the peak zero-drop performance with deduplication. The performanceis almost halved from the peak zero-drop performancewithout deduplication (in Figure 6), but its performancereaches 75% of the peak zero-drop performance ofn2disk10g. Even if we amortize the latency of SHA-1hashing over time, even a short spike of SHA-1 com-putation delay induces random packet drop, which dragsthe zero-drop performance. Nevertheless, we believe thatone can monitor a full 10 Gbps link with deduplicationwithout worrying about packet drop.

Finally, we estimate how many deduplication table en-tries are needed for a good deduplication rate. This ques-tion is difficult to answer given that we have only 67GB of real traffic workload. For this workload, we drawa graph that shows the deduplication rates over variousdeduplication table size. Figure 8 presents the resultsover the entire period of traffic replay at 13.5 Gbps. Thetrend we find here is that (i) even a small cache worksvery effectively: A table with only 64K entries (or just 4MB of content cache on disk) provides about 14% dedu-plication rate, (ii) it requires almost eight times more en-tries to double the deduplication rate, (iii) once the entriesare filled up, the deduplication rate stabilizes over time.We obtain 34.5% of deduplication rate with 4 million en-tries, which is enough to suppress the actual redundancyin the original data. We do note that our trace is too smallto draw any definitive conclusions, but we observe thata much larger workload shares a few similarities as ourworkload [31]. Given that 512 GB of content cache with4KB chunks (or 16 million entries in the deduplicationtable) gives 30 to 35% of deduplication rate at a busy 10Gbps link, we believe that a few GB of entries would beenough to achieve a good performance in practice.

8 Related WorkWe briefly discuss the related work here. Due to a largebody of works, comprehensive coverage is difficult, sowe attempt to provide a summary of representative works.

10

High-speed packet acquisition and writing: Scal-able and fast packet capture heavily affects the per-formance of network security monitoring systems suchas firewalls and intrusion detection/prevention systems(IDS/IPS). High-end security systems typically adopt aparallel packet acquisition architecture that exploit theparallelism in modern CPUs and network cards. For ex-ample, software-based IDSes such as SnortSP [4], Suri-cata [5] and Para-Snort [14] deploy parallelized pipelin-ing architectures suitable for a multi-core environment.Gnort [20], MIDeA [30] and Kargus [23] adopt a sim-ilar parallelized architecture for fast packet capture, butuse general-purpose GPUs to offload the heavy patternmatching operations for extra performance improvement.n2disk10g [17] is a recent high-speed packet capture andstoring system that adopts PF RING DNA for packetacquisition, and uses direct disk I/O for scalable diskthroughput. FloSIS shares the parallel packet capture ar-chitecture with these systems, but it goes beyond scalablepacket capture and shows how one should map hetero-geneous tasks of different computation budget (e.g., en-gine, writing, indexing threads) to computing resourcesfor high zero-drop performance.

Intelligent indexing for fast retrieval: It is of criticalimportance to have an intelligent indexing structure forfast query response. pcapIndex [19] uses compressedbitmap indexes to index pcap files at off-line, and sup-ports user queries with the BPF syntax. Hyperion [18]presents a stream file system optimized for sequentialimmutable streaming writes to store high-volume datastreams, and proposes a two-level index structure basedon Bloom filters. Gigascope [16] supports SQL-likequeries on a packet stream, however, without archivinglong-duration data. n2disk10g [17] supports per-packetindexing based on the Bloom filter and a packet metadatacalled packet digest. However, the per-packet indexingstructures are used only for rough filtering of the entiredata and it still requires linear search for query process-ing, often causing a long delay. FloSIS adopts flow-levelindexing and query processing, which eliminates lineardisk search and minimizes disk access in query process-ing. The required storage space for indexing is muchsmaller than that of packet-level indexing.

Network Traffic Compression: There have been manyworks on network traffic deduplication [9, 10, 11, 12, 29,31]. These works show that typical Internet traffic hassignificant content-level redundancy. In terms of dupli-cate suppression, a smaller chunking unit and content-based chunking algorithm like Rabin’s fingerprinting [27]generally leads to a higher deduplication rate at the costof a larger computation overhead. To the best of ourknowledge, FloSIS is the first traffic capture system thatemploys deduplication, and our choice of whole-flow

deduplication is reasonable given the trade-off of com-putation overhead and the level of storage savings.

Cooke et al. present a multi-format storage that storesdetailed information for recent data, but maintains onlythe summary of old data as time goes by [15]. Hori-zon Extender [21] proposes Web traffic classification andstoring based on white-listing, considering the Web ser-vice popularity. It mainly focuses on compressing storedHTTP data for storage savings while minimizing the lossof data required to find the evidence of data leakage.Time Travel [25] stores only the first part of the largeflows considering the heavy-tailed nature of network traf-fic. While these works reduce the storage requirement atthe cost of losing a fraction of the data, FloSIS focuses onlossless storing of full flow data at a high-speed networkfor accurate traffic monitoring. However, we believe ahybrid approach is possible for further storage savings.

9 ConclusionAs the network edge bandwidth exceeds 10 Gbps, thedemand for scalable packet capture and retrieval, usedfor attack analysis, network troubleshooting and perfor-mance debugging, is rapidly increasing. However, exist-ing software-based packet capture systems neither pro-vide high performance nor support flow-level indexingfor fast query response. In this paper, we have proposed ahighly scalable, software-based flow storing and indexingsystem, FloSIS, characterized by three features: exercis-ing full parallelism, flow-aware processing for minimiz-ing expensive disk access for user queries, and dedupli-cation for efficient storage usage.

We have demonstrated that FloSIS achieves up to30 Gbps of zero-drop performance without deduplica-tion, and 15 Gbps with deduplication with real trafficstoring and indexing, at the indexing storage cost of only0.25% of the stored data. The two-stage flow-level index-ing of FloSIS completes searching and reading 2.7 KBof flow data from 10 TB in 330.2 ms. Finally, our flowcontent deduplication reduces 34.5% of storage space for67 GB of the real traffic trace with 256 MB of additionalmemory consumption.

Acknlowdgments

We would like to thank Shinjo Park and Sanghyeok Chunfor their help on the initial design of the system. Wealso thank the anonymous reviewers and our shepherd,Keith Adams for valuable comments on the paper. Thiswork was supported in part by National Security Re-search Institute grants #N04130037, #N04140063, Na-tional Research Foundation of Korea (NRF/MSIP) [NRF-2013R1A2A2A01067633], and the ICT R&D programof MSIP/IITP, Republic of Korea [B0101-15-1368, De-velopment of an NFV-inspired networked switch and anoperating system for multi-middlebox services].

11

References

[1] Intel DPDK: Data Plane Development Kit. http://dpdk.org/.

[2] n2disk User’s Guide. http://www.ntop.org/.

[3] netsniff-ng. http://netsniff-ng.org/.

[4] SnortSP (Security Platform). http://www.snort.org/snort-downloads/snortsp/.

[5] Suricata Intrusion Detection System. http://www.openinfosecfoundation.org/index.php/download-suricata.

[6] Tcpdump & Libpcap. http://www.tcpdump.org/.

[7] Tcpdump: netmap support for lipcap. http://seclists.org/tcpdump/2014/q1/9/.

[8] Wireshark. http://www.wireshark.org/.

[9] AGARWAL, B., AKELLA, A., ANAND, A., BAL-ACHANDRAN, A., CHITNIS, P., MUTHUKRISH-NAN, C., RAMJEE, R., AND VARGHESE, G. En-dre: An end-system redundancy elimination servicefor enterprises. In Proceedings of USENIX NSDI(2010).

[10] ANAND, A., GUPTA, A., AKELLA, A., SESHAN,S., AND SHENKER, S. Packet caches on routers:The implications of universal redundant traffic elim-ination. In Proceedings of ACM SIGCOMM (2008).

[11] ANAND, A., MUTHUKRISHNAN, C., AKELLA,A., AND RAMJEE, R. Redundancy in network traf-fic: Findings and implications. In Proceedings ofACM SIGMETRICS (2009).

[12] ANAND, A., SEKAR, V., AND AKELLA, A.SmartRE: an architecture for coordinated network-wide redundancy elimination. In Proceedings ofACM SIGCOMM (2009).

[13] BLOOM, B. H. Space/time trade-offs in hash cod-ing with allowable errors. Communications of ACM13, 7 (July 1970), 422–426.

[14] CHEN, X., WU, Y., XU, L., XUE, Y., AND LI,J. Para-Snort: A Multi-thread Snort on Multi-coreIA Platform. In Proceedings of Parallel and Dis-tributed Computing and Systems (PDCS) (2009).

[15] COOKE, E., MYRICK, A., RUSEK, D., AND JA-HANIAN, F. Resource-aware multi-format networksecurity data storage. In Proceedings of ACM SIG-COMM workshop on Large-scale attack defense(2006).

[16] CRANOR, C., JOHNSON, T., SPATASCHEK, O.,AND SHKAPENYUK, V. Gigascope: a streamdatabase for network applications. In Proceedingsof ACM SIGMOD (2003).

[17] DERI, L., CARDIGLIANO, A., AND FUSCO, F. 10gbit line rate packet-to-disk using n2disk. In Pro-ceedings of IEEE INFOCOM Workshop on TrafficMonitoring and Analysis (2013).

[18] DESNOYERS, P. J., AND SHENOY, P. Hyper-ion: high volume stream archival for retrospectivequerying. In Proceedings of USENIX ATC (2007).

[19] FUSCO, F., DIMITROPOULOS, X., VLACHOS, M.,AND DERI, L. pcapIndex: an index for networkpacket traces with legacy compatibility. ACM SIG-COMM Computer Communications Review 42, 1(Jan. 2012), 47–53.

[20] G. VASILIADIS, S. ANTONATOS, M. POLY-CHRONAKIS, E. P. MARKATOS, AND S. IOANNI-DIS. Gnort: High Performance Network IntrusionDetection Using Graphics Processors. In Proceed-ings of the International Symposium on Recent Ad-vances in Intrusion Detection (RAID) (2008).

[21] GUGELMANN, D., SCHATZMANN, D., ANDLENDERS, V. Horizon extender: long-term preser-vation of data leakage evidence in web traffic. InProceedings of ACM CCS (2013).

[22] HAN, S., JANG, K., PARK, K., AND MOON, S. B.PacketShader: a GPU-accelerated software router.In Proceedings of ACM SIGCOMM (2010).

[23] JAMSHED, M. A., LEE, J., MOON, S., YUN, I.,KIM, D., LEE, S., YI, Y., AND PARK, K. Kar-gus: A highly-scalable software-based intrusion de-tection system. In Proceedings of ACM CCS (2012).

[24] JANG, K., HAN, S., HAN, S., MOON, S. B., ANDPARK, K. SSLShader: Cheap SSL Accelerationwith Commodity Processors. In Proceedings ofUSENIX NSDI (2011).

[25] MAIER, G., SOMMER, R., DREGER, H., FELD-MANN, A., PAXSON, V., AND SCHNEIDER, F. En-riching network security analysis with time travel.In Proceedings of ACM SIGCOMM (2008).

12

[26] NTOP. Libzero for DNA: Zero-copy flex-ible packet processing on top of DNA.http://www.ntop.org/products/pf_ring/libzero-for-dna/.

[27] RABIN, M. O. Fingerprinting by random polyno-mials. Tech. Rep. TR-15-81, Harvard University,1981.

[28] RIZZO, L. netmap: a novel framework for fastpacket I/O. In Proceedings of USENIX ATC (2012).

[29] SPRING, N. T., AND WETHERALL, D. A protocol-independent technique for eliminating redundantnetwork traffic. In Proceedings of ACM SIGCOMM(2000).

[30] VASILIADIS, G., POLYCHRONAKIS, M., ANDIOANNIDIS, S. MIDeA: A Multi-Parallel IntrusionDetection Architecture. In Proceedings of ACMCCS (2011).

[31] WOO, S., JEONG, E., PARK, S., LEE, J., IHM, S.,AND PARK, K. Comparison of caching strategies inmodern cellular backhaul networks. In Proceedingsof ACM MobiSys (2013).

13