IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 6, DECEMBER 2016 3477

Accurate and Efficient Per-Flow LatencyMeasurement Without Probing and Time Stamping

Muhammad Shahzad and Alex X. Liu

Abstract— With the growth in number and significance ofthe emerging applications that require extremely low latencies,network operators are facing increasing need to perform latencymeasurement on per-flow basis for network monitoring andtroubleshooting. In this paper, we propose COLATE, the firstper-flow latency measurement scheme that requires no probepackets and time stamping. Given a set of observation points,COLATE records packet timing information at each point so thatlater, for any two points, it can accurately estimate the averageand the standard deviation of the latencies experienced by thepackets of any flow in passing the two points. The key idea is thatwhen recording packet timing information, COLATE purposelyallows noise to be introduced for minimizing storage space,and when querying the latency of a target flow, COLATE usesstatistical techniques to denoise and obtain an accurate latencyestimate. COLATE is designed to be efficiently implementable onnetwork middleboxes. In terms of processing overhead, COLATEperforms only one hash and one memory update per packet.In terms of storage space, COLATE uses less than 0.1-b/packet,which means that, on a backbone link with half a million packetsper second, using a 256-GB drive, COLATE can accumulatetime stamps of packets traversing the link for over 1.5 years.We evaluated COLATE using three real traffic traces, namely,a backbone traffic trace, an enterprise network traffic trace, anda data center traffic trace. Results show that COLATE alwaysachieves the required reliability for any given confidence interval.

Index Terms— Passive measurement, per-flow latency.

I. INTRODUCTION

A. Motivation

ALTHOUGH traditionally throughput has been the pri-mary focus of network engineers, nowadays latency

has seen growing importance because a wide variety ofemerging applications and architectures require extremely low(in microseconds) and stable (low jitter) latencies. First,many emerging applications, such as financial tradingapplications [22], storage applications utilizing Fiber Channelover Ethernet, and real-time data center and cloud applications

Manuscript received May 27, 2015; revised December 23, 2015; acceptedJanuary 21, 2016; approved by IEEE/ACM TRANSACTIONS ON NETWORK-ING Editor S. Chen. Date of publication August 11, 2016; date of currentversion December 15, 2016. This work was supported in part by theNational Science Foundation under Grant CNS-1318563 and Grant CNS-1524698, the National Natural Science Foundation of China under Grant61472184 and Grant 61321491, and the Jiangsu Innovation and Entrepre-neurship (Shuangchuang) Program. The preliminary version of this paper,titled “Noise Can Help: Accurate and Efficient Per-flow Latency Measurementwithout Packet Probing and Time Stamping,” was published in the Proceed-ings of ACM SIGMETRICS, pp. 207–219, Austin, TX, USA, June 2014.(Corresponding author: Alex X. Liu.)

M. Shahzad was with the Department of Computer Science, Michigan StateUniversity, East Lansing, MI 48824 USA. He is now with the Department ofComputer Science, North Carolina State University, Raleigh, NC 27695 USA(e-mail: mshahza@ncsu.edu).

A. X. Liu is with the Department of Computer Science, Michigan StateUniversity, East Lansing, MI 48824 USA (e-mail: alexliu@cse.msu.edu).

Digital Object Identifier 10.1109/TNET.2016.2533544

such as multiplayer games and media streaming [5], andothers ([7], [12], [19], [24], [27]–[29], [31]), demand lowlatency. A small increase in latency may cause violationsof service level agreements and result in significant revenuelosses. For example, a one-millisecond advantage in financialtrading applications can be worth $100 million a year for majorbrokerage firms [22]. Second, many emerging architectures,such as content delivery networks (CDNs) and mission-criticaldata center networks, demand low latency. CDN providers aremostly evaluated and ranked by content publishers based onlatency. Companies such as Cedexis and Turbobytes constantlyevaluate and rank CDN providers mostly based on latency.A one-millisecond disadvantage could put one CDN providerbehind others and result in loss of business with content pub-lishers. Similarly, the transit providers are primarily evaluatedand ranked by CDN providers based on latency. For datacenters running mission-critical applications, latency guaranteeis a key requirement for the underlying networks.

In managing networks with stringent latency demands,operators often need to measure the latency between twoobservation points for a particular flow. An observation pointis either a port in a middlebox (such as a router or a switch) ora network card in an end host. Per-flow latency measurementcan be used reactively by network operators to perform taskssuch as detecting and localizing delay spikes in a network,isolating offending flows that are responsible for causing delaybursts, and rerouting them through other paths. It can also beused proactively by network operators to continuously monitorlatencies between observation points for locating bottlenecklinks and replace them with higher capacity links.

Existing routers and switches provide little help for latencymeasurement and monitoring. SNMP counters measure thenumber of packets passing through a port. NetFlow measuresbasic statistics, such as the numbers of packets and bytes,of a flow. Both provide no measurement on latency. Networkoperators often rely on injecting probe packets to measure end-to-end delays. However, to achieve latency measurement withextremely high accuracy, the required number of probe packetswill be extremely large; consequently, the probe packets willconsume too much bandwidth and the measured latency doesnot reflect the real latency without probe packets. Althoughsome specialized latency monitoring devices are commerciallyavailable, they are costly. For example, Tokyo stock exchangeuses latency monitoring devices manufactured by Corvilcosting around USD 180,000 for a 2 × 10Gbps box [3].

B. Problem Statement

This paper addresses the fundamental problem of per-flowlatency measurement: for any flow that passed through any two

1063-6692 © 2016 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

3478 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 6, DECEMBER 2016

observation points, measure (or say estimate) the average andstandard deviation of the latencies experienced by the packetsof that flow in passing through the two observation points.Formally, given a confidence interval β ∈ (0, 1] and a requiredreliability α ∈ [0, 1), for any flow f that passed throughany two observation points S and R, obtain estimates μ̃f

of the average μf and σ̃f of the standard deviation σf ofthe latencies experienced by the packets of f in passingthrough S and R so that P {|μ̃f − μf | ≤ βμf} ≥ α andP {|σ̃f − σf | ≤ βσf} ≥ α.

An accurate per-flow latency measurement scheme shouldfurther satisfy the following two requirements. (1) No packetprobing: As probe packets may use up a significant portion ofnetwork bandwidth, the latency measured with the insertion ofprobe packets may significantly deviate from the real latency.Thus, the estimates obtained with probe packets may notsuffice for microsecond level accuracy. (2) No time stamping:First, IP headers do not have a time stamp field and the TCPtime stamp option is meant for measuring end-to-end laten-cies. Embedding time stamps at observation points requiresmodifications to packet header formats, which further requiresmodifications to the data forwarding paths of existing routersand middleboxes. Furthermore, the added packet header fieldsmay consume a significant portion of network bandwidth.Second, the process of attaching time stamps to each packettakes a non-negligible amount of time at observation points.Thus, the latency measured with time stamping may signifi-cantly deviate from the real latency.

C. Limitations of Prior Art

To the best of our knowledge, there are only two per-flow latency measurement schemes, namely RLI [16] andMAPLE [17]. However, neither of them satisfies both require-ments because RLI uses packet probing and MAPLE attachestime stamps to every packet. Other than RLI and MAPLE, theclosest work is LDA [14], which performs aggregate latencymeasurement, i.e., given any two observation points, measure(or say estimate) the average and standard deviation of thelatencies experienced by all the packets that passed throughthe two observation points, regardless of the flow that eachpacket belongs to. Aggregate latency measurement is useful;however, it does not provide fine grained per-flow latencyinformation. Here is an important fact: for all flows that passedthrough two arbitrary observation points S and R, the latenciesexperienced by the packets of different flows in passingthrough S and R can be quite different. First, there may bemultiple paths from S to R and different flows may be routedvia different paths. Second, at each intermediate middleboxalong a path from S to R, packets of different flows may takedifferent amount of processing time due to mechanisms suchas QoS. As aggregate latency measurement does not reflectthe latency of every flow, it falls short in engineering latencysensitive networks. On one hand, when the aggregate latencybetween two points appears normal, the latency experiencedby an individual flow may be wildly abnormal. On the otherhand, when the aggregate latency between two observationpoints appears abnormal, aggregate latency measurement doesnot provide operators the per-flow latency information needed

Fig. 1. Counter vector and counter subvectors.

to identify the flows being hurt. Another set of techniques thatinfer aggregate latency measurement use network tomography.These schemes inject probe packets to first measure end-to-end delays and then use tomography to infer link and hopdelays [8], [10], [30]. Tomography based techniques fallshort from two aspects. First, to achieve latency measurementwith high accuracy, the required number of probe packets isextremely large, (see [14], [16]–[18] for brief discussions).Second, tomography based techniques can not measure laten-cies on per-flow basis.

D. Proposed Approach

In this paper, we propose COLATE, a Counter based Per-flow Latency Estimation scheme. The key idea of COLATE isthat it records timing information of packets at each observa-tion point and purposely allows noise to be introduced in therecorded information for minimizing storage. When queryingthe latency of a target flow, COLATE statistically denoises therecorded information to obtain an accurate latency estimate.COLATE has two phases: recording phase and querying phase.Next, we give an overview of these two phases.

Recording Phase: In this phase, at each observation point,COLATE maintains a vector of counters in RAM calledcounter vector, which it uses to record the timing informationof each packet arriving at or departing from that observationpoint. For each flow with a unique ID, COLATE maps it to aunique subset of these counters, which we call counter sub-vector. The timing information of any given flow is recordedonly in the counters of its own counter subvector. The ID ofa flow can be a flow identifier such as the standard five-tuple(i.e., source IP, destination IP, source port, destination port, andprotocol type). To make the mapping memoryless, COLATEuses a hash function to map each flow to a random subvector.Due to this random mapping, a counter may belong to countersubvectors of multiple flows. Figure 1 shows an examplecounter vector and its three counter subvectors, from whichwe see that counters 5 and 8 belong to counter subvectorsof multiple flows. For each arriving or departing packet atthe observation point, COLATE executes two simple steps:(1) randomly maps the packet to a counter in the countersubvector of the flow that the packet belongs to; (2) adds thecurrent time to that counter. Before any counter overflows,COLATE dumps the counter vector to a permanent storage(such as a solid state drive (SSD)) and resets counters tozero. We call a dumped counter vector a counter epoch, whichhas two attributes: the time stamp of the first recorded packetand the time stamp of the last recorded packet. The recordingmodule can be implemented in hardware to keep up with wirespeed. For the hashing function, we can use hardware hashimplementations such as those proposed in [13] and [26].For each counter, we can store its less significant bits as a

SHAHZAD AND LIU: ACCURATE AND EFFICIENT PER-FLOW LATENCY MEASUREMENT 3479

counter in SRAM and the more significant bits as a counter inDRAM – when the counter in SRAM overflows, we incrementthe corresponding counter in DRAM.

Querying Phase: In this phase, given a latency measurementquery of a flow f , which contains the flow ID, the startingand ending time of the flow, and two observation points thatthe flow passed through within the time frame, COLATEfirst finds all the counter epochs whose time frame overlapswith the starting and ending time of the flow f at each ofthe two given observation points. Second, for each counterepoch, COLATE applies statistical techniques to estimate thesum of time stamps contributed only by flow f from eachcounter in the counter subvector of f . COLATE uses theseextracted values to estimate the average and standard deviationof the latencies experienced by the packets of flow f inpassing through the two observation points. COLATE requiresthe clocks of different observation points to be accuratelysynchronized; otherwise, the measured latencies will contain aconstant offset. This synchronization can be simply achievedby the standard time synchronization protocol IEEE 1588,which provides microsecond level time synchronization.

Key Intuition: The intuition behind mapping a flow tomultiple counters (in a counter subvector), instead of a singlecounter, is to mitigate counter overflow for elephant flows.The motivation behind sharing counters among multiple flows,instead of allocating unique counters for each individual flow,is to save memory. Due to the sharing of counters amongmultiple flows, the counter subvector of a flow f contains notonly the timing information of the packets in f but also thatof the packets in other flows. The intuition behind allowingthis “mixing” is that later in the querying phase, we canextract the timing information of only the packets in the flowf using statistical techniques by treating the mixed-in timinginformation of the packets from other flows as noise.

Deployability: COLATE is designed to be efficiently imple-mentable on network middleboxes from both processing over-head and storage space perspectives. In terms of processingoverhead, COLATE performs only one hash and one memoryupdate per packet. In modern memory architecture used inhigh speed routers (such as the smart memory architectureby Huawei [21] and the bandwidth engine by MoSys [23]),where each memory location has built-in circuitry for handlingupdates, one update (such as incrementing by up to a 64-bitnumber) requires only one memory access. In terms of storagespace, COLATE uses less than 0.1 bit per packet. To get anidea of the length of time for which COLATE can accumulatetime stamps using commodity permanent storage devices,consider the 10GigE backbone link in San Jose monitored byCAIDA. An interactive tool at CAIDA’s website reported thaton average approximately 0.484 million packets traversed thislink per second between Nov. 01 and Nov. 29, 2013 [1]. With0.1 bits per packet, on a commodity 256GB SSD, COLATEcan accumulate time stamps of packets traversing this linkfor about 1.5 years, which means that a network operator canmeasure average and standard deviation of up to 1.5 years oldflows. This gives not only enough time to identify and debugany problems, but also enough information to study otheraspects related to packet delays such as diurnal patterns in

flow latencies. On a 12TB SSD, COLATE can accumulate timestamps of packets traversing this link for more than 69 years.

E. Technical Challenges and Solutions

The first challenge is to denoise the recorded information toextract the sum of the time stamps of the packets in the targetflow from the counter subvector of the flow. To address thischallenge, we first show that for each counter in the countersubvector of the target flow, the value contributed by the timestamps from the packets in the target flow and that from thepackets in other flows can both be modeled with binomialdistributions. We then derive an expression to calculate theexpected value of each counter in the counter subvector ofthe target flow. From this expression, we estimate the sum ofthe time stamps of all the packets of the target flow. Using thisestimate in conjunction with maximum likelihood estimation,we extract the sum of the time stamps of the packets in thetarget flow from each counter in the counter subvector.

The second challenge is to calculate the sum of the squaresof the latencies of each packet in a target flow, which isneeded for calculating the standard deviation of packet laten-cies. To address this challenge, we first use the time stampsum extracted from counter subvectors to construct a virtualdeviation vector and then use this vector to estimate this sumof the squares of the latencies.

F. Advantages of COLATE Over Prior Art

COLATE brings forward the state-of-the-art in per-flowlatency measurement on the following fronts: reliability,passiveness, scalability, memory, efficiency, and flexibility.For reliability, COLATE takes the required reliability andconfidence interval as input whereas existing schemes don’t.For passiveness, COLATE neither sends probe packets norattaches time stamps. For scalability, COLATE maintains onlyone counter vector at each observation point regardless ofhow many other observation points are sending and receivingpackets from it; in contrast, LDA and RLI have to maintainseparate vectors of counters for each pair of sender andreceiver. For memory, COLATE uses less than 0.1 bit ofstorage space per packet, which is over 128 times improvementcompared to 12.8 bits of storage space per packet used byMAPLE. Due to this, on a commodity 256GB SSD, whereCOLATE can accumulate time stamps of packets traversing theSan Jose backbone link for about one and a half year, MAPLEcan accumulate the time stamps for only 4.1 days. For effi-ciency, COLATE performs only 1 hash and 1 memory updateper packet whereas MAPLE uses 9 hashes and 12 memoryaccesses per packet. For flexibility, COLATE allows differentobservation points to allocate different amount of RAM basedon their available resources whereas LDA requires both thesender and receiver to allocate the same amount of RAM.

II. RELATED WORK

To the best of our knowledge, there are only two per-flow latency measurement schemes, namely MAPLE [17]and RLI [16]. In MAPLE, for any packet passing throughtwo observation points S and R, S attaches a time stamp to thepacket and R calculates the latency of the packet from S to R

3480 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 6, DECEMBER 2016

by subtracting the time stamp from its current time. To reducespace for storing latency values of all packets, MAPLE mapsthe calculated latency of each packet to the closest valuein a set of predetermined latency values. Thus, instead ofstoring the latency of every packet, MAPLE only storesthese predetermined latency values and for each predeterminedlatency value MAPLE uses a Bloom filter to store all packetsmapped to it. To query the latency of a given packet, MAPLEfirst finds the predetermined latency value that the packet wasmapped to by querying Bloom filters and uses that value asthe estimated latency of the packet. Compared with COLATE,MAPLE falls short from a few perspectives. First, MAPLEis based on a strong assumption that packet latencies betweentwo observation points are tightly clustered around a set of pre-determined latency values. We have not found any theoreticalor empirical validation of this assumption in prior literature.Second, MAPLE requires attaching time stamps to packetsand thus has the limitations we pointed out in Section I-B for time stamping based latency measurement schemes.Furthermore, time stamping individual packets can consumeup to 10% of the available bandwidth [14]. Third, MAPLE haslarge memory overhead (12.8 bits/packet at each observationpoint) and large processing overhead (9 hash computations and12 memory accesses per packet: 9 for hash functions, 2 forupdating counters, and 1 for determining the cluster a packet’slatency belongs to), whereas COLATE uses 0.1 bits/packet andperforms 1 hash and 1 memory update per packet.

In RLI, for a flow passing through two observation points Sand R, S inserts probe packets with time stamps into the flowand R calculates the latency of each probe packet similarlyto MAPLE. To calculate the latency of the regular packetsbetween two probe packets whose latency has been calculatedas l1 and l2, R simply uses the straight line equation to calcu-late the latency of these regular packets based on their arrivaltime and the latency of the two probe packets. Comparedwith COLATE, RLI has the following limitations. First, RLIis based on the strong assumption that packet latency betweenS and R increases or decreases linearly in the time intervalbetween receiving any two probe packets at R. When the timeinterval between two probe packets is extremely small, thisassumption may practically hold but the extremely small timeinterval implies that the number of probe packets is extremelylarge. For example, to achieve an accuracy of only 81%,RLI inserts, on average, 1 probe packet every 4.78 regu-lar packets. Furthermore, as we mentioned in Section I-B,latency measured with a large number of probe packets maysignificantly deviate from the real latency when there are noprobe packets. When the interval between two probe packetsis large, this assumption is nonintuitive and may not holdin practice. Similarly, we have not found any theoretical orempirical validation of this assumption in prior literature.

The other latency measurement schemes are LDA [14] andFineComb [18], which provide aggregate, not per-flow, latencymeasurement between a sender and a receiver. In LDA, boththe sender and the receiver maintain several counter vectorswhere each element is a pair of counters: time stamp counterfor accumulating packet time stamps and packet counter forcounting the number of arriving/departing packets. Each vector

has a sampling probability and a sampling function. For eacharriving or departing packet, LDA first maps the packet to acounter vector such that in the long run, the fraction of packetsthat LDA maps to any counter vector is equal to its samplingprobability. It then randomly maps the packet to a counter pairin this counter vector and adds the time stamp of the packetto the time stamp counter and increments the packet counterby one. To obtain the aggregate latency estimate between thesender and receiver, for each counter pair in each vector, LDAchecks whether they have the same packet counter value andselects all counter pairs that have the same packet countervalue for both the sender and receiver. Then, LDA can easilycalculate the total number of successfully delivered packetsand the sum of their time stamps at both sides. Finally, toobtain the aggregate average latency between the sender andreceiver, it subtracts the sum of time stamps at the senderside from that at the receiver side and divides it with the totalnumber of successfully delivered packets.

III. COLATE—RECORDING PHASE

Now we present the recording phase and the statisticalmodeling of COLATE. Table I lists the symbols used in paper.

A. Noisy Accumulation of Time Stamps

At each observation point X , COLATE maintains a vec-tor CX of n counters where each counter CX [i] (1 ≤ i ≤ n)has b bits with initial value 0. When a packet arrives ator departs from an observation point X , COLATE extractsits flow ID f , chooses a random number j from a uniformdistribution in the range [1,m] where m � n, calculatesthe hash function H(f, j) whose output is uniformly dis-tributed in range [1, n], and adds the time stamp of thispacket (i.e., the current time at observation point X) tothe counter CX [H(f, j)]. Thus, the time stamp of all pack-ets in flow f will be uniformly distributed to m coun-ters: CX [H(f, 1)],CX [H(f, 2)], · · · ,CX [H(f,m)]. These mcounters constitute the counter subvector of flow f , which isdenoted by Sf

X where SfX [j] = CX [H(f, j)] for j ∈ [1,m].

In this recording phase, for each packet, COLATE performsone memory update to update the value of CX [H(f, j)]. Fordifferent flows, the probability that COLATE maps them to thesame counter subvector is m!

nm (=2.4 × 10−62 for n = 10000and m = 20, for example), which is practically zero. Note thatan observation point is a port of a middlebox, not a middleboxitself, because the arriving and departing time of a packet ata middlebox are different due to the non-negligible packetprocessing time within the middlebox.

B. Analysis of Noisy Accumulation

First, by Lemma 1, we show that in any counter epoch,on average, a flow contributes the same amount to eachcounter in its counter subvector. We further show thatthe amount contributed by a flow to each counter in itscounter subvector can be modeled by a binomial distribu-tion. Second, we derive expressions for the expected valueand variance of a counter in counter vector in Theorem 1.In Section IV, we use the expression of expected value to

SHAHZAD AND LIU: ACCURATE AND EFFICIENT PER-FLOW LATENCY MEASUREMENT 3481

TABLE I

SYMBOLS USED IN THE PAPER

estimate the average and standard deviation of packet latenciesfor any given flow. In Section V, we use the expression ofvariance to determine the parameter values that can ensurethat the actual reliability achieved by COLATE is no less thanthe required reliability.

Lemma 1: Let Cf be the random variable representing the

sum of time stamps contributed by flow f to a counter SfX [j],

1 ≤ j ≤ m, in its counter subvector of lengthm at observationpoint X . Let pf be the number of packets in flow f thatcontributed time stamps to the current counter epoch CX . LetTf be an independent random variable representing the timestamp contributed by each packet of flow f to CX . Then, we

have E[Cf ] = pf×E[Tf ]m .

Proof: Let Pf,j be a random variable representing thenumber of packets in flow f that contributed time stamps to

Fig. 2. CDF: observed Cf /E[Cf ].

counter SfX [j]. Therefore, E[Cf ] = E

[∑Pf,j

Tf

]. Apply-

ing Wald’s Lemma, we get E[Cf ] = E[Pf,j ] × E[Tf ].As the output of hash function H is uniformly distributedin range [1, n], its output is also uniformly distributed in[1,m] for the packets in flow f . Thus, the probability thatCOLATE adds the time stamp of a packet of the flow fto counter Sf

X [j] is 1/m. The random variable Pf,j followsthe binomial distribution, i.e., Pf,j ∼ Binom(pf , 1/m). Thus,E[Cf ] = pf

m × E[Tf ]. �Figure 2 plots the CDF of the ratio of the observed values

of Cf from simulations of our ICSI traffic trace to the E[Cf ]from Lemma 1. We observe from this figure that there is asteep rise in the CDF when the value of this ratio is around 1.We also observe from simulations that the mean and medianof the ratio are both equal to 1. This empirically establishesthe result in Lemma 1.

Lemma 1 shows that the sum of time stamps of all packetsof flow f is equally divided among all counters in its countersubvector, which conforms to binomial distribution. Thus, weapproximate the distribution of Cf with a binomial distributionas Cf ∼ Binom(tf,X , 1/m), where tf,X represents sum ofall times-stamps contributed by flow f to the counters in itscounter subvector at observation point X . Let Cr be the ran-dom variable representing the sum of time stamps contributedby packets of all flows other than f to counter Sf

X [j]. Usingsimilar reasoning as for Cf , we approximate the distribution ofCr with a binomial distribution. The probability that a packetof a flow f̄ �= f contributes a time stamp to counter Sf

X [j] isthe product of the probability that H maps the packet to Sf

X [j]given that Sf

X [j] ∈ Sf̄X , which is 1/m, and the probability

that counter SfX [j] is in the counter subvector of f̄ , which is

denoted by P{SfX [j] ∈ Sf̄

X} and calculated as

P{SfX [j] ∈ Sf̄

X} = 1 −{(

m

0

)(1n

)0(1 − 1

n

)m}

= 1 −{

1 − m

n+

(m)(m− 1)n2 × 2!

− . . .

}

≈ m

n∵ m� n

Thus, the probability that a packet of a flow f̄ �= f contributesa time stamp to counter Sf

X [j] is 1m × m

n = 1n . Thus,

Cr ∼ Binom(tX − tf,X , 1/n).Theorem 1: Let C be the random variable representing

the value of a counter SfX [j], 1 ≤ j ≤ m, in the counter

subvector of a flow f . Let tf,X be the sum of all time stampscontributed by packets of flow f to all counters in its countersubvector and tX be the sum of all time stamps contributed

3482 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 6, DECEMBER 2016

by packets of all flows in the counter epoch at observationpoint X . Let Cf ∼ Binom(tf,X , 1/m) represent the sum oftime stamps contributed by packets of flow f to Sf

X [j] and letCr ∼ Binom(tX−tf,X , 1/n) represent the sum of time stampscontributed by packets of all flows other than f to counterSf

X [j]. Let Cf and Cr be independent of each other. Theexpected value and variance of C are calculated as follows

E[C] =tf,X

m+tX − tf,X

n(1)

Var(C) =(tf,X

m

)(1 − 1

m

)+(tX − tf,X

n

)(1 − 1

n

)(2)

Proof: As C = Cf +Cr, we get, E[C] = E[Cf ]+E[Cr] =tf,X

m + tX−tf,X

n . As Cf and Cr are assumed to be independent,Var(C) = Var(Cf )+Var(Cr). As variance of Binom(v, w) isvw(1−w), we get Var(Cf ) =

( tf,X

m

)(1− 1

m

)and Var(Cr) =( tX−tf,X

n

)(1 − 1

n

). �

In Theorem 1, we assumed Cf and Cr to be independent ofeach other, which is true whenever tf,X � tX . In practice tf,X

indeed is much smaller than tX because tf,X is the sumof the time stamps added by a single flow in the counterepoch while tX is the sum of the time stamps added byall flows (in the order of tens and hundreds of thousands).Furthermore, in Theorem 1, we approximate the distributionsof Cf and Cr with binomial distributions because when thereare a large number of packets that contribute time stamps to thecounters in the counter vector, we can approximate each timestamp to be smeared over the counters, i.e., as if instead ofadding time stamp tp of a packet to one counter in its countersubvector, a value of tp/y was added to y counters in itscounter subvector. This approximation makes the formal devel-opment of variance of C and its use in calculating COLATE’sparameters tractable. If the exact equation for variance of C isdesired, it can be obtained as follows. Consider any packetwith time stamp tp not belonging to a flow f . This packethas a probability 1

n of being mapped to a counter in thecounter subvector of the flow f . Thus, the time stamp foreach such packet contributes a variance of t2p(

1n )(1 − 1

n ) tothe overall variance of C. In Theorem 1, tf,X/m modelsthe average sum of the time stamps contributed by flow f ,and (tX − tf,X)/m models the average noise contributed byall other flows to Sf

X [j].

IV. COLATE—QUERYING PHASE

In this section, we present the methods that COLATE usesto estimate the average and standard deviation of the latenciesof the packets of a flow in passing any two observation points.

A. Estimating Latency Average

For a flow f passing through observation point S and thenobservation point R, we want to calculate μ̃f , the estimate ofaverage latency μf of flow f . For a packet z in flow f , letuf,S [z] be its time stamp at observation point S and uf,R[z] beits time stamp at observation point R. The delay experiencedby this packet in traveling from S to R is thus uf,R[z] −uf,S [z]. Let tf,S and tf,R be the sums of all time stamps of

the packets in flow f at S and R, respectively. Recall that pf

denotes the number of packets in f . Thus,

μf =1pf

∑

∀z

(uf,R[z] − uf,S[z]

)

=1pf

(∑

∀z

uf,R[z] −∑

∀z

uf,S [z])

=1pf

(tf,R − tf,S)

Note that the value of pf can be measured by tools such asNetFlow available on Cisco routers or by schemes proposedin [15] and [20]. Thus, to estimate the value of μf , we needto obtain the estimated values t̃f,S and t̃f,R for tf,S and tf,R,respectively. Then, we can calculate

μ̃f =1pf

(t̃f,R − t̃f,S) (3)

We present two methods to estimate the sum of time stamps ofthe packets in a flow at any observation point. The first methoduses an expectation based estimator (EBE) while the secondmethod uses a maximum likelihood based estimator (MLE).EBE is computationally simple but the estimate it provideshas a higher variance. Consequently, it requires comparativelymore SRAM at the observation points to achieve the requiredreliability. MLE is computationally slightly more complex butthe estimate it provides has a smaller variance. Consequently,it requires comparatively less SRAM at the observation pointsto achieve the required reliability. We propose to use EBE toestimate the per-flow latency at runtime. Whenever, a problemis identified by EBE, we propose to use MLE to obtainmore accurate per-flow latency measurements for further trou-bleshooting. Next we describe EBE and MLE is more detail.

1) Expectation Based Estimator: Theorem 2 shows how toobtain the estimated value t̃f,X for the sum of time stamps ofthe packets in flow f at observation point X using EBE.

Theorem 2: Given a counter epoch CX of length n atobservation point X , where each counter subvector is oflength m, let tX denote the sum of all counters in CX , theestimate t̃f,X of the sum of all time stamps of the packets inflow f is calculated as follows

t̃f,X =1

n−m

{n

m∑

j=1

SfX [j] −mtX

}(4)

Proof: Given CX and flow f , we can easily obtain the

values of every counter in the counter subvector SfX of f .

Thus, we can calculate E[C] as E[C] = 1m

∑mj=1 Sf

X [j]. Sub-

stituting E[C] in Equation (1) by 1m

∑mj=1 Sf

X [j], replacingtf,X by t̃f,X , and solving for t̃f,X , gives Equation (4). �

2) Maximum Likelihood Based Estimator: Theorem 3shows how to obtain the estimated value t̃f,X using MLE.

Theorem 3: Given a counter epoch CX of length n atobservation point X , where each counter subvector is oflength m, let tX denote the sum of all counters in CX ,let S̃f

X [j] denote the observed value of counter SfX [j], the

estimated value t̃f,X of the sum of time stamps of packetsin flow f is given by the numerical solution of the followingequation:

m∑

j=1

⎧⎨

⎩

∑S̃fX [j]

sf =0 {X[j] × Z[j]}∑S̃f

X [j]sf =0 {X[j] × Y[j]}

⎫⎬

⎭= 0 (5)

SHAHZAD AND LIU: ACCURATE AND EFFICIENT PER-FLOW LATENCY MEASUREMENT 3483

where

X[j] =(

1m

)sf(1− 1

m

)−sf(

1n

)S̃fX [j]−sf

(1− 1

n

)tX−S̃fX [j]+sf

Y[j] =(tf,X

sf

)(tX − tf,X

S̃fX [j] − sf

)(1 − 1

m

)tf,X(

1 − 1n

)−tf,X

Z[j] = Y[j][log

{1− 1

m

}−log

{1− 1

n

}−ψ(0){1+tX −tf,X}

+ψ(0){1 + sf − S̃f

X [j] + tX −tf,X

}

+ ψ(0) {1 + tf,X} − ψ(0) {1 − sf + tf,X}]

Here ψ(0){.} is the 0th order polygamma function.Proof: The probability or likelihood of getting the obser-

ved value S̃fX [j] of a counter Sf

X [j] in the counter subvectorof flow f is given by the following equation.

P {C = s} =s∑

sf =0

{(tf,X

sf

)(1m

)sf(

1 − 1m

)tf,X−sf

×(tX−tf,X

s− sf

)(1n

)s−sf(

1− 1n

)tX−tf,X−s+sf}

(6)

Thus, the likelihood of getting the observed values of allcounters in the counter subvector of the flow is as follows.

L=m∏

j=1

P{C = S̃f

X [j]}

=m∏

j=1

⎧⎨

⎩

S̃f[j]X∑

sf =0

[(tf,X

sf

)(1m

)sf(1− 1

m

)tf,X−sf(tX − tf,X

S̃fX [j] − sf

)

×(tX−tf,X

S̃fX [j]−sf

)(1n

)S̃fX [j]−sf

(1− 1

n

)tX−tf,X−S̃fX [j]+sf

]}

(7)

Note that in the right hand side (R.H.S) of the equation above,there is only one unknown, i.e., tf,X . We use the maximumlikelihood estimation (MLE) method to estimate the valueof tf,X using this equation, i.e., we determine the value of tf,X

that maximizes the likelihood calculated by this equation.Formally, the estimated value t̃f,X of the sum of time stampsof packets in flow f at observation point X is given by:

t̃f,X = argmaxtf,X

{L} (8)

To make the mathematical development tractable, we firstapply natural logarithm on both sides of the equation above.

ln {L} =m∑

j=1

ln

⎧⎨

⎩

S̃fX [j]∑

sf =0

[(tf,X

sf

)(1m

)sf(

1 − 1m

)tf,X−sf

×(tX − tf,X

S̃fX [j] − sf

)(1n

)S̃fX [j]−sf

(1− 1

n

)tX−tf,X−S̃fX [j]+sf

]⎫⎬

⎭

Next, we differentiate ln {L} with respect to tf,X and equateit to 0, i.e., d

dtf,Xln {L} = 0. This gives us the following:

m∑

j=1

d

dtf,Xln

⎧⎨

⎩

S̃fX [j]∑

sf=0

[(tf,X

sf

)(1m

)sf(

1 − 1m

)tf,X−sf

×(tX − tf,X

S̃fX [j] − sf

)(1n

)S̃fX [j]−sf

×(

1 − 1n

)tX−tf,X−S̃fX [j]+sf

]⎫⎬

⎭= 0 (9)

Let

X[j] =(

1m

)sf(

1 − 1m

)−sf(

1n

)S̃fX [j]−sf

×(

1 − 1n

)tX−S̃fX [j]+sf

Y[j] =(tf,X

sf

)(tX − tf,X

S̃fX [j] − sf

)(1 − 1

m

)tf,X(

1 − 1n

)−tf,X

Thus, Equation (9) becomes

m∑

j=1

d

dtf,Xln

⎧⎨

⎩

S̃fX [j]∑

sf =0

[X[j] × Y[j]]

⎫⎬

⎭= 0

As X[j] is not a function of tf,X , the equation above becomes

m∑

j=1

⎧⎪⎨

⎪⎩

∑S̃fX [j]

sf =0

{X[j] × d

dtf,XY[j]

}

∑S̃fX [j]

sf =0 {X[j] × Y[j]}

⎫⎪⎬

⎪⎭= 0 (10)

To calculate ddtf,X

Y[j], we use the following identity.

d

dw

(v

w

)=(v

w

)(ψ(0) {v − w + 1} − ψ(0) {w + 1}

)(11)

Using this identity and standard algebraic operations, we getthe following equation for d

dtf,XY[j].

d

dtf,XY[j] = Y[j]

[log

{1 − 1

m

}− log

{1 − 1

n

}

−ψ(0) {1 + tX − tf,X}+ψ(0)

{1 + sf − S̃f

X [j] + tX − tf,X

}

+ψ(0) {1 + tf,X} − ψ(0) {1 − sf + tf,X}]

Note that the R.H.S of this equation is the value ofZ[j] in Theorem statement. Replacing d

dtf,XY[j] by Z[j] in

Equation (10), we get Equation (5) in Theorem statement. �

B. Estimating Latency Standard Deviation

Section IV-A2 does something nice:Let Df be the random variable representing the latency

experienced by a packet in flow f . The standard deviation

3484 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 6, DECEMBER 2016

of Df can be calculated by σ̃f =√

Var(Df ), where Var(Df )can be calculated as follows:

Var(Df )= E[D2

f ] − E2[Df ]

={

1pf

∑

∀z

(uf,R[z]− uf,S[z]

)2}

−{

1pf

∑

∀z

(uf,R[z]− uf,S[z]

)}2

={

1pf

∑

∀z

(uf,R[z] − uf,S [z]

)2}−{

1pf

(tf,R − tf,S

)}2

(12)

We can calculate the second term { 1pf

(tf,R − tf,S)}2 inEquation (12) based on Theorem 2. Now the key challengeis to calculate the first term { 1

pf

∑∀z(uf,R[z]− uf,S[z])2} in

Equation (12). Our solution to this challenge is based on thestatistical technique proposed by Alon et al.in [4]. The mainidea is to introduce a random variable Gz where the value gz

that this random variable takes on is either +1 or −1 withequal probability. Before adding the time stamp of a packet zto a counter, if we randomly multiply the time stamp with gz

and then add it to the counter, then we will get Equation (13).

E

[{∑

∀z

(gzuf,R[z]−gzuf,S[z]

)}2]=∑

∀z

(uf,R[z]−uf,S[z]

)2

(13)

This can be proven as follows:

E

[{∑

∀z

gz

(uf,R[z] − uf,S [z]

)}2]

= E

[∑

∀z

g2z

(uf,R[z] − uf,S [z]

)2

+∑

∀z �=z

gzgz

(uf,R[z] − uf,S[z]

)(uf,R[z] − uf,S[z]

)]

Using the well known result that expectation of sum of randomvariables is the sum of their individual expectations and thatg2

z = 1, we get:

E

[{∑

∀z

gz

(uf,R[z] − uf,S[z]

)}2]

=∑

∀z

(uf,R[z] − uf,S [z]

)2

+∑

∀z �=z

(uf,R[z]− uf,S [z]

)(uf,R[z]− uf,S [z]

)× E[GzGz]

Note that {G1, G2, G3, . . . } is a set of independent andidentically distributed random variables. So, E[GzGz] =E[Gz ]×E[Gz ]. As E[Gz] = 0 for all values of z, this impliesE[GzGz] = 0. Thus, the second term in the equation aboveis 0, which proves Equation (13).

We now present our method for calculating the first term inEquation (12). First, for each counter in the counter subvectorof f , whose value is contributed by the time stamps of thepackets in flow f and those of the packets in other flows,we use Theorem 4 to extract an estimate of the value thatis contributed by only the time stamps of the packets in f .In other words, we eliminate the noise introduced by the

packets of flows other than f from the counter subvector of f .Second, we statistically simulate the process of multiplyingthe time stamp of a packet in f with the random variable Gz

and then adding the multiplication result to the correspondingcounter in the counter subvector of f . By repeating this processa statistically sufficient number of times, we obtain an accurateestimate of

∑∀z(uf,R[z] − uf,S[z])2 based on Theorem 5.

Note that this simulation does not require any changes to therecording phase. Next, we present our denoising solution andstatistical simulation process.

1) Denoising Counter Subvectors: Our denoising solution isbased on Theorem 4. The numerical solution of Equation (14)gives us the estimate of the value that is contributed by onlythe time stamps of the packets in f for each counter in f ’ssubvector.

Theorem 4: Let wf,X [j] (1 ≤ j ≤ m) be the sum of thetime stamps of flow f ’s packets that are mapped to counterSf

X [j] at observation point X . Let t̃f,X be the estimate ofsum of all time stamps contributed by packets of flow f to allcounters in f ’s counter subvector and tX be the sum of all timestamps contributed by packets of all flows in the counter epochat observation point X . The maximum likelihood estimatew̃f,X [j] of wf,X [j] satisfies the following equation:

ln {n− 1} = ψ(0){tX − t̃f,X − Sf

X [j] + w̃f,X [j] + 1}

−ψ(0){

SfX [j] − w̃f,X [j] + 1

}(14)

where ψ(0){.} is the 0th order polygamma function.Proof: The maximum likelihood estimate of wf,X [j] is

the value of wf,X [j] that maximizes the probability that thecounter Sf

X [j] takes the observed value.

arg maxwf,X [j]

P{C = Sf

X [j]∣∣wf,X [j]

}

This value of wf,X [j] can be obtained by differentiatingP{C = Sf

X [j]∣∣wf,X [j]

}w.r.t wf,X [j] and equating to 0.

d

d(wf,X [j])P{C = Sf

X [j]∣∣wf,X [j]

}= 0

As C = Cf + Cr and Cf = wf,X [j], the L.H.S. becomes

d

d(wf,X [j])P{Cr = Sf

X [j] − wf,X [j]}

For simplicity, let ξ = SfX [j] − wf,X [j] and τ = tX − tf,X .

As Cr is a binomial random variable, this derivative becomes

d

d(wf,X [j])

(τ

ξ

)(1n

)ξ(1 − 1

n

)τ−ξ

=

{(τ

ξ

)(1n

)ξ(1 − 1

n

)τ−ξ}{

ψ(0) {ξ + 1}

− ψ(0) {τ − ξ + 1} + ln{

1 − 1n

}− ln

{1n

}}

(15)

Due to space limitations, we have skipped the intermediatederivation steps, which uses the identity given in Equation 11.By replacing tf,X with t̃f,X , which is calculated usingTheorem 2, in τ = tX − tf,X and further in the R.H.S ofEquation (15) and equating it to zero, we obtain the maximum

SHAHZAD AND LIU: ACCURATE AND EFFICIENT PER-FLOW LATENCY MEASUREMENT 3485

likelihood estimate w̃f,X [j] of wf,X [j]. As(τξ

)(1n

)ξ(1− 1n

)τ−ξ

is P{Cf = Sf

X [j]|wf,X [j]}

, which is �= 0, we have{

ψ(0){ξ+1}−ψ(0){τ − ξ +1} + ln{1− 1

n

}− ln

{1n

}}

= 0

Simplifying the ln{.} terms results in Equation (14). �2) Statistical Simulations: We have obtained the m values

extracted using Theorem 4 from f ’s counter subvector atobservation point X . For flow f that passes observationpoint X , each unique permutation of the m distinct integersfrom 1 to m, denoted by vector Q, defines a unique deviationvector vf,X of size m/2 as follows. To ensure that m/2 is aninteger, we choose m to be an even number.

vf,X [l] = w̃f,X

[Q[2l]

]− w̃f,X

[Q[2l− 1]

]1 ≤ l ≤ m/2

(16)

Each unique permutation of the m distinct integers from 1 tom is essentially a unique simulation of the aforementionedstatistical process of multiplying half the time stamps withGz = +1 and the other half with Gz = −1. Theorem 5gives us the way to estimate

∑∀z(uf,R[z]− uf,S [z])2, which

is needed for calculating Var(Df ) based on Equation (12).Theorem 5: Given any two observation points S and R, for

any permutation Q of the m distinct integers from 1 to m,let vf,S and vf,R be the corresponding deviation vectorsof flow f at observation points S and R, respectively. Thefollowing equation holds:

∑

∀z

(uf,R[z] − uf,S [z]

)2 = E

[ m2∑

l=1

(vf,R[l] − vf,S [l])2]

(17)

Proof: Let YlS be the set of all the time stamps con-tributed by the packets of flow f to counters Sf

S [Q[2l]] andSf

S [Q[2l − 1]]. Similarly, let YlR be the set of all the timestamps contributed by the packets of flow f to countersSf

R[Q[2l]] and SfR[Q[2l − 1]]. Let ylS [i] be the i-th element

of YlS , where 1 ≤ i ≤ |YlS |. Similarly, let ylR[i] be thei-th element of YlR, where 1 ≤ i ≤ |YlR|. Starting from theR.H.S of Equation (17), we have:

=

m2∑

l=1

E[(vf,R[l] − vf,S [l])2

]

=

m2∑

l=1

E

[{ |YlR|∑

i=1

gi.ylR[i] −|YlS |∑

i=1

gi.ylS [i]}2]

=

m2∑

l=1

E

[{ |YlR|∑

i=1

(gi.ylR[i] − gi.ylS [i])}2]

∵ |YlR| = |YlS |

=

m2∑

l=1

|YlR|∑

i=1

(ylR[i] − ylS [i]

)2, using Equation (13)

=∑

∀z

(uf,R[z] − uf,S[z]

)2

The last equality follows from the fact that each ylX [i] isactually the value of some time stamp uf,X [z] at observationpoint X . �

For any permutation Q of the m distinct integers from1 to m, we calculate vf,R[l] and vf,S [l] for each 1 ≤ l ≤ m/2based on Equation (16), and then calculate

∑m2

l=1(vf,R[l] −vf,S [l])2, which is one estimate of

∑∀z

(uf,R[z] − uf,S[z]

)2

according to Theorem 5. As∑m

2l=1(vf,R[l] − vf,S [l])2 is a

random variable, its variance can be reduced by γ timesif we repeat the above process γ times using a differentrandom permutation Q each time and use the average ofthe γ values of

∑m2

l=1(vf,R[l] − vf,S [l])2 as the estimate of∑

∀z

(uf,R[z]−uf,S[z]

)2. As the R.H.S. of Equation (17) is an

expected value, to get an accurate estimate of∑

∀z

(uf,R[z]−

uf,S[z])2

, we need to calculate∑m

2l=1(vf,R[l]− vf,S[l])2 for a

statistically sufficient number of unique permutations of the mdistinct integers from 1 to m. The process of calculating∑m

2l=1(vf,R[l] − vf,S [l])2 for different permutations of Q is

essentially simulating the aforementioned random statisticalprocess of multiplying the time stamp of each packet withrandom variable Gz that takes the value of +1 and −1 withequal probability without having to perform this process inthe recording phase. We name this process of calculating∑m

2l=1(vf,R[l] − vf,S [l])2 using different permutations of Q

as virtual repetitions. The number of distinct ways in whichwe can repeat this process is

∏m2

i=1

(m−2(i−1)

2

), which is

large enough for us to obtain any required reliability α forestimating the standard deviation. For example, when m = 20,∏m

2i=1

(m−2(i−1)

2

)= 2.38 × 1015.

3) Steps of Estimating Standard Deviation: To summarize,COLATE performs the following six steps to estimate thestandard deviation of the latencies that the packets in flowf experienced in traversing from observation points S to R.(1) Obtain the number of packets in flow f , denoted by pf ,using NetFlow or the schemes proposed in [15] and [20]. (2)Obtain the estimates of tf,S and tf,R, which are the sumof the time stamps of all packets in flow f at observationpoints S and R respectively, using Theorem 2. (3) Extract thevalues of wf,S [j] and wf,R[j], which are the sum of the timestamps of flow f ’s packets that are mapped to counter Sf

S [j]at observation point S and to counter Sf

R[j] at observationpoint R, respectively, for all 1 ≤ j ≤ m, using Theorem 4.(4) Randomly choose γ permutations of the m distinct integersfrom 1 to m. For each permutation Q, first calculate vf,R[l]and vf,S [l] for all 1 ≤ l ≤ m/2 using Equation (16) and thencalculate

∑m2

l=1(vf,R[l]−vf,S[l])2. (5) Calculate the average ofthe γ values of

∑m2

l=1(vf,R[l]−vf,S[l])2, which is the estimatedvalue of

∑∀z

(uf,R[z]−uf,S[z]

)2. (6) Estimate of the standard

deviation using Equation (12).

C. Discussion

COLATE, as described above, makes two assumptions. Thefirst assumption is that there are no packet losses. To handlepacket losses in COLATE, we can easily adapt the strategyproposed in [14] for handling packet losses in the aggregatelatency measurement scheme LDA. According to this strategy,instead of maintaining a single counter vector, COLATE canmaintain a set of counter vectors at each observation point,where each counter vector has a sampling probability and a

3486 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 6, DECEMBER 2016

packet counter associated with it. The sum of the samplingprobabilities of all counter vectors is 1. The sampling prob-ability of a counter vector is the fraction of packets whosetime stamps COLATE will add to this counter vector. Thepacket counter of a counter vector keeps track of the numberof packets whose time stamps have been added to this countervector. In the recording phase, when a packet arrives at ordeparts from an observation point, COLATE first uses a hashfunction to map the packet to a counter vector such that inthe long run, the fraction of packets that it maps to anycounter vector is equal to its sampling probability. Then,COLATE adds the time stamp of the packet to this vector asdescribed earlier. Note that all observation points use the samehash function to guarantee that the same packet is mappedto the same counter vector at all observation points. In thequerying phase, for a target flow between two observationpoints, COLATE compares the packet counter of each countervector at one observation point with that of the correspondingcounter vector at the other observation point. Then, COLATEselects those two counter vectors at the two observation pointsthat have the highest sampling probability associated withthem and equal values of packet counters. After this, COLATEfollows the procedure of the querying phase described earlier.

The second assumption is that if a network supports pro-tocols such as multi-path TCP, which forward packets of thesame flow over different paths, then to measure the latency ofa flow f , COLATE should only be used on the observationpoints that are the source and destination of the flow f .To understand the reason behind this, consider a simpletriangular topology with three nodes A, B, and C, where A hastwo paths to C: A→C and A→B→C. Suppose node A has tosend a flow f to node C. If node A sends packets of thisflow f to node C via the two paths, A→C and A→B→C,then COLATE should only be used to measure latency of thisflow f between nodes A and C and not between nodes A and Bbecause not all the packets of f that leave A arrive at B.All such packets of f that were directly sent to C on the pathA→C behave as lost packets if COLATE is used to measurelatency of f between nodes A and B. To handle such settings,we will need to develop, adapt, and incorporate the potentialsolution proposed above when discussing the first assumption.

V. COLATE—RELIABILITY

COLATE has four parameters: (1) the total number of coun-ters denoted by n, (2) the number of counters in each countersubvector denoted by m, (3) the number of bits in each counterdenoted by b, and (4) the vector threshold denoted by T . Notethat when the sum of all n counters in a counter vector reachesT , COLATE dumps the counter vector into permanent storageas a counter epoch and then resets all counter values to be zero.In this section, we present solutions to find the values for theseparameters so that our estimated average latency achieves therequired reliability α ∈ [0, 1) for the given confidence intervalβ ∈ (0, 1]. Note that for standard deviation, we have alreadypresented a method in Section IV that can achieve arbitrarilyhigh required reliability. Recall μ̃f = 1

pf(t̃f,R − t̃f,S) (in

Equation (3)), which shows that the estimate μ̃f depends ontwo other estimates t̃f,S and t̃f,R. Next, we find the confidence

interval B and required reliability A that the estimate t̃f,X

for tf,X at each observation point X must satisfy so that theestimate μ̃f for μf satisfies the confidence interval β andrequired reliability α. That is, we want to find the valuesof B and A so that if for every observation point X wehave P

{|t̃f,X − tf,X | ≤ Btf,X

}≥ A, then we will have

P {|μ̃f − μf | ≤ βμf} ≥ α. After we find the values for Band A, we present a solution to calculate the optimal valuesof the four parameters n, m, b, and T .

A. Individual Reliability Requirements

Individual Required Reliability: The maximum fractionof estimated values μ̃f that can violate the requirement of|μ̃f − μf | ≤ βμf , while the overall estimate still satisfies therequired reliability α, is 1 − α. Thus, the maximum fractionof estimates t̃f,X at either observation points of S and R thatcan violate the requirement |t̃f,X − tf,X | ≤ Btf,X must be nogreater than (1 − α)/2. Thus,

A = 1 − (1 − α)/2 = (1 + α)/2 (18)

Individual Confidence Interval: The estimate μ̃f obtained byCOLATE needs to satisfy the requirement of |μ̃f −μf | ≤ βμf

with probability of at least α. As μ̃f = 1pf

(t̃f,R − t̃f,S) and

μf = 1pf

(tf,R − tf,S), the confidence interval requirement|μ̃f − μf | ≤ βμf becomes:∣∣(t̃f,R − tf,R) − (t̃f,S − tf,S)

∣∣

=∣∣(t̃f,R − t̃f,S) − (tf,R − tf,S)

∣∣ ≤ β(tf,R − tf,S)

The largest value of∣∣(t̃f,R − tf,R)− (t̃f,S − tf,S)

∣∣ is Btf,R +

Btf,S , which must be no greater than β(tf,R − tf,S).

Btf,R +Btf,S ≤ β(tf,R − tf,S)

Thus, we get

B ≤ β

(tf,R − tf,S

tf,R + tf,S

)(19)

To determine B for a given network, we conduct measurementof (tf,R − tf,S)/(tf,R + tf,S) on the network to find theappropriate value so that Equation (19) statistically holds.

B. Reliability Centered Parameter Selection

As we have four unknown parameters (i.e., n, m, b, andT ), we need at least four equations so that we can calculatethe values of these parameters by solving the four equations.Next we develop these four equations.

Equation 1 : Let M be the total number of bits of the RAMthat an observation point can allocate for storing the countervector, which requires n× b bits. Thus,

n× b = M (20)

Equation 2 : Based on Lemma 1, the sum of all the timestamps of all packets of all flows is equally divided amongall n counters on average. Thus, the value of each counter onaverage can go up to T/n. Thus, the number of bits in eachcounter, which is b, needs to satisfy the following equation.

b = log2

{T

n

}+ 1 (21)

SHAHZAD AND LIU: ACCURATE AND EFFICIENT PER-FLOW LATENCY MEASUREMENT 3487

Note that we add 1 in the R.H.S of this equation to doublethe capacity of each counter to avoid overflows.

Equation 3 : As the expected value of a counter in a countersubvector, which is specified in Equation (1), should neverexceed the maximum capacity of the counter, we have

tf,X

m+T − tf,X

n≤ 2b − 1

Let tmaxf,X be the maximum value of tf,X for all flows on a

network. Thus, the value of b should satisfy the followingequation:

tmaxf,X

m+T − tmax

f,X

n= 2b − 1 (22)

Here tmaxf,X can be obtained by some measurement on the sum

of the time stamps of all packets on a per-flow basis on thegiven network. The three equations derived here are used forboth EBE and MLE. Next, we derive the fourth equation forEBE. For MLE, we only present the fourth equation and skipits derivation due to limitation of space.

Equation 4 for EBE: To achieve the required reliability,P{|t̃f,X − tf,X | ≤ Btf,X

}should at least be equal to its

lower bound A, i.e.,

P{(1 −B)tf,X ≤ t̃f,X ≤ (1 +B)tf,X

}= A (23)

Based on Equation (1), we can represent E[C] as a function oftf,X ; denoting this function by g, we have E[C] = g{tf,X}.Thus, tf,X = g−1{E[C]}. Let C̃ be the observed value ofE[C]. Then, we have t̃f,X = g−1{C̃}. Equation (23) becomes

A = P{(1 −B)tf,X ≤ g−1{C̃} ≤ (1 +B)tf,X

}

= P{g {(1 −B)tf,X} ≤ C̃ ≤ g {(1 +B)tf,X}

}(24)

Similarly, based on Equation (2), we can represent standarddeviation of C as a function of tf,X ; denoting this functionby h, we have Var(C) = h2{tf,X}. Based on the fact thatthe variance of a random variable reduces by m times if therandom event is repeated m times, by observing the valuesof C from m counters, the variance of C becomes h2{tf,X}

m

and the standard deviation of C becomes h{tf,X}√m

. Let Z

denote C̃−g{tf,X}h{tf,X}/

√m

. Thus, Equation (24) becomes

P{g {(1 −B)tf,X} − g {tf,X}

h {tf,X} /√m

≤ Z ≤ g {(1 +B)tf,X} − g {tf,X}h {tf,X} /

√m

}= A (25)

By the central limit theorem, Z approximates a standardnormal random variable. The area under the standard normalcurve gives the success probability, which is the required reli-ability. As our confidence interval requirement is symmetricon both the upper and lower sides of tf,X , we can representthe required reliability A in terms of a constant k as follows:

P {−k ≤ Z ≤ k} = A (26)

Let Φ be the cumulative distribution function (CDF) of astandard normal distribution and erf {.} be the standard error

function, we get

P {−k ≤ Z ≤ k} = Φ(k) − Φ(−k) = erf

{k√2

}(27)

From Equations (26) and (27), we get

k =√

2 erf−1 {A}

We observe that the absolute values of the upper and lowerbounds of Z in Equation (25) are the same. Thus, equatingthe lower bound with −k or the upper bound with k resultsin the following equation:

B2t2f,X =k2n2m

n−m

{( tf,X

m

)(1− 1

m

)+(T − tf,X

n

)(1− 1

n

)}

(28)

By rearranging Equation (28), we get.

B2 =1tf,X

[k2n2m

n−m

{( 1m

)(1 − 1

m

)−( 1n

)(1 − 1

n

)}]

+1t2f,X

[k2n2m

n−m

( 1n

)(1 − 1

n

)T

]

This equation shows that B is inversely proportional to tf,X

when other parameters are fixed. This makes intuitive sensebecause the more packets in flow f in passing observationpoint X (i.e., the larger tf,X is), the more timing informationwe can obtain from the packets, and the smaller confidenceinterval can be achieved. Thus, we should use the statisticallyminimum observable value of tf,X , denoted tmin

f,X , for the givennetwork, in Equation (28). Here tmin

f,X can be obtained by somemeasurement on the sum of the time stamps of all packets ona per-flow basis on the given network. The parameter valuesobtained using tmin

f,X in Equation (28) will ensure that theestimates for all flows whose sum of time stamps are ≥ tmin

f,X

satisfy P{|t̃f,X − tf,X | ≤ Btf,X

}≥ A. By replacing tf,X

by tminf,X in Equation (28), we get

B2tminf,X

2

=k2n2m

n−m

{( tminf,X

m

)(1− 1

m

)+(T − tmin

f,X

n

)(1− 1

n

)}(29)

Equation 4 for MLE: As stated earlier, we skip the deriva-tion of the fourth equation for MLE due to space limitation.The final expression for the fourth equation for MLE is:

2

((kmn2 × Var(C))(m− n)(Btmin

f,X )

)2

= m2{1 − 2n+ n2(1 + 2Var(C))

}

+n2 + 2mn {1 − n} (30)

Solving Equations: COLATE takes M , α, β, tminf,X , and tmax

f,X

as input. The values of required reliability α and confidenceinterval β are provided by network operators. The value ofRAM space M depends on the amount of RAM available atan observation point. For tmin

f,X and tmaxf,X , network operators

can obtain them by measurements on targeted flows in thegiven network. With the values of M , α, β, tmin

f,X , and tmaxf,X ,

COLATE simultaneously solves the four equations (i.e., (20),(21), (22), and (29)) (or (30)) to obtain the appropriate values

3488 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 6, DECEMBER 2016

Fig. 3. Permanent storage vs. RAM.

Fig. 4. Threshold T vs. RAM size.

of the four parameters n, m, b, and T . To simultaneously solvethese equations, we express m, b, and T in terms of n usingEquations (20), (21), and (22) and replace them in Equation(29) (or (30)). This results in an expression with only oneunknown parameter n. We numerically solve this expressionto obtain n and then the other three unknown parameters.

RAM Space vs. Storage Space: From the simultaneous solu-tion of these four equations, we have an interesting observationthat COLATE requires smaller amount of permanent storagespace for storing counter epochs when it is allocated with moreRAM for storing counter vectors. Figure 3 plots an examplegraph of the permanent storage size vs. RAM size.

While this observation seems surprising, it makes intu-itive sense because the sum of the two maximum values oftwo b-bit numbers is 2 × 2b = 2b+1 whereas the maximumvalue of a 2b-bit number is 22b � 2b+1. As we increase thetotal number of bits in a counter vector, i.e., M , the countervalue threshold T increases as shown in Figure 4. This impliesthat the frequency of writing the counter vector into permanentstorage is reduced, and although each counter epoch takesmore space, the overall required storage space is reduced asshown in Figure 3. The M value at the knee of the curvein Figure 3 represents the best tradeoff point between RAMspace and permanent storage space.

Flexibility in Parameter Selection: If we only measureaverage per-flow latency, each observation point can chooseits own values for the parameters of n, m, b, and T basedon its available resources and traffic condition without globalcoordination among observation points. If we also need tomeasure standard deviation, then all observation points need touse the same value of m because Theorem 5 requires that thetwo sets YlS and YlR contain the time stamps from the sameset of packets at the two observation points, which is possibleonly when the value of m is the same at both points. Theremaining three parameters can still be chosen independentlyat each observation point.

Fig. 5. CDFs of flow sizes.

VI. PERFORMANCE EVALUATION

We implemented COLATE in Matlab. We also implementedRLI [16] in Matlab for comparison purposes. We did notimplement LDA [14] and MAPLE [17] because LDA can notprovide per-flow latency measurement and MAPLE requiresattaching time stamps to every packet, i.e., MAPLE is nota latency estimation scheme but a storage scheme. In thissection, we present our evaluation results of COLATE incomparison with RLI. We first give details of the three networktraces that we used. Second, we evaluate the accuracy ofCOLATE as well as the impact of RAM space on permanentstorage space used by COLATE. Last, we compare COLATEwith RLI and with Count-Min (CM) sketch [9], a summarizingdata structure for queries on data streams.

A. Network Traces

To evaluate COLATE, we need real packet traces withhigh-resolution time stamps collected simultaneously from atleast two observation points. Unfortunately, no such traces arepublicly available. Thus, we resort to three real network traceswhere each is collected at a single observation point at a time.These traces include CHIC [2], ICSI [25], and DC [6]. CHIC isa backbone header trace, published by CAIDA, which includesthe arrival times of packets at a 10GigE link interface. We usedtraces generated from 5 minutes of packet capture. Note thatthe authors of RLI and MAPLE also evaluated their schemeson the header trace of this backbone link collected by CAIDA.ICSI is an enterprise network traffic trace, collected at amedium-sized site, which includes the arrival times of packetson an ethernet link for a duration of over 41.1 hours. ICSI isavailable in the form of 41 trace files collected at 17 differentports in an enterprise network. DC is a data center traffic trace,collected at a university data center, which includes the arrivaltimes of packets on an ethernet link for a duration of a littlemore than an hour. DC is available in the form of 20 trace filescollected at the same port. Figure 5 shows the CDFs of sizesof flows in each trace. We observe that the traces contain bothmice flows as well as elephant flows. Table II reports the totalduration, number of packets, number of flows, and averagedata rate of each trace.

As these network traces contain only the arrival time stampsof packets, we adopt the simulation strategy used by RLIand MAPLE, which simulates the traversal of packets in eachtrace through a queue to get a departure time stamp for eachpacket and uses random early detection (RED) [11] as thequeue management strategy because RED is most popular.As modern routers typically use a queue size that can hold0.5 seconds of traffic at their maximum line rates, we also use

SHAHZAD AND LIU: ACCURATE AND EFFICIENT PER-FLOW LATENCY MEASUREMENT 3489

TABLE II

SUMMARY OF NETWORK TRACES

Fig. 6. CDF of observed β in average estimate using EBE (1S, 1R).(a) α = 0.99, β = 0.01. (b) α = 0.90, β = 0.10.

the same queue size. For the remaining parameters of REDqueue management strategy, we use minth = 0.475×queuesize, maxth = 0.95×queue size, wq = 0.002, and maxp = 1

50as per the guidelines in [11].

B. COLATE Accuracy

Now we evaluate the accuracy of the average and standarddeviation of the flows estimated by COLATE.

1) Average Latency Using Expectation Based Estimator:We evaluated COLATE for both the scenario of only twoobservation points (where one sender sends and one receiverreceives) and that of more than two observation points (wheremultiple senders send and multiple receivers receive). For thescenario with more than two observation points, we choosethree observation points forming a triangle topology whereeveryone sends to and receives from everyone else. We choosethree observation points and the triangle topology for the sakeof simplicity as the number of senders and receivers does notaffect the accuracy of COLATE. For a triangle topology, thereare 6 unidirectional links. We used the largest 6 of the 41 tracefiles from ICSI data set, each trace file representing the trafficon one of the 6 links. This choice is arbitrary.

We performed our evaluation for two different accuracyrequirements: low (α = 0.90, β = 0.10) and high (α = 0.99,β = 0.01). For each of these three accuracy requirements,we evaluated COLATE using three values of available RAM:small (M = 1MB), medium (M = 10MB), and large(M = 100MB). We obtained the values of tmin

f,X and tmaxf,X

from simple measurement of the network traces. We calculatedthe values of the parameters b, m, n, and T using the methoddescribed in Section V. For example, for M = 1MB, α =0.95, and β = 0.05, typical values of these parameters areb = 19, m = 20, n = 455334, and T = 8 × 1010.

Our results show that COLATE always achieves the requiredreliability using EBE. Figures 6(a) and 6(b) show the CDFs ofthe observed values of β, i.e., |μ̃f − μf |/μf , for the averagelatency estimated by COLATE for the three traces under thescenario of one sender and one receiver using the low andhigh accuracy requirements, respectively. Figures 7(a) and 7(b)show the CDFs of the observed values of β, i.e., |μ̃f −μf |/μf ,

Fig. 7. CDF of observed β in average estimate using EBE (multiple S, R).(a) α = 0.99, β = 0.01. (b) α = 0.90, β = 0.10.

Fig. 8. CDF of observed β in average estimate using MLE (1S, 1R).(a) α = 0.99, β = 0.01. (b) α = 0.90, β = 0.10.

Fig. 9. CDFs of relative errors in standard deviation.

for the average latency estimated by COLATE for the six linksunder the scenario of multiple senders and receivers usingthe low and high accuracy requirements, respectively. Thehorizontal line in each of these figures shows the requiredreliability. We see that every plot of CDF always crosses thehorizontal line for an observed value of β that is smaller thanthe required confidence interval. This shows that COLATEalways achieves the required reliability. Due to lack of space,we only show plots for M = 10MB. Observations fromM = 1MB and M = 100MB are the same.

2) Average Latency Using Maximum Likelihood BasedEstimator: Our results show that COLATE always achievesthe required reliability using MLE. Figures 8(a) and 8(b) showthe CDFs of the observed values of β, i.e., |μ̃f − μf |/μf ,for the average latency estimated by COLATE for the threetraces under the scenario of one sender and one receiver usingthe low and high accuracy requirements, respectively. We seethat every plot of CDF always crosses the horizontal linefor an observed value of β that is smaller than the requiredconfidence interval. This shows that COLATE always achievesthe required reliability. Due to lack of space, we only showplots for M = 10MB. Observations from M = 1MB andM = 100MB are the same.

3490 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 6, DECEMBER 2016

Fig. 10. Relative error in standard deviation vs. number of repetitions.

Estimates obtained using MLE are more accurate comparedto the estimates obtained using EBE. By visually comparingFigure 6(a) with 8(a) and Figure 6(b) with 8(b), we observethat the CDF plots in Figures 8(a) and 8(b) cross the horizontallines at lower value of observed β compared to the CDF plotsin Figures 6(a) and 6(b). This shows that the estimates obtainedusing MLE are more accurate compared to the estimatesobtained using EBE.

3) Standard Deviation: Our results show that the rel-ative error in the standard deviation estimates of over91% flows is less than 0.05 with only 1000 virtual rep-etitions. Relative error is defined as (actual value −estimated value)/actual value. Figure 9 plots theCDFs of the relative errors in standard deviation estimated byCOLATE for each of the three traces.

Our results also show that the percentage of flows, forwhich the relative error is less than 0.05, increases with theincrease in the number of virtual repetitions. Figure 10 plotsthis percentage versus the number of virtual repetitions for thethree traces. With 105 iterations, this percentage reaches 98%.Although 105 iterations may take some time depending on theavailable computing power, it is not an issue as the estimationof standard deviation is an offline process and does not haveto keep up with high line rates.

C. RAM and Storage Size

Our results show that COLATE uses less than 0.1 bit ofpermanent storage per packet. Figure 11 shows the bar graphof the number of bits per packet used by each of the threetraces for all three values of M , when α = 0.99 and β = 0.01.This small size of storage required per packet results in a verylow frequency of transferring the counter vectors from RAMto permanent storage. For example, for ICSI network trace,COLATE transfers a counter vector to SSD every 24.6 hourswhen M = 1MB, α = 0.95, and β = 0.05. Transferring 1MBof content from RAM to SSD once a day is trivial for moderndevices. The number of bits per packet in permanent storagedecreases with the increase in RAM size M , which confirmsour analysis based on Figures 3 and 4. However, this decreaseis hard to observe from Figure 11 because the difference issmall. Nevertheless, this difference becomes significant forlonger time durations (on the order of say days and weeks).

D. Comparison With Prior Art

Our results show that COLATE always achieves higheraccuracy than RLI. RLI requires two inputs, namely the lowerand upper limits of the Probe packet Injection Rate (PIR).The authors of RLI used the lower limit as 1 probe packet

Fig. 11. Storage bits per packet.

Fig. 12. Delay estimates comparison.

TABLE III

AVG. # OF REG. PACKETS AFTER WHICH RLI INSERTS A PROBE PACKET

per 300 regular packets and the upper limit as 1 probepacket per 10 regular packets in [16]. We first evaluatedRLI using this pair of PIR values. Because the accuracy ofRLI increases as PIR increases, to improve the estimationaccuracy of RLI, although at the cost of larger bandwidthusage, we also evaluated RLI using much higher PIRs – 1probe packet per 10 regular packets as the lower limit and1 probe packet per 2 regular packets as the upper limit.Figure 12 plots the CDFs of the observed value of β in theaverage latency estimated by RLI in the three traces for thesetwo configurations of PIRs. For comparison, we also plot theCDF of the observed value of β in the estimates obtained usingCOLATE for high accuracy requirement (α = 0.99, β = 0.01).Note that the observed value of β is essentially the relativeerror in the estimated values of average latency. Figure 12shows that the relative error of COLATE is much smaller thanthat of RLI. This relative error can be made arbitrarily smallby specifying smaller values of β and larger values of α. Thisfigure further shows that the relative error of RLI is smallerwhen PIR is larger. With the PIR proposed by the authors,on average, only 77% flows have relative error less than 5%.At this rate, on average, RLI inserts one probe packet after21.66 regular packets. With our high PIR configuration, onaverage, only 81% flows have a relative error less than 5%,but at this rate, on average, RLI inserts one probe packet every4.78 regular packets in the three traces. Table III shows theaverage number of regular packets after which RLI insertsa probe packet for each trace. In contrast, COLATE doesnot insert any probe packet at the cost of small memory atobservation points.

SHAHZAD AND LIU: ACCURATE AND EFFICIENT PER-FLOW LATENCY MEASUREMENT 3491

Count-Min (CM) sketches can theoretically be used forlatency measurement but practically, there is a fundamentallimitation. Let tfi represent the sum of time stamps of allpackets in flow fi and let there be j flows in total whose timestamps need to be stored for latency measurements. We can usea CM-sketch to store time stamps of multiple flows and obtainthe estimate t̃fi of the sum of time stamps of any flow fi as perthe method described in [9]. The estimate t̃fi obtained throughCM-sketch can satisfy the condition t̃fi ≤ tfi +B ×

∑∀j tfj

with probability α. This is problematic because we requirethe estimate to satisfy the condition t̃fi ≤ tfi + B × tfi withprobability α. Therefore, to achieve the required reliabilityusing CM-sketch, we need to ensure that

∑∀j tfj ≤ tfi ,

which is possible if and only if we do not add time stampsof any flow other than fi to the CM-sketch. Consequently weneed to maintain a CM-sketch for each flow, which results inlarge memory requirements. For example, for α = 0.95 andβ = 0.05, CM-sketch requires 165 counters per flow and 3hash functions and 3 memory accesses per packet. Assumingthe same counter size of 19 bits as for COLATE in thisscenario, CM-sketch requires 165 × 19 = 3135 bits per flow.In comparison, COLATE requires 0.1 bit per packet. The mem-ory requirement of CM-sketch matches that of COLATE onlyif each flow has at least 31350 packets, which is impractical asseen in Figure 5. The number of memory accesses and hashfunctions per packet for CM-sktech are always greater thanthose for COLATE.

VII. CONCLUSION

The key contribution of this paper is in proposing anaccurate and efficient per-flow latency measurement schemewithout packet probing and time stamping. The key noveltyof this paper is that we purposely allow noise to be introducedin recording packet timing information for minimizing storagespace and use statistical techniques to denoise the recordedinformation to obtain accurate latency estimates when latencyof a target flow is queried. The key technical depth of thispaper is in the mathematical development of the estimationtheory that our scheme is based upon. Our theoretical analysisand experimental results show that our scheme always achievesthe required reliability. Our scheme has a much smallerprocessing overhead in terms of number of hash computationsand memory updates compared to existing schemes, whichfurther require sending probe packets or attaching time stampsto every packet. Our scheme is scalable in that the amount ofmemory required at each observation point is only dependenton the number of packets and not on the number of sendingand receiving observation points.

REFERENCES

[1] CAIDA Network Monitors, accessed on Jul. 1, 2013. [Online]. Available:http://www.caida.org/data/realtime/passive/.

[2] The CAIDA UCSD Anonymized 2011 Internet Traces, accessed onJul. 1, 2013. [Online]. Available: http://www.caida.org/data/passive/passive_2011_dataset.xml.

[3] Corvil Claims to Minimize Network Latency, 2007. [Online]. Available:http://www.pcworld.idg.com.au/article/196828/corvil_claims_minimize_network_latency/.

[4] N. Alon, Y. Matias, and M. Szegedy, “The space complexity of approxi-mating the frequency moments,” in Proc. ACM STOC, 1996, pp. 20–29.

[5] S. K. Barker and P. Shenoy, “Empirical evaluation of latency-sensitiveapplication performance in the cloud,” in Proc. ACM SIGMM Conf.Multimedia Syst., 2010, pp. 35–46.

[6] T. Benson, A. Akella, and D. A. Maltz, “Network traffic characteristicsof data centers in the wild,” in Proc. IMC, 2010, pp. 267–280.

[7] Y. Cai, F. Yu, C. Liang, B. Sun, and Q. Yan, “Software defined device-to-device (D2D) communications in virtual wireless networks withimperfect network state information (NSI),” IEEE Trans. Veh. Technol.,still not published.

[8] Y. Chen, D. Bindel, H. Song, and R. H. Katz, “An algebraic approachto practical and scalable overlay network monitoring,” in Proc. ACMSIGCOMM, 2004, pp. 55–66.

[9] G. Cormode and S. Muthukrishnan, “An improved data stream summary:The count-min sketch and its applications,” J. Algorithms, vol. 55, no. 1,pp. 58–75, Apr. 2005.

[10] N. Duffield, “Simple network performance tomography,” in Proc. ACMIMC, 2003, pp. 210–215.

[11] S. Floyd and V. Jacobson, “Random early detection gateways forcongestion avoidance,” IEEE/ACM Trans. Netw., vol. 1, no. 4,pp. 397–413, Aug. 1993.

[12] P. Guo, J. Wang, X. H. Geng, C. S. Kim, and J.-U. Kim, “A variablethreshold-value authentication architecture for wireless mesh networks,”J. Internet Technol., vol. 15, no. 6, pp. 929–936, 2014.

[13] N. Hua, E. Norige, S. Kumar, and B. Lynch, “Non-crypto hardware hashfunctions for high performance networking ASICs,” in Proc. ACM/IEEEANCS, Oct. 2011, pp. 156–166.

[14] R. R. Kompella, K. Levchenko, A. C. Snoeren, and G. Varghese, “Everymicrosecond counts: Tracking fine-grain latencies with a lossy differenceaggregator,” in Proc. ACM SIGCOMM, 2009, pp. 255–266.

[15] A. Kumar, J. Xu, J. Wang, O. Spatschek, and L. Li, “Space-codebloom filter for efficient per-flow traffic measurement,” in Proc. IEEEINFOCOM, Mar. 2004, pp. 1762–1773.

[16] M. Lee, N. Duffield, and R. R. Kompella, “Not all microsecondsare equal: Fine-grained per-flow measurements with reference latencyinterpolation,” in Proc. ACM SIGCOMM, 2010, pp. 27–38.

[17] M. Lee, N. Duffield, and R. R. Kompella, “MAPLE: A scalablearchitecture for maintaining packet latency measurements,” in Proc.IMC, 2012, pp. 101–114.

[18] M. Lee, S. Goldberg, R. R. Kompella, and G. Varghese, “Fine-grainedlatency and loss measurements in the presence of reordering,” in Proc.ACM SIGMETRICS, 2011, pp. 289–300.

[19] H. Li, K. Wu, Q. Zhang, and L. M. Ni, “CUTS: Improving channelutilization in both time and spatial domain in WLANs,” IEEE Trans.Parallel Distrib. Syst., vol. 25, no. 6, pp. 1413–1423, Jun. 2014.

[20] Y. Lu, A. Montanari, B. Prabhakar, S. Dharmapurikar, and A. Kabbani,“Counter braids: A novel counter architecture for per-flow measure-ment,” in Proc. ACM SIGMETRICS, 2008, pp. 121–132.

[21] B. Lynch and S. Kumar, “Smart memory for high performance networkpacket forwarding,” in Proc. Hot Chips Symp., 2010.

[22] R. Martin, “Wall Street’s quest to process data at thespeed of light,” Inf. Week, vol. 4, no. 21, 2007. [Online]:http://www.informationweek.com/wall-streets-quest-to-process-data-at-the-speed-of-light/d/d-id/1054287?

[23] M. J. Miller, “Bandwidth engine serial memory chip breaks 2 billionaccesses/sec,” in Proc. Hot Chips Symp., 2011, pp. 1–23.

[24] Z. Pan, Y. Zhang, and S. Kwong, “Efficient motion and disparityestimation optimization for low complexity multiview video coding,”IEEE Trans. Broadcast., vol. 61, no. 2, pp. 166–176, Jun. 2015.

[25] R. Pang et al., “A first look at modern enterprise traffic,” in Proc. IMC,2005, pp. 15–28.

[26] M. V. Ramakrishna, E. Fu, and E. Bahcekapili, “Efficient hardware hash-ing functions for high performance computers,” IEEE Trans. Comput.,vol. 46, no. 12, pp. 1378–1381, Dec. 1997.

[27] Z. Xia, X. Wang, X. Sun, and Q. Wang, “A secure and dynamic multi-keyword ranked search scheme over encrypted cloud data,” IEEE Trans.Parallel Distrib. Syst., vol. 27, no. 2, pp. 340–352, Feb. 2016.

[28] S. Xie and Y. Wang, “Construction of tree network with limiteddelivery latency in homogeneous wireless sensor networks,” WirelessPers. Commun., vol. 78, no. 1, pp. 231–246, 2014.

[29] Q. Yan and F. Yu, “Distributed denial of service attacks in software-defined networking with cloud computing,” IEEE Commun. Mag.,vol. 53, no. 4, pp. 52–59, Apr. 2015.

[30] Y. Zhao, Y. Chen, and D. Bindel, “Towards unbiased end-to-end networkdiagnosis,” in Proc. ACM SIGCOMM, 2006, pp. 219–230.

[31] Y. Zhou, T. Z. J. Fu, and D. M. Chiu, “A unifying model and analysis ofP2P VoD replication and scheduling,” IEEE/ACM Trans. Netw., vol. 23,no. 4, pp. 1163–1175, Aug. 2015.

3492 IEEE/ACM TRANSACTIONS ON NETWORKING, VOL. 24, NO. 6, DECEMBER 2016

Muhammad Shahzad received the Ph.D. degreein computer science from Michigan State Uni-versity in 2015. He is currently an AssistantProfessor with the Department of Computer Science,North Carolina State University, USA. His researchinterests include design, analysis, measurement,and modeling of networking and security systems.He received the 2015 Outstanding Graduate StudentAward, the 2015 Fitch Beach Award, and the 2012Outstanding Student Leader Award at MichiganState University.

Alex X. Liu received the Ph.D. degree in computerscience from The University of Texas at Austinin 2006. His research interests focus on networkingand security. He received best paper awards fromICNP-2012, SRDS-2012, and LISA-2010.He received the IEEE and IFIP William C. CarterAward in 2004, the National Science FoundationCAREER Award in 2009, and the Michigan StateUniversity Withrow Distinguished Scholar Award in2011. He is an Associate Editor of the IEEE/ACMTRANSACTIONS ON NETWORKING, an Editor of

the IEEE TRANSACTIONS ON DEPENDABLE AND SECURE COMPUTING,and an Area Editor of Computer Communications.