1

Traffic Engineering of High-Rate Large-sized Flows

Acknowledgment: UVA work is supported by DOE ASCR grants DE-SC002350 and DE-SC0007341, and NSF grants, OCI-1038058, OCI-1127340, and CNS-1116081, and ESnet work is supported by DOE grant DE-AC02-05CH11231

Tian Jin, Chris Tracy, Malathi Veeraraghavan, Zhenzhen Yan University of Virginia and ESnet

mvee@virginia.edu, ctracy@es.netJuly 8-11, 2013

Outline

• Problem statement & Motivation– Example of ESnet measured load– Adverse effects of “alpha flows”

• Hybrid Network Traffic Engineering System (HNTES)

• HNTES evaluation– NetFlow data collection– Effectiveness– Afflicted-flow packet percentage

2

Problem statement

• Flows generated by high-rate large-sized file transfers are called alpha flows– thresholds used in this paper: 1 GB in 1 min

• Previous work shows that alpha flows– are the cause of burstiness of IP traffic

• Experiment shows adverse effects of alpha flows on real-time A/V flows

• Problem: How can a provider identify such alpha flows within their network and direct them to separate QoS-controlled VCs?

3

Motivation: ESnet4 Core network for US Dept. of Energy Labs

StarLight

MAN LAN(32 A of A)

PNNL

FNL

ORNL

LLNL

GA

BNL

LANL

IP router

Lab

Optical node

SDN router Lab Link

MANNLR 10G

30/40/50G SDNIP

50

50

50

5040

3030

30

40

50

30

5050

40

40

4040

40

40

40

4040

40

Steve Cotter, Chin Guok, Joe Metzger, Bill Johnston

Brookhaven National Laboratory

Traffic surges on ESnet interface

5

Link rate: 10 Gbps

Outgoing traffic

Incomingtraffic

9 Gbps

Jan. 12, 2013

Motivation: Adverse effects of alpha flows

• Used DOE 100G testbed• Hosts: high-performance diskpts

6BNL

NEWY

ping flow(delay-sensitive)

TCP (alpha) flow

UDP flow(background)

buffer buildups

Impact of alpha flows on real-time flows

7

• Impact on ping flow delay – significant in 1-

queue configuration

– negligible in 2-queue configuration

• Need separate virtual queue for alpha flow packets

Pings: 1 per secDelay: 60 ms in 1-queue case

Delay: 2.1 ms in 2-queue case

UDP flow

TCP flow

3 Gbps

6 Gbps

Outline

• Problem statement & MotivationHybrid Network Traffic Engineering

System (HNTES)• HNTES evaluation

– NetFlow data collection– Effectiveness– Afflicted-flow packet percentage

8

Hybrid network traffic engineering system (HNTES)

- Intradomain identification/redirection of alpha flows

9

•Three steps– Analysis of NetFlow

reports from ingress routers to identify address prefixes of completed alpha flows

– IDC creates L3 circuits between ingress-egress router pairs and configures QoS

– IDC sets firewall filters to direct future alpha flows with matching address prefixes to L3 circuits

Aging parameter (A): age out rules corresponding to prefixes for which no alpha flows have been observed

Outline

• Problem statement & Motivation• Hybrid Network Traffic Engineering

System (HNTES)• HNTES evaluation

– NetFlow data collection– Effectiveness– Afflicted-flow packet percentage

10

Data collection for HNTES evaluation: NetFlow data from 4 routers were collected for 7 months (214

days)

11

router-1 & router-2: provider-edge (PE) routersrouter-3: core router (REN peering)router-4: core router (commercial peering)

OP: observation point

Effectiveness Analysis

• Two types of effectiveness– Cumulative effectiveness (Ci): percent of

alpha bytes (bytes reported in alpha NetFlow reports) that would have been redirected in period (1,i)

– Daily effectiveness (Ei): percent of alpha bytes that would have been redirected on day i

• Choose aging parameter for: – High effectiveness– Stability in firewall-filter size

12

Aging parameter: tradeoff effectiveness with size of firewall filter

• graphs for router 1 (similar for other routers)• 30 days is good compromise for aging parameter

13

Firewall filter size stable with aging parameter 30 Cumulative effectiveness > 90%

Cumulative effectiveness (/24)

14

Provider edge routers(single customers) Peering routers

(router-3: REN;router-4: commercial) Why is cumulative

effectivness lower for peeringrouters, esp. router-4?

Boxplots for 214 values each router-1 omitted as it is similar to router-2

Cum

ula

tive e

ffect

iveness

Effectiveness comparisons

15

• Obs. 1: higher effectiveness for /24 than for /32• Obs. 2: higher effectiveness for router-1 and router-2 than

for router-3 and router-4• Obs. 3: fewer alpha prefix IDs for router-3 and router-4

Explanations

16

• Obs. 1: data-transfer node clusters are typically located in the same /24 subnet; thus, repetition is greater with /24 than /32

• Obs. 2 and obs. 3: • Higher effectiveness for routers 1 & 2:

downloads from supercomputing facilities are repetitive (a scientist accesses the same data transfer nodes)

• Lower effectiveness for routers 3 & 4:• fewer uploads to DoE labs than

downloads from DOE labs• expect few, if any, scientific data

transfers from commerical peers (router-4)

Outline

• Problem statement & Motivation• Hybrid Network Traffic Engineering

System (HNTES)• HNTES evaluation

– NetFlow data collection– Effectiveness– Afflicted-flow packet percentage

17

Afflicted-flow packets

• B: set of non-alpha NetFlow reports for flows that share alpha prefix IDs

• Divide B into four subsets in sequence– C: non-alpha reports of alpha flows– D B-C: data-transfer reports (heuristic)– W B-C-D: well-known ports– L: leftover = B-C-D-W

• Afflicted flows: W+L

18

Afflicted-flow packets

• Tradeoff: /24 vs /32– /32 has lower effectiveness: large % of afflicted-flow packets

will be impacted when an alpha flow is not redirected– /24 has higher afflicted-flow packet percentage: small % of

afflicted-flow packets are adversely impacted

• Recommend /24 address prefixes for firewall filters

19

Percentage of afflicted-flow packets in samples of beta-flow (non-alpha flow) packets; across the 214-day period

Conclusions

• Hypothesis: Most high-speed data transfer nodes have static IP addresses, and alpha flows are created repeatedly between the same source-destination subnets– Validated for flows generated by dataset downloads as

observed at edge routers

• HNTES solution of determining src-dest address prefixes of completed alpha flows & using these prefixes to set firewall filters for future alpha-flow redirection is effective for downloads from DOE labs

• Less effective for uploads esp. from commercial peering links – But alpha-flow causing uploads are fewer

20