LHCONE Research Data Flows - L2 vs. L3

Anita Nikolich – anikolich@uchicago.eduBrent O’Keeffe – bokeeffe@uchicago.edu

1

Internet2 Fall 2011 Member Meeting

Agenda• Current LHC model • UChicago case study• LHCONE Overview• Layer 2 at the Core• Layer 3 at the Edge• Load Distribution across multiple paths• Technical Solution to Address Design Concerns

LHC Experiments• ATLAS - one of two general-purpose detectors at the

LHC. It will investigate a wide range of physics, including the search for the Higgs boson, extra dimensions, and particles that could make up dark matter. ATLAS will record sets of measurements on the particles created in collisions - their paths, energies, and their identities. 3000 scientists from 174 institutes in 38 countries.

• CMS - Same scientific goals as the ATLAS experiment, but it uses different technical solutions. More than 3000 scientists collaborate in CMS, coming from 183 institutes in 38 countries

3

original Site roles in LHC Computing

MONARC• Tier 0 - CERNRaw Data, First pass reconstruction & Filter; Distribution

• Tier 1 - in US, Brookhaven LabReprocessing

• Tier 2Simulation + physics analysis

4

LHC Overview• LHC Worldwide Computing Grid

• >140 sites• >250 CPU cores• >150 PB disk

• Creates 6-7 PB Raw Data per Year• All 4 experiments together; more than

120PB/year of data products that are created and stored world-wide.

5

LHC Network challenges

• LHC data stresses the R&E general purpose Internet • Data will grow more than linearly in next 2 years• Processing & analysis capability at Tier-2s will

continue to increase with computing improvements• Current ATLAS model now allows Tier-2s to pull data

from Tier-1s from other clouds & other Tier 2s to meet production schedules, and transfers outputs back to those Tier-1s.   (CMS model allows Tier-2s to pull data from any Tier-1)

6

Uchicago & Atlas

7

• Midwest Tier2: (one of 5 in US ATLAS)– University of Chicago– Indiana University– U Illinois/NCSA (’12)

• 417 worker nodes • 13 storage servers (1470 TB)• 15 head nodes • Capacity: – 36,840 HEP-Spec2006– 4236 job slots, 2GB/slot

8

Uchicago & Atlas

Atlas tier 0, 1, 2 transfers

9

ATLAS Tier1 and 2’s

10

US ATLAS Tier1 and Tier 2s, and transfers between them and to Tier 1's in other clouds.  

LHCONE Objective• Build an open and neutral global unified service platform for the

LHC community. • Provide a collection of access locations that are effectively entry

points into a network that is private to the LHC T1/2/3 sites. • Improve transfers between Tier-1s and Tier-2s and make

efficient Tier-2 to Tier-2 • Data should be able to be pulled from any Tier-1 but, more

importantly, also from any Tier-2 – or even, if possible, from several Tier-2s simultaneously

• Bos-Fisk report1 – “unstructured” T2 traffic will be the norm• LHCONE is not intended to replace the LHCOPN but rather to

complement it.

111 - https://twiki.cern.ch/twiki/bin/view/LHCOPN/T2sConn

LHCONE requirements*• Bandwidth: 1 Gb (small site) to multiple 10Gb (leadership)• Scalability: Bandwidth growth: Minimal = 2x/yr, Nominal &

Leadership sites = 2x/2yr• Connectivity: Facilitate good connectivity to under-served

sites• Flexibility: Ability to include or remove sites at any time• Organic activity, growing over time according to needs• Initial deployment uses predominantly static configuration

(shared VLAN & Lightpaths)• A Framework for Traffic Engineering in order to “Empower

users”

12*courtesy of Artur Barcyzk & David Foster

LHCOnE architecture*• 3 recurring themes:• Flat(ter) hierarchy: Any site can use any other site

as source of data

• Dynamic data caching: Analysis sites will pull datasets from other sites “on demand”, including from Tier2s in other regions. Possibly in combination with strategic pre-placement of data sets

• Remote data access: jobs executing locally, using data cached at a remote site in quasi-real time. Possibly in combination with local caching

13

*courtesy of Artur Barcyzk & David Foster

LHCONE Logical

14

LHCONE Physical

15

From Bill Johnson’s 2011/09/26 Update:https://indico.cern.ch/getFile.py/access?contribId=8&resId=4&materialId=slides&confId=149042

Layer 2 at the Core• Layer 2 Domains act as multipoint, fully-meshed core• Domains joined at the edge• Two VLANs for redundancy and pseudo-load balancing

• VLAN 2000 configured on GEANT/ACE transatlantic segment• VLAN 3000 configured on US LHCNet transatlantic segment

• Intent is to allow sites to use both transatlantic segments, providing resiliency

• 2 route servers per VLAN• Each connecting site peers with all 4 route servers

• Interim Solution – Architecture Group is still working on final solution

16

L2 limitations• A single spanning tree may result in sub-optimal

routes within the domain• Limited in scaling globally• Potential for broadcast storm affecting entire domain

• Mitigated with storm suppression techniques (custom to each vendor’s implementation)

• Broadcast traffic limited/eliminated to prevent storms• Multicast restricted

• Building dynamic end-to-end circuits entails more burdensome support model

17

L3 advantages• Path diversity• Ability to spontaneously collaborate without

necessarily needing central coordination• Enables traffic engineering by the end site;

LHC data is one of many science flows• Scalable – Can be grown beyond the two

transatlantic links today

18

LHCONE Success Metrics

Following are factors used to measure the effectiveness of the LHCONE design• Improved throughput and latency

• USAtlas Tier2Ds & European Tier 1 & CERN• European Tier 2’s and BNL & Fermi

• US Tier 2 – European Tier 3• Improving Tier 3 connectivity to Tier 2’s

19

Current LHCONE Status• Launched on March 31st• Started shared VLAN service including two initial

sites: CERN, Caltech• Active Open Exchange Points: CERNLight,

NetherLight, StarLight (and MANLAN)• Four route servers for IP-layer connectivity installed

and operational at CERN, GEANT and MANLAN• Midwest pilot locations to include University of

Chicago and University of Michigan

20

uChicago connectivity• Dual 10GE Connections to Regional Networks

• MREN• CIC OmniPOP

• Layer 2 Peer Mapping to • Brookhaven National Labs for Tier 1• Indiana University for Midwest Tier 2 • Additional peering needed for new Midwest Tier2

member UI-UC

• Dual Layer 3 peering to Internet 2 for fallback ATLAS connectivity

21

22

UChicago Current Connectivity

BNL / OmniPop

23

Iu / MREN

24

Our Design Principles• Isolate high bandwidth science flows from the general purpose IP

flows• Allow LHC traffic to seamlessly move between IP and dynamic

circuit infrastructures such as DYNES• Allow end site users to manage their part of the end-to-end path in

order to optimize traffic. • Allow diversity in case of routing failure• Expand beyond the 10Gb site limitation today.• Boost network capacity utilization by load sharing VPLS, L3VPNs,

whatever protocol is best for your network (BGP, OSPF, ISIS)• Connectivity at Layer 3, where appropriate and compatible• No cost increases for network

25

Future Architecture• Larger Data Flows• Bandwidth Availability• Campuses Beyond 10Gb / Up to 20Gb

possibly? • MREN versus OMNIPOP• UChicago/UIUC Direct Physical Connection• 100Gb Roadmap

• Upgrading Ciena DWDM to 100G Capable – Jan. 2012

26

27

Standard Connectivity

Investigating Layer 3• Connect to LHCONE using two paths

• VLAN 2000 via MREN / Starlight• VLAN 3000 via CIC OmniPOP

• Routed border to LHCONE Network• Implement Performance Routing (PfR)

• Utilizing BGP Local Pref and AS Prepending to prefer one path vs. another

• Decision may be based on many factors (Latency, jitter, probe data)

28

29

Investigating Layer 3

Performance Routing Overview

• Cisco Proprietary Protocol• Enhances classic routing with additional

serviceability parameters• Reachability, Delay, Cost, Jitter, MOS score• Can use interface parameters like load,

throughput and monetary cost

• Based on/Enhancement to Optimized Edge Routing (OER)

30

PfR Components• PfR Border Routers

• PfR Master Controller

• Performance Measuring• Passive Monitoring• Active Monitoring• Combined

31

Passive MonitoringMode Comparison Table

32

Comparison Parameter Active Mode

Passive Mode

Combined Mode

Fast Failover Mode

Active/IP SLA Yes No Yes Yes

Passive/NetFlow No Yes Yes Yes

Monitoring of Alternate Paths On Demand

On Demand On Demand Continuous

Best Failover Time 10 seconds

~ 1 minute ~ 1.1 minute 3 seconds

Support for Round Trip Delay Yes Yes Yes Yes

Support for Loss Only with Jitter probe

Only for TCP traffic

Only for TCP traffic

Only for TCP traffic and Jitter probe

Support for Reachability Yes Only for TCP traffic

Only for TCP traffic

Yes

Support for Jitter Yes No No Yes

Support for MOS Yes No No Yes

Passive MOnitoring• Uses NetFlow to gather statistics

• Delay• Packet Loss• Reachability• Throughput

33

Policy Application• BGP Route advertisements

• AS Path Prepend (Preferred)• AS Community

• Exit Link Selection• BGP Local Preference (Preferred)• Static Route Injection• Policy Route• Protocol Independent Route Optimization (PIRO)

34

uChicago Implementation

35

uCHicago Implementation

• Fully utilize dual Regional Network connectivity

• Control path utilization based upon shared link performance

• Optimize path selection based on NetFlow performance statistics

• Provide measurable data to report on success factors

36

PFR Pros/ConsPROS• Simple / Automated

Solution for Traffic Engineering requirement currently missing in LHCONE Architecture

• Works with current LHCONE Design

• Provides statistics on traffic patterns

CONS• Cisco Proprietary

• Requires multiple exit paths to LHCONE VLANS• Though can be applied to VLAN

sub-interfaces

• Short-term solution to address current LHCONE Architecture Limits

• May create complex routing advertisements depending on granularity

• Cisco Proprietary……..

37

38

Proof of ConceptUtilize GNS3 to Model the Routing

• Simulate LHCONE Core

• Simulate Two remote LHCONE Connected Sites• MIT• MSU

• Only use two Route Servers

Pre-PFR FlowWithout Tuning

• Ingress traffic selecting single path (VLAN)

• Egress traffic traversing both VLANs via ECMP. • Flows asymmetric• Packets out of order

10.0.0.0/8 is variably subnetted, 4 subnets, 2 masksO E2 10.14.0.0/24 [110/1] via 192.168.16.2, 00:13:57, GigabitEthernet0/0 [110/1] via 192.168.16.1, 00:13:54, GigabitEthernet0/0O E2 10.15.0.0/24 [110/1] via 192.168.16.2, 00:13:57, GigabitEthernet0/0 [110/1] via 192.168.16.1, 00:13:54, GigabitEthernet0/0

39

10.0.0.0/24 is subnetted, 3 subnetsB 10.14.0.0 [20/0] via 192.16.157.14, 00:18:47C 10.15.0.0 is directly connected, GigabitEthernet1/0B 10.16.0.0 [20/0] via 192.16.157.16, 00:19:14

PfR Flow

40

With Tuning

• Ingress traffic selecting single path (VLAN)

• Egress traffic selecting single path (VLAN). • Symmetric Flow• Packets in order

• Automatic redirection in case of congestion

10.0.0.0/8 is variably subnetted, 4 subnets, 2 masksO E2 10.14.0.0/24 [110/1] via 192.168.16.2, 00:13:57, GigabitEthernet0/0O E2 10.15.0.0/24 [110/1] via 192.168.16.2, 00:13:57, GigabitEthernet0/0

Network Next Hop Metric LocPrf Weight Path* i10.14.0.0/24 192.16.161.14 0 100 0 20641 229 i*> 192.16.157.14 150 0 20641 229 i* i10.15.0.0/24 192.16.161.15 0 100 0 20641 3 i*> 192.16.157.15 150 0 20641 3 i

10.0.0.0/24 is subnetted, 3 subnetsB 10.14.0.0 [20/0] via 192.16.157.14, 00:18:47C 10.15.0.0 is directly connected, GigabitEthernet1/0B 10.16.0.0 [20/0] via 192.16.157.16, 00:19:14

Network Next Hop Metric LocPrf Weight Path*> 10.14.0.0/24 0.0.0.0 0 32768 i* 10.15.0.0/24 192.16.161.15 0 20641 3 i*> 192.16.157.15 0 20641 3 i* 10.16.0.0/24 192.16.157.16 0 20641 160 160 160 160 ?*> 192.16.161.16 0 20641 160 ?

Performance Impact

Additional Info• Minimum for ISR and 7200 Routers – 12.4(9)T• 7600 Series – 12.2(33)SRB• Cat6500 Requires IOS 12.2(33)SXH• ASR1000 – 3.3.0S• Additional PfR Netflow Data Export in IOS XE 3.4S

http://www.cisco.com/en/US/docs/ios-xml/ios/pfr/configuration/xe-3s/pfr-netflow-v9.pdf

• FAQ:http://docwiki.cisco.com/wiki/Performance_Routing_FAQs

41

Alternatives to CiscoJuniper

• Real-Time Performance Monitoring (RPM)

• Closest Alternative Found

• Uses probes to gather performance data

• Unable configure routing protocols

http://www.juniper.net/us/en/local/pdf/app-notes/3500145-en.pdf

INTERNAP

• Flow Control Platform (FCP)

• Appliance Based

• Passively Monitors via Tap

• Actively reconfigures routing

http://www.internap.com/wp-content/uploads/FCP-Flow-Control-Platform_DS_0511.pdf

42

Addressing Success Metrics

Success Factors

• Improved throughput and latency • USAtlas Tier2Ds &

European Tier 1 & CERN• European Tier 2’s and

BNL & Fermi

• US Tier 2 – European Tier 3

• Improving Tier 3 connectivity to Tier 2’s

Proposed Solution

• Provides ability to actively modify routes to best path (VLAN) based on live statistics – Delay and Throughput

• Ensure best path is utilized

• Ensure best path is utilized

43

Review• Science Data Flows are getting Larger• Flows are Becoming Less “Predictable”• Current LHCONE Architecture is short-term• There is a need to address Traffic Engineering

• Not easily satisfied at end-sites

• PfR solution provides an automated method to address TE need to select the best path (VLAN)• Not only during outage, but during congestion

• Provides statistics to help measure LHCONE success factors

44

Q & A

45