global networking for the lhc artur barczyk california institute of technology ecoc conference...
TRANSCRIPT
![Page 1: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/1.jpg)
Global Networking for the LHC
Artur Barczyk
California Institute of Technology
ECOC Conference
Geneva, September 18th, 2011
1
![Page 2: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/2.jpg)
INTRODUCTION
First Year of LHC from the network perspective
2
![Page 3: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/3.jpg)
WLCG Worldwide Resources
3
Today >140 sites>250k CPU cores>150 PB disk
Today we have 49 MoU signatories, representing 34 countries:Australia, Austria, Belgium, Brazil, Canada, China, Czech Rep, Denmark, Estonia, Finland, France, Germany, Hungary, Italy, India, Israel, Japan, Rep. Korea, Netherlands, Norway, Pakistan, Poland, Portugal, Romania, Russia, Slovenia, Spain, Sweden, Switzerland, Taipei, Turkey, UK, Ukraine, USA.
WLCG Collaboration StatusTier 0; 11 Tier 1s; 68 Tier 2 federations
In addition to WLCG, O(300) Tier-3 sites, not shown
![Page 4: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/4.jpg)
Data and Computing Models
4
The Evolving MONARC Picture: Circa 2003
From Ian Bird, ICHEP 2010
Variations by experiment
The models are based
on the MONARC
model
Now 10+ years old
Circa 1996
![Page 5: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/5.jpg)
The LHC Optical Private Network Serving Tier0 and Tier1 sites
• Dedicated network resources for Tier0 and Tier1 data movement• Layer 2 overlay on R&E
infrastructure• 130 Gbps total Tier0-Tier1
capacity• Simple architecture
– Point-to-point Layer 2 circuits– Flexible and scalable topology
• Grew organically– From star to partial mesh
•Open to technology choices• have to satisfy requirements• OC-192/SDH-64, EoMPLS,
OTN-3• Federated governance model
– Coordination between stakeholders
5
![Page 6: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/6.jpg)
6
MB/sper day
6 GB/s
Peaks of 10 GB/s reached
~2 GB/s(design)
Grid-based analysis in Summer 2010: >1000 different users; >15M analysis jobs
The excellent Grid performance has been crucial for fast release of physics results. E.g.: ICHEP: the full data sample taken until Monday was shown at the conference Friday
2010 Worldwide data distribution and analysis (F.Gianotti)Total throughput of ATLAS data through the Grid: 1st January November.
![Page 7: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/7.jpg)
CMS Data Movements (2010) (All Sites and Tier1-Tier2)
7
1 hour average: to 3.5 GBytes/s
Th
rou
gh
pu
t
[GB
y/s]
3
4
2
Th
rou
gh
pu
t
[GB
y/s]
1
2
0
1.5
2.5
0.5
Daily average total rates reach over
2 GBytes/s
1
0
1
2
0
1.5
0.5
120 Days June-October 2010
Daily average T1-T2 rates reach
1-1.8 GBytes/s
132 Hours in Oct. 2010
6/19 7/03 7/17 7/31 8/14 8/28 9/11 9/25 10/9 6/23 7/07 7/21 8/4 8/18 9/1 9/15 9/29 10/13
10/6 10/7 10/8 10/9 10/10
Tier2-Tier2 ~25% of Tier1-Tier2
Traffic
To ~50% during Dataset
Reprocessing & Repopulation
![Page 8: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/8.jpg)
THE RESEARCH AND EDUCATION NETWORKING LANDSCAPE
Selected representative examples
8
![Page 9: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/9.jpg)
GEANT Pan-European Backbone
9
Dark Fiber Core Among 19 Countries:
Austria Belgium Croatia Czech Republic Denmark Finland France Germany Hungary Ireland Italy Netherlands Norway Slovakia Slovenia Spain Sweden Switzerland United Kingdom
34 NRENs, ~40M Users; 50k km Leased Lines 12k km Dark Fiber; Point to Point ServicesGN3 Next Gen. Network Started in June 2009
![Page 10: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/10.jpg)
SURFNet & NetherLight: 8000 Km Dark FiberFlexible Photonic Infrastructure
5 Photonic Subnets
λ Switching 10G, 40G; 100G Trials
Fixed or Dynamic
Lightpaths for
LCG, GN3, EXPRES
DEISALOFAR CineGrid
5 Photonic Subnets
λ Switching 10G, 40G; 100G Trials
Fixed or Dynamic
Lightpaths for
LCG, GN3, EXPRES
DEISALOFAR CineGrid Cross Border Fiber: to Belgium, on to CERN
(1650km); to Germany: X-Win, On to NORDUnet;Cross Border Fiber: to Belgium, on to CERN
(1650km); to Germany: X-Win, On to NORDUnet; Erik-Jan Bos
![Page 11: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/11.jpg)
GARR-X in Italy: Dark Fiber Network Supporting LHC Tier1 and Nat’l Tier2 Centers
GARR-X
10G Links Among Bologna Tier1
& 5 Tier2s
Adding 5 More Sites at 10G
2 x 10G Circuits to the LHCOPNOver GEANT
and to Karlsruhe Via Int’l Tier2 – Tier1 Circuits
GARR-X
10G Links Among Bologna Tier1
& 5 Tier2s
Adding 5 More Sites at 10G
2 x 10G Circuits to the LHCOPNOver GEANT
and to Karlsruhe Via Int’l Tier2 – Tier1 Circuits
M. MarlettaCross Border Fibers to Karlsruhe (Via CH, DE)Cross Border Fibers to Karlsruhe (Via CH, DE)
![Page 12: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/12.jpg)
US: DOE ESnet
12
Current ESnet4 Topology: Multi-10G backbone
SDN nodeIP router node10G linkMajor site v
![Page 13: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/13.jpg)
DOE Esnet – 100Gbps Backbone Upgrade
13
100G nodeRouter node100G linkMajor site v
ESnet5 100G Backbone, Q4 2012First deployment started Q3 2011
![Page 14: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/14.jpg)
US LHCNetNon-stop Operation; Circuit-oriented Services
14
Performance enhancingStandard
Extensions:VCAT, LCAS
USLHCNet, ESnet, BNL
& FNAL:Facility,
equipment and link
redundancy
Core: Optical multiservice
Switches
Dynamic circuit-oriented network services with BW guarantees, with robust fallback at layer 1: Hybrid optical network
![Page 15: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/15.jpg)
Dark Fiber in NREN Backbones 2005 – 2010Greater or Complete Reliance on Dark Fiber
TERENA Compendium 2010: www.terena.org/activities/compendium/
2005 2010
![Page 16: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/16.jpg)
Cross Border Dark Fiber in EuropeCurrent and Planned: Increasing Use
TERENA Compendium
2010
![Page 17: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/17.jpg)
Global Lambda Integrated Facility
A Global Partnership of R&E Networks and Advanced Network R&D Projects Supporting HEP
http://glif.is
GLIF 2010 Map – Global View
![Page 18: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/18.jpg)
GLIF 2010 Map North America
~16 10G Trans- Atlantic Links
in 2010
![Page 19: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/19.jpg)
GLIF Open Lightpath Exchanges:MoscowLight, CzechLight, CERNLight, NorthernLight
NetherLight, UKLight
GLIF Open Lightpath Exchanges:MoscowLight, CzechLight, CERNLight, NorthernLight
NetherLight, UKLight
GLIF 2010 Map: European ViewR&E Networks, Links and GOLEs
![Page 20: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/20.jpg)
Open Exchange Points: NetherLight Example
3 x 40G, 30+ 10G Lambdas, Use of Dark Fiber
Convergence of Many Partners on Common Lightpath Concepts Internet2, ESnet, GEANT, USLHCNet; nl, cz, ru, be, pl, es, tw, kr, hk, in, nordicConvergence of Many Partners on Common Lightpath Concepts
Internet2, ESnet, GEANT, USLHCNet; nl, cz, ru, be, pl, es, tw, kr, hk, in, nordic
![Page 21: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/21.jpg)
LHC NETWORKING - BEYOND LHCOPN
21
![Page 22: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/22.jpg)
Computing Models Evolution
• Moving away from the strict MONARC model• Introduced gradually since 2010• 3 recurring themes:
– Flat(ter) hierarchy: Any site can use any other site as source of data
– Dynamic data caching: Analysis sites will pull datasets from other sites “on demand”, including from Tier2s in other regions
• Possibly in combination with strategic pre-placement of data sets– Remote data access: jobs executing locally,
using data cached at a remote site in quasi-real time
• Possibly in combination with local caching
• Variations by experiment
22
![Page 23: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/23.jpg)
LHC Open Network Environment
• So far, T1-T2, T2-T2, and T3 data movements have been using General Purpose Network infrastructure– Shared resources (with other science fields)– Mostly best effort service
• Increased reliance on network performance need more than best effort• Separate large LHC data flows from routed R&E GPN
• Collaboration on global scale, diverse environment, many parties– Solution to be Open, Neutral and Diverse – Agility and Expandability
• Scalable in bandwidth, extent and scope• Organic activity, growing over time according to needs• Architecture:
– Switched Core, Routed Edge– Core: Interconnecting trunks between Open Exchanges– Edge: Site Border Routers, or BRs of regional aggregation networks
• Services: Multipoint, static point-to-point, dynamic point-to-point
23
![Page 24: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/24.jpg)
LHCONE High-Level Architecture Overview
24
LHCONE Conceptual
Diagram
![Page 25: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/25.jpg)
LOOKING FORWARD: NEW NETWORK SERVICES
25
![Page 26: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/26.jpg)
Characterization of User Space
26
Cees de Laat; http://ext.delaat.net/talks/cdl-2005-02-13.pdf
This is where LHC users are
![Page 27: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/27.jpg)
27
David Foster, 1st TERENA ASPIRE Workshop, May 2011
![Page 28: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/28.jpg)
The Case for Dynamic Circuits in LHC Data Processing
• Data models do not require full-mesh @ full-rate connectivity @ all times• On-demand data movement will augment and partially replace static pre-
placement Network utilisation will be more dynamic and less predictable• Performance expectations will not decrease
– More dependence on the network, for the whole data processing system to work well!
• Need to move large data sets fast between computing sites– On-demand: caching– Scheduled: pre-placement– Transfer latency is important
• Network traffic far in excess of what was anticipated• As data volumes grow rapidly, and experiments rely increasingly on the
network performance - what will be needed in the future is– More bandwidth – More efficient use of network resources– Systems approach including end-site resources and software stacks
• Note: Solutions for the LHC community need global reach28
![Page 29: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/29.jpg)
Dynamic Bandwidth Allocation
• Will be one of the services to be provided in LHCONE• Allows to allocate network capacity on as-needed basis
– Instantaneous (“Bandwidth on Demand”), or – Scheduled allocation
• Significant effort in R&E Networking community– Standardisation through OGF (OGF-NSI, OGF-NML)
• Dynamic Circuit Service is present in several advanced R&E networks – SURFnet (DRAC)– ESnet (OSCARS)– Internet2 (ION)– US LHCNet (OSCARS)
• Planned (or in experimental deployment)– E.g. GEANT (AutoBahn), RNP (OSCARS/DCN), …
• DYNES: NSF funded project to extend hybrid & dynamic network capabilities to campus & regional networks – In first deployment phase; fully operational in 2012
29
![Page 30: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/30.jpg)
US Example: DYNES Project
• NSF-funded project: DYnamic NEtwork System• What is it?
– A nationwide cyber-instrument spanning up to ~40 US universities and ~14 Internet2 connectors
– Extends Internet2s ION service into regional networks and campuses, based on ESnet’s OSCARS implementation of IDC protocol
• Who is it?– A collaborative team including Internet2, Caltech, University of Michigan, and
Vanderbilt University – Community of regional networks and campuses– LHC, astrophysics community, OSG, WLCG, other virtual organizations
• The goals– Support large, long-distance scientific data flows in the LHC, other leading
programs in data intensive science (such as LIGO, Virtual Observatory, and other large scale sky surveys), and the broader scientific community
– Build a distributed virtual instrument at sites of interest to the LHC but available to R&E community generally
30http://www.internet2.edu/dynes
![Page 31: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/31.jpg)
DYNES System Description
• AIM: extend hybrid & dynamic capabilities to campus & regional networks – A DYNES instrument must provide two basic capabilities at the Tier 2s, Tier3s and
regional networks:
1. Network resource allocation such as bandwidth to ensure transfer performance
2. Monitoring of the network and data transfer performance
• All networks in the path require the ability to allocate network resources and monitor the transfer. This capability currently exists on backbone networks such as Internet2 and ESnet, but is not widespread at the campus and regional level– In addition Tier 2 & 3 sites require: 3. Hardware at the end sites capable of
making optimal use of the available network resources
31
Two typical transfers that DYNES supports: one Tier2 - Tier3 and
another Tier1-Tier2.
The clouds represent the network domains involved in such a transfer.
![Page 32: Global Networking for the LHC Artur Barczyk California Institute of Technology ECOC Conference Geneva, September 18 th, 2011 1](https://reader035.vdocuments.us/reader035/viewer/2022070415/56649f0e5503460f94c232e2/html5/thumbnails/32.jpg)
Summary
• LHC Computing models rely on efficient high-throughput data movement between computing sites (Tier0/1/2/3)
• Close collaboration between the LHC and R&E networking communities– Regional, National, International
• LHCOPN (LHC Optical Private Network):– Layer 2 overlay network, dedicated resources for Tier0 and Tier1 centres– Very successful operation
• LHCONE (LHC Open Network Environment):– New initiative to provide reliable services to ALL LHC computing sites (Tier 0-3)– Being developed as collaboration between LHC community and the Research
and Education Networks world-wide– User driven, organic growth– Current architecture is built on switched core with routed edge– Will provide advanced network services with dynamic bandwidth allocation
32