Download - U.S. ATLAS Tier-1 Network Status
![Page 1: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/1.jpg)
Michael Ernst [email protected]
U.S. ATLAS Tier-1 Network Status
Evolution of LHC Networking – February 10, 2014 1
![Page 2: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/2.jpg)
Tier-1 Production Network Connectivity• As to the Tier-1: the maximum usable b/w is 70 Gbps
• 50 Gbps dedicated circuits/unshared plus 20 Gbps general IP service shared across all departments at BNL)
• Currently available for Tier-0 Tier-1 and Tier-1 Tier-1: 17 Gbps via OPN/USLHCNet + 2*10 Gbps ESnet/GEANT shared by researchers in US
• One dedicated 10 Gbps circuit for LHCONE (LHC Open Network Environment, connecting the Tier-1 at MANLAN in New York)
• DOE/ESnet “dark fiber project” has brought abundant physical fiber infrastructure into the lab
• BNL connected to ESnet at 100G• Have T1 facility connected to ESnet at 100G for R&D (ANA TA link)• In the process of moving BNL/T1 production environment to 100G
![Page 3: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/3.jpg)
OPN
R&E +Virtual Circuits
LHCONE
ATLAS Software and Computing Week - October 24, 2013
3
2.5 Gbps
3.5 Gbps
![Page 4: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/4.jpg)
4
SNLL
PNNL
LANL
SNLA
JLAB
PPPL
MIT/
PSFC
BNL
AMES
LLNL
GA
JGI
LBNL
SLAC
NERSC
OR
NL
ANL
FNAL
Salt Lake
INL
GFDLPU Physics
SEAT
ST
AR
CHICWASH
commercial peering points
NASH ATLA
HO
US
NEWY AOFA
BOST
KANS
DEN
V
ALBQ
LASV
BOIS
SA
CR
ELP
A
SDSC
LOSA
Routed IP 100 Gb/sRouted IP 4 X 10 Gb/s3rd party 10Gb/sExpress / metro 100 Gb/sExpress / metro 10GExpress multi path 10GLab supplied linksOther linksTail circuits
Major Office of Science (SC) sitesLBNL
CLE
V
ESnet optical transport nodes (only some are shown)
MANLAN
ESnet managed 10G router10
ESnet managed 100G routers100
Site managed routers100 10
1010
10
10
100 10
10
10
1
100
100
100
100
10
10
100
100
10
10
10
10
10
100
100 10
10
100 10
100
SUNN ESnet PoP/hub locations
LOSA ESnet optical node locations (only some are shown)
10
100
100
100
100
100
100 E
QC
H
Geography isonly representational
ESnet5 March 2013
1
100
10
SUNN
100
100 100
10
100
CH
AT
![Page 5: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/5.jpg)
BNL’s LHCOPN Connectivity is provided by USLHCNet
H. Newman
![Page 6: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/6.jpg)
12PB
WNs
20 DataTransfer Nodes
![Page 7: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/7.jpg)
12PB
WNs
![Page 8: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/8.jpg)
![Page 9: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/9.jpg)
CERN/T1 -> BNL Transfer Performance via ANA 100G
• Regular ATLAS Production + Test Traffic
• Observations (all in the context of ATLAS)– Never exceeded ~50 Gbits/sec– CERN (ATLAS EOS) -> BNL limited at ~1.5 GB/s
• Achieved >8 GB/s between 2 Hosts @ CERN and BNL– Each T1 (via OPN/CERN) -> BNL limited to ~0.5 GB/s
![Page 10: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/10.jpg)
![Page 11: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/11.jpg)
Evolving Tier-2 Networking • All 5 US ATLAS Tier-2 (10 Sites) sites are currently
connected at rate of at least 10 Gbps– This has proven not sufficient to efficiently utilize the resources
at federated sites (CPU & Disk at different sites)
• US ATLAS Facilities have recognized the need to develop network infrastructure at sites– A comprehensive, forward looking plan exists– Additional Funding was provided by US ATLAS Mgmt & NSF
• Sites are in the process of upgrading their Connectivity to 100 Gbps– 6 Sites will have completed upgrade by end of April– All others will be done by the end of 2014
11
![Page 12: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/12.jpg)
From CERN to BNL
12
![Page 13: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/13.jpg)
From BNL to T1s
13
![Page 14: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/14.jpg)
From BNL to T1s and T2s
14
![Page 15: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/15.jpg)
T1s vs. T2s from BNL (2013 Winter Conference Preparations)
15
CA T1
CA T2s
DE T1
DE T2s
UK T1
UK T2s
FR T1
FR T2s
T2s in several regions are getting ~an order of magnitude more Data from BNL than the associated T1s
![Page 16: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/16.jpg)
From T1s to BNL
16
![Page 17: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/17.jpg)
From T1s and T2s to BNL
17
![Page 18: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/18.jpg)
From BNL to CERN
18
![Page 19: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/19.jpg)
T1s vs. T2s to BNL
19
DE T1
DE T2FR T1
FR T2
UK T1
UK T2
CA T1
CA T2s
![Page 20: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/20.jpg)
From BNL to non-US T2s
20
![Page 21: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/21.jpg)
From non-US T2s to BNL
![Page 22: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/22.jpg)
Remote Access – A possible Game-Changer
• Data access over the WAN at job runtime– Today tightly coupled with Federated Data
Access• Automatic data discovery with XrootD redirector• Unpredictable network/storage bandwidth
requirement–Possible issues include hotspots, campus
network congestion, storage congestion, latency– Totally synchronous: time to completion within
minutes/seconds (or less)
![Page 23: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/23.jpg)
Worldwide FAX Deployment
![Page 24: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/24.jpg)
Jobs Accessing Data Remotely w/ FAX
![Page 25: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/25.jpg)
Traffic Statistics - Observations
• Traffic Volume To/From BNL– From CERN to BNL: ~500 TB/month during ATLAS data taking – To BNL: 1,400 TB/month (Peak 1,900 TB/month)– From BNL: 1,900 TB/month (Peak 2.200 TB/month)
• T1 Traffic Volume To/From BNL via LHCOPN– To BNL: 400 TB/month (Peak 1,200 TB/month)– From BNL: 400 TB/month (Peak 600 TB/month)– BNL to T2 Volume during conference preparation order of magnitude
higher than BNL to T1 Volume• Traffic Volume From/To BNL via LHCONE and GIP
– To BNL from non-US T2s: 200 TB/month (Peak 500 TB/month)– From BNL to non-US T2s: 1000 TB/month (~400MB/s)– Traffic clearly driven by analysis activities
25
![Page 26: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/26.jpg)
Trends
• Looking at 2012 and 2013 statistics data, from BNL’s perspective Traffic statistics suggests BNL/T1 to T2 Traffic is dominating– Traffic to non-US T2s doubled to 1 PB/month in
September 2012• ~Constant since then, potential to grow w/ new data• Largely driven by analysis
– BNL Traffic Volume from/to T1s via LHCOPN staying fairly constant for 2 years at ~500 TB/month
• Largely independent from data taking
26
![Page 27: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/27.jpg)
Conclusion• Rather than maintaining distinct networks the
LHC Community should aim at unifying its network infrastructure – In ATLAS Tiers are becoming more and more
meaningless– We are thinking about optimizing usage of CPU and
Disk and we also need to think about optimizing usage of Network resources
– Load-balanced links – Traffic prioritization, if necessary
• Traffic aggregation on fewer Links 27
![Page 28: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/28.jpg)
R&E Transatlantic Connectivity (5/2013)
Dale Finkelson
Not showingLHCOPNCircuits
![Page 29: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/29.jpg)
Looking at it generically …
![Page 30: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/30.jpg)
Concerns• With the T1 and the T2s in the US upgrading now to
100G the global infrastructure needs to follow• LHCONE Evolution
– Currently the LHCONE side-by-side w/ general R&E infrastructure
– Traffic segregated, but what is actually the benefit?• Is anyone looking at the flows for optimization, steering?• Is it really true that our ‘Elephant’ flows interfere w/ traffic from other
science communities?
• P2P/Dynamic Circuit Infrastructure– Are the interface definitions and components mature enough
to serve applications?– What would happen if the experiments started to use dynamic
circuits extensively, in multi-domain environment?– Would there be sufficient infrastructure in the system? 30
![Page 31: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/31.jpg)
Backup Material
31
![Page 32: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/32.jpg)
From US T2s to BNL
![Page 33: U.S. ATLAS Tier-1 Network Status](https://reader035.vdocuments.us/reader035/viewer/2022062802/5681449d550346895db14f46/html5/thumbnails/33.jpg)
From BNL to US T2s