experiences with the qwest/scinet bandwidth challenge 2000-2002
DESCRIPTION
Experiences with the QWEST/SCinet Bandwidth Challenge 2000-2002. John Shalf, Wes Bethel LBNL. Shaky City. Data Caches. HPC Resources. Simulations. Sensor Nets. STM. Collaborators. Handheld Devices. Dr Jane. Backend. Source Volume. Viewer Component. Visapult Architecture. Source - PowerPoint PPT PresentationTRANSCRIPT
Experiences with theQWEST/SCinet Bandwidth
Challenge2000-2002
John Shalf, Wes BethelLBNL
Shaky City Data Caches HPC Resources
Sensor Nets
Simulations
STM
HandheldDevices
Dr Jane
Collaborators
Visapult Architecture
SourceVolume
Backend
Viewer Component
Visapult Architecture
SourceVolume
3D 2D
Visapult Architecture
SourceVolume
3DGigabits
2DMegabits
Visapult Architecture
SourceVolume
3DGigabits
2DMegabits
SC2000 Demo Configuration
NTON
8 node StorageCluster (DPSS)
Network Throughput: 5 sec peak 1.48 Gbits/sec (72 streams: 20.5Mbits/stream); 60 minute sustained average: 582 Mbits/sec
Compute Cluster (8 nodes)
Berkeley Lab:.75 TB, 4 server DPSS
ANL Booth Linux Cluster
OC-48OC-48
2 x 1000 BT
HSCC
SGI Origin (8 CPU)
1.5 Gb/s4 x 1000BT
QwestASCI Booth:
SGI Origin (8 CPU)
4 x 1000BT
Visapult VisualizationApplication
File Transfer Application
SC2000 Network Throughput
Refactoring the Design• Congestion avoidance
- Good for internet- Bad bad baaaad for PVCs and other dedicated networks.
(switched lambdas?)• Multistream TCP
- Erratic performance- Requires a lot of tuning- Unfriendly to other users- Unfriendly to visualization applications
• We want full control of the “throttle”- Very much like network video
Refactoring the Design• TCP is the wrong thing for interactive vis!
- Layer 3 latency/jitter (all buffering effects)- Poor response to bursty traffic- Vis needs interactivity and minimal latency!
• Network Video / UDP streams- Present packets to app. immediately (low latency) - Full control of data rate- Lossy, but effects of loss can be managed
• SOCK_RDM
Effect of Loss on Visapult
Evolving Binary Black Hole Merger Simulation (100 timesteps)
SC2001 BWC Testbed
SC2001 BWC Results
•3.3 Gigabits Peak (MRTG of ESNet/QWEST OC-48)•Lots of time spent tuning (packet pacing is nontrivial)
SC2002 Planning• Better UDP packet pacing with poor clock
granularity- Error diffusion algorithm from dithering
• Omnidirectional viewing for Visapult
• Nascent Grid Infrastructure- Global Grid Testbed Collaboration (GGTC)- GGTC won 3 out of 4 HPC challenge awards
Global Testbed
Bandwidth Scavenging ServiceCzech Republic
90% utilization on path from Brno
To GEANT (to SC) via Praha
Steady @ 16+ Gigabits!Visapult/Cactus SC02 Bandwidth Challenge Results
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
20000
0 100 200 300 400 500 600 700 800
Time (seconds)
Megabits
10G-Link-1
10G-Link-2
10G-Link-3
Cumulative
Why do we do this??• Opportunity to test the next generation network capacity (2
years before production)• Opportunity to test software ideas that will be required to
make use of the next generation network (2 year lead)• Opportunity to test future network engineers (give them a 2
year lead on the problems with future networks)• Prescient
- I-Way 1995 (presaged vBNS experiences and “The Grid”)- 1997 multi-stream tests (presaged multi-stream TCP protocols)- 2001 10GigE and UDP-based protocols
• 10GigE backbone for SC2002• RUDP, SABUL, TSUNAMI user-space UDP-based protocols
What should you know for 2003• Manual throttle (UDP-based protocols) are here to stay.
- Hopefully SOCK_RDM will cover most needs- Whaaa? Those idiots are going to burn down the network! Next big thing:
resource management• RSVP & DiffServ were developed to manage this very situation
with regard to network video• RSVP & DiffServ are never going to happen
- Gregory Bell, “Failure to Thrive: QoS and the Culture of Operational Networking,” Proceedings of the ACM SIGCOMM 2003 Workshops, RIPQoS Workshop.
• Next Big Thing? : Pluggable Congestion Management- AIMD for internet (can even mimic multistream TCP behavior)- Fixed rate for PVCs and switched lambdas- Get hints from monitoring services (NWS) to manage flow end-to-end
Concern about the future of BWC• The future and eternal reigning champion
of the BWC?
Concern about the future of BWC• The future and eternal reigning champion
of the BWC?
iperf --udp -b <infinity>
Concern about the future of BWC• The future and eternal reigning champion of the BWC?
- How do we prevent this?
Concern about the future of BWC• The future and eternal reigning champion of the BWC?
- How do we prevent this?• There is not technical solution to this problem
- It is a social problem, not a technical one
Concern about the future of BWC• The future and eternal reigning champion of the BWC?
- How do we prevent this?• There is not technical solution to this problem
- Can’t ban UDP- Can’t eliminate high bandwidth category
Concern about the future of BWC• The future and eternal reigning champion of the BWC?
- How do we prevent this?• There is not technical solution to this problem
- Can’t ban UDP- Can’t eliminate high bandwidth category
• Must answer- Does this advance science or art?- Is this a glimpse of the future?
Concern about the future of BWC• The future and eternal reigning champion of the BWC?
- How do we prevent this?• There is not technical solution to this problem
- Can’t ban UDP- Can’t eliminate high bandwidth category
• Must answer- Does this advance science or art?- Is this a glimpse of the future?- What the heck are judges for anyways?
Acknowledgements• Wes Bethel• Brian Tierney• Dan Gunter• Jason Lee• Eli Dart• Mike Bennett• Chip Smith• John Christman• Raju Shah• Ed Seidel• Gabrielle Allen• Tom Goodale• Thomas Radke• Mirek Ruda• Ludek Matskya
• NERSC• LBNL• NCSA• PSC• ANL• AEI/Potsdam• University of Amsterdam• ESNet• QWEST• Abilene• StarLight• Teragrid • Force10 Networks• Scinet
More Information• Visapult:
- V1.0: http://vis.lbl.gov/projects/visapult- V2.0: http://vis.lbl.gov/projects/visapult2
• Cactus- http://www.cactuscode.org
• VisPortal (Grid Portal for managing Visapult)- http://frost.lbl.gov/visportal/
• SC2002 Global Grid Testbed- http://scb.ics.muni.cz/static/SC2002/Testbed.html
• Related Publications- Special Issue on Visualization on the Grid, IEEE Computer Graphics
and Applications, Volume 23, Number 2, March/April 2003.