Download - UDT.pptx
![Page 1: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/1.jpg)
UDT: High Performance Data Transfer
Presented by Dan Sullivan 10/22/2013
![Page 2: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/2.jpg)
The UDT Team
• Dr. Yunhong Gu – Architect and lead developer, an expert in high speed networking and high performance compuKng. Was a research scienKst at UIC, is now a soOware engineer at Google
• Dr. Robert Grossman – CRIO of the U of C’s Center for Research InformaKcs
![Page 3: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/3.jpg)
Overview
• 3x winner of the SuperCompuKng Bandwidth challenge
• Uses UDP for data transfer with internal reliability and congesKon control mechanisms
• With emerging bandwidth technologies (i.e. 10Gb/s and 40Gb/s Ethernet, a new transmission protocol is needed to address scalability problems with bandwidth delay product (BDP).
![Page 4: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/4.jpg)
Bandwidth Delay Product Overview
• DefiniKon – the mulKplicaKon of a data link’s capacity and it end-‐to-‐end delay.
• Result is a amount of bits or bytes, data that is on the circuit that has been transmiZed but not acknowledged.
• BDP is the expression of full circuit uKlizaKon
![Page 5: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/5.jpg)
TCP: Reliable Streaming Unicast • But first, UDP: It’s not reliable, so, we’re not going to talk any more about it.
• TCP uses AIMD for its windowing algorithm (addiKve increase, mulKplicaKve decrease). Increases window size by 1 per RTT, halves it when packet loss is detected.
• This means for high bandwidth networks it takes a long Kme to reach the maximum network throughput.
• It is difficult to realize the full throughput of TCP on a network that has packet loss.
![Page 6: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/6.jpg)
TCP CongesKon Performance CharacterisKcs
![Page 7: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/7.jpg)
In an HPC Environment, We’d Want Something That Looks Like This:
![Page 8: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/8.jpg)
UDT Protocol SpecificaKons, ObjecKves, Design CharacterisKcs:
– Designed for distributed data intensive applicaKons using high speed networks
– Easy to deploy – Open Source (BSD & LGPLv2 licenses) – A configurable protocol framework and custom congesKon control
algorithm – Consists of a C++ library (containing the UDT API). Currently on
version 4. – Portable (works on Linux, Windows, OS X, and BSD).
![Page 9: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/9.jpg)
UDT vs. TCP Reliable Stream Socket AbstracKon
udt.sourceforge.net
UDP
Socket API
ApplicaKons
TCP
Socket API
ApplicaKons
ApplicaKons
UDT
UDT Socket
![Page 10: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/10.jpg)
• sendfile/recvfile for efficient file transfer • sendmsg/recvmsg: messaging with parKal reliability • selectEx: a more efficient version of select() • kept the tradiKonal UNIX socket interface to lessen the
learning curve for developers • Under certain condiKons, applicaKons can be ported from TCP
to UDT without any modificaKons. • Uses a AIMD control algorithm proporKonal to bandwidth
with a random decrease between 1/2 and 1/8. • Exposes variables related to congesKon control, allows
modificaKon through callbacks
The UDT API
![Page 11: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/11.jpg)
![Page 12: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/12.jpg)
UDT Source Tree
![Page 13: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/13.jpg)
udt.sourceforge.net
A UDT Instance
![Page 14: UDT.pptx](https://reader031.vdocuments.us/reader031/viewer/2022022202/5880a6211a28abd8158b772b/html5/thumbnails/14.jpg)
• InstantWebP2P/node-‐hZpp Run HTTP over UDP with Node.js for P2P Web.
• AhemoFS • EMC VPLEX • nabto • iTwin • Kaseya • GroupLogic MassTransit 7.0 • Center for ComputaKon and Technology, Louisiana State University
CCT, LSU uses UDT for data transmission in high speed interacDve and distributed visualizaDon and exploraDon of very large dataset.
• HSCP by InsKtute for Molecular Science, Japan HSCP (Hybrid scp) is a high speed variant of the "scp" file copy tool developed to help transferring large chemistry research data at IMS Japan and is now made open source to public users.
• Naval Research Laboratory Naval Research Laboratory (NRL) implemented the UDT protocol on FPGA and used it to support long distance infiniband.
Technical ImplementaKons of UDT
udt.sourceforge.net