internet 2 members meeting 20 april 2004 arlington, va mohan ramamurthy unidata program center ucar...
TRANSCRIPT
![Page 1: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/1.jpg)
Internet 2 Members Meeting20 April 2004Arlington, VA
Mohan RamamurthyUnidata Program CenterUCAR Office of Programs
Boulder, CO
CRAFT and Unidata: A Natural CRAFT and Unidata: A Natural PartnershipPartnership
![Page 2: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/2.jpg)
UnidataUnidataA Program in UCAR*A Program in UCAR*
Unidata Mission Statement:
Provide data, tools, and community leadership for enhanced Earth-system education and research.
At the Unidata Program Center, we
• Facilitate [Real-time] Data AccessFacilitate [Real-time] Data Access
• Provide ToolsProvide Tools
• Support Faculty and StaffSupport Faculty and Staff
• Build and Advocate for a CommunityBuild and Advocate for a Community
*University Corporation for Atmospheric Research (UCAR) is a nonprofit consortium of 68 universities committed to advancing our understanding of atmosphere and other Earth systems.
![Page 3: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/3.jpg)
Internet Data DistributionInternet Data Distribution(IDD)(IDD)
Source
LDM
Source
Source
LDM LDM
LDMLDM
LDM LDM
LDM
LDM
Internet
Radar
Model
Satellite
About 150+ sites are participating in Unidata Internet Data Distribution (IDD) system
![Page 4: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/4.jpg)
Internet2 Traffic Internet2 Traffic (Week of 4/5/04)(Week of 4/5/04)
ApplicationApplication OctetsOctets OctetsOctets
1)1) IperfIperf 21.43%21.43% 102.3TB102.3TB
2)2) NNTPNNTP 8.88%8.88% 42.42TB42.42TB
3)3) HTTPHTTP 7.80%7.80% 37.24TB37.24TB
4)4) FTPFTP 3.29%3.29% 15.73TB15.73TB
5)5) BitTorrentBitTorrent 8.14%8.14% 38.87TB38.87TB
6)6) ShoutcastShoutcast 2.73%2.73% 13.04TB13.04TB
7)7) LDMLDM 2.02%2.02% 9.639TB9.639TB
Unidata IDD/LDM uses more of the Internet2 than any other advanced application;
![Page 5: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/5.jpg)
The Unidata LDMThe Unidata LDM
The Local Data Manager (LDM)The Local Data Manager (LDM)• is a collection of cooperating programs that is a collection of cooperating programs that
select, capture, manage, and distribute arbitrary select, capture, manage, and distribute arbitrary data products. data products.
• is designed for event-driven data distribution, and is designed for event-driven data distribution, and is currently used in is currently used in Internet Data Distribution (IDD).
• includes network client and server programs and includes network client and server programs and their shared protocols. their shared protocols.
• supports flexible, site-specific configuration, supports flexible, site-specific configuration, multiple sources of data products, and user-multiple sources of data products, and user-customizable actions on received data productscustomizable actions on received data products
![Page 6: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/6.jpg)
Data-Feed TopologyData-Feed Topology
SourceLDM
LDM
LDM LDM
LDM LDM
Primary Feed
Secondary Feed
… …
(Relay node)(Relay node)(Relay node)(Relay node)
(Leaf node)(Leaf node) (Relay node)(Relay node) (Leaf node)(Leaf node)
![Page 7: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/7.jpg)
Real-time data distribution via IDD/LDM6
Unidata Unidata community community now extends now extends internationally internationally on several on several continentscontinents
![Page 8: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/8.jpg)
Technology Transfer: Operational LDM Use in the NWS
Recently, the Korean Meteorological Administration has started using the LDM for some of their internal data distribution to/from nearly 40 weather service offices.
![Page 9: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/9.jpg)
IDD TopologiesIDD Topologies
Lightning dataLightning data
Radar dataRadar data
Satellite dataSatellite data
Surface/Upper-air dataSurface/Upper-air data
![Page 10: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/10.jpg)
SuomiNet and LDMSuomiNet and LDM
A network of GPS receivers to provide real-time atmospheric precipitable water vapor measurements and other geodetic and meteorological information
SuomiNet collects data from 100+ GPS receivers distributed throughout the world.
The observations are sent to Boulder, CO for processing and analysis and then redistributed to the community using the LDM.
![Page 11: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/11.jpg)
Generic LDM InstallationGeneric LDM Installation
ProductQueue
IngesterLDMServer
SendingLDM
ReceivingLDM
pqact DecoderExecution
Data Flowpqact
![Page 12: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/12.jpg)
LDM SpecificsLDM Specifics Uses registered port 388Uses registered port 388 Uses TCP and ONC RPCUses TCP and ONC RPC Actions driven by configuration-fileActions driven by configuration-file Arbitrary data of arbitrary size (0 – 2Arbitrary data of arbitrary size (0 – 23232-1 bytes)-1 bytes) One process per downstream (upstream) LDMOne process per downstream (upstream) LDM MetadataMetadata
• Size of data in bytesSize of data in bytes• Creation timeCreation time• Creation hostnameCreation hostname• Feed-type (32 bit integer)Feed-type (32 bit integer)• Product identifier (255 byte string)Product identifier (255 byte string)• MD5 checksumMD5 checksum
![Page 13: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/13.jpg)
LDM StrengthsLDM Strengths Robust operation in the face of network Robust operation in the face of network
congestion and outagescongestion and outages EfficientEfficient
• Event drivenEvent driven• Capable of transmitting 2 TB/d from a well-connected, Capable of transmitting 2 TB/d from a well-connected,
upstream hostupstream host Highly user-configurableHighly user-configurable
• Authentication of feed requestsAuthentication of feed requests• Requested productsRequested products• Processing of received productsProcessing of received products
Proven technology: operational since 1994Proven technology: operational since 1994
![Page 14: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/14.jpg)
LDM LimitationsLDM Limitations
Current implementation is UNIX-onlyCurrent implementation is UNIX-only Data-product availability is limited by size Data-product availability is limited by size
of product-queue on upstream LDMof product-queue on upstream LDM Upstream sites are fixed for duration of Upstream sites are fixed for duration of
LDM session (i.e., static routing)LDM session (i.e., static routing) Performance over unreliable connection is Performance over unreliable connection is
limited by inherent receive-timeouts of limited by inherent receive-timeouts of ONC RPCONC RPC
Adaptability is limited by single-threaded, Adaptability is limited by single-threaded, multiple-process implementationmultiple-process implementation
![Page 15: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/15.jpg)
LDM-6 UpgradeLDM-6 Upgrade
Use TCP guaranteed delivery instead of Use TCP guaranteed delivery instead of RPCsRPCs
User-selectable product “chunking”User-selectable product “chunking” Real-time statistics gathering and displayReal-time statistics gathering and display Code cleanup for greater efficiencyCode cleanup for greater efficiency
![Page 16: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/16.jpg)
LDM-6 Performance LDM-6 Performance Model Data: Highest Volume DatastreamModel Data: Highest Volume Datastream
U. AlbanyU. Albany U. IllinoisU. Illinois
U. WashingtonU. Washington U. UtahU. Utah
![Page 17: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/17.jpg)
NLDM Benefits Over LDM Flooding algorithm: Flooding algorithm:
• automated routing through redundancyautomated routing through redundancy Virtually unlimited number of hierarchically structured Virtually unlimited number of hierarchically structured
newsgroups: newsgroups: • finer granularity of product categorization and subscriptionfiner granularity of product categorization and subscription
Cross posting: Cross posting: • Multiple “views” of same data productsMultiple “views” of same data products
Multiple numbers and types of storage buffersMultiple numbers and types of storage buffers Dynamic construction and destruction of connectionsDynamic construction and destruction of connections Backlog handlingBacklog handling Protocol supports both push and pull transmissionProtocol supports both push and pull transmission
NLDM:News Server Technologyto Relay Data in Near Real Time
![Page 18: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/18.jpg)
NLDM ResultsNLDM Results
INN used “out of the box” with one modification:INN used “out of the box” with one modification:• Message ID generation and handling modified to use product Message ID generation and handling modified to use product
signaturessignatures Default approach is to create message ID based on host nameDefault approach is to create message ID based on host name Doesn't allow duplicate detection based on product contentDoesn't allow duplicate detection based on product content
Robust delivery with latencies comparable to current LDMRobust delivery with latencies comparable to current LDM• May be improved further with additional tuningMay be improved further with additional tuning
Local management functionality comparable to current LDMLocal management functionality comparable to current LDM Robust automated, dynamic routingRobust automated, dynamic routing Automated connection managementAutomated connection management
![Page 19: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/19.jpg)
NLDM Routing StatisticsCONDUIT, Boulder to D.C.
● Isabel hit Washington, D.C.Two day window, 30 second bin size
Average Latencies
Maximum Latencies Boulder → U.Oregon.2 → D.C.
Boulder → U.Oregon.1 → D.C.
Direct path: Boulder → D.C.
![Page 20: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/20.jpg)
MeteoForumMeteoForum
MeteoForum NetworkMeteoForum Network
AMPATHAMPATH
CLARACLARA
Buenos Aires
Belem
San Jose
Rio de Janeiro
Caracas
Bridgetown
MeteoForum is a joint project between Unidata and COMET, two programs in UCAR
![Page 21: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/21.jpg)
IDD-BrazilIDD-Brazil Connects to Unidata IDD Connects to Unidata IDD
Current ParticipantsCurrent Participants• UnidataUnidata• University of MiamiUniversity of Miami• UFRJ (Brazil)UFRJ (Brazil)• UFPA (Brazil)UFPA (Brazil)• CPTEC/INPE (Brazil)CPTEC/INPE (Brazil)• USP (Brazil)USP (Brazil)
InfrastructureInfrastructure• Internet2Internet2• AMPATH (NSF)AMPATH (NSF)• FIUFIU• Global CrossingGlobal Crossing• RNPRNP• ANSPANSP
![Page 22: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/22.jpg)
IDD-BrazilIDD-Brazil - Stress Testing - Stress Testing
IDD relay of all open feeds to UFRJIDD relay of all open feeds to UFRJ• Sustained 1.6 GB/hr over 10 day period at end of Sustained 1.6 GB/hr over 10 day period at end of
DecemberDecember• Peak rates routinely over 2.3 GB/hr over same periodPeak rates routinely over 2.3 GB/hr over same period• Typical latencies range from 1 to several secondsTypical latencies range from 1 to several seconds• Impact on relay machine negligibleImpact on relay machine negligible
![Page 23: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/23.jpg)
Visualization SoftwareVisualization SoftwareGEMPAK, IDV, McIDASGEMPAK, IDV, McIDAS
![Page 24: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/24.jpg)
Shaping the Future of Data Use in the Geosciences
We are moving from an era of data provision towards one in which data- and related web-services are important;
Multidisciplinary integration and synthesis are emphasized.
![Page 25: Internet 2 Members Meeting 20 April 2004 Arlington, VA Mohan Ramamurthy Unidata Program Center UCAR Office of Programs Boulder, CO CRAFT and Unidata: A](https://reader036.vdocuments.us/reader036/viewer/2022081519/56649e845503460f94b8576c/html5/thumbnails/25.jpg)
User applications: e.g., McIDAS, IDV,
LAS, IDL, MatLab...
DLESEDigital Library for
Earth-System Education
HydrologyData, e.g.
GeophysicalData, e.g.
Satellite Images, e.g.Satellite
Images, e.g.Satellite
Images, e.g.Satellite
Imagery...
OpenDAP, ADDE, &
FTP protocols
IDD
DLinterchange
protocol
IDD
Discovery
IDD IDD IDD
User applications: e.g., McIDAS, IDV,
LAS, IDL, MatLab...
DLESEDigital Library for
Earth-System Education
HydrologyData, e.g.
GeophysicalData, e.g.
Satellite Images, e.g.Satellite
Images, e.g.Satellite
Images, e.g.Satellite
Imagery...
OpenDAP, ADDE, &
FTP protocols
IDD
DLinterchange
protocol
IDD
Discovery
IDD IDD IDD
PeoplePeople
DocumentsDocuments DataData
Catalog
Generation Tools
Analysis andVisualization Tools
Data Services
Discovery andPublication Tools
Discovery and Publication Services
Dat
a C
atal
ogS
ervi
ces
PeoplePeople
DocumentsDocuments DataData
Catalog
Generation Tools
Analysis andVisualization Tools
Data Services
Discovery andPublication Tools
Discovery and Publication Services
Dat
a C
atal
ogS
ervi
ces
Thematic Real-time Environmental Distributed Data Servers (THREDDS)
• Combines IDD “push” with several forms of “pull” and DL discovery
• About 25 data providers are partners in THREDDS
• To make it possible to publish, locate, analyze, visualize, and integrate a variety of environmental data
• Connecting People with Documents and Data
THREDDSMiddleware