unidata 2008: shaping the future of data use in the geosciences expanding horizons: using...
Post on 28-Dec-2015
217 Views
Preview:
TRANSCRIPT
Unidata 2008:Shaping the Future of Data Use in the
Geosciences
Expanding Horizons: Using Environmental Data for Education, Research, and Decision Making
23 June 200323 June 2003Boulder, COBoulder, CO
Mohan RamamurthyMohan RamamurthyUnidata Program CenterUnidata Program CenterUCAR Office of ProgramsUCAR Office of Programs
Boulder, COBoulder, CO
Thank you, Dave
I wish to take this opportunity to extend my sincerest gratitude to Dave Fulker, the founding director of Unidata, for his distinguished service to the Unidata Community for nearly 20 years.
Unidata would not be what it is today without his vision, leadership, energy and his many extraordinary qualities.
And thank you to Ben Domenico for his excellent stewardship during the transition.
The Word of the Day for Jun 23rd is
The Word of the Day for Jun 23 is:
bloviate \BLOH-vee-ayt\ verb
: to speak or write verbosely and windily
[Courtesy: Jo Hansen, Unidata Program Center]
Example sentence:
Mohan can bloviate on a par with the windiest of professors, but he's also capable of being concise and getting right to the point. (yeah, right)
Expanding Horizons
New Strategic Plan New Director New 5-year proposal Many new and
exciting initiatives New logo!
UnidataUnidata
Mission Statement:Mission Statement:
Provide data, tools, and community leadership for enhanced Earth-system education and research.
At the Unidata Program Center, we
• Facilitate Data AccessFacilitate Data Access
• Provide ToolsProvide Tools
• Support Faculty and StaffSupport Faculty and Staff
• Build and Advocate for a Community Build and Advocate for a Community user user workshops come under this activityworkshops come under this activity
Technology PortfolioTechnology Portfolio
1) McIDAS: A client/server analysis and display package, originally developed by U. Wisconsin/SSEC, that emphasizes image processing of data from satellite-borne sensors;
2) GEMPAK: An analysis, display, and product generation package for meteorological data;
3) Local Data Manager: Software for capturing, disseminating, and organizing data in near-real time; It is the heart of the Internet Data Distribution (IDD) system;
4) NetCDF: A software interface for platform-independent access to self describing datasets;
5) Integrated Data Viewer: Java-based, platform-independent data analysis and 3D visualization tools;
6) THREDDS: A project to facilitate remote access to thematic, distributed, interdisciplinary data servers;
Unidata as a Diverse CommunityUnidata as a Diverse Community
About 150+ sites are participating in About 150+ sites are participating in Unidata Internet Data Distribution Unidata Internet Data Distribution (IDD) system(IDD) system• 120 or so of those sites are in academia 120 or so of those sites are in academia
and the rest in government and and the rest in government and research labsresearch labs
User community is interdisciplinary - 2/3rd of sites have users outside atmospheric sciences
Internet Data DistributionInternet Data Distribution
Source
LDM
Source
Source
LDM LDM
LDMLDM
LDM LDM
LDM
LDM
Internet
Radar
Model
Satellite
Approximately 2 GB of data injected/hour from distributed sources;
Unidata IDD/LDM uses more of the Internet2 than any other advanced application;
Approx. 5 Terabytes of data transmitted each week. (Amount varies with weather)
By design, the system has no data center.
Education Drivers (a.k.a. A Community-Articulated Need)
Active, student-centered learning
Earth-system science or “holistic” approach to education
Learning science by doing science• Observations (data)• Tools (models,
visualization)• Discovery
NSF Director Rita Colwell, 1998: "Interdisciplinary connections are absolutely fundamental. They are synapses in this new capability to look over and beyond the horizon. Interfaces of the sciences are where the excitement will be the most intense... ."
Science DriversScience Drivers
Grand Challenges in Environmental Sciences National Research Council
Multidisciplinary ProblemsMultidisciplinary Problems
Fire Danger determination requires taking into account past, present and future weather, fuel types, and the state of both live and dead fuel moisture.• Dead Fuel Moisture• Live fuel moisture (NDVI)Live fuel moisture (NDVI)• Drought conditionsDrought conditions• Atmospheric stabilityAtmospheric stability• Lightning mapsLightning maps• Lightning ignition efficiencyLightning ignition efficiency• AirflowAirflow• Recent rainfallRecent rainfall• Rainfall forecastRainfall forecast
Dual-Polarization Radar use in Fire Dual-Polarization Radar use in Fire Weather ManagementWeather Management
The differential The differential reflectivity (ZDR) reflectivity (ZDR) values are values are noteworthy in the noteworthy in the smoke signalsmoke signal
Many regions show Many regions show ZDR >+6 dB.ZDR >+6 dB.
Suggests flattened Suggests flattened ash particles (like ash particles (like corn flakes)corn flakes)
Source: CHILL Radar Group, CSU
Flooding due to Tropical Storms Flooding due to Tropical Storms Tropical Storm Allison
Research studies and emergency management of hurricane-induced flooding involve integrating data from atmospheric sciences, oceanography, hydrology, geology, geography, and social sciences.
Multidisciplinary SynthesisMultidisciplinary Synthesis Requires integration of disparate datasets and Requires integration of disparate datasets and
databases from diverse sources that are databases from diverse sources that are distributed geographically and disciplinarily;distributed geographically and disciplinarily;
Needs integration of Scientific Information Needs integration of Scientific Information Systems with Geographic Information SystemsSystems with Geographic Information Systems
The integration poses numerous challenges;The integration poses numerous challenges; However, such integration is critical to solving However, such integration is critical to solving
societal problems and advancing science.societal problems and advancing science. Metadata is crucial to achieving integrationMetadata is crucial to achieving integration
Remote Sensing & Data Explosion In the next 10 years, about 100 new
satellite instruments will be launched to monitor the environment
Five-order magnitude increase in satellite data is expected during that period• GIFTS (Geostationary Imaging Fourier
Transform Spectrometer) will have about 1700 channels and a resolution of 4 km
• Each NPOESS satellite will generate one terabyte of data each day
Advances in Radar technology• 28 fold increase in WSR-88D data
volume in 5 years• Phased-array radars will generate 100
fold increase data
By 2004, NOAA will ingest more data in one year than was contained in the total archive in 1998.
Advances in ModelingAdvances in Modeling
Shift from a purely deterministic to a more probabilistic approach, requiring the use of ensemble modeling techniques.
Growing emphasis on multidisciplinary studies, requiring coupled models:
• e.g., Hurricane landfall flooding problem: Atmospheric model (WRF/MM5), Ocean model (ROMS), Hydrologic model (HMS)
Local Modeling: A Notable Trend
Over 30 universities are now running mesoscale models locally.
One can think of this aggregation as a national forecasting instrument
However, only one or two groups initializing their model runs with local observations
As the scale of these local model runs becomes finer, there is a natural desire to integrate their output with information from other sources (e.g., hydrology, infrastructure, societal datasets in GIS form)
Iowa St. Linux Cluster
Technology DriversTechnology Drivers
Object-oriented programmingObject-oriented programming Open Standards, Interoperability and Open Source MovementOpen Standards, Interoperability and Open Source Movement
• Metcalfe's Law: the usefulness, or utility, of a network increases as the square of the number of users. .
Web services (HTTP, Java, XML, SOAP, UDDI, …)Web services (HTTP, Java, XML, SOAP, UDDI, …) Digital libraries (Metadata, discovery, information services…)Digital libraries (Metadata, discovery, information services…) Grid environments and distributed computingGrid environments and distributed computing Commodity microprocessorsCommodity microprocessors Cluster computingCluster computing High bandwidth networks: 10GigE, Fast IP, …High bandwidth networks: 10GigE, Fast IP, … Broadband accessBroadband access Wireless networks: 802.11 networks, GPRS, 3GWireless networks: 802.11 networks, GPRS, 3G IPv6: Next-generation internet protocolIPv6: Next-generation internet protocol Collaborative computingCollaborative computing Scientific data mining and knowledge discoveryScientific data mining and knowledge discovery
Users
Collections (Data, tools, educational materials)
Metadata repository
Services
Users
Collections (Data, tools, educational materials)
Metadata repository
Services
Web services is a technology and process for discovery and connection.
The eXtended Markup Language, XML, is accepted as THE emerging standard for data interchange on the Web.
XML allows authors to create their own markup, which has led to the proliferation of “MyOwn Markup Language”
Web Services and the Wild and Wooly World of Markup Languages
Title: Unidata 2008: Shaping the Future of Data Use in the Geosciences
Six endeavors are proposed, focusing on Community and Support Services and Data Services, Systems, and Tools
The proposed endeavors will enable the community to advance scientific exploration, education, and decision-making.
We are moving from an era of data provision to one in which data- and related web-services are emphasized
Five-year Core Funding NSF ProposalFive-year Core Funding NSF Proposal
“The unanimous finding of the panel is that the Unidata Program Center program be supported as fully as possible by NSF for the years 2003-2008.”
Proposed EndeavorsProposed Endeavors
Endeavor 1.Endeavor 1. Responding to a broader and more diverse community.• Respond to increased emphasis on Earth-system
science (e.g., bring new data sets to the community)• Establish new partnerships with related communities
(e.g. with Hydrology via CUAHSI)• Support new tools in technically less-sophisticated
institutions (e.g., community colleges)
Endeavor 2Endeavor 2.. Comprehensive support services • Deploy web-based training modules Deploy web-based training modules • Simplify installation and maintenance for all supported Simplify installation and maintenance for all supported
packages packages • Explore new technologies (e.g., Access Grid) to facilitate Explore new technologies (e.g., Access Grid) to facilitate
remote collaborationremote collaboration
Endeavor 3: Endeavor 3: Real-time, self-managing Real-time, self-managing data flowsdata flows
More flexibility and control• Many more feed types for finer control over routing and subsetting• Configurable product priorities
Self-managing data flows (automatic dynamic routing)• Application-level multicast looks promising for hundreds of sites (IP multicast not suitable due to limitations)• NLDM: data flooding via Usenet protocols may provide practical routing solution (needs more testing)
Support for new standards• Use of IP version 6 protocols • Internet2, Grid and e-services standards (authentication, resource use, ...) • Location-transparency for data
LDM-5 Vs. LDM-6 LatenciesLDM-5 Vs. LDM-6 Latencies
CONDUIT ExperienceAverage delivery time : ~20 seconds to top-tier sites
Endeavor 4. Software to analyze and visualize geoscience data
Integrate diverse datasets Support analysis and
visualization of local and climate modeling efforts
Develop collaborative tools to make effective use of shared visualizations
Allow customized user experiences
Adapt to GIS frameworks
– Cloud water isosurface from COMMAS storm model data Cloud water isosurface from COMMAS storm model data
(courtesy Adam Houston and Dan Bramer, NCSA/UIUC)(courtesy Adam Houston and Dan Bramer, NCSA/UIUC)
PeoplePeople
DocumentsDocuments DataDataC
atalog
Generation Tools
Analysis andVisualization Tools
Data Services
Discovery andPublication Tools
Discovery and Publication Services
Dat
a C
atal
ogS
ervi
ces
THREDDSTHREDDSMiddlewareMiddleware
THREDDS, GIS, DL InteroperabilityTHREDDS, GIS, DL InteroperabilityGIS Client
ApplicationsTHREDDS Client
Applications
OpenGIS Protocols:WMS, WFS, WCS
OGC or proprietary GIS
protocols
OGC or OPeNDAPADDE. FTP…
protocols
GIS ServerGIS ServerGIS Servers
Demographic, infrastructure, societal impacts, …
datasets
THREDDS ServerTHREDDS Server
THREDDS ServersSatellite, radar,
forecast model output, … datasets
Digital Library Discovery Systems
Metadatacrosswalk
Open Archives Initiative (OAI) Metadata Harvesting
Metadatacrosswalk
Current Implementation
netCDF
Application
Parallel file
system
Proposed Implementation
HDF5 (serial and/or parallel)
netCDF
Application
Network or to/from another
application
Stream
POSIX I/O
File Meta-data
Rawdata
User-defined device
CustomMPI-
I/O
Split files
POSIX I/O
File
NetCDF-HDF IntegrationNetCDF-HDF Integration
Extend netCDF to high-performance computing environment
Implement parallel I/O, large grids, etc.
Work will directly benefit WRF and CCSM communities
Endeavor 6: Improved data access infrastructure
The Visual Geophysical Exploration The Visual Geophysical Exploration Environment (VGEE)Environment (VGEE)
The VGEE is an integrated framework in which students use visualization tools, data, and curricular materials to learn basic physical principles of atmospheric science
It includes:• A learner interface to the IDVA learner interface to the IDV• Java-based concept models to Java-based concept models to
support physical insightsupport physical insight• A curriculum to guide inquiryA curriculum to guide inquiry• A catalog of data (THREDDS)A catalog of data (THREDDS)
Students notice that the Western Pacific is considerably warmer than the East.
Identify RelateExplainIntegrate
VGEE: An Integrated FrameworkVGEE: An Integrated Framework
Concept Models, which are used to explore relations in an idealized context.
Concluding Remarks
We live in an exciting moment in the history of the Earth sciences.
Workshops like this and the diversity of representation from academia are testimony to the vibrancy of the community and the program.
The portfolio of tools and technologies within Unidata, coupled with the energies of a creative and collaborative community, puts us in an ideal position to meet the important challenges facing the education and research communities in the atmospheric and related sciences.
top related