the usace datanet · pdf file2/15/2005 denise martin 1 the usace datanet providing efficient...
Post on 13-Mar-2018
219 Views
Preview:
TRANSCRIPT
2/15/20052/15/2005 Denise MartinDenise Martin 11
The USACE The USACE DataNetDataNet
Providing efficient data access Providing efficient data access through a netthrough a net--centric frameworkcentric framework
2/15/20052/15/2005 22
AuthorsAuthors
Denise MartinDenise MartinJames StinsonJames StinsonDavid StuartDavid StuartCary ButlerCary Butler
2/15/20052/15/2005 33
AbstractAbstract
The U.S. Army Corps of Engineers (USACE) relies on interactive The U.S. Army Corps of Engineers (USACE) relies on interactive computercomputer--based systems to identify and assess alternatives, make based systems to identify and assess alternatives, make decisions, and solve problems. Data is the principle component decisions, and solve problems. Data is the principle component that drives the decision making process. Much of the data needethat drives the decision making process. Much of the data needed d to support USACE are available from other federal agencies. to support USACE are available from other federal agencies. Acquisition and use of these data is often inefficient and Acquisition and use of these data is often inefficient and inconsistent. Moreover, data are provided in a myriad of disparinconsistent. Moreover, data are provided in a myriad of disparate ate formats and structures while the applications that consume theseformats and structures while the applications that consume thesedata require differing formats as well. The goal of the USACE data require differing formats as well. The goal of the USACE DataNetDataNet is to streamline the acquisition and dissemination of data is to streamline the acquisition and dissemination of data across all USACE business areas. This presentation describes 1)across all USACE business areas. This presentation describes 1)the development and implementation of a common netthe development and implementation of a common net--centric centric framework that provides a consistent interface to data sources framework that provides a consistent interface to data sources internal and external to USACE, and 2) client applications that internal and external to USACE, and 2) client applications that use use the the DataNetDataNet for data acquisition and delivery, including ArcGIS for data acquisition and delivery, including ArcGIS extensions.extensions.
2/15/20052/15/2005 44
ProblemProblem
USACE uses a multitude of heterogeneous USACE uses a multitude of heterogeneous computerized systems to assist in decisioncomputerized systems to assist in decision--making. making. Data is the principle component that drives Data is the principle component that drives the decisionthe decision--making process.making process.The efficient handling of data is critical in The efficient handling of data is critical in making timely, costmaking timely, cost--effective decisions.effective decisions.S&E spend more time acquiring, S&E spend more time acquiring, manipulating, and organizing data than manipulating, and organizing data than performing analyses to support decisions.performing analyses to support decisions.
2/15/20052/15/2005 55
Scientific work flow Scientific work flow
~70%time
~30%time
Distributed Heterogeneous Data Sources
Locate, Assemble and
Load Data
Scientific Simulations and Experiments
Scientific Analysis and Discovery
Data Preprocessing
Locate, Assemble and
Load Data
Scientific Simulations and Experiments
Scientific Analysis and Discovery
~70%time
~15%time
Distributed Heterogeneous Data Sources
Data Preprocessing
~15%time
70-15-15 Rule70-15-15 Rule
2/15/20052/15/2005 66
Problem Problem (continued)(continued)
Decentralized computing across Decentralized computing across USACE resulted in vertically focused USACE resulted in vertically focused technology/information.technology/information.How can technology assist in How can technology assist in integrating disparate heterogeneous integrating disparate heterogeneous information and making it available to information and making it available to employees, customers, decision employees, customers, decision makers?makers?
2/15/20052/15/2005 77
Guiding principles for managing dataGuiding principles for managing data
Avoid duplication in data acquisition. Share data Avoid duplication in data acquisition. Share data wherever possible via networks and partnerships.wherever possible via networks and partnerships.Look for existing datasets before performing data Look for existing datasets before performing data collection.collection.Adhere to government and industry data content, Adhere to government and industry data content, access, and delivery standards.access, and delivery standards.Manage data to maximize its use by multiple Manage data to maximize its use by multiple processes.processes.Manage data at the owner level and negotiate access Manage data at the owner level and negotiate access arrangements.arrangements.Require the use of metadata for every dataset.Require the use of metadata for every dataset.
- Natural Resources Information Mgmt ToolkitConcise Guide for Technical Managers
2/15/20052/15/2005 88
RequirementsRequirements
Develop, promote, and deploy a Develop, promote, and deploy a common netcommon net--centric framework that centric framework that provides a consistent interface to data provides a consistent interface to data sources internal and external to sources internal and external to USACEUSACEFormalize connectivity to Formalize connectivity to USACEUSACE’’ss data data sharing partnerssharing partners
2/15/20052/15/2005 99
SolutionSolution
The The DataNetDataNet employs a netemploys a net--centric centric approach to streamline and approach to streamline and standardize the acquisition, standardize the acquisition, dissemination, and management of dissemination, and management of data across all USACE business areas.data across all USACE business areas.
2/15/20052/15/2005 1010
DataNetDataNet FrameworkFrameworkInteragency
ProvisioningReplicate Warehouse Proxy
Convert TransformIntegration
USACE applications
Sec
urity
Man
agem
ent
Infr
astr
uctu
re
Sec
urity
Man
agem
ent
Infr
astr
uctu
re
Data Sources
Databases
Accessibility
Fuse/Aggregate
Intranet Internet Extranet LAN
Standard API
GIS
tableImages
Intraagency
Met
adat
a
2/15/20052/15/2005 1111
ServiceService--oriented oriented architecturearchitecture
ServiceProvider
ServiceRequestor
ServiceRegistry
Publish Interact
FindClient
SOAPService
WSDL
UDDISLA
2/15/20052/15/2005 1212
Our Web Service VisionOur Web Service Vision
Enable the Enable the use of distributed resourcesuse of distributed resources. . –– From this we expect to see fundamental changes in From this we expect to see fundamental changes in
how scientists and engineers access powerful how scientists and engineers access powerful computing systems, multicomputing systems, multi--scale data archives, scale data archives, scientific instruments, and collaboration tools.scientific instruments, and collaboration tools.
Provide services that are Provide services that are integrated with the integrated with the useruser’’s work environments work environment–– and that provide uniform and highly capable access and that provide uniform and highly capable access
to computers, data, and instruments, regardless of to computers, data, and instruments, regardless of the locations or exact nature of these resourcesthe locations or exact nature of these resources
2/15/20052/15/2005 1313
WebWeb--Based Integrated Based Integrated Development Environment Development Environment (WIDE)(WIDE)
Business Partners
Gov’t/Industry Services
Gov’t/IndustryStandards
Developers
CDFTeam
Configuration Management
DevelopmentEnvironment
S&E Requirements
RequirementsTechnology &
Services
Repository(Operational)
Web Farm MonitoringHosting
Monitor
Guidance/Feedback Transition
Web Framework External Forces
Operations
Users
ApplicationBuild
Leverage
2/15/20052/15/2005 1414
WebWeb--centric Implementation centric Implementation
Web services technology offers a platform neutral approach for integrating applications across four domains:
• Internal to USACE• Government to
Government• Government to
Business• Government to
Citizen
ArcInfoArcInfo XMSXMSOKCOKC
Tools and Models
USACE
Industry Federal and State PartnersFederal and State Partners
USGSUSGS
Geospatial One-Stop
Geospatial One-Stop
NOAANOAA
USGS LULC dataUSGS LULC dataLandUseLandUse
USDA STATSGOUSDA STATSGOSoilsSoils
USACE NIDUSACE NIDInfrastructureInfrastructure
CWMS CWMS dssdss datadata
CWMS CWMS RealtimeRealtime Gage dataGage data
USGS Stream FlowUSGS Stream FlowHydroHydro
NCDC PrecipitationNCDC Precipitation
METAR current surface METAR current surface condcond..
Univ. of Utah Univ. of Utah MesoMeso West West Precipitation/WeatherPrecipitation/Weather
WeatherWeather
NOAA Estuarine BathymetryNOAA Estuarine Bathymetry
USGS NEDUSGS NEDElevationElevation
Data SourceData SourceCategoryCategory
2/15/20052/15/2005 1515
AccessibilityAccessibility
•. Centralized and secure Web development and Web hosting environment that serves as the operational platform for all local DataNet services.
Web Farm Approach•Centralizes interactions with DREN and CEEIS•Provides 24x7x365 operational capabilities –higher availability•Conducts backups•Provides connectivity to Internet, Intranet, and Extranet•Supports both Windows and Unix•Monitors resource access•Provides DoD certified computing environment•Shares cost among all users•Recycles hardware every 3 years
2/15/20052/15/2005 1616
MetadataMetadata
Technical Technical documentationdocumentationService RegistryService Registry
2/15/20052/15/2005 1717
Accessibility
DataNetDataNet Framework Framework --ApplicationsApplications
COTSCOTS
desktopdesktop
WebWeb--basedbased
HPC Systems
The challenge is to develop a framework that will support data accessibility by all of these applications.
A common approach toaccessing
and sharingdata – “One Data World”
2/15/20052/15/2005 1818
Locating and assembling dataLocating and assembling dataLocating and assembling data
USGSUSGS
EPAEPA
USDAUSDA
NationalNationalWeatherWeatherServiceService
Browse1
Assemble2
ESRIESRI
Download3
SEDB is a common gateway interface that serves data from a variety of information sources and scientific formats. SEDB accesses these sources via the DataNet.
2/15/20052/15/2005 1919
CDF Connection to ArcGISCDF Connection to ArcGIS
ArcGIS Application
S&E Toolbar provides simple interface to DataNet Services
2/15/20052/15/2005 2020
CDF Connection to ArcGISCDF Connection to ArcGISCDF Web ServiceTo EPA STORETdata
2/15/20052/15/2005 2121
Web Applications
connect to data via CDF Services
2/15/20052/15/2005 2222
Legacy applications
connect to data via CDF Services
2/15/20052/15/2005 2323
SummarySummary
The The DataNetDataNet framework:framework:–– avoids duplication in data acquisitionavoids duplication in data acquisition–– facilitates the sharing of data, both internal and facilitates the sharing of data, both internal and
external, via networks and partnershipsexternal, via networks and partnerships–– adheres to standardsadheres to standards–– promotes ownerpromotes owner--level management and servicelevel management and service--
level agreementslevel agreements–– requires metadata for data and servicesrequires metadata for data and services–– is accessible by distributed, heterogeneous is accessible by distributed, heterogeneous
applicationsapplications
Consistent with the basic guiding principles Consistent with the basic guiding principles of data managementof data management
2/15/20052/15/2005 2424
SummarySummary
The The DataNetDataNet provides a oneprovides a one--stopstop--shop for shop for data acquisition from within USACE, from data acquisition from within USACE, from other federal and state agencies, and from other federal and state agencies, and from industry.industry.S&E applications can access heterogeneous S&E applications can access heterogeneous data sources via the data sources via the DataNetDataNet in a timely and in a timely and secure manner. secure manner. The The DataNetDataNet framework truly provides a framework truly provides a standardsstandards--based realization of the servicebased realization of the service--oriented computing paradigm. oriented computing paradigm.
2/15/20052/15/2005 2525
Scientific work flow Scientific work flow improvementimprovement
Locate, Assemble and
Load Data
Scientific Simulations and Experiments
Scientific Analysis and Discovery
~70%time
~15%time
Distributed Heterogeneous Data Sources
Data Preprocessing
~15%time
~43%time
~42%time
DataNet
Locate, Assemble and
Load Data
Scientific Simulations and Experiments
Scientific Analysis and Discovery
Data Preprocessing ~15%
time
2/15/20052/15/2005 2626
QuestionsQuestions
Denise MartinDenise MartinAssociate Technical DirectorAssociate Technical DirectorInformation Technology LaboratoryInformation Technology LaboratoryEngineer Research & Development CenterEngineer Research & Development CenterU.S. Army Corps of EngineersU.S. Army Corps of Engineers3909 Halls Ferry Rd.3909 Halls Ferry Rd.Vicksburg, MS 39180Vicksburg, MS 39180--61996199Phone: (601) 634Phone: (601) 634--45744574Fax: (601) 634Fax: (601) 634--26382638Email: Email: denise.b.martin@erdc.usace.army.mildenise.b.martin@erdc.usace.army.mil
top related