NOAA Challenges in

Environmental Data and Information Management Prepared for National Research Council

Board on Research Data and Information

Deirdre JonesNOAA’s National Weather Service and

Deputy Chair, Environmental Data Management Committee

November 30, 2010

Prepared by Environmental Data Management Committee for BRDI 2

Outline

Background Recent High Visibility Drivers• Ocean Policy Task Force Recommendations• NOAA Climate Service• Deep Water Horizon Incident

Near Term Plans• Data Management Plan Guidance• Improve Data Documentation• Archive and Architecture ConOps• Data Access

Summary of FY 2011 Activities Next Steps

Broad Scope for Environmental Data Stewardship•~150 Research & Operational Observing Systems•~4-5 Petabytes of data/year (~15 Pb total)

Prepared by Environmental Data Management Committee for BRDI 3

Data Management Challenges are Changing• No longer just about data volume• Data discovery and integration• Data stewardship and information

NOAA’s Environmental Information Management Challenges

Background

2

Ocean Policy Task Force

Prepared by Environmental Data Management Committee for BRDI 4

9

ACTIONS: Data and Tools Theme Team1.Develop and implement a comprehensive strategy for identifying and integrating relevant NOAA data sets to support regional Coastal and Marine Spatial planning.2.Contribute to the design and creation of the interagency NIMS.3.Work with NOAA and partners to assess, enhance and provide access to robust and flexible analytical tools to evaluate alternative ocean use scenarios.

ACTIONS:1.Develop source to repository data management system.2.Adopt and implement common standards for data transport, metadata, data models, and data discovery.

NOAA Climate Service

Prepared by Environmental Data Management Committee for BRDI 5

NCS Core Capability 1: Observing Systems, Data Stewardship, and Monitoring

•NCS users will obtain easy and timely access to the nation’s trusted data and information about the current state of the climate system in context with the past.

Deep Water Horizon Lessons Learned

Prepared by Environmental Data Management Committee for BRDI 6

Pre-incident (routine) data management challenges

Post-incident data management challenges

• Volume and diversity of data

• Collected for one-use; potential for multi-use

• Infrastructure needed to support multiple access mechanisms

• Releasability of data (organizational vs. technical)

• Coordination of collection and distribution

• Comparison and utilization of modeling outputs and raw observations

Clear and consistent data management policyNeed for better data documentation (metadata)

Overarching response plan

• Quickly identify applicable / available data

Immediate post-incidentData management challenge

Prepared by Environmental Data Management Committee for BRDI 7

FY 2011 EDMC Operating Plan Activities

• Develop Data Management Plan Guidance

• Improve Data Documentation

• Develop Archive Architecture ConOps

• Enhance Data Access Activities

• Develop Data Sharing Policy for NOAA Grants

• Maintains NOAA’s policy of “full and open access” to environmental data

• Provides mechanism for EDMC to develop procedural directives for more detailed guidance

• Presents an end-to-end lifecycle framework for the management of environmental data

• Next step: Develop Data Management Plan Guidance– Leveraging work underway by number of NOAA

programs (e.g. IOOS, Coral Reef Conservation Program, GOES-R)

Prepared by Environmental Data Management Committee for BRDI 8

Environmental data will be visible, accessible and independently understandable to users, except where limited by law, regulation, policy or by security requirements.

Revision of NOAA-wide Policy (NAO) 212-15

Prepared by Environmental Data Management Committee for BRDI 9

FY 2011 EDMC Operating Plan ActivitiesImprove Data Documentation

• The primary focus in FY 2011 will be to establish a metadata standard for NOAA and to provide detailed implementation guidance

• NOAA will be encouraging the use and convergence on International Standard Organization (ISO) Metadata Standard

• Detailed guidance to be built on NMFS Data Documentation Procedural Directive which is nearing completion

• NOAA’s Rolling Deck to Repository (R2R) is another example of end-to-end data documentation

Rolling Deck to Repository (R2R)

Improve Data Documentation Data Documentation Challenge

• Transition to ISO standards essential to documenting data for access and exploitation

• Barriers to this transition – Ownership of the responsibility to document data – Documentation standards are voluminous - much more than legacy protocols preceding

the current standards– Data providers, data stewards, and data users all have a stake – Boils down to resources--why up front planning is important

• Exemplary practices in the application of ISO-compliant documentation are starting to emerge from a growing and diverse set of mission- and discipline-specific contexts

• Exploring formation of a consortium with role to– Identify and/or develop exemplary practices in data documentation, especially the

application of ISO metadata standards, and– Offer training and education opportunities that facilitate and promote the replication

and even the advancement of these practices.• Expertise needed in three areas:

– Metadata creation– Metadata translation– Metadata extension

Prepared by Environmental Data Management Committee for BRDI

Improving Data Documentation Prototype Metadata Metric

Prepared by Environmental Data Management Committee for BRDI

Prepared by Environmental Data Management Committee for BRDI 12

FY 2011 EDMC Operating Plan Activities Develop Archive Architecture ConOps

Open Archival Information System Reference Model (OAIS-RM) Functions

• NOAA is following the International Standard for information preservation (OAIS-RM)

• This standard identifies the functions required to provide long-term preservation

• NOAA has multiple organization and systems fulfilling these responsibilities

• The Concept of Operations will establish the enterprise process and procedures for determining who does what

Data.gov

GEOSS PortalNOAA Portals / Websites

Prepared by Environmental Data Management Committee for BRDI 13

FY 2011 EDMC Operating Plan Activities Enhance Data Access Activities

NOAA MetadataRecords

NOAA’s Environmental Data &Information Infrastructure

Client Tools / Applications

Standard Access Services

Geospatial One-Stop

Prepared by Environmental Data Management Committee for BRDI 14

FY 2011 EDMC Operating Plan Activities Develop Data Sharing Policy for NOAA Grants

Prepared by Environmental Data Management Committee for BRDI 15

Summary

• NOAA’s revamped Environmental Data Management Committee has full work plan for the next few years

• Already leveraging partnerships, e.g. short course in collaboration with ESIP, DM Plan guidelines with IWGDD, etc.

• Policy will provide the structure to make EDM truly end to end throughout NOAA Line Offices and Programs from planning through preservation

• Outreach and education key to ensuring tangible results

• BRDI involvement in any of these areas useful and welcome

Questions

Improving Environmental Data and Information Management in NOAA

Prepared for NOAA Leadershipby the Environmental Data Management Committee

BACKUP SLIDES

Prepared by Environmental Data Management Committee for BRDI 18

External Engagement

BRDI Board on Research Data and Information

DAARWG NOAA Science Advisory Board's Data Archive and Access Requirements Working Group

EISWG NOAA Science Advisory Board's Environmental Information Services Working Group

ESIP Federation of Earth Science Information Partners

IOC Intergovernmental Oceanographic Commission

IWGDD Interagency Working Group on Digital Data

NSTC National Science and Technology Council

OGC Open Geospatial Consortium

USGEO/GEO United States Group on Earth Observations/Group on Earth Observations

WMO World Meteorological Organization

NAO

212

-15

Life

cycl

e C

ompo

nent

s

FY 2011 EDMC Activities

FY 2011 Activities Mapped to Lifecycle Components

Prepared by Environmental Data Management Committee for BRDI 19

= work in progress