1

Winter 2009 ESIP Federation Meeting

NOAA Data Users Workshop

Data Stewardship at the NOAA Data Centers

Data Stewardship Data Stewardship at the NOAA Data Centersat the NOAA Data Centers

2

Scientific Data Stewardship (SDS)Scientific Data Stewardship (SDS)

• Value Added Products– Putting all our data sources together, to understand

today’s environment in the context of the past

• Data Quality Monitoring for Long Term Applications– Critically examining data quality from our unique,

integrated, and long-term perspective

• Long Term Preservation of Data and Information– Ensuring information survives and remains

understandable to our children and beyond

• Data Archaeology and Rescue– Finding, saving, ingesting, and restoring data from all

over the world

• Data and Information Access - Tomorrow’s Session– Providing information to users and ensuring the

feedback loop contributes to its continual improvement

Increa

sing L

evel o

f Stew

ardship

3

Scientific Data Stewardship (SDS)Scientific Data Stewardship (SDS)

Your input is critical!

Increa

sing L

evel o

f Stew

ardship

4

Winter 2009 ESIP Federation Meeting

NOAA Data Users Workshop

Data Stewardship at the NOAA Data Centers

Data Archaeology Data Archaeology and Rescueand Rescue

5

Data Archaeology and RescueData Archaeology and Rescue

• Data centers routinely seek to acquire and preserve information that is in danger of being lost forever, from causes as diverse as deteriorating storage media to retiring principal investigators who never submitted their data to an archive. Participants are encouraged to discuss with the Data Centers their needs, areas of focus, and new technologies to support these rescue efforts.

• What are the key gaps the Data Centers should focus on?

6

Global Ocean Data Archaeology and Rescue (GODAR) Project

Global Ocean Data Archaeology and Rescue (GODAR) Project

• GODAR – begun in 1993 with endorsement from the Intergovernmental Oceanographic Commission (IOC)

• Goals– Increase the volume of historical oceanographic data available

– Locate and digitize ocean profile and plankton data sets not yet in digital form

– Rescue data on electronic media that are at risk of loss due to media degradation

– Submit these data to a national data center

• Accomplishments– > 3.5 million temperature profiles, >100,000 chlorophyll profiles,

and >600,000 plankton observations added to data center holdings

– Increased awareness of at-risk data collections

– World Ocean Database project established to stimulate international exchange of modern oceanographic data

7

Climate Data Modernization ProgramClimate Data Modernization Program

• CDMP - a multi-year partnership begun in 2000 between NCDC and private industry

• Goals– scan, digitize, preserve and provide access to diverse types

of climate data and information.

• Accomplishments– >52 million weather and environmental records digitized

• Digital capture (images and data) of 19th century surface weather observations

• Environmental proxy records, e.g., Common murre nesting site images and notes

• Digitize 7,200 negatives of killer, minke whales and other mammals, scan 15,000 pages of notes, keypunch 3,000,000 characters

8

Over 10 terabytes of Climate data now only a mouse click away

• Fifty two million weather & environmental images online• Hundreds of million of records digitized & now online • International data access and rescue activities

1842 Hourly Weather Data from Washington, DC Imaged and Digitized through the CDMP ProgramNorth Carolina Cooperative

Observer Images -OnlineMissing images 1950-1980 were made available in 2007

African RecordsRescue Project

Climate Data Modernization ProgramClimate Data Modernization Program 2000-20082000-2008

9 03/13/2006 9

Forecast Warning Analysis

NWS Coop Observers

Global Synoptic Reports

NCEP Charts

Ship, Buoy Reports

Rocketsonde

Radiosonde

Storm Data

Doppler Radar

(GOES, POES, NPOESS, many other) Satellites

Aircraft

Profiler

ASOS

USCRN

Data Received From Many SourcesData Received From Many Sources

10

Geosat Tape RescueGeosat Tape Rescue

• From March 1985 until January 1990, the U.S. Navy satellite Geosat generated a new data set with unprecedented spatial and temporal coverage of the global oceans – In the first 18 months, Geosat’s classified Geodetic Mission (GM)

succeeded in mapping the marine geoid– From November 1986 until December 1989, Geosat conducted its

Exact Repeat Mission (ERM) to collect observations of sea level, wave height, and wind speed

• Goals– Recover original Sensor Data Records (SDRs) from deteriorating

tape media before data were lost• Accomplishments

– Recovered original Sensor Data Records (SDRs) from the GM– NODC partnering with NOAA’s Lab for Satellite Altimetry and a tape

recovery specialist to recover original SDRs from the ERM from deteriorating tape media

• To date, 670 of 1047 9-track tapes have been rescued• An amazingly small loss-rate has been achieved - only 95 seconds of data from

670 days of data

11

Geosat Tape RescueGeosat Tape Rescue

• An amazingly small loss-rate has been achieved - only 95 seconds of data from 670 days of data!!!!

Example tape damage

12

Metadata ArchaeologyMetadata Archaeology

• Many historic data collections are poorly documented, which limits the usefulness of the data– Data formats, codes used– Methods, instruments, calibrations, precisions

• Challenge is to discover, document, and connect robust, relevant metadata in gray literature, published works, or PI manuscripts and notes to existing observation data

13

Discussion QuestionsDiscussion Questions

• What are the priority types of data in need of archaeological recovery, rescue, and preservation?– Physical, chemical, biological variables in need of data

rescue– Other types of observations in need of rescue

• Still images

• Video transects and time series observations

• Model outputs, source code or other elements

• What resources should data centers provide or obtain to develop better data rescue partnerships?

• Other questions…