June 8, 2003 NOAA’s National Climatic Data Center

Exploratory Streaming Data and Climate Analysis Tools for

Environmental Satellite and Weather Radar Data

John J. Bates, ChiefRemote Sensing Applications DivisionNOAA’s National Climatic Data Center151 Patton Ave., Asheville, NC 28801

John.J.Bates@noaa.gov

June 8, 2003 NOAA’s National Climatic Data Center

Outline Introduction – NOAA NESDIS Data ServicesClimate observing system performance

monitoringDetection of long-term climate trends using

environmental satellite data Time-space analysis of massive observational

data sets Extreme event detection using weather radar

data Conclusions

June 8, 2003 NOAA’s National Climatic Data Center

NESDIS

MISSION: The NOAA NESDIS mission is to provide and ensure timely access to global environmental data from satellites and other sources to promote, protect, and enhance the Nation’s economy, security, environment, and quality of life. To fulfill its responsibilities NESDIS acquires and manages the Nation’s operational environmental satellites, provides data and information services, and conducts related research.

June 8, 2003 NOAA’s National Climatic Data Center

NOAA Climate Observations and Services

OARClimate Research

Long-Term Climate Modeling

Monitoring of Atm Composition

Ocean Obs

NWSClimate Prediction

Regional/Local Forecasting

Outreach

In Situ Obs

Climate Obs & Services

Sustained Obs

Assessments/ Predictions

Trans. to Operations

NESDISOperational Satellites

Climate Data & Inf Mgmt

Climate Monitoring

June 8, 2003 NOAA’s National Climatic Data Center

Climate

Climate research and monitoring capabilities should be balanced with the requirements for operational weather observation and forecasting within an overall U.S. strategy for future satellite observing systems1

1 NAS/NRC Report on Integration of Research and Operational Satellite Systems for Climate Research (2000)

June 8, 2003 NOAA’s National Climatic Data Center

Geostationary Operational Environmental Satellite (GOES)

Polar-Orbiting Operational Environmental Satellite (POES)

In Situ Surface and Upper Air Observations

NEXRAD Weather Radar

National Polar-Orbiting Environmental Satellite System (NPOESS)

Environmental Data Management National Climatic Data Center National Oceanographic Data Center National Geophysical Data Center

Applications Research and Development

NESDIS Programs that Support Monitoring the Earth-Climate

System

June 8, 2003 NOAA’s National Climatic Data Center

Managing the Nation’s Operational Environmental Satellite Systems

Polar Orbiting Satellites Geostationary Satellites

June 8, 2003 NOAA’s National Climatic Data Center

Geostationary Satellites

Warnings to U.S. Public -- Detect, track and characterize

Hurricanes Severe or possibly tornadic storms

Flash flood producing weather systems

Imagery and soundings for weather forecasting

Winds for aviation and NWS numerical models

Environmental data collection – Platforms including buoys, rain gauges…

June 8, 2003 NOAA’s National Climatic Data Center

GOES Program Overview

On-Orbit Storage

Operational Spacecraft

• GOES satisfies National Weather Service (NWS) requirements for 24 hour observation of weather and Earth’s environment to support storm-scale weather forecasting by forecasters and numerical models

• To meet requirements, GOES continuously maintains operational satellites at two locations (75 degrees West and 135 degrees West), with an on-orbit spare ready in case of failure

June 8, 2003 NOAA’s National Climatic Data Center

To provide UNINTERRUPTED flow of global environmental information in support of operational requirements for: Global Soundings Global Imagery and Derived Products Global and Regional Surface & Hydrological Obs Direct Readout, Data Collection, Search and Rescue Space Environment and Ozone Obs

This requires two satellites on-orbit to allow for

continuous coverage during the inherent time it

takes to launch and checkout a replacement satellite.

POES Program

June 8, 2003 NOAA’s National Climatic Data Center

In Situ – Surface and Upper Air

Surface in situ data are ingested from automatic weather reporting stations in remote locations, airports, and weather service field sites

Upper air observations are ingested from weather balloons that are launched twice a day to provide detailed temperature and moisture profiles

June 8, 2003 NOAA’s National Climatic Data Center

NEXRAD Weather Radar Observations

Over 100 NEXRAD weather radars operate continuously to detect both rain and doppler velocity (for tornado vortex signatures

Data was originally recorded on tape at each weather service office

About half the sites are now transmitting data in real-time to the archive via the Abelene and the remaining sites wil by the end of the year

June 8, 2003 NOAA’s National Climatic Data Center

A Presidentially Directed, Tri-agency Effort to Leverage and Combine Environmental Satellite Activities

National Polar-Orbiting Operational Environmental Satellite System – Next Generation System

Mission Statement

To provide a single, national, operational, polar remote-sensing capability to acquire, receive and disseminate global and regional environmental data

To achieve National Performance Review (NPR) cost savings through the convergence of DoD and NOAA environmental satellite programs

To incorporate, where appropriate, technology transition from NASA’s Earth Science Enterprise (ESE)

0530

13300930

June 8, 2003 NOAA’s National Climatic Data Center

Unique Role of NOAA’s National Data Centers

Acquire data from U.S. and foreign sources Preserve the Nation’s environmental data

assets Assemble data into easy to use long-term data

sets Provide access to environmental data for

business, federal and science users Describe the environment

June 8, 2003 NOAA’s National Climatic Data Center

NOAA’s Data System Capability

Manages 3 National Data Centers and 7 World Data Centers

Archives over 450 terabytes of data and responds to over 4,000,000 requests per year from over 70 countries

Maintains some 1300 data bases containing over 2400 environmental variables

Maintains over 535,000 tapes 375, 000,000 film records 140,000,000 paper records

NODCSilver Spring, MD

NCDCAsheville, NC

NGDCBoulder, CO

June 8, 2003 NOAA’s National Climatic Data Center

More Data to Manage

Volume growth of new data is outstripping the ability to ingest and process the data sets

• NOAA’s cumulative digital archive grew

130 terabytes from 1978-1990

• Grew another 130 terabytes from 1990-1995

• Grew another 130 terabytes in 1996 alone

• Currently approximately 800 terabytes

By 2004, NOAA will ingest and process more new data in one year than was contained in the total digital archive in 1998.

June 8, 2003 NOAA’s National Climatic Data Center

Introduction – Massive Environmental Data Volumes

MAJOR SYSTEMS PROJECTED GROWTH 2002 - 2017

0

10000

20000

30000

40000

50000

60000

70000

80000

90000

TE

RA

BY

TE

S

GOES NEXRAD DMSP POESEOS METOP NPP NASA NPPNPOES Future NASA Missions GIFTS

June 8, 2003 NOAA’s National Climatic Data Center

June 8, 2003 NOAA’s National Climatic Data Center

Application of consistent cloud detection, navigation, error check, retrieval algorithm

Data are checked swath by swath

Data are composited on global grids and also checked

Orbit statistics are saved as metadata for further analysis

June 8, 2003 NOAA’s National Climatic Data Center

Monitoring histogram distribution of mean, 10th and 90th percentile radiances over water

Monitoring the quantiles of the frequency distribution is helpful in determining the calibration stability of instruments

We need ultrafast software to perform these calculations on the massive data rates expected in the future

We could also use ultrafast code for computing clustering or classification information

June 8, 2003 NOAA’s National Climatic Data Center

POES Data Characterization and Bias Monitoring

Limb correction and cloud detection schemes must be assessed and applied

Numerous statistical tools are then applied to assess characteristics of the data

Forward and inverse radiative transfer methods must be applied

Multiple different techniques for intersatellite bias adjustment should be tried

June 8, 2003 NOAA’s National Climatic Data Center

Detection of long-term climate trends using environmental satellite data

Creation of seamless time series – nominal, normalized, and absolute calibration

Application of consistent cloud detection, navigation, error check, retrieval algorithm

Exploratory data analysis techniquesHypothesis formulation and testingAncillary data analysis to confirm hypothesis

and long-term trend analysis

June 8, 2003 NOAA’s National Climatic Data Center

Creation of seamless time series

Similar instruments on different satellites give systematic biases

Individual satellites drift later in local time

Individual channels sometimes change over time

Lifetime of satellites varies greatly

June 8, 2003 NOAA’s National Climatic Data Center

Exploratory data analysis techniques –Area average time series/indices, empirical

orthogonal function analysis

June 8, 2003 NOAA’s National Climatic Data Center

Hypothesis formulation and testing

Extremes in upper level water vapor occur most frequently in Northern winter and spring

Extremes also occur synchronous with extremes in El Niño events

For La Niña cold events (top), strong westerlies lead to strong eddy activity and high water vapor amounts

For El Niño warm events (bottom), deep convection along the equator leads to no eddies

June 8, 2003 NOAA’s National Climatic Data Center

Ancillary data analysis to confirm hypothesis and long-term trend analysis

Upper tropospheric humidity climatology shows distribution of tropical monsoon-desert system

20-year trend shows increasing UTH along equator and east Asia, decreasing UTH in subtropics

Confidence levels show only largest trends are significant – confidence intervals are computed using linear scatter, lag-1 autocorrelation, and length of record vs. trend

June 8, 2003 NOAA’s National Climatic Data Center

Time-space analysis of massive observational data sets – radar reflectivity and rainfall

Atmospheric wave motions and phenomena propagate east and west with characteristic speeds

Identification of these phenomena is critical to understanding and forecasting

High spatial and temporal coverage is required to fully sample these phenomena

Several examples are used to illustrate diagnosis and application of this technique

June 8, 2003 NOAA’s National Climatic Data Center

Monitoring the tropical Pacific and El Niño

June 8, 2003 NOAA’s National Climatic Data Center

Time-space to wavenumber-frequency analysis

Analyze twice daily satellite radiance data for the global tropics

Apply FFT in both the time and space dimensions

Subtract background red noise spectrum as a function of wavenumber and frequency

Contour resulting spectrum energy

Relate distinctive maximum to idealized equations of motion atmospheric wave solutions

June 8, 2003 NOAA’s National Climatic Data Center

Applying time-space analysis to weather-climate interactions

Outgoing longwave radiation (OLR) anomalies are used to track the propagation of large tropical cloud clusters

Madden-Julian oscillations (MJOs) have been related to changes in North American winter flow pattern regimes and El Niño onset

MJOs and easterly Kelvin waves have also been related to regimes that favor or suppress monsoons and hurricanes

June 8, 2003 NOAA’s National Climatic Data Center

Extreme event detection using remotely sensed data – radar tornado vortex

June 8, 2003 NOAA’s National Climatic Data Center

Evaluating tornado vortex signature classifiers

Bayesian classifier is optimal with respect to minimizing the classification error probability

Multiple Prototype Minimum Distance Classifier (Mpmd) learns a set of one or more prototypes for each class that are meant to represent the patterns in that class. It classifies patterns by finding the prototype with the minimum distance to the pattern

Self Partitioning Neural Network (SPNN) is a special kind of back-propagation network. It is designed to work with two class (Usually a target class and a non-target class) problems

ADaM Reader

WSR-88D NEXRAD data

Data in internal ADaM format

1D shear feature detection

2D shear feature detection

3D shear feature detection

Classifiers

Classifiers training

Training data generation

Classified 3D features

“ground truth” features

Classifiers parameters

June 8, 2003 NOAA’s National Climatic Data Center

Real-time data streaming of weather radar data

When no precipitation is present, weather radar are kept on ‘clear sky’ mode

Clear sky mode can reveal a number of other atmospheric backscatter phenomena – bugs, smoke, thermal boundaries

Debris from the Columbia disaster were picked up on several radars

Data from the NCDC archive were available immediately for the accident investigation

June 8, 2003 NOAA’s National Climatic Data Center

Conclusions Data streams from environmental satellites and

weather radar are projected to increase geometrically over the next 10-15 years

Statistical tools to analyze these data range from simple to complex, but simple tools remain most useful because the phenomena we are trying to analyze are highly complex

The outlook for hardware to process and store massive amounts of data is good

Additional investment in people is required to ensure future generations have the technical skills required to fully exploit the massive data sets available

We need to collaborate with other researchers in the development and application of tools to mine streaming data