2013 NOAA MADIS PlansMADIS – The Meteorological Assimilation Data Ingest System

Greg Pratt, Leon Benjamin, Thomas Kent, Gopa Padmanabhan, and Leigh CheatwoodNOAA/Oceanic and Atmospheric Research/Earth System Research Laboratory/Global Systems Division

Tim Mcclung, Steven Pritchett, Curtis Marshall, Ben Kyger, Daniel Starosta, Rebecca Cosgrove, and Wen MengNOAA/National Weather Service/Office of Science and Technology & National Centers for Environmental Prediction

John Bates, Drew Saunders, and Philip JonesNOAA/National Environmental Satellite, Data and Information Service/National Climate Data Center

MADIS GoalA more usable, complete, accurate, timely, and higher density observational infrastructure for use in local weather warnings and products, numerical weather prediction, and use by the greater meteorological community.

MADIS Provides• Access to real-time and archived data sets• Uniform data formats, observation units, and time stamps• Observational Quality Control (QC)• Network-enabled distribution with server-site sub-setting• Authentication for proprietary data• User documentation and help desk support

NOAA MissionNOAA’s mission increasingly demands advanced data management processes, including data integration, to achieve interoperable, accessible, and readily usable observational data.

MADIS Data Scope• 66,127 stations from over 160 surface networks

producing nearly 13 million observations per day• 154 Profiler sites (>200,000 observations per day)• >450,000 aircraft observations per day• Plus global radiosonde and satellite observations

MADIS Data Sets

Available Observations

Surface

Aircraft

Radiosonde

Profiler

GOES Satellite

POES Satellite

Radiometer

-

-

-

-

-

-

-

+o

X

P

R

MADIS observations covering North America

The MADIS Team

NWS/OST&NCEP

Transition/OperationsTim Mcclung, Steven Pritchett, Curtis Marshall, Ben Kyger,

Daniel Starosta, Rebecca Cosgrove, and Wen Meng

MADIS Future Plans

Additional data sets – (FAA 1 minute ASOS , Clarus, …)

Advanced data query and web services Improved data/metadata standards Improved QC and station monitoring Open source development environment Extend QC algorithms to meet operational and

research needs

• Reach Full Operating Capability (FOC) for real-time processing within the NWS

• Work with NESDIS to transition the MADIS archive capabilities into operations at NCDC

• Work with Federal Highway Administration Clarus team to fully transition Clarus capabilities into MADIS

• FAA NextGen/NWS AWIPS data discovery/dissemination capable

• Conduit for efficient transfer of research and development to operations

Decoders

Integrationand Quality

Control

NCEP

WOC

Ingest

Distribution

Providers

Users

MADIS Final Operating Capability

ESRL/GSD MADIS software is being ported to a distributed environment hosted at NOAA’s WOC, NCEP, and NCDC with GSD

supporting a research-to-operations test bed hosted at GSD.

NESDIS/NCDC

ArchiveJohn Bates, Drew Saunders, and Philip Jones

OAR/ESRL/GSD

Transition/R&D/Conduit to OperationsGreg Pratt, Leon Benjamin, Thomas Kent, Gopa

Padmanabhan, and Leigh Cheatwood

Providers

Observations and Metadatahttp://madis.noaa.gov/mesonet_providers.html

National Mesonet (NM)Mobile Platform Environmental Data (MoPED)Clarus

NCDC

Archive

Internet

MADIS Users Include…

• International meteorological centers

• >200 universities• Public

• NWS Forecast Offices and National Centers

• OAR, NESDIS, NOS• NCAR, NASA, DOE, FAA, DOT• Hundreds of private companies

MADIS Google Displays

Surface Temperatures

Metadata SensorML

A 2011 Update to the NOAA MADIS SystemMADIS – The Meteorological Assimilation Data Ingest System

Greg Pratt, Leon A. Benjamin, Randall S. Collander, Thomas B. Kent, and Gopa PadmanabhanNOAA/Oceanic and Atmospheric Research/Earth System Research Laboratory/Global Systems Division

Patricia A. Miller and Michael F. BarthNOAA/Oceanic and Atmospheric Research/Earth System Research Laboratory/Global Systems Division (Retired)

Steven Pritchett, Curtis Marshall, David Helms, Rebecca Cosgrove, Daniel Starosta, Michelle Mainelli, Mahnaz Dean, and Robert BungeNOAA/National Weather Service

MADIS GoalTo integrate other-agency observations with NOAA obser-vations and make them easily accessible and useable to the meteorological community.

MADIS Provides• Access to real-time and saved data sets• Uniform data formats, observation units, and time stamps• Observational Quality Control (QC)• Network-enabled distribution with server-site subsetting• Authentication for proprietary data• User documentation and help desk support

NOAA MissionNOAA’s mission increasingly demands advanced data management processes, including data integration, to achieve interoperable, accessible, and readily usable observational data.

MADIS Data Scope• 69,917 stations from over 160 surface networks

producing nearly 13 million observations per day• 154 Profiler sites (>200,000 observations per day)• >450,000 aircraft observations per day• Plus global radiosonde and satellite observations

MADIS Data Sets

Available Observations

Surface

Aircraft

Radiosonde

Profiler

GOES Satellite

POES Satellite

Radiometer

-

-

-

-

-

-

-

+o

X

P

R

MADIS observations covering North America

MADIS Overall BenefitsA more usable, complete, accurate, timely, and higher density observational infrastructure for use in local weather warnings and products, model predictions, and hazardous situations.

MADIS Central U.S. Observations

Standard Surface Observations Additional MADIS

Surface Observations All MADIS Observations

MADIS Provides QC and Station Monitoring Support

Estimated observation errors for a station inGalveston, Texas. The time series shows a bias

detected by MADIS and corrected by the data provider.

Data Provider Notified Data Restored

MADIS Users Include…• International

meteorological centers

• >200 universities

• NWS Forecast Offices and National Centers

• OAR, NESDIS, NOS• NCAR, NASA, DOE, FAA• Hundreds of private companies

-40

-20

0

20

40

60

80

100

4:41

:23

4:48

:18

4:55

:43

5:08

:11

5:21

:51

5:38

:24

5:54

:45

6:09

:35

6:27

:29

6:44

:04

7:01

:19

7:13

:09

7:25

:09

7:33

:33

7:40

:39

7:46

:53

8:01

:56

8:19

:05

8:29

:37

8:40

:08

8:56

:41

9:24

:17

9:39

:40

9:55

:12

10:0

7:33

10:2

6:52

10:3

6:24

10:4

4:02

10:5

5:34

11:0

7:38

11:1

7:49

11:2

6:37

11:3

4:45

11:4

9:38

12:0

2:08

12:1

1:16

12:2

2:35

12:3

7:58

12:5

7:35

13:1

8:52

13:3

7:58

13:5

9:19

14:1

7:39

14:2

9:00

14:4

3:04

14:5

4:26

15:1

9:29

16:0

7:31

16:3

8:47

16:5

9:05

17:3

8:58

Vehicle Speed (mph) Temperature (F) Error (F)

MADIS Future Plans

Additional data sets Advanced data query and web services Expanded metadata fields Improved QC and station monitoring

• Reach Full Operating Capability (FOC) for real-time processing within the NWS

• Work with NESDIS to transition the MADIS archive capabilities into operations at NCDC

• Continue research and development enhancements

Decoders

Integrationand Quality

Control

NCEP

TOC

Ingest

Distribution

Internet

MADIS NWS Compute Environment

MADIS Real-Time Processing is Now Operational Within the NWS

ESRL/GSD MADIS software has been ported to an integrated NWS TOC and NCEP distributed environment with a supporting

backup and research-to-operations test environment at GSD.

MADIS – New Datasets

Observations from a mobile platform reporting via the Mobile Platform Environmental Data (MoPED) System on

December 24, 2010. The plot shows low temperature errors at high platform speed, but higher errors at lower speeds.

New MADIS datasets are now in development at GSD:• National Mesonet observations with extended metadata;• Mobile Vehicle observations.