11

Towards A 35 Year Earth Science Data Record of Gridded IR Atmospheric Radiances

byM. Halem, D. Chapman, P. Nguyen

University of Maryland, Baltimore CountyMulticore Computational Center (UMBC/MC2)

Presentation at NESDIS CDR workshop 11/18/08halem@umbc.edu Supported by NASA ACCESS and IBM SUR

grants

2

OverviewOverview

► MotivationMotivation

► SOAR gridding systemSOAR gridding system

► Spatial observation based geo-location gridding Spatial observation based geo-location gridding

systemsystem

► Hadoop cloud computing processing approachHadoop cloud computing processing approach

► Neural network approach for orbit/SRF adjustments Neural network approach for orbit/SRF adjustments

► MODIS & AIRS long term IR radiance comparisonsMODIS & AIRS long term IR radiance comparisons

► Climate ApplicationsClimate Applications

► SummarySummary

3

MotivationMotivation

► NASA, NOAA DOD and ESA have collected large volumes (Petabytes) of satellite Infrared Radiance (IR) data for the past 39 years which are stored in various archives and formats and on different storage media.

► Gridded data records of Level 1B IR measurements for AIRS/HIRS/VTPR/SIRS and MODIS/AVHRR data have many advantages for climate studies but are not available today as products

► The basis of our approach to forming a HIRS/AVHRR gridded FCDR is to The basis of our approach to forming a HIRS/AVHRR gridded FCDR is to calibrate against AIRS/MODIS which is expected to be operational until 2013. If calibrate against AIRS/MODIS which is expected to be operational until 2013. If AIRS/MODIS instruments continue to operate, we show this data can be a AIRS/MODIS instruments continue to operate, we show this data can be a Fundamental Decadal Data Record (FDDR)Fundamental Decadal Data Record (FDDR)

► AIRS is hyperspectral with 2378 channels spanning 3.7um to 15.4 um ~14km and 4 Vis channels at 0.41 - 0.44 μm, 0.58 - 0.68 μm, 0.71 - 0.92 μm and 0.49 - 0.94 μm at ~1km. AVHRR overlap are 0.58 - 0.68 and0.725 - 1.00um.

► MODIS is hyperspatial with 16 IR spectral bands 3.6 3.6 um 14.3 14.3 um at 1km and 20 at 1km and 20 Vis channels several corresponding to AVHRR.Vis channels several corresponding to AVHRR.

http://www-airs.jpl.nasa.gov/Technology/HistoricalContext/

Service Oriented Atmospheric Radiances Service Oriented Atmospheric Radiances (SOAR)(SOAR)

• Developed a web based system with an interface for accessing and Developed a web based system with an interface for accessing and invoking gridding and analysis services on level 1B Infra-red radiance invoking gridding and analysis services on level 1B Infra-red radiance data data

• Employs SOA technologies to discover and select services for use with Employs SOA technologies to discover and select services for use with multi -sensor infra-red radiances on UMBC/MC2 clustermulti -sensor infra-red radiances on UMBC/MC2 cluster

• Serves up gridded (lat-lon) AIRS and MODIS IR spectral radiances on-Serves up gridded (lat-lon) AIRS and MODIS IR spectral radiances on-demand and being extended to serve up HIRS/AVHRR and requested demand and being extended to serve up HIRS/AVHRR and requested images and/or structured data formats images and/or structured data formats

• Provides a multi instrument framework that precisely accounts for the Provides a multi instrument framework that precisely accounts for the scanned radiances emitted from each grid cell using the instruments own scanned radiances emitted from each grid cell using the instruments own spatial response function spatial response function

• Provides a platform for users to exploit historical IR data for generating Provides a platform for users to exploit historical IR data for generating climate analysis with tools from traditional methodologiesclimate analysis with tools from traditional methodologies

5

SOAR Web-based System SOAR Web-based System http://bluegrit.cs.umbc.edu/soar

ID: soar Pwd:soar1234!ID: soar Pwd:soar1234!► SOAR system provides:SOAR system provides:

6 +years AIRS IR global pre-gridded at 0.56 +years AIRS IR global pre-gridded at 0.500x1x1.0.000 and and processing 4 Vis channels at 0.125processing 4 Vis channels at 0.12500 x 0.25 x 0.2500 km res km res

3+ years of non-continuous MODIS gridded datasets 3+ years of non-continuous MODIS gridded datasets User request of subsets, arbitrary regions resolutions, User request of subsets, arbitrary regions resolutions,

visualizations, structured data sets, statistical routines, for visualizations, structured data sets, statistical routines, for selected channelsselected channels

---------------------------------------------------------------------------------------------------------------------------- Under beta test development: release 2/1/09Under beta test development: release 2/1/09 IR gridded datasets from multiple sensors AIRS/HIRS and IR gridded datasets from multiple sensors AIRS/HIRS and MODIS/AVHRR on demand for arbitrary grid resolutionsMODIS/AVHRR on demand for arbitrary grid resolutions Gridding algorithm employs a spatial ray casting scan Gridding algorithm employs a spatial ray casting scan

angle adjustment for zenith angle dependence and neural angle adjustment for zenith angle dependence and neural networks for orbital and spectral function response networks for orbital and spectral function response adjustments.adjustments.

Serves up anomalies, MJOs and OLR products on demand.Serves up anomalies, MJOs and OLR products on demand.

Maps arbitrary level 1B granules ontoMaps arbitrary level 1B granules onto

level 3 grids with desired statistics.level 3 grids with desired statistics.

Advantages:Advantages:► Gridding greatly reduces data volume (lossy compression)Gridding greatly reduces data volume (lossy compression)► Provides analysis services Provides analysis services based directly on observations based directly on observations ► Contains a spatial raycasting framework (Gridder) for Contains a spatial raycasting framework (Gridder) for

mapping multi instrument obs accurately onto gridsmapping multi instrument obs accurately onto grids► Uses highly efficient parallel computing paradigm, Hadoop.Uses highly efficient parallel computing paradigm, Hadoop.► No level 3 gridded radiance products available for No level 3 gridded radiance products available for

AIRS/HIRS and MODIS/AVHRR instrument teamsAIRS/HIRS and MODIS/AVHRR instrument teams► Increases usability by scientists (modeling & visualization Increases usability by scientists (modeling & visualization

climate analysis studies, etc.)climate analysis studies, etc.)

Why we use SOARWhy we use SOAR

SOARs Generic GridderSOARs Generic Gridder

► Philosophy: Common gridding algorithms for many Philosophy: Common gridding algorithms for many instruments.instruments. Spatial calibration with recursive ray casting algorithmSpatial calibration with recursive ray casting algorithm Spectral calibration with neural network algorithm.Spectral calibration with neural network algorithm.

► Framework developed for gridding many radiative scanning Framework developed for gridding many radiative scanning instruments.instruments. Currently implemented for AIRS, MODIS, AIRS VisibleCurrently implemented for AIRS, MODIS, AIRS Visible Being extended to HIRS3, HIRS4 and AVHRRBeing extended to HIRS3, HIRS4 and AVHRR Incorporating artificial neural network algorithm into Incorporating artificial neural network algorithm into

Gridder.Gridder.

Radiances travelling from Earth to instrument are weighted by the sensor's spatial response function.

The shape and distribution of the footprint can be calculated by projecting the spatial response function onto Earth

Actual observation is a weighted sum, IE integral, of all radiation from the footprint

(a) through (d) are simplified examples of spatial response functions

(a) uniform response function, similar to AIRS

(b) triangular response function, similar to MODIS

(c) ideal Gaussian response function

(d) arbitrary response function

Determine what percentage of a footprint lies in a grid cell.

Given: the footprint is a distribution of spatial response

Thus, we must integrate the footprint within the grid cell.

Numerical integration allows this technique to be used with any spatial response function.

Algorithm: Footprint is recursively subdivided until it is completely within a gridcell. Raycasting is used in recursion to geolocate points.

Images between -90 and -80 degree latitudinal bandImages between -90 and -80 degree latitudinal band

Corrects errors caused by lat-lon singularities.

(a) Missing data (blue) over most of the south pole(b) South pole correctly gridded with numerical footprint

Jan 1st/2nd 2008Jan 1st/2nd 2008(a) No Obscov (b) (a) No Obscov (b) 128x128128x128

AIRS OLR at 0.5x1 Lat Lon grid 12 micron window channel:

Errors up to or exceeding 1K BT are corrected from all over the planet.

The global daily average is corrected by 0.3K BT.

Neural Networks Neural Networks

►NNeural networks are algorithms used for either eural networks are algorithms used for either classification or function approximationclassification or function approximation

► A commonly used type of neural network is the Multi-Layer A commonly used type of neural network is the Multi-Layer Perceptron, of which Kalman filters are one typePerceptron, of which Kalman filters are one type

► A neural network with A neural network with x input nodes, one hidden layer with x input nodes, one hidden layer with y hidden nodes and z output nodesy hidden nodes and z output nodes

► In supervised mode, time series of several spectral bands In supervised mode, time series of several spectral bands per data sample presented to the network and desired per data sample presented to the network and desired output is used to modify the weights so that the deviation output is used to modify the weights so that the deviation between actual and obtained output is minimized.between actual and obtained output is minimized.

► main drawback is they require experience in selecting main drawback is they require experience in selecting values for the numerous parameters that need to be setvalues for the numerous parameters that need to be set

14

Neural Networks (Cont.)Neural Networks (Cont.)

15

The Generic GridderThe Generic Gridder

► Issues:Issues: Cannot readily obtain spatial response Cannot readily obtain spatial response

functions for most instruments.functions for most instruments.►Implicit approximations (circle, triangle, etc) Implicit approximations (circle, triangle, etc)

can be used, and well documentedcan be used, and well documented

Missing documentation HIRS3 spectral Missing documentation HIRS3 spectral multipliermultiplier

►Last resort use neural network to solve for Last resort use neural network to solve for missing constantmissing constant

Compare HIRS to known data to “calibrate” the Compare HIRS to known data to “calibrate” the missing constantmissing constant

17

18

19

20

21

22

23

24

25

AIRS/MODIS comparisonAIRS/MODIS comparison

► AIRS/MODIS Brightness Temperatures for AIRS/MODIS Brightness Temperatures for OCT. 1, 2007 at 0.5OCT. 1, 2007 at 0.5oox1x1oo ► Consistent stable instrument measurements after 6+ years!Consistent stable instrument measurements after 6+ years!► Avg differences between MODIS and AIRS are ~1 K. Many channels

show agreement of ~ 0.1 K

GREEN are AIRS observations

RED are MODISobservations

Difference between MODIS & MODIS convolvedwith AIRS

26

AIRS Monthly anomaly 0.5AIRS Monthly anomaly 0.5oox1x1oo at 12.18 at 12.18

µmµm

27

AIRS Monthly anomaly 0.5AIRS Monthly anomaly 0.5oox1x1oo at at

12.18 12.18 µmµm

► Year to year variancesYear to year variances► Cold radiances Feb 05Cold radiances Feb 05 (strong El (strong El

Nino year) convective cloudNino year) convective cloud, , Warm radiances Feb 07 cloud Warm radiances Feb 07 cloud clear surface in Western Pacificclear surface in Western Pacific

► Similar in Indian Ocean and Similar in Indian Ocean and West Pacific areaWest Pacific area

► Feb 05 warmer than other 2 Feb 05 warmer than other 2 year in East US (hurricanes)year in East US (hurricanes)

► Variances in Variances in Intertropical Convergence Zone

Feb 2006 anomaly

Feb 2005 anomaly Feb 2007 anomaly

28

MJO- resultsMJO- results

C)

Variances color code

BT color code

Variances color codeDec1506Dec1506

Dec1706Dec1706

Dec1906Dec1906

Dec2106Dec2106

Dec2306Dec2306

Dec2506Dec2506

Dec2706Dec2706

Dec2906Dec2906

Jan0107Jan0107

Jan0307Jan0307

Jan0507Jan0507

Jan0707Jan0707

Jan0907Jan0907

Jan1107Jan1107

Jan1307Jan1307

Jan1507Jan1507

Jan1707Jan1707

lag1lag1

lag2lag2

lag3lag3

lag4lag4

lag5lag5

The first EOF explaining about explaining about 14.3% variance14.3% variance

B)

2 day running mean of MODIS channel 2 day running mean of MODIS channel 3232 (Surface/Cloud Temperature) (Surface/Cloud Temperature) at 0.5at 0.5oox1 11.7 x1 11.7 µmµm -12.2 -12.2 µmµm 5S-5N 0-180E Brightness Temperature 5S-5N 0-180E Brightness Temperature descending orbit from Dec 15 2006 to descending orbit from Dec 15 2006 to Jan 17 2007.Jan 17 2007.

A)The extended EOF which The extended EOF which captures captures the dynamics using a the dynamics using a temporal lag of 2 day running temporal lag of 2 day running mean.mean.

29

AIRS/MODIS total OLR AIRS/MODIS total OLR 0.50.5oox1x1oo

► AIRS/MODIS total OLR AIRS/MODIS total OLR isentropic assumption isentropic assumption

► Compare with CERES/Erbe Compare with CERES/Erbe OLROLR

30

AIRS/MODIS Aqua total OLR AIRS/MODIS Aqua total OLR 0.50.5oox1x1oo

AIRS, MODIS Feb. 2005 global average 230, 233 W/m2

respectively

31

SummarySummary• Moving towards implementing an FCDR of gridded Moving towards implementing an FCDR of gridded

HIRS & AVHRR data sets in the next few monthsHIRS & AVHRR data sets in the next few months SOAR system provides precise mult-instrument spatial gridding options for SOAR system provides precise mult-instrument spatial gridding options for

generating arbitrary spatial/spectral level 3 radiance resolutions generating arbitrary spatial/spectral level 3 radiance resolutions Relative AIRS and MODIS IR spectral radiance measurements have not Relative AIRS and MODIS IR spectral radiance measurements have not

degraded in over six years and have potential to provide long term degraded in over six years and have potential to provide long term (>10 year) Fundamental Climate Data Record(>10 year) Fundamental Climate Data Record Leverages off AIRS and MODIS to train Neural net to maintain consistencyLeverages off AIRS and MODIS to train Neural net to maintain consistency aacross HIRS and AVHRR platformscross HIRS and AVHRR platforms Employs an extremely efficient parallel computing paradigm to conduct data Employs an extremely efficient parallel computing paradigm to conduct data

intensive processing and reprocessingintensive processing and reprocessing Incorporates a variety of instrument science analysis products as servicesIncorporates a variety of instrument science analysis products as services

Tracking MJO directly from calibrated observations Tracking MJO directly from calibrated observations indicates fewer indicates fewer uncertainties for climate inference compared with MJO anomaly exploited uncertainties for climate inference compared with MJO anomaly exploited from reanalysisfrom reanalysis

Work supported by NASA ACCESS grant and IBM SUR grant Work supported by NASA ACCESS grant and IBM SUR grant

32

►BACK UP SLIDESBACK UP SLIDES