Guang Yu Zhao and Larry Di Girolamo

Department of Atmospheric SciencesUniversity of Illinois at Urbana-Champaign

ACCESS to Terra Fusion Products

The Terra Data Fusion Project Department of Atmospheric Sciences, University of Illinois

Larry Di Girolamo Guangyu Zhao Yizhe Zhan Landon Clipp Shashank BansalYat Long Lo Dongwei Fu Brandon Chen

Department of Geography and GIS, University of IllinoisShaowen Wang Yan Liu Yizhao Gao

The HDF GroupMuQun (Kent) Yang H Joe Lee

National Center for Supercomputing Applications, University of IllinoisJohn Towns Kandace Turner Michelle Butler Sean Stevens David Ralia

Jonathan Kim Donna Cox Stuart Levy Robert Patterson Andrew Christiensen

Department of Atmospheric Sciences, Texas A&MPing Yang Hioki Souichiro Yi Wang

NASA Langely/SSAILusheng Liang

NASA Goddard Space Flight CenterRalph Kahn Jim Limbacher

AcknowledgementsNASA Advancing Collaborative Connections for Earth System Science (ACCESS) program

NSF Blue Waters ProjectNSF ROGER MRI

NASA Langley Research Center Atmospheric Sciences Data Center NASA Atmosphere Archive & Distribution System Distributed Active Archive Center

NASA Land Processes Distributed Active Archive Center

Terra, in 2015 alone…

More than 360 million files…

Totaling more than 3.4 PB data…

Delivered to more than 100,000 users around the world.

More than 1,800 peer-reviewed publications (over 15,000 to date)

Results from Terra cited more than 49,000 times (over 250K to date)

“The high publication rate includes an increasing number of papers capitalizing on fusion of data among Terra sensors”

– NASA Senior Review 2017: Terra

Geology

Glaciology

Ocean Biology

Hydrology

VolcanologyMeteorology

Earth’s Radiation Budget and Climate

Agriculture and Land Use

Atmospheric Dynamics, Transport,and Air Pollution

• There exists inadequate cyberinfrastructure to tackle whole-mission data fusion and mining problems.

• Instrument data reside at different locations, in different file formats, with different granularity, and using different projections.

Current Problems:

• > 1 PB and counting (~2 TB/day)

• A range of expertise, not normally had at an investigator’s institution, is required to get the job done

• Distribution of fusion datasets.

The challenges for individual investigators for mission-scale Terra analyses are great just from a computing stand point.

These problems greatly limits/discourages scientists to take full advantage of the Terra data.

The Terra Data Fusion Project

1) How do we efficiently generate and deliver Terra data fusion products?

2) How do we facilitate the use of Terra data fusion products by the communityin generating new products and knowledge through national computing facilities, and disseminate these new products and knowledge through national data sharing services?

We aim to provide a significant community-element addition to NASA’s system of systems infrastructure for scientists requiring mission-scale access, processing, analytics, and distribution.

“ACCESS to Terra Data Fusion Products” tackles the following questions:

Instrument Size (TB) DAAC Protocol Rate (MB/s) Days NCSA’s Help Completion

MISR 240 ASDC GridFTP160-210

stable14 Yes 100%

MODIS 331 LAADS FTP0 – 100

unstable210 No 100%

ASTER 400 LPDAAC HTTP0-20

unstable>300 Yes 100%

CERES 1.5 ASDC GridFTP

160-210

stable 0.2 Yes 100%

MOPITT 0.8 ASDC GridFTP160-210

stable0.1 Yes 100%

• Result: a 2.5x increase in data transfer rates out of the Langley ASDC.

• Worked with NASA networking (CNOC) and ASDC teams to diagnose and remedy performance obstacles for data transfer out of the ASDC.

Step 1: Transferring All L1B Data from DAACs to NPCF

9Presentation Title

Lessons Learned

• Mission scale data transfers require active support from the DAACs,

which is not a scalable (A special shout out to the ASDC… amazing!)

• GridFTP provides much more efficient and reliable protocol than ftp and

http for bulk data transfer of the kinds we are dealing with.

Data Transfer

Issues that need to be addressed by the DAACs

• Complete lists of the original data files along with their sizes staged on

the DAACs’ servers are unavailable, making it difficult to verify the

integrity of data transfers.

• Corrupted data files exist on the DAACs’ servers.

• Mission-wide L1B ASTER data could not be filled by the LPDAAC, so we

had to settle with L1T data, which doesn’t contain radiance data for

ASTER’s oblique camera (3B).

• Goal: Merge the radiance fields from the L1B granules for all the Terra Instruments into one common granule, design it to be easy to use, and optimize it for large scale processing .

• Granularity: One Terra Orbit

• File Format: HDF5 (conformed to NetCDF4 CF conventions)

• File Content : 1. Unscaled Radiance (floating-point format) 2. Radiance Data Quality3. Geolocation for each FOV at its native resolution 4. Sun-view Geometry5. Observational Time 6. Other Attributes/Metadata

Keep instrument contents as is

Step 2: Develop the Basic Fusion Product

File Naming Convention: :

TERRA_BF_L1B_OXXXX_YYYYMMDDHHMMSS_F000_V000.h5

File Size:One granule: 20-120 GB without compression; 20-75 GB with compression, depending on the number of available ASTER granules for one orbit

Processing Time (node hours):

One granule: ~30 min without compression~45 min with compression and chunking

(Opening movie from TERRA_BF_L1B_O3571_20000819133750_F000_V001.h5)

Step 2: Develop the Basic Fusion Product

• nearest neighbor resampling – for source and destination grids having similar pixel resolutions (e.g., MISR -> MODIS)

• statistical resampling – for source and destination grids having distinct pixel resolutions (e.g., ASTER -> MISR)

• Regularly spaced grid defined by any map projection• User-friendly user interface• HDF5 and GeoTIFF output

Functional Requirements:A community, open source tool to resample/reproject the radiance fields for one Terra instrument onto the grids used by another Terra instrument, a regularly spaced grid, or any other user specified grid.

Accomplishments:

Performance:• Developed in C and C++ and optimized for HPC • It takes 40 seconds to resample MISR 1.1km radiance data onto MODIS 1.0km

grid for a whole orbit and 4 minutes to resample MISR 275m radiance data onto MODIS 250m grid using 32 threads (1 node) on Blue Waters.

Development of the Resampling and Reprojection Tool(Next Level of Fusion: Step 3)

13Presentation Title

MIS

R b

lue

band

radi

ance

resa

mpl

ed o

n th

e M

ODIS

grid

MOD

IS B

lue

Band

Development of the Resampling and Reprojection Tool(Next Level of Fusion: Step 3)

TERRA_BF_L1B_O69626_20130119123228_F000_V000.h5

14Presentation Title

Development of the Resampling and Reprojection Tool(Next Level of Fusion: Step 3)

MOD

IS B

and

8

ASTE

R Ba

nd 3

N

ASTE

R re

sam

pled

on

the

MOD

IS g

rid

15Presentation Title

Metadata Generation

Metadata files:

• 1 collection-level XML file

• 85302 granule-level XML files (We do provide metadata for no-data BF files

caused by instrument anomalies, PGE failure and et al.)

Granule-level Metadata include:

• CDL metadata generated from BF granules

• Metadata from original input granules used to generate BF

• Metadata can only be obtained through the NASA CMR queries

⏤ We did 20,000 granule queries per hour via CMR APIs and then: Ø “It was bringing CMR to it's knees” from the CMR Operations

LeadØ Earthdata Search experienced transient search timeouts for a

whole weekend.Ø Our IPs were blocked for a few days.

16Presentation Title

Zhao, G., L. Di Girolamo, D.J. Diner, C.J. Bruegge, K. Mueller, and D.L. Wu, 2016: Regional changes in Earth’s color and texture as observed from space over a 15-year period.IEEE Trans. Geosci. Remote Sens., 54(7), 4240-4249, doi:10.1109/TGRS.2016.2538723

Fu, D., L. Di Girolamo, L. Liang, and G. Zhao, 2017: The observed behavior of the bias in MODIS-retrieved cloud drop effective radius through MISR-MODIS data fusion. American Geophysical Union 2017 Fall Meeting, December 10-15, New Orleans, LA.

Wang, Y., S. Hioki, P. Yang, L. Di Girolamo, and D. Fu, 2017: Seasonal bias of retrieved ice cloud optical properties based on MISR and MODIS measurements. American Geophysical Union 2017 Fall Meeting, December 10-15, New Orleans, LA.

Di Girolamo, L., et al., 2017: The Terra data fusion project: an update. American Geophysical Union 2017 Fall Meeting, December 10-15, New Orleans, LA.

Example Science Use Cases to Aid Development

First true color image of the Earth’s climate at ~ 10:30 AM.

Aka: Climate MarbleTM

Climate MarbleTM

MISR RGB 15-year average, daily time step, 20-day averageing window

© University of Illinois

18Presentation Title