an idl-basedvalidation toolkit: extensions to use the hdf-eos swath format
DESCRIPTION
An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format. Ken Stone, Center for Lower Atmospheric Studies - University of Colorado, Boulder with help from. Acknowledgements. Larry L. Gordley - GATS, Inc., Newport News, Virginia - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/1.jpg)
An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS
Swath Format
Ken Stone, Center for Lower Atmospheric Studies -
University of Colorado, Boulder
with help from...
![Page 2: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/2.jpg)
Acknowledgements
• Larry L. Gordley - GATS, Inc., Newport News, Virginia
• The HALogen Occultation Experiment (HALOE) Team
• Cheryl Craig, NCAR
• Joe McInerney, University of Colorado
![Page 3: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/3.jpg)
Overview
• Overview of the toolkit– History– Motivation– Some examples
• HDF-EOS extensions– Problems & Limitations– Strengths– Lessons learned
• Summary
![Page 4: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/4.jpg)
Motivation
• Problem: It takes considerable time and effort to make the many multiple comparisons, internal and external, needed during validation phase.
• Goal: A simple set of tools that allow validation staff to select, line-up, compare and plot profiles and cross sections.
• Latest twist: HDF-EOS files are now the archive format of choice.
![Page 5: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/5.jpg)
History
• LIMS - Early 80’s– Fortran routine for comparisons and plotting– Fast, low memory usage– Input: multiple files– Output: text summaries and plots
• HALOE - Early 90’s– Perl Scripts and Standard I/O file format (bsel format)– Fortran routine ported to IDL– New functionality and visualization tools.
• HIRDLS– HDF-EOS Swath capability– Additional Tools (dot-plots, map differences)
![Page 6: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/6.jpg)
Toolkit Overview
• IDL procedures with Perl scripts for control– Complex scenarios developed both with interactive use (storing results
in “journal” file) and stringing scripts together with Unix shell.
• Analytical Visualization tools– Mapping, cross sections, line plots.– Focus on quality and simplicity rather than complex visualizations.
• Analysis tools– Coincidence finder, statistical computations.
• Standardized file format(s) for I/O• “Profile” based format hinges on vectors of geo-referenced
quantities.
![Page 7: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/7.jpg)
Toolkit Detail
• Prior to HDF-EOS extension the Toolkit used a single file type for input (BSEL format)– Similar in some ways to HDF 4 SDS but limited to sets of 1-d
arrays stored as records.
• User input for any given IDL scenario can be recorded in a journal (text) file for later use.
• Perl “wrapper” scripts create this file as output.– Keyword substitutions in a “mask” file.
– Depending on the mode (mood?) the user is in, they could modify the journal file directly, or change the inputs on the script.
![Page 8: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/8.jpg)
Toolkit Script Methodology
Perl Script (e.g., 2dlat)
IDL procedureor C, or Fortran...
Mask File
Input File
User Input: -f filename -I id, -o outputfile -g -h, etc
Postscript
BSEL File
Perl helper lib
Journal file creation from IDL scripts Analysis routines
![Page 9: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/9.jpg)
Toolkit Data Flow: Archive and Working Formats
Selection Analysis Display
Data
Source
HDF-EOS
Bsel
Toolkit Extension
“Standard” Data Format
Archive
“working” data format
![Page 10: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/10.jpg)
![Page 11: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/11.jpg)
![Page 12: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/12.jpg)
![Page 13: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/13.jpg)
![Page 14: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/14.jpg)
Toolkit Data Model (BSEL Format)BSEL File
Event (1) Event (i)
Plot InfoRange, axis Label, legend Label, Scale, Logflag
Event HeaderReal(8)Long(8)
char*60(6)
Data (j,k)
. . .
Data HeaderData order (=2)J (Profile count)K (levels/event)
![Page 15: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/15.jpg)
HIRDLS HDF-EOS Swath
Metadata
Data Fields
HDF-EOS Swath File
Inst. Name,Ranges, Labels, Scales, Missing Values, Profile count
Time
Global Attributes
Geolocation Fields
Pressure Ozone Mixing Ratio
nLev
els
nTimes
Dimensions:nTimes, nLevels etc.
![Page 16: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/16.jpg)
HDF-EOS Extension Strengths
• Fast! Ten times faster than native (Bsel) format
• Easier to identify field names (and more flexible)
• Standard way of identifying missing data
• IDL/Fortran/C I/O interfaces
• Potentially more tools available
![Page 17: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/17.jpg)
HDF-EOS Extension Difficulties
• Lots of different ways to store data and still meet the standard.– Aura standard format guidelines helps here.
• HDF-EOS Swath isn’t “packet-like” – You can’t just “cat” files together!
• IDL/Fortran/C I/O interfaces– Some trouble with stability of certain I/Fs.
• Point Interface (and raw Vdata) failed from the start due to lack of “id” resources.
![Page 18: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/18.jpg)
Lessons Learned
• Developed general purpose swath reader in IDL – usage: data = get_sw_data(filename, swathname, varname)– Generally reads any variable or attribute.– http://www.eos.ucar.edu/hirdls/get_sw_data.pro
• Working out the best way to “easily select” variables from swaths is a work-in-progress.
• To map HDF to the internal data structure a small external data table was needed...
![Page 19: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/19.jpg)
Mapping Table HDF-EOS - Internal
SwathName VarName Label ID Min Max Log Scale UnitHIRDLS H2O H2O Mixing Ratio 2 0.0 10.0 0 1.0e-06 ppmvHIRDLS NO2 NO2 Mixing Ratio 3 0.0 10.0 0 1.0e-09 ppbvHIRDLS O3 O3 Mixing Ratio 4 0.0 10.0 0 1.0e-06 ppmvHIRDLS CH4 CH4 Mixing Ratio 6 0.0 2.0 0 1.0e-06 ppmvHIRDLS CFCl3 CFCl3 Mixing Ratio 151 0.0 0.5 0 1.0e-09 ppbvCH4 L2gpValue CH4 Mixing Ratio 6 0.0 2.0 0 1.0e-06 ppmvO3 L2gpValue O3 Mixing Ratio 4 0.0 10.0 0 1.0e-06 ppmv
Internal to toolkitInternal to HDF-EOS
Designations for MLS Files
![Page 20: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/20.jpg)
Summary
• HDF-EOS Swath format provides a good storage facility for atmospheric profile data.– Current swath implementation (a la Aura) provides solid
groundwork for future sophisticated software development.
• Interface instability can contribute to major headaches.
• Ancillary tools to “view and edit” or simply compute additional fields from existing ones could prove quite useful.
![Page 21: An IDL-BasedValidation Toolkit: Extensions to use the HDF-EOS Swath Format](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681514f550346895dbf7287/html5/thumbnails/21.jpg)
Future
• Extend data base table for mapping fields.
• Develop interface for Grid format.– Already tested in limited cases.
• Testing, testing, testing…
• Perhaps incorporate new IDL features– Improved color contouring
• Develop an easier to use interface for the scripting tools.