how to meet the cf conventions with ncml for nasa hdf/hdf-eos

The HDF Group

1 www.hdfgroup.orgJuly 8, 2014 2014 Summer ESIP Federation Meeting

How to Meet the CF Conventions with NcML for NASA HDF/HDF-EOS

Hyo-Kyung Joe Lee and Ted HabermannThe HDF Group

www.hdfgroup.org2014 Summer ESIP Federation Meeting 2

The CF Conventions

Big and complex!

Just focus on a few key conventions:• coordinate variables & attributes (bald)• valid_range / _FillValue (fat)• scale_factor / add_offset (short)• units ($ vs. ₵)

www.hdfgroup.org2014 Summer ESIP Federation Meeting 3

Why the key CF conventions matter

If a data product doesn’t follow them, your NetCDF visualization tool like IDV is useless.

www.hdfgroup.org2014 Summer ESIP Federation Meeting 4

But if you follow them carefully…

you can visualize data instantly & correctly.

www.hdfgroup.org2014 Summer ESIP Federation Meeting 5

OBPG L3 NetCDF-4 Example

The previous screenshots are created from a NASA OBPG beta product.

The product doesn’t work with IDV as is. But we made the product work by augmenting data with NcML.

www.hdfgroup.org2014 Summer ESIP Federation Meeting 6

NcML

NetCDF Markup Language

XML representation of netCDF metadata

It can be an input file for editing metadata.

HDF

www.hdfgroup.org2014 Summer ESIP Federation Meeting 7

Why NcML?

Isn’t CDL enough? No.

NcML has more features and flexibility.

Plus, NcML works with THREDDS Data Server.

www.hdfgroup.org2014 Summer ESIP Federation Meeting 8

What OBPG NC4 missed - #1

Coordinate variables are not provided.

Dataset { String palette[rgb = 3]; Int16 /geophysical_data/sst[rows = 4320][columns = 8640]; Byte /geophysical_data/sst_qual[rows = 4320][columns = 8640]; Float32 /geophysical_data/Lat[rows= 4320]; Float32 /geophysical_data/Lon[columns = 8640]; } testAll/A20030602003090.L3m_MO_AT109_SST_4.nc4;

The CF convention expects lat/lon variables.

www.hdfgroup.org2014 Summer ESIP Federation Meeting 9

Add new coordinate variables in NcML.

<netcdf location="c:/tomcat/webapps/thredds/share/testdata/cdmUnitTest/it/content/thredds/public/testdata/A20030602003090.L3m_MO_AT109_SST_4.nc4" xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"><dimension name="_geophysical_data_Lat" length="4320"/> <dimension name="_geophysical_data_Lon" length="8640"/> <group name="geophysical_data"> <variable name="Lat" shape="_geophysical_data_Lat" type="float"> <values start="90.0" increment="-0.0416"/> <attribute type="string" name="units" value="degrees_north" /> </variable> <variable name="Lon" shape="_geophysical_data_Lon" type="float"> <values start="-180.0" increment="0.0416"/> <attribute type="string" name="units" value="degrees_east" /> </variable> …

www.hdfgroup.org2014 Summer ESIP Federation Meeting 10

Now IDV can visualize data but…

No data on Sea w/ Range = [-1, -1]?Examine units.

www.hdfgroup.org2014 Summer ESIP Federation Meeting 11

What OBPG NC4 missed - #2

“units” attribute has “degrees-C” value. Accepted values in the CF Convention:Celsius / celsius / degree_centigrade / degC /degreeC / degree_C / degree_c / deg_C / deg_c

Remove “s-” from “degrees-C”.

www.hdfgroup.org2014 Summer ESIP Federation Meeting 12

Modify attribute value with NcML.

… <variable name="_sst" shape="_geophysical_data_Lat _geophysical_data_Lon" orgName="sst"> <attribute type="string" name="units" value="degreesC" /> </variable>…

Before:

After:

www.hdfgroup.org2014 Summer ESIP Federation Meeting 13

What OBPG NC4 missed - #3

The actual fill value is -1 according to HDFView.

Attributes {/geophysical_data/sst { Int16 _FillValue -32767; Float64 valid_min -5.0; Float64 valid_max 50.0; }

HDFView

www.hdfgroup.org2014 Summer ESIP Federation Meeting 14

Modify _FillValue attribute with NcML.

… <variable name="_sst" shape="_geophysical_data_Lat _geophysical_data_Lon" orgName="sst"> <attribute type="string" name="units" value="degreesC" /> <attribute type=“short" name=“_FillValue" value=“-1" /> </variable>…

Now every value is a fill value – “missing” in IDV.

www.hdfgroup.org2014 Summer ESIP Federation Meeting 15

What OBPG NC4 missed - #4

valid_min / valid_max doesn’t match the range of actual data.

Attributes {/geophysical_data/sst { Int16 _FillValue -32767; Float64 valid_min -5.0; Float64 valid_max 50.0; }

No scale/offset attributes are defined either.

www.hdfgroup.org2014 Summer ESIP Federation Meeting 16

Add scale/offset attributes with NcML.

<variable name="_sst" shape="_geophysical_data_Lat _geophysical_data_Lon" orgName="sst"> <attribute type="string" name="units" value="degreesC" /> <attribute type="float" name="scale_factor" value="0.000717185"/> <attribute type="float" name="add_offset" value="-2.0"/> <attribute type="float" name="valid_min" value="-5.0"/> <attribute type="float" name="valid_max" value="50.0"/> <attribute type="short" name="_FillValue" value="-1"/>

</variable>

Please note that all types match as float._FillValue’s type matches the data type.

www.hdfgroup.org2014 Summer ESIP Federation Meeting 17

Finally, OBPG met the CF conventions.

www.hdfgroup.org2014 Summer ESIP Federation Meeting 18

But tools behave differently.

The same NcML doesn’t work for GoDiva2. NC4 “Group” is the reason. GoDiva2: lat/lon outside groupIDV/Panopy: lat/lon inside group

Is CF/NcML ready for “Group”?

www.hdfgroup.org2014 Summer ESIP Federation Meeting 19

GoDiva2

www.hdfgroup.org2014 Summer ESIP Federation Meeting 20

Panoply

www.hdfgroup.org2014 Summer ESIP Federation Meeting 21

Acknowledgement

This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA). Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.