beat - the basic envisat atmospheric toolbox

Sander Niemeijer <[email protected]>

BEAT - The Basic ENVISAT Atmospheric Toolbox

QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.QuickTime™ and a

TIFF (Uncompressed) decompressorare needed to see this picture.

June 14, 2008Copenhagen

2 of 21

Introducing the BEAT softwareBEAT: the Basic ENVISAT Atmospheric Toolbox

Goal: to provide the necessary software to support scientific usage of atmospheric data

• One of the ESA ENVISAT Toolboxes• Open Source and freely available• Cross Platform (Linux, Windows, Mac OS X, etc.)


3 of 21

BEAT components

• BEAT package– BEAT-I direct product interface

– BEAT-II simplified data interface

• VISAN visualization & analysis application

• Scientific Tutorial• Scientific Modules (GeoFit)• Sample data


4 of 21

BEAT-I Overview

• BEAT Layer-I provides a ‘direct product interface’ to all data contained in a product file.

• Support for:– GOMOS, MIPAS, SCIAMACHY (ENVISAT)– GOME (ERS-2)– OMI, TES, MLS (Aura)– GOME-2, IASI (MetOp)– Any HDF4/5-based data (e.g. ground-based observations)– Most XML files (new since BEAT v4.2, future use for e.g. Aeolus)

• Directly ingest data from C, Fortran, Python, IDL, MATLAB• Command-line tools: beatcheck, beatdump, beatfind


5 of 21

BEAT-I: How does it work? (1)

Single Interface to all product data

ASCII/BIN

BEAT data dictionary

ENVISAT(PDS)

…GOME

(binary, ASCII)

netCDF

HDF4,HDF-EOS2

(Cal/Val data)

HDF5,HDF-EOS5

(Aura)

XML HDF4 HDF5

XML data dictionary

XML(Earth Explorer)

4 back-ends

MetOp(EPS)


6 of 21


Both self-describing and non-self-describing products are hierarchically structured (i.e., like a tree):

– records have fields, identified by name– arrays have entries, identified by index

BEAT-I understands this tree structure, and allows you to gointo branches by specifying…

– field names - e.g. “mph”, “dsr_time”, …– array indices - e.g. [1,2,3], [17], or just 17.


7 of 21


• BEAT-I includes a data dictionary containing electronic descriptions of all data formats that are not ‘self-describing’ and uses this to dynamically interpret a product

• Documentationof data dictionarydescription (i.e. fullformat descriptions)included with BEAT


8 of 21

The BEAT-I interface - examples

pf = beat_open(“SCI_OL__2P_…….N1”)

product = beat_fetch(pf)mph = beat_fetch(pf, “mph”)abs_orbit = beat_fetch(pf, “mph”, “abs_orbit”)time = beat_fetch(pf, “nad_uv0_o3”, [12], “dsr_time”)time = beat_fetch(pf, “nad_uv0_o3”, 12, “dsr_time”)all_times = beat_fetch(pf, “nad_uv0_o3”, -1, “dsr_time”)

result = beat_close(pf)


9 of 21

BEAT-II OverviewWhy a second layer?• Many data products have a very complicated structure that

reflect the way the data-processors work, not the way end-users look at the data.

• Correct matching of measured spectra/retrievals to geolocation (“ground pixels”) can be hard to get right.

• For typical analysis work, only a fraction of the data within the file is useful.

• It is difficult to compare data from different instruments.

BEAT-II addresses these issues by providing a simplified (but still valid!) view on the data.


10 of 21

BEAT-II Features (1)

• Only one call needed to ingest the most important data from a single product or multiple products;

• Ingest product data into ‘flat’ BEAT-II records;• BEAT-II records use native data type for each interface (e.g.

‘struct’ for MATLAB/IDL, ‘object’ for Python);• Each field corresponds with a specific scientific parameter;• Fields use standardized naming convention and provide

data in standardized units, thus allowing easy comparison;


11 of 21

BEAT-II Features (2)

• Powerful ingestion filter options to determine which parameters to ingest and to put restrictions on time, location, etc.;

• Provides useful algorithms for analysis:– filter data;– find co-located data;– import/export record to ASCII, binary, HDF4, and HDF5;

• Support for IDL, MATLAB, C, Fortran, and Python;• Command-line tool: beatl2dump;• BEAT-II is build on top of BEAT-I, but also supports:

– ACE-FTS, ODIN/OSIRIS


12 of 21

Reading SCIAMACHY data with BEAT-II

• The ingested SCIAMACHY data is returned to you using the same structure as you would get for GOME or OMI data, even though those data was stored very differently in the data files.

• Also, the call to the ingestion function is almost identical to the one you would use for GOME or OMI.

>>> scia = beatl2.ingest(‘SCI_OL__2P*’, ‘data=nadir_uv_0, longitude_max=-120, latitude_min=50, latitude_max=75, scan_direction=forward, convert_to_DU=1’)

>>> print scia

type: ’SCIAMACHY_L2_Offline_nadir_uv_0'

time: [2132 double]

latitude: [2132 double]

longitude: [2132 double]

corner_latitude: [2132 x4 double]

corner_longitude: [2132 x4 double]

o3_column: [2132 double]

o3_column_unit: 'DU'

o3_column_error: [2132 double]

o3_column_error_unit: ’%'

integration_time: [2132 double]

scan_direction: [2132 string]

>>> beat.time_to_string(scia.time[0])

’16-SEP-2005 05:27:49.339949'


13 of 21

The VISAN program

• Ingest data using BEAT-I and BEAT-II;

• Python language for command input and performing calculations and manipulations;

• Product Browser to inspect product structure and contents;

• With one function call, create interactive 2D and World plot visualizations of your data;

• Open Source and Cross-Platform: supports Windows, Linux, and Mac OS X.


14 of 21

Main Window & Product Browser

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.


15 of 21

Plotting SCIAMACHY data

>>> wplot(scia, colorrange=[150,500])

• Only a single command is needed to turn the ingested SCIAMACHY data into a visualization


16 of 21

VISAN - different projections


17 of 21

VISAN - GOMOS Level 1

QuickTime™ and a decompressor

are needed to see this picture.


18 of 21

Example: Co-locating data

• Co-locations can be performed based on time, earth surface distance, and/or altitude (the last three parameters of the beatl2.find_colocated_data function).

• After ingestion of SCIAMACHY and OMI data as show before, we us a distance parameter of 100 km to co-locate the data.

>>> [scia2, omi2] = beatl2.find_colocated_data(scia, omi, -1, 100,-1)


19 of 21

Example: VISAN Overplot

>>> w = wplot(scia2, colorrange=[150,500])

>>> wplot(omi2, colorrange=[150,500], window=w)

• VISAN overplots can be very helpful when you want to perform a quick visual comparison of data


20 of 21

Future of BEAT

More missions!

Aeolus

EarthCARE

…

But your feedback counts as well!


21 of 21

More Information?

http://www.stcorp.nl/beat/

[email protected]

beat - the basic envisat atmospheric toolbox

Documents