world meteorological organization ra-ii/vi regional training on table driven codes muscat, oman, 10...

WORLD METEOROLOGICAL ORGANIZATION

RA-II/VI Regional Training on Table Driven CodesMuscat, Oman, 10 - 14 December, 2005

An Introduction to GRIB2An Introduction to GRIB2Simon Elliott

[email protected]



Saturday Afternoon AgendaSaturday Afternoon Agenda

• Background (i.e. my background!)• Overview (codes, GRID, GRIB, GRIB2)• Code Structure (Sections like others, iterations of

sections)

• GRIB2 Code Templates (what kinds are there … examples)

• Tables (Template tables, code and flag tables, examples)

• Identification of parameters (?)• GRIB2 compression (methods, explanations of some)



Sunday Morning AgendaSunday Morning Agenda

• Procedure for code form update (addns to tables, addition of tables/templates, new editions)

• Changes not requiring code change (adding table/template entries for decoders … down stream code change)

• Changes requiring code change (always for data provider, plus new editions)

• Case studies of Cloud Mask and Precipitation Data (bespoke Encoder and Decoder, example product)

• GRIB2 Interface for NMC (example of NCEP code)• Review/Discussion



OverviewOverview• Intention is to share information efficiently and

unambiguously• Data should be accessible by anyone, not encrypted• Format should allow all information to be included (not

rounded or skipped)• Format should allow addition of new data types• WMO “codes” developed per data type, TEMP, SYNOP,

SATOB, et c.

FM 13 SHIPFM 42 AMDARFM 41 CODAR

FM 35 TEMP FM 88 SATOBFM 87 SARADFM 86 SATEMFM 85 SAREP



GRID, GRIB, GRIB2GRID, GRIB, GRIB2• GRID was developed for gridded data (forecast, analysis)

– ASCII -> human readable?– ASCII -> big files, slow transfer

– Symbolic letters and code tables: F1F2NNN in Section 0

• F1F2 is originating centre as per C-1,

• NNN is catalogue number of grid used by centre

• GRIB introduced for binary exchange (storage and transmission efficiency).– Edition 0, 1985– Edition 1, 1990

• GRIB Edition 2 (i.e. GRIB2) because Code Table 2 was full, EPS data, ...



GRIB uses the concept of Template:GRIB uses the concept of Template:« Description of the standardized layout of a set « Description of the standardized layout of a set

of data items »of data items »

GRIB1 structureGRIB1 structure• Section 0 Indicator section: “GRIB”• Section 1 Product definition section: unique standard

template• Section 2 Grid description section: one of the standard

templates describing a type of grid• Section 3 Bit map section• Section 4 Binary data section• Section 5 End section: “7777”



GRIB usageGRIB usage• On the GTS• By the WAFS Centres (« ICAO » products)• For archiving of fields• on MDD

Main limitation is:• One parameter on one level for one grid per

GRIB field



GRIB1 weaknessesGRIB1 weaknesses Transmission and archiving of:• Spectral data• Multi-dimension data• Long-range and climate products• Ensemble products (EPS)

Also:• No convention for missing data• IEEE not used for floating point data• No support for small time steps• No cross-sections, no time-sections• No Hovmöller Diagrams (ex. Time-longitude)• Limited support for images



Need for “object oriented” GRIB2 Need for “object oriented” GRIB2 (more modules = modularity)(more modules = modularity)

• Modularity: code and parameter tables referred to through templates

• Flexibility: new tables and templates can be added

• ALSO:– All GRIB1 fields can be described in GRIB2– More compression schemes (e.g.

introduction of JPEG 2000 and PNG)



GRIB2 calendarGRIB2 calendar

• Experimental GRIB2 presented at CBS 98• Finalized in Spring 2000• Approved by CBS in Autumn 2000• Operational in November 2001• First products in 2003



Generic structureGeneric structure

• Identification: GRIB/BUFR/CREX• Header: Date, time, originator,

table versions ...• Optional section: Metadata

(potentially XML), private data …• Data description: What sort of data

follows• Actual data: here• Closure: “7777”

Table driven codes

generally have this structure



GRIB2 structureGRIB2 structure• Section 0 Indicator section: “GRIB”

• Section 1 Identification section: data characteristics

• Section 2 Optional (local) section: anything or nothing

• Section 3 Grid definition section: geometry of grid used, potential reference to pre-defined grid

• Section 4 Product definition section: description of data type, e.g. satellite data from spectral bands xx, yy, zz

• Section 5 Data representation section: packing method used for data, reference value, scale factor et c.

• Section 6 Bit map section: data present indicators (if used)

• Section 7 Data section: data values themselves

• Section 8 End section: “7777” Note: Sections 1 to 7 start withsection length and number



Section 0Section 0

Section 0: Indicator section– Octets 1 - 4: “GRIB” in ASCII (i.e. 71 82 73 66)– Octets 5 - 6: Reserved (normally set to 0)– Octet 7: Discipline (CT 0.0, e.g. 3 for space)– Octet 8: GRIB edition number (2)– Octets 9 - 16:Total length in octets

Messages of length up to ~18x1018 (or 264 - 1) bytes, i.e. ~18,000,000 Tb



Section 1Section 1

Section 1 Identification section– Octets 1 - 4: Section length– Octet 5: Section number (1)– Octets 6 - 9: Originating centre and sub-centre– Octets 10 - 11: Master and local table version numbers– Octet 12: Time significance (CT 1.2, e.g. 3 for

observation time)– Octets 13 - 19: Date / time– Octet 20: Production status of data (CT 1.3, e.g. 0

for operational data) – Octet 21: Type of data (CT 1.4, e.g. 7 for

processed radar observations)



Section repetitionSection repetitionGRIB2 allows some groups of sections to be repeated for

efficiency (not in GRIB1, one field per message).

Non-repeated sections stay in effect

• Section 0 (Indicator)• Section 1 (Identifier)• Section 2 (Local use)• Section 3 (Grid)• Section 4 (Product)• Section 5 (Data Repn)• Section 6 (Bit map)• Section 7 (Data)• Section 8 (Ending)

Rep

eata

ble

Rep

eata

ble

Rep

eata

ble



Section repetition (example 1)Section repetition (example 1)

Forecast from same centre on same grid for same parameter at same level but from but for different validity times

Validity time is in Section 4 (Product Definition)

Section 4 to 7 repeated in message

• Section 0 (Indicator)• Section 1 (Identifier)• Section 3 (Grid)• Section 4 (Product)• Section 5 (Data Repn)• Section 6 (Bit map)• Section 7 (Data)• Section 4 (Product)• Section 5 (Data Repn)• Section 6 (Bit map)• Section 7 (Data)• Section 4 (Product)• Section 5 (Data Repn)• Section 6 (Bit map)• Section 7 (Data)• Section 8 (Ending)

0600Z

1200Z

1800Z



• Section 0 (Indicator)• Section 1 (Identifier)• Section 3 (Grid)• Section 4 (Product)• Section 5 (Data Repn)• Section 6 (Bit map)• Section 7 (Data)• Section 3 (Grid)• Section 4 (Product)• Section 5 (Data Repn)• Section 6 (Bit map)• Section 7 (Data)• Section 3 (Grid)• Section 4 (Product)• Section 5 (Data Repn)• Section 6 (Bit map)• Section 7 (Data)• Section 8 (Ending)

Section repetition (example 2)Section repetition (example 2)

Forecast from same centre for same validity time for same parameter at same level but from but on different grids

Grid is defined in Section 3 (Grid Definition Section)

Section 3 to 7 repeated in message

GRID A

GRID B

GRID C



Grid definition templatesGrid definition templates• Grid definition is in Section 3 (GRIB1, Section 2)• Can refer to centre’s own grid or use Grid Definition

Template (GDT) to specify details• Many GDTs have been developed, e.g.

– 3.0: Latitude/Longitude (or equidistant cylindrical, or Plate Carrée)

– 3.1: Rotated Latitude/Longitude (or equidistant cylindrical, or Plate Carrée)

– 3.2: Stretched Latitude/Longitude (or equidistant cylindrical, or Plate Carrée)

– 3.3: Stretched and Rotated Latitude/Longitude (or equidistant cylindrical, or Plate Carrée)

– 3.20: Polar Stereographic

– 3.90: Space view perspective, or orthographic



A grid definition template exampleA grid definition template exampleGDT 3.90 - Space view perspective or orthographic

Floating point numbers stored in 5 bytes as scale factor (one byte) and scaled value (4 byte IEEE floating point)

OctetNo.

Contents

15 Shape of Earth (CT 3.2: Spherical, oblate sphere, ICAOshape)

16-20 Radius of spherical Earth

21-30 Major and minor radii of oblate spherical Earth

31-38 Number of pixel columns and rows in grid

39-46 Latitude and longitude of sub-satellite point

47 Resolution and component flags (FT 3.3: Referencedirection for vector components, grid or east/north)



OctetNo.

Contents

48-55 Apparent diameter of Earth in each direction

56-63 Co-ordinates of sub-satellite point

64 Scanning mode (FT 3.4: location of consecutive scan pointand lines, e.g. 1st row in –x dirn, all scan rows in same dirn)

65-68 Orientation of grid (i.e. skeweness relative to longitudemeridien)

69-72 Height of camera in equatorial radii units, scaled by 106

73-80 Co-ordinates of origin of image

GDT example continuedGDT example continued

Repeating sections 4 to 7 means this information need not be repeated for multiple fields in one GRIB2 message



Product definition templatesProduct definition templates• Product definition is in Section 4 (GRIB1, Section 1)• Many PDTs have been developed, e.g.

– 4.0: Analysis or forecast on horizontal level or layer at a point in time– 4.1 to 4.4: Various information pertaining to ensemble forecast

systems– 4.5: Probability forecast on horizontal level or layer at a point in time– 4.7: Analysis or forecast error on horizontal level or layer at a point in

time– 4.20: Radar products– 4.30: Satellite products

• Hybrid sigma levels can be specified in Section 4



A product definition template A product definition template exampleexample

PDT 4.30 - Satellite products

OctetNo.

Contents

10 Parameter category (CT 4.1: For space products, i.e.discipline 3, 0 = image, 1 = data are in physical units)

11 Parameter number (CT 4.2: radiance, albedo, brightnesstemperature, skin temperature, cloud mask, …)

12 Type of generating process (CT 4.3: analysis, forecast,observation, …)

13 Observation generating process identifier (locally defined)

14 Number of contributing spectral bands, NB



PDT example continuedPDT example continued

OctetNo.

Contents

15-16 Satellite series of band 1 (defined by originating centre)

17-18 Satellite number of band 1 (defined by originating centre)

19 Instrument type of band 1 (defined by originating centre)

20-24 Central wave number of band 1 (m-1)

Note: Octets 25 to 24 are given for band 1, but are repeated for each contributing band up to NB, as specified in octet 14



Code table 4.3: Type of generating Code table 4.3: Type of generating processprocess

Code figure012345678

192 - 254255

MeaningAnalysis

InitialisationForecast

Bias corrected forecastEnsemble forecastProbability forecast

Forecast errorAnalysis errorObservation

Reserved for local useMissing



Data representation templatesData representation templates

• Data representation is in Section 5 (GRIB1, within Section 4)• Some DRTs have been developed, e.g.

– 5.0: Grid point data - simple packing– 5.1: Matrix values at grid point - simple packing– 5.2: Grid point data - complex packing– 5.3: Grid point data - complex packing and spatial differencing– 5.50: Spectral data - simple packing– 5.51: Spherical harmonics data - complex packing

• Section 5 also gives total number of data values to be found in Section 7



A data representation template A data representation template exampleexample

DRT 5.0 - Grid point data - simple packing

OctetNo.

Contents

12-15 Reference value, R (IEEE floating point number)

16-17 Binary scale factor, E

18-19 Decimal scale factor, D

20 Number of bits used for each packed value

21 Type of original field values (CT 5.1: Floating point orinteger)

IEEE floating point numbers in 32 bits:seeeeeee emmmmmmm mmmmmmmm mmmmmmmm



Simple and complex packingSimple and complex packing• Simple packing

– Y x 10D = R + (X1 + X2) x 2E

– Y is original value, D is decimal scale, R is reference, E is binary scale, X1 is 0, and X2 is encoded value

– In template R, E, D and bits per value are stored– Field values follow sequentially

• Complex packing (intended to decrease message size)– Y x 10D = R + (X1 + X2) x 2E

– R, E, D are as for simple packing– X1 is reference for group, X2 is scaled value in group (X1

removed)– Data are split into groups with similar values per group



Use of D and EUse of D and E D E Increment 0 0 1.000-1 -3 1.250 1 4 1.600 0 1 2.000-1 -2 2.500-2 -5 3.125 1 5 3.200 0 2 4.000-1 -1 5.000-2 -4 6.250 0 3 8.000

D E Increment-1 0 10.000-2 -3 12.500 0 4 16.000-1 1 20.000-2 -2 25.000-3 -5 31.250 0 5 32.000-1 2 40.000-2 -1 50.000-3 -4 62.500-1 3 80.000

D D - 1 Increment Increment x 10

Here R = 0, X1 = 0, X2 = 1Increment = Y = original value



R, E and D selection exampleR, E and D selection example

Temperature on balcony: -15.0 to 35.0, ±0.2

• Set D to give increments in range 1 - 10– D = 1, i.e scale by 10– data are now: -150 to 350, ±2

• Set R to minimum– R = -150.0 (IEEE floating point)– data are now: 0 to 500, ±2

• Set E to give required precision– E = 1, i.e. data are in increments of 2

• Set bit width to give required range– Required range = scaled range / increment– Required range = 500 / 2 = 250 so 8 bits

R = -150, E = 1, D = 1, bit width = 8



Spatial differencingSpatial differencing

• Saves space for data which vary smoothly (order 1), or data whose variation varies smoothly (order 2), et c.

• For order 1, data values, f, define new values, g:

– g1 = f1, g2 = f2 - f1, …, gn = fn - fn-1

– 1, 2, 3, 5, 7, 8, 9, 11 becomes 1, 1, 1, 2, 2, 1, 1, 2• For order 2, g values are replaced by h:

– h1 = f1, h2 = f2, h3 = g3 - g2, …, hn = gn - gn-1

– 1, 2, 4, 10, 17, 27, 40, 57 becomes 1, 2, 1, 4, 1, 3, 3, 4• Minimum value is subtracted to keep values positive

Order 1Complex packing

4 groups

Order 2Complex packing

3 groups



Section 6Section 6

Section 6: Bit map section– Octets 1 - 4: Length of section– Octet 5: Section number (6)– Octet 6: Bit map indicator (CT 6.0)– Octets 7 - xx:Bit map (if present)

Code figure0

1 - 253254255

MeaningBit map is present and follows in this sectionUse bit map predefined by originating centre

Re-use previously defined bit mapMissing (no bit map present)

Code Table 6.0: Bit map section



Bit map case studyBit map case study

Conclusion: bit map may not always save space, but if more than two bits per data point (typically true) it usually will …

consider cloud top height data, space corners, et c.

2000

2000

No bit map

2000 x 2000 x 8= 32,000,000 bits

No data

Data (8 bits per pixel)

With bit map

bit map(2000 x 2000 x 1)= 4,000,000 bits

data( x 1000 x 1000 x 8) 25,130,000 bits

total 29,130,000 bits



Updates not requiring code changeUpdates not requiring code change• Type of changes concerned

– Addition of code and or flag table entries

• Agreement and implementation– “Fast Track” can be used– Table entries can be approved for pre-operational use within a few

months of consideration and successful validation– Full formal CBS approval follows, available around 2 years later

• Application updates– Encoder needs new code / flag table available– Encoder software needs updating to insert field– Decoder needs new code / flag table available– Down-stream application (beyond decoder) needs to be updated to

process new field– Decoder software can remain unchanged



Updates requiring code changeUpdates requiring code change• Type of changes concerned

– Changes to code structure (new edition), or new GDT, PDT or DRT

• Agreement and implementation– ET/DRC considers a request and proposes update– Multilateral validation of proposed update– Full formal CBS approval follows, available around 2 to 4 years later

• Application updates– Encoder needs new software and /or template available– Encoder software needs updating accordingly– Decoder needs new software and /or template available– Down-stream application (beyond decoder) needs to be updated to

process new data

• Long lead time necessary … consider PUMA example



Interfacing to third party softwareInterfacing to third party software• Use of third party generic software simplifies

implementation• NOAA provides GRIB2 software for free download (look at

http://www.nws.noaa.gov/tdl/iwt)• ECMWF provides GRIBEX software - will be / is being

updated for GRIB2 (look at http://www.ecmwf.int/products/data/software/grib2.html)

• Typically software package is set of functions compiled into a library

• User’s application sets up required parameters and calls function from library, which are referenced at run time

• Specific (not generic) encoders and decoders are quite simple but less flexible



Some GRIB2 data from EUMETSATSome GRIB2 data from EUMETSAT

• Cloud mask, CLM, every 15 minutes, ~3km resolution, via EUMETCast or archive request.

• Cloud top height, CTH, every 60 minutes, ~9km resolution, via EUMETCast or archive request.

• Cloud analysis image, CLAI, every 180 minutes, ~9km resolution, via EUMETCast or archive request.

• Clear sky reflectance map, CRM, twice per week, ~3km resolution, via EUMETCast or archive request.

• Multi-sensor precipitation estimate, MPE, every 30 minutes, ~5km resolution, via web site.

• Fire detection, FIR, every 15 minutes, ~3km resolution, via anonymous FTP (evolving algorithm).



Case study A, precipitation dataCase study A, precipitation data



Case study A, precipitation dataCase study A, precipitation data

• GRIB2 encoded file

• Specific decoder program for GRIB2 precipitation data

• Decoder output

• Dump of GRIB2 file



Case study B, cloud mask dataCase study B, cloud mask data



Case study B, cloud mask dataCase study B, cloud mask data


• Specific en/decoder program for EUMETSAT cloud mask data

• Decoder output

• Dump of GRIB2 file

• Original data to be encoded



Case study C, cloud analysis image Case study C, cloud analysis image datadata

GRIB2 Cloud analysis image



Case study C, cloud analysis image Case study C, cloud analysis image datadata



• Bit map used



Case study D, cloud top height dataCase study D, cloud top height data

GRIB2 Cloud top height



Case study D, cloud top height dataCase study D, cloud top height data



• Bit map used• Two repeats of Sections 4, 5,

6 and 7



Sources of GRIB2 informationSources of GRIB2 information

The information used in preparing this presentation is based on:

Guide to WMO Table Driven Code Form Used for the Representation and Exchange of Regularly Spaced

Data in Binary Form: FM 92-XII GRIB

Written by Dr Cliff Dey

WMO Manual on Codes, WMO Publication No. 306, Vol. 1, Part B

&

world meteorological organization ra-ii/vi regional training on table driven codes muscat, oman, 10...

Documents