fmi open data interface and usage
TRANSCRIPT
Open Meteorological DataUsage and Best Practices
Finnish Meteorological InstituteRoope Tervo
05/02/2023 2
Finnish Meteorological Institute opened its data in 2013.
Basically everything FMI has rights was opened.
Data is provided freely in machine readable format.
FMI Open Meteorological Data Services | Roope Tervo
FMI Open Data
https://en.ilmatieteenlaitos.fi/open-data
05/02/2023 3
FMI Open Data Portal follows INSPIRE requirements.
FMI Open Data Portal
Meta data
Data Models
Services
The very same data portal works as Open Data and INSPIRE portal.
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 4
Data set Description Time Interval
Estimated publish date
Weather Observations
Temperature, Wind, Humidity, Ground Temperature…
10 min Open,older data to be added
Sun Radiation UV, Short and Long Term Radiation…
1 min Open
Marine Observations
Waves, Sea Temperature, Sea Level…
1 h Open
Weather Radars Precipitation Rate, Precipitation Amount…
5 min Open, older and more data to be added
Lightning Thunder Strikes in Finland
5 min Open
Example of Data Sets
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 5
Example of Data SetsData set Description Time Interval Estimated
publish date
Real Time Observations
Real Time Observations from specific location(s)
AWS 2010 – Soundings 1959 –Flashes 1998 – Sea Level 1971 – Waves 2005 –
Openolder data will be added
Climatological Observations
Daily and monthly temperature mean and extreme values from weather stations
1959 - Open
Climatological Observations
Monthly temperature and precipitation rate mean values interpolated to grid
1961 - Open
Climatological Reference
Climatological Reference. Temperature, humidity, pressure, precipitation amount and snow depth.
Reference seasons:1971-2000 1981-2010
Open
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 6
Example of Data SetsData set Description Time Interval Estimated
publish dateWeather forecast model HIRLAM RCR
Point forecasts and grid data
Latest model run(4 times a day) 0…54 h
Open
Sea forecast models Sea level point forecasts, Wave (WAM) and current (HBM) as grid data
Latest model run(4 times a day) 0...54 h
Open
Environmental Monitoring Facilities
Weather observation stations, radars…
- 2017
Aviation Observations
METAR/IWXXM 30 min Open
Mast observations Special observations from masts
open
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 7
Example of Data SetsData set Description Time Interval Estimated
publish dateAir Quality Observations
Air Quality Observations 1h open
Silam Model Dispersion Model for Air Quality, Forest Fire and Pollen
Latest model run (once a day)0…96h
2017
HELMI Ice Model Ice forecast model Latest model run(4 times a day) 0...54 h
open
Soundings Temperature, Humidity, Pressure, Wind from ground to 25 km height
2 times a day open
Quality information
Quality information for observations
open
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 8
Registrationo Registration is required to use View and Download
Services o Working email address is the only mandatory
informationo After registration the user gets an API key which have to
be added into all requestso GET parameter fmi-apikey=…&o Header fmi-apikey; …o Part of url http://wms.fmi.fi/fmi-apikey/…/wms?
o One can create several API keys with one email
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 9
Usage LimitsWith one API key it’s allowed to
o do at most 20 000 requests per day to Download Serviceo do at most 10 000 requests per day to View Serviceo do at most 600 requests per 5 minutes to both services
o If all observations from one time step is calculated to as one, little over 17 000 new data sets are published daily
o So, with one API key it’s allowed load everything onceo View service can be used for testing but can not be used as a
back end for popular clients
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 10
Catalog Service (CSW)
o Based on GeoNetworko High-level metadatao Provides UIo Maybe more useful sources:
o http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services (based on WFS DescribeStoredQueries)
o http://www.paikkatietoikkuna.fi/web/fi/metatietohaku (based on GeoNetwork)
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 11
View Service (WMS)o Based on GeoServero Visualize observation and
forecast data and provides radar images
o Only the most common layers are published
o Can be used for prototyping - not for production use
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 12
Download Service (WFS 2.0)o Web Feature Service (WFS) 2.0
Simple Profileo Based on stored queries
o Predefined data sets with possibility for additional parameters (i.e. time and area)
o In-house production
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
13
GetCapabilities:• Describes capabilities of the service• Normally start point for client applications• http
://data.fmi.fi/fmi-apikey/0017a4fc-b1e9-4851-850c-0f9ff66f8cae/wfs?request=GetCapabilities
WFS Basicsobseravtions and point forecasts
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
14
XSLT
DescribeStoredQueries• List and describes available stored queries (pre-defined
data sets)• Stored query name and description
• Possible additional parameters for the data query
• http://en.ilmatieteenlaitos.fi/open-data-manual-fmi-wfs-services
WFS Basicsobseravtions and point forecasts
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
15
GetFeature• Actual data query• In FMI service, stored query id is compulsory
• &storedquery_id=..
• Other possible parameters depend on the stored query• Possible values can be fetched with DescribeStoredQueries
request.
• Location has to be often defined some how.
http://data.fmi.fi/fmi-apikey/.../wfs?request=GetFeature&storedquery_id=fmi::observations::weather::cities::timevaluepair&bbox=21,62,22,64
WFS Basicsobseravtions and point forecasts
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
16
Location parameters in stored queries& bbox area
• By default, in same srs with the (requested) srs. Specify as fifth element if needed (i.e. &bbox=132249,6433579,631999,6933329,epsg:3067)
& fmisid FMI id for observation station& wmo WMO id for observation station
WFS Basicsobseravtions and point forecasts
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
17
Location parameters in stored queries& geoid http://geonames.org id for location& place human readable name
• use region to accurate your query (i.e. &place=kumpula,tampere)
• can be used several times in one query (i.e. &place=kumpula,helsinki&place=heinola)
& latlon coordinates (used only in marine data sets)
WFS Basicsobseravtions and point forecasts
05/02/2023 18
GetPropertyValue• Responses only requested part of the GetFeature response• Use ValueReference=xpath to define part of interest• Same additional parameters for stored query with GetFeature
request are still valid • For example: http://data.fmi.fi/fmi-apikey/…/wfs
?request=GetPropertyValue&storedquery_id=fmi::observations::weather::multipointcoverage&place=kumpula,tampere&ValueReference=wfs:FeatureCollection/wfs:member/omso:GridSeriesObservation/om:result
FMI Open Meteorological Data Services | Roope Tervo
WFS Basicsobseravtions and point forecasts
05/02/2023 19
Data Modelso Observations and point
forecasts as GMLo The same data is published in
two different formats.o Gridded data is provided in
appropriate binary format (grib, NetCDF, HDF…)o WFS members contains the
metadata ‘envelope’ with a link to a actual data
FMI Open Meteorological Data Services | Roope Tervo
o Every XML document as self-explanatory as possivble
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
20
om:phenomenonTime• Tells start and end time of returned data
Data Modelsmost important elements
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
21
om:procedure• Describes data producing procedureom:parameter• Changing parameters in the process
• i.e. analysis time of the numerical model
Data Modelsmost important elements
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
22
om:featureOfInterest• Describes location(s) or area of returned data• Several different kind of identifiers
• target:SurfaceWeatherTargetArea area (grid data)
• fmisid FMI id for observation station
• wmo WMO id for observation station
• geoid http://geonames.org id for location
• name Human readable name
• region county of the station
Data Modelsmost important elements
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
23
om:observedProperty• Describes meteorological parameters in returned data• Defined as xlink• Describes label, basePhenomenon and units of measure of
every parameter in the reponse• Tip: use the same service without parameter definitions to
query all possible parameters• http://data.fmi.fi/fmi-apikey/.../meta?observableProperty=forecast
• Note that all parameters are not available in all data sets
Data Modelsmost important elements
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
24
om:result• Data in required format in observations and point forecasts and link to a
external binary data in grid formatted data
Data Modelsmost important elements
05/02/2023 25
Data Modelsgmlcov:MultiPointCoverage
gml:rangeSetgml:doubleOrNilReasonTupleList
The data is listed for every point defined in domain set.
gml:domainSetgmlcov:simpleMultiPoint
The coverage is defined as a list of
points in 4 dimensional grid (lat,
lon, height, time).
gmlcov:rangeType
The parameters listed in range set
are defined in separate element.
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 26
Cons- Not intuitive- No natural
structure of XML XSLT and Xpath don’t work
Pros+ Compact+ Efficient+ Small file size+ Works for many
data types
Data Modelsgmlcov:MultiPointCoverage
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 27
Data Modelswml2:MeasurementTimeseries
wml2:MeasurementTimeseries
One member contains time series of one parameter and
one location
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 28
Cons- Lots of repetition- Large file size- Heavy for DOM-
based parsers- Don’t work i.e. for
thunder strikes
Pros+ Intuitive+ Easy to use+ XSLT & XPath
works
Data Modelswml2:MeasurementTimeseries
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 29
Data ModelsSimpleFeature
SimpleFeature
One member contains one time step of one parameter
and one location
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 30
Cons- Huge file size- Heavy for DOM-
based parsers
Pros+ Intuitive+ Easy to use+ Wide application
support
Data ModelsSimpleFeature
FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 31
Data Models file size comparison
Simple
Featur
e
Measu
remen
tTim
eseri
es
MultiPoin
tCov
erage
0
20
40
60
80
10081.7
52.9
1.81.3 1.2 0.2
Document Size [MB]
Compressed DocumentSize [MB]
o 138 o
bser
vatio
n stat
ions
o 11 pa
ramete
rs
o 72 tim
e step
s (12
hour
s)
FMI Open Meteorological Data Services | Roope Tervo
Data Type Data Format
Observations wml2:MeasurementTimeseries
gmlcov:MultiPointCoverage
SimpleFeature
Point Forecasts wml2:MeasurementTimeseries
gmlcov:MultiPointCoverage
SimpleFeature
Lightning Observations gmlcov:MultiPointCoverage
SimpleFeature
Grid Forecasts XML Envelope + Grib2/NetCDF
Radar Images GeoTiff / PNG images
METAR IWXXM
05/02/2023 32FMI Open Meteorological Data Services | Roope Tervo
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
33
• Aviation weather reposts are delivered as IWXXM• New data model coming into use in aviation
• Consists of the same elements than other messages• om:phenomenonTime, om:procedure, om:featureOfInterest,
om:result
• Content of the METAR is in om:result part as• extracted into XML elements
• original, “old fashion”, METAR string
Data Modelsaviation observations IWXXM / METARS
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
34
• Once published, content and structure of IWXXM messages will not change
• Messages will contain a digital signature
• Use GetPropertyValue to download only original METAR string
• http://data.fmi.fi/fmi-apikey/…/wfs?request=GetPropertyValue&storedquery_id=fmi::avi::observations::finland::latest::iwxxm&valuereference=wfs:FeatureCollection/wfs:member/avi:VerifiableMessage/avi:metadata/avi:MessageMetadata/avi:source/avi:Process/avi:input
Data Modelsaviation observations IWXXM / METARS
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
35
• Messages have been automatically created based on the original METAR/SPECI code messages.
• Messages are NOT an authoritative aviation weather report. They SHOULD NOT be used as a weather report to be used in flight planning or other aviation related use.
Data Modelsaviation observations IWXXM / METARS
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
36
• WFS response contains the same meta data information than in observations and point forecasts
• om:result contains gmlcov:RectifiedGridCoverage• Basically the same with multipointcoverage
• domainSet defines the grid (now as regular grid)
• rangeSet contains data as an external link to the binary content• rangeType defines the parameters
Data Modelsbinary data
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
37
• Note that fileReference points often to an other service• Can be used directly, but
• It is always good practice to consult WFS for available times and parameters
Data Modelsbinary data
Client
WFS
WMSFMI Data Server
Available radar images?Available weather model outputs?
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
38
• Area, time and weather parameters may be defined in the request
• For weather models, intersection of requested area and available data area is returned
• For radar images, all images which intersects requested spatial and temporal space are returned
Data Modelsbinary data
Requested area
Radar images
Radar images Requested area
Model coverage
Weather models
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
39
Radar images• The reference points to a “original” gray scale GeoTiff image• Images can also be downloaded as color images
• Remove &styles=raster from the data request
• But then information is lost
• Used SLD-files can be downloaded from https://github.com/fmidev/opendata-resources/tree/master/sld
• Consult om:parameter element for single radar measurement parameters
• Scanning angle, bin count and bin length
Data Modelsbinary data
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
40
Numerical models• The reference points to (often a subset) the model output in
appropriate format• GRIB2 for weather models
• NetCDF for marine models
• Note that whole model output can be large• Up to 17 GB
• Download always only area, parameters and time range of interest
Data Modelsbinary data
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
41
For binary data• PanoPly
• http://www.giss.nasa.gov/tools/panoply/
• Simple
• Handle only very basic data
• IDV• http://www.unidata.ucar.edu/software/idv/
• Handle most of the data
Helpful Software
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
42
For exploring the data• Google Earth & QGIS• Start from GetCapabilities, UI guides thereafterhttp
://wms.fmi.fi/fmi-apikey/.../geoserver/wms?service=WMS&version=1.3.0&request=GetCapabilities
Helpful Software
05/02/2023 FMI Open Meteorological Data Services | Roope Tervo
43
Libraries• MetoLib (JS) https://github.com/fmidev/metolib• Weather Downloader (Python)https
://github.com/Tumetsu/FMI-weather-downloader• API R client (R) https://github.com/rOpenGov/fmi
Helpful Software
05/02/2023 44
Please contribute!
https://github.com/fmidev/
MetoLibo Open source JavaScript library produced by Finnish
Meteorological Instituteo Helps users to load and use the data
o Supports multi point coverage data formato Python version is on the list
Easy requests Cache
Parse the data to as JSON
FMI Open Meteorological Data Services | Roope Tervo
www.fmi.fi