open weather data as part of big data

Open Meteorological Data as

Part of Big Data

Finnish Meteorological Institute

Roope Tervo, Mikko Visa, Tarja Riihisaari

Finnish Meteorological Institute opened its data in 2013.

Basically everything that FMI has property rights was opened.

Data is provided in freely in machine readable format.

30.1.2015 Open Meteorological Data as part of Big Data / Roope Tervo 2

FMI Open Data

https://en.ilmatieteenlaitos.fi/open-data


Data set Description Time

Interval

Estimated

publish date

Weather

Observations

Temperature, Wind,

Humidity, Ground

Temperature…

10 min Open,

older data to be

added

Sun Radiation UV, Short and Long

Term Radiation…1 min Open

Marine

Observations

Waves, Sea

Temperature, Sea

Level…

1 h Open

Weather Radars Precipitation Rate,

Precipitation Amount…5 min Open,

older data to be

added

Lightning Thunder Strikes in

Finland5 min Open


Example of Data Sets



Data set Description Time Interval Estimated

publish date

Real Time

Observations

Real Time Observations from

specific location(s)

AWS 2010 –

Soundings 1959 –

Flashes 1998 –

Sea Level 1971 –

Waves 2005 –

Open

older data will

be added

Climatological

Observations

Dayly and monthly

temperature mean and

extreme values from weather

stations

1959 - Open

Climatological

Observations

Monthly temperature and

precipitation rate mean

values interpolated to grid

1961 - Open

Climatological

Reference

Climatological Reference.

Temperature, humidity,

pressure, precipitation

amount and snow depth.

Reference seasons:

1971-2000 1981-

2010

Open




publish date

Weather forecast

model HIRLAM RCR

Point forecasts and grid

dataLatest model

run

(4 times a day)

0…54 h

Open

Sea forecast models Sea level point

forecasts, Wave (WAM)

and current (HBM) as

grid data

Latest model

run

(4 times a day)

0...54 h

Open

Environmental

Monitoring Facilities

Weather observation

stations, radars…2015

Aviation

Observations

METAR 30 min open

Ground & mast

observations

Special observations

from ground and masts2015




publish date

Air Quality

Observations

Air Quality Observations 1h 2015-2016

Silam Model Dispersion Model for Air

Quality, Forest Fire and

Pollen

Latest model

run (once a day)

0…96h

2015

HELMI Ice Model Ice forecast model Latest model

run

(4 times a day)

0...54 h

open

Soundings Temperature, Humidity,

Pressure, Wind from

ground to 25 km height

2 times a day 2015

FMI Open Data Portal follows INSPIRE requirements.


FMI Open Data Portal

Meta data

Data Models

Services

The very same data portal works as Open Data and

INSPIRE portal.

Catalog Service

(CSW)

o Based on GeoNetwork

30.1.2015 8Open Meteorological Data as part of Big Data / Roope Tervo

View Service (WMS)

o Based on GeoServer

o Only the most common layers

published

Download Service

(WFS 2.0)

o Web Feature Service (WFS) 2.0 Simple Profile

o Based on stored queries

o Predefined data sets with possibility for additional parameters (i.e. time and area)

o In-house production


Data Models

o Observations and point forecasts as GML

o The same data is published in two different formats.

o Gridded data is provided in

appropriate binary format (grib,

NetCDF, HDF…)

o WFS members contains the metadata ‘envelope’ with a link to a actual data

o Every XML document as self-

explanatory as possible


Weather Data as

Part of Big Data

• Important constituent

• Weather impacts to everything and each one of us.

• New technology makes us even more fragile of weather.


Traffic

Advertising

Energy

Retail trade

Outdoor

activities

Weather data – is it big?


Snijders, C.; Matzat, U.; Reips, U.-D. (2012). "'Big Data': Big gaps of knowledge in the field of Internet". International Journal of Internet Science 7:

1–5



“Data sets with sizes beyond the ability of commonly used software tools to capture,

curate, manage, and process data within a tolerable elapsed time.“Snijders, C.; Matzat, U.; Reips, U.-D. (2012). "'Big Data': Big gaps of knowledge in the field of Internet". International Journal of Internet Science 7:

1–5



• In-situ weather measurement 1 TB

• Weather radar data 50 TB

• NWP model gridded data (FMI)

• HIRLAM 120 TB

• AROME 230 TB

• Satellite image data (FMI)

• Globsnow 32 TB

• Other 50 TB

• Climate model data (FMI) 29 TB

• Other models

• SILAM 100 TB

• Tuuliatlas 21 TB

• Other 300 TB

FMI Open Data as

Part of Big Data

• Tried to tackle big data challenges

• Machine readable

• Follows standards

• Rich meta data• Every XML document as self-

explanatory as possible

• Users don’t need to download everything, they can fetch only the parts of their interest

• parameters, area, time, etc…

• Quality information included (where applicable)



MetoLib

o Open source JavaScript library produced by Finnish Meteorological Institute

o Helps users to load and use the data

o Python version is on the list

Easyrequests Cache

Parse the data to as JSON


Data Contents

Popularity Comparison

91

5 40

102030405060708090

100

Radar Images(WMS + WFS)

Forecast (WFS) WeatherObservations

(WFS)

Downloads[%]

o Based on year 2014

o All data sets combined

o All radar image queries treated as data queries

o Note that some data is updated more often than others

And a little over300 000 data downloads

per day(3,7 req/s)

At the moment about 7200

registered users


Some Experiences

Practicallyno clientsupportscomplexfeatures

Although standards are followed, there’s

a gap between provided data model

and clients’ capabilities


Some Experiences

GeoServer is modified to support

stored queries in WFS 2.0 (will be

integrated into the standard package

Q1/2015)

FMI is going to open the same data as simple features to

support clients


Some Experiences

Industry ishappy to usestandardized

services

Amateur andfreelancer coders

would prefer simple JSON API


Some Experiences

So far maybe

even more

professional

interest than

private

Quite many

expected a user

interface to load data

to i.e. to Excel

instead of machine

readable interface


Some Experiences

For now,

very few have

been interested in

forecast models

as a grid data

Point forecasts,

observations and

radar images are the

most interesting data

types

30.1.2015 25

Some Experiences

Open Meteorological Data as part of Big Data / Roope Tervo

Finally we have

all the data behind

one access point

behind standard

interfaces and

in harmonized

formats

The Open Data

project has been a

great boost for

ourselves both

technically and for

working culture

30.1.2015 26

Some Experiences

Open Meteorological Data as part of Big Data / Roope Tervo

www.fmi.fi

http://www.slideshare.net/tervo/


http://www.slideshare.net/tervo/


open weather data as part of big data

Technology

openolder data

day2015fmi open data

sea temperature

pollenlatest model

silam modeldispersion

height2 times

precipitation rate mean

web feature service