open data and open source software - online registrationmeetings.wmo.int/ec-69/presentations/ec-69...
TRANSCRIPT
Open Data and Open Source Software
@Finnish Meteorological Institute
WMO EC SE, 11.5.2017
Harri Pietarila, Head of Expert Services, [email protected]
Roope Tervo, Development Manager, [email protected]
Motivation
• Ecosystem is changing
• NMHS will not control the whole value chain anymore
• Large companies like IBM and Google are becoming more and more
relevant players
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
2
Charge for data
Charge for Service
Closed ecosystem
Open ecosystem
Motivation
• A role of NMHS is also changing
• Challenges:
• Ensure authoritative voice in warnings
• More efficiency in development and operations
• Ensure the impact of produced information
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
3
Finnish Meteorological
Institute opened its data
11/05/2017 4
FMI Open Source Software
https://en.ilmatieteenlaitos.fi/open-data
FMI starts to open it’s
software
2013 2016
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
• Finnish Meteorological Institute
opened its data in 2013.
• Basically everything that FMI has
property rights was opened.
• Both (near) real-time and historical and climatological data.
• Data is provided in freely in machine
readable format.
11/05/2017 5
FMI Open Data
https://en.ilmatieteenlaitos.fi/open-data
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
Data set Description Time
Interval
Estimated
publish date
Weather
Observations
Temperature, Wind, Humidity, Ground Temperature…
10 min Open,
older data to be
added
Weather forecast
model HIRLAM RCR
Point forecasts and grid data
Latest
model run
(4 times a
day) 0…54
h
Open
Aviation Observations METAR 30 min open
Weather Radars Precipitation Rate, Precipitation Amount…
5 min Open,
older data to be
added
Lightning Thunder Strikes in Finland
5 min Open
Some Example Data Sets
11/05/2017 6 Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Mikko Visa
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
Finnish Meteorological
Institute opened its data
11/05/2017 7
FMI Open Source Software
https://en.ilmatieteenlaitos.fi/open-data
FMI starts to open it’s
software
2013 2016
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
Motivation
Boost Product Development of the Software
• NMHSs are well connected concerning NWP development
and GTS/WIS
• Why not extend collaboration to upstream as well?
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
8
NWP Post-
processing Interpretation Visualization Applic
ations
Applications Observations
Collaboration and OSS Proprietary software
Motivation
Streamline collaboration with partners
• Open Source Software can boost the development
• A very powerful way to support developing countries
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
9
NWP Post-
processing Interpretation Visualization Applic
ations
Applications Observations
Collaboration and OSS Proprietary software
Motivation
Increase appropriability of
weather and climate data
• NMHSs are successful when
weather and climate don’t
cause unanticipated
unwanted impact to society
• Data is valuable only when it
reach relevant audience
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
10
Motivation
Increase appropriability of
weather and climate data
• Maximal coverage requires
several different channels
and services
• One organization can’t handle
them all
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
11
Motivation
Support Research
• Open Source Software empowers research as well
• Easy and open methods to access and analyze the data
• Methods are repeatable when anyone can access the
tools
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
13
Motivation
Public-private-partnership
• Open Source Software is a great tool to ease public-private
partnership
• OSS provides seamless access to weather and climate
information from multiple NMHS providers
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
14
Motivation
• Boost and streamline collaboration between NMHSs
• Streamline collaboration with partners
• Support developing countries
• Boost product development of the software
• Support research activities
• Increase appropriability of weather and climate data
• Support companies
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
15
Potential Users
• Other National Meteorological Services
• International agencies (such as ECMWF, EUMETSAT…)
• Other environmental agencies
• Research Institutes
• Environmental companies
• Private developers
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
16
License
• MIT license is recommended for all FMI
OSS…
• …but every license is considered
separately.
• For example, other members of the project and/or
used third party projects may require some
specific license.
Commercial Use
Distribution
Modification
Private Use
License and Copyright Notice required
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
17
What
• About 30 different software under
review
• Open source is the baseline…
• … but every software is considered
separately
• How easy the software is to isolate from
production system?
• Is the project vital? Is the software still
developed and maintained?
• Does the software potentially benefit
someone?
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
18
What
Examples
• SmartMet Server (open) High-capacity and high-availability data and product server for
MetOcean data.
• SmartMet Workstation (2018) Tool for visualizing and editing meteorological data
• SmartMet Alert (2019) Tool for creating and disseminating (supports CAP) weather
warnings
• HIMAN (open) Post-processing suite for meteorological data
• PyTroll (open) Set of tools for the reading, interpretation, and writing of weather satellite data.
• SILAM (open) Airquality and dispersion model
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
19
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
20
FMI Open Data
Portal & FMI
INSPIRE Data
Services
Backend for
clients’ web
services
Integration to
clients’
systems Backend for
mobile
applications
Backend for
FMI Client
Services
Backend for
FMI public
pages
• Basis of FMI product generation
SmartMet Server
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
21
• High-performing high-availability C++
data and product server for all kind
of MetOcean data
• FMI installation responses to over 30
million requests per day
• Over 99,95 % availability
• Also used at ECMWF Copernicus
Climate Data Store
SmartMet Server
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
22
Web pages
with warning
maps and
texts
Integration to
clients’
systems
Automatic
text
messaging
Mobile phone
application
• SmartMet Alert provides
• A user interface for the forecasters to input warning information
• Automatic warning texts
• Several possible (automatic) dissemination channels
• CAP output
SmartMet Alert
CAP
TV
RSS Feeds Print
media
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
23
• Used in operational warning production in Finland since February 2015
• Has been installed and trained in three Caribbean countries (Cuba,
Antigua and Barbuda, Jamaica)
• April-November 2016 has been installed and trained also in 6 Pacific
countries: Samoa, Vanuatu, Fiji,
Tonga, PNG and Solomon Islands
• Is going to be installed and trained
in Bahamas in 2017
SmartMet Alert
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
24
• Tool for visualizing and editing meteorological data
• Support for all kind of meteorological data
• NWP, Satellites, Weather Radars, Observations, Lightning
Detection, Soundings…
SmartMet Workstation
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
25
• Post-processing and grid manipulation suite
• Runs automatic post-processing scripts like statistical correlations or
derivative parameters
• Support GRIB1, GRIB2, CSV and QueryData formats
• Utilize GPU
HIMAN
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
26
• Free and open source python
modules for the reading,
interpretation, and writing of
weather satellite data.
• Project started by SMHI (Sweden)
and DMI (Denmark)
• FMI is collaborating with many
others
• User base is global
PyTROLL
Adding Value for Satellite Users
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
27
• Operational Emergency and Air Quality Modelling
System
• a global-to-beta-meso-scale (down to 1 km
resolution) dispersion model
• applied for wide range of atmospheric composition,
air quality, and emergency decision support
applications, as well as for inverse dispersion
problem solution
• Applications for nuclear accidents, wildland fires
and volcanic ash as well as for regulatory air
quality assessments and forecasting
SILAM
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
28
• Data format conversion tools
• Product generation tools
• Observation network software and
quality control
• TAC IWXXM converters
• Radar composite and analyzing
software
• …
Many Others
Some Open --- Some Under Investigation
Policies and Guidelines
• Source code and documentation in
GitHub: https://github.com/fmidev
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
29
Examples
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
30
Project Ongoing Project in Preparation
Completed cooperation or consulting project in the country
Scientific Collaboration
International co-operation & expert services
Pacific SIDS
Caribbean SIDS
Central Asia
Andean region
S-E Europe
11/05/2017 Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
11/05/2017 Open Source Software @ Finnish Meteorological Institute | Roope
Tervo, Harri Pietarila, Mikko Rauhala 32
SmartMet installed
SmartMet – installations
• Used in operational warning production in Finland since February 2015
• Has been installed and trained in three Caribbean and six Pacific countries: • Cuba
• Antigua and Barbuda
• Jamaica
• Samoa
• Papua New Guinea
• Fiji
• Tonga
• Vanuatu
• Solomon Islands
SmartMet Alert installations
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
33
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
34
Andean region
• SmartMet installations:
Efficient data display and
editing tool for the
forecaster & automated
weather production for
web and mobile
- Peru SENAMHI
- Ecuador INAMHI
- Bolivia SENAMHI
- Columbia IDEAM
• ICI projects with Peru
Ecuador, Bolivia and
Columbia
• Regional project PRASDES
SmartMet products example
SEMANHI-Bolivia
http://www.senamhi.gob.bo/
New Android APP SENAMHI Bolivia
the alerts and forecast in Aymara language
new color coded Alert protocol of SENAMHI
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
35
SmartMet in SIDS Caribbean
• FMI cooperation in SIDS Caribbean from 2000
• Establishment of Caribbean wide SmartMet (forecaster workstation and forecast production system) network (10 countries)
• Improvement in forecast quality
• Improvement in forecast production and end-user products
11/05/2017 36
SmartMet installed
Dominican Republic Cuba
Jamaica
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
SmartMet Alert in SIDS Caribbean
• Development of Early Warning services
• Installation of SmartAlert software in Jamaica, Cuba and Antigua & Barbuda
• Improvement in warning delivery and launching of modern end-user products such as maps etc…
• Systems in pilot phase (goal: operational after COPS project)
11/05/2017 37
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
Google map visualization works also as a mobile phone app - Example from Cuba.
SmartMet Alert
Warning map visualization – google map
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
38
• FINPAC - Finnish-Pacific Project to Reduce Vulnerability of the Pacific Island Countries' Livelihoods to the Effects of Climate Change
• Project covering the entire value chain from improved observations to services and benefits to communities
• Funding of 3.8M€ to SPREP with Technical Assistance from FMI
• Partnership with SOPAC, Red Cross, USP and WMO
• Beneficiaries are the NMHSs from 14 Pacific Island Countries
• Kick-off on 1 July 2013 at the Pacific Meteorology Council meeting in Fiji
11/05/2017 39
SmartMet in SIDS Pacific
FINPAC (13-16)
MFA
FMI Project
Coordinator
Project Manager
Steering Committe
e
SPREP Manageme
nt Team
Financial reporting
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
1.4: Implement weather forecasting tools and production systems at selected NMSs
• SmartMet workstation for editing and viewing meteorological data was installed in six countries (Fiji, PNG, Samoa, Solomon Islands, Tonga and Vanuatu).
11/05/2017 40
SmartMet in SIDS Pacific
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
• Examples:
11/05/2017 41
SmartMet in SIDS Pacific
PNG met service web page
Tonga met service web page
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
SmartAlert in SIDS Pacific
forecaster interface to issue warnings
Forecaster
view - example
from Samoa
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
42
• Examples (google map visualization of the warnings):
11/05/2017 43
SmartMet in SIDS Pacific
PNG met service web page
Solomon Islands met service
web page
Samoa met
service
web page
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
Social media integration – Facebook
(Papua New Guinea Weather Service)
• Forecast data produced with SmartMet
• Facebook messenger inquiry to receive text format local
weather forecast
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
44
Social media integration – Twitter
(Tonga Meteorological Service)
• Forecast data produced with SmartMet
• Automatized tweeting of forecast information in text
format
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
45
Training to ensure sustainability
Tonga Vietnam
Papua New Guinea Cuba Peru
11/05/2017
Open Source Software @ Finnish Meteorological
Institute | Roope Tervo, Harri Pietarila, Mikko Rauhala
46
www.fmi.fi
http://en.ilmatieteenlaitos.fi/open-source-code
https://en.ilmatieteenlaitos.fi/open-data
https://github.com/fmidev