title of presentation - meteoschweiz.admin.ch · climate grid software for creating national...
TRANSCRIPT
![Page 1: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/1.jpg)
Climate Grid Software for creating national climate data products and services
Dan Hollis, Ian Edwards, Mark McCarthy
Data Management Workshop, St Gallen, Switzerland, 28-30 October 2015
![Page 2: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/2.jpg)
Climate Grid
Aim:
Develop a portable, modular and traceable code base
following open software standards to provide a tool kit for
the generation, exploration and visualisation of UK climate
statistics.
![Page 3: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/3.jpg)
ArcView
FORTRAN
Oracle
Excel
Excel
AVENUE
Fireworks
intranet & external
web
Dream-weaver & CMS
Grid Archive
![Page 4: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/4.jpg)
Development process
Test-driven development
Iris 1.8 Python library for analysing and visualising meteorological and oceanographic data sets. http://scitools.org.uk/iris/
Code refactoring
![Page 5: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/5.jpg)
Climate Grid Components
Grid Archive
Gridding Product Creation
![Page 6: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/6.jpg)
Gridding Overview
many dates
many variables
iris cube
(points) interpolation
iris cube
(raster) netCDF
![Page 7: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/7.jpg)
Gridding Process
Convert to anomalies
Regression analysis
IDW interpolation
Re-combine regn model &
residuals
Convert to actuals
![Page 8: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/8.jpg)
Gridding Process
Convert to anomalies
Regression analysis
IDW interpolation
Re-combine regn model &
residuals
Convert to actuals
idw_actuals()
idw_anomalies()
idw_regression_residual()
idw_regression_residuals_anomalies()
![Page 9: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/9.jpg)
System Configuration
Indexed via ‘short_name’ = a string combining the temporal resolution and variable name • e.g. monthly_maxtemp • e.g. daily_rainfall
• Paths to system resources • CF metadata • Grid definition (extent, projection, resolution etc) • Gridding method (by variable, month and run type) • Legends and colours for maps
•System resources: • Station metadata • Product templates • Region definitions (shapefiles and raster masks) • Grids of the independent regression variables
![Page 10: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/10.jpg)
Climate Grid Components
Grid Archive
Gridding Product Creation
![Page 11: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/11.jpg)
Grid Archives
Provisional
Pre-QC Test
Final LTA
Historic
\grid\variable\year\mm.nc
\station\variable\year\mm.nc
my_archive = GridArchive(path, ...)
combined_archive = GridArchiveHierarchy(
final_archive, provisional_archive, historic_archive)
![Page 12: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/12.jpg)
Climate Grid Components
Grid Archive
Gridding Product Creation
![Page 13: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/13.jpg)
Products and Processes
Maps
Regional values
Grids
Point values
Monthly, provisional
Daily, final
Services
Ad hoc products
![Page 14: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/14.jpg)
Automation
![Page 15: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/15.jpg)
Automation
extract_station_data()
gridding()
product_creation()
product_delivery()
csv file
netCDF file
png file
cron
Rose: A framework for managing and running meteorological suites.
http://metomi.github.io/ rose/doc/rose.html
![Page 16: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/16.jpg)
Quality Control
![Page 17: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/17.jpg)
Current status
• Software has been used for July, August and September summaries
• No major problems but system is still bedding in (various small issues have needed fixing...)
• Work is ongoing in various areas: •Memory issues when working with large networks •Automation •Quality control •Batch processing e.g. multi-month runs •Additional system tests •Refactoring for open source •Documentation
![Page 18: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/18.jpg)
Next Steps: Sharing
![Page 19: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/19.jpg)
ClimateGrid
Next Steps: End-to-end
FORTRAN
MIDAS
Excel
Excel
Python
Fireworks
Metnet & external
web
Dream-weaver & CMS
Grid Archive
![Page 20: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/20.jpg)
ClimateGrid
Next Steps: Portability
A.N.Other
Python
Prototypes
Grid Archive
![Page 21: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/21.jpg)
Next Steps: Methods
Convert to anomalies
Regression analysis
IDW interpolation
Re-combine regn model &
residuals
Convert to actuals
Alternate method 1
Alternate method 2
![Page 22: Title of Presentation - meteoschweiz.admin.ch · Climate Grid Software for creating national climate data products and services Dan Hollis, Ian Edwards, Mark McCarthy Data Management](https://reader030.vdocuments.us/reader030/viewer/2022041207/5d5f7ca188c99399358bacb9/html5/thumbnails/22.jpg)
Summary
Successful transition
Work in progress
Maintained continuity
Greater flexibility
Improved skills
Future collaboration