first online hangout sc5 - big data europe first pilot-presentation-hangout
Post on 15-Apr-2017
444 Views
Preview:
TRANSCRIPT
SOCIETAL CHALLENGE 5 PILOT PRESENTATION
NCSR “DEMOKRITOS”, GREECE1st SC5 hangout, January 12, 2016
Climate 1st Pilot case: Supporting data-intensive climate research
Pilot under development by NCSR “Demokritos” Impact on / basis for synergies with:
o SCs that need feedback on the potential effects of climate change for considering adaptation, mitigation and prevention measuresSC1 –HealthSC2 -Food and AgricultureSC3 -Secure, clean and efficient EnergySC4 -Smart, green and integrated TransportSC6 -Europe in a changing world - inclusive, innovative and reflective societiesSC7 -Secure societies - protecting freedom and security of Europe and its citizens
Background & Problem Statement
The extended scientific effort over the last few decades in understanding and studying the earth’s climate and climate change has resulted in a huge volume production of model (e.g. Global Circulation Models, Regional Models) and observational (e.g. satellite, aircraft, station) data
The provision of such model data satisfies an important objective, that of assessing the potential impacts of climate change on well being for adaptation, prevention and mitigation measures and supporting other policy making decisions
The climate research and impact assessment communities need to interface with useful data resources to satisfy the requirement of extracting data efficiently and timely. One of the methodologies applied is that of dynamical downscaling to obtain values of physical atmospheric variables in smaller spatial and temporal scales.
The present BDE pilot aims to facilitate the process of dynamical downscaling from global climate data to regional / local scales with the support of tools aggregated on the Big Data Europe platform
Dynamical downscaling
http://hosted.verticalresponse.com/1022925/c3c95b44ec/TEST/TEST/
Output of Global Circulation Models (GCMs) run at coarse spatial resolution drives Regional or Local Models run at higher resolution
Aim: To obtain local weather and climate variables needed for impact assessment and planning by decision makers
Assumption: local climate depends on large-scale atmospheric characteristics and local-scale features
Purpose of the Pilot
Provide an intuitive interface between researchers and specific climate data portals and providers.
Search and download climate model and, optionally, observational data, according to user requirements, such as geographic coverage and / or experiments (scenarios).
Setup and orchestrate the execution of the dynamic downscaling process on institutional computational resources, while gathering and managing data products.
Establish a workflow for useful metadata mappings and data lineage.
Scenario
The user accesses the climate data portal and defines the geographic area and scenarios.
The user then selects and downloads data on the BDE platform. The user defines the case-specific parameters for the dynamical downscaling process
o several of the above options will be predefined for the present pilot The platform initiates and controls the dynamical downscaling process by invoking the
modules that will perform the preprocessing and main computation steps.o on institutional resources provided and used by the partnero data mapping and necessary transformations will be addressed at the pre-processing stage,
to ensure compatibility of the data with the modelling software The intermediate and final data products are stored on the BDE platform, while data
lineage is also tracked. The process described can be repeated either from any intermediate step or from the
beginning, using data from different climatic models
Current practice
no generally agreed workflow to perform the dynamical downscaling process data acquisition, handling and management is often performed in an ad-hoc manner.
o observational and analysis data from external services are transferred to local filesystems Researchers then preprocess the data using third-party or custom-made scripts. The preprocessed data (i.e. the resulting set of files) are then fed into the regional
model of choice along with a set of suitable parameters. The regional model is usually run on local computational infrastructure The resulting files are used for visualisation and further analysis either on local or,
data-size and task-permitting, on the researchers’ machines. Difficulties can be encountered in managing and archiving the data produced at the
various stages. Difficult to track progress and intermediate data products as well as to encourage
reuse and collaboration across disciplines
Primary content / data involved
This Pilot will make extensive use of publically available climate data Earth System Grid Federation (ESGF) http://esgf.llnl.gov
o international collaboration for the management, dissemination and analysis of model output and observational data
o complete set of CMIP5 data (global climate model simulations)o most CORDEX data (regional climate model simulations)o more than 15 portals and more than 45 data nodes worldwideo IS-ENES operates the European part of ESGF (https://verc.enes.org/ISENES2)
European Centre for Medium range Weather Forecasting (ECMWF)o intergovernmental organisation supported by 21 European Member States and 13
cooperating Stateso ECMWF’s public data http://apps.ecmwf.int/datasets/data/interim-full-dailyo Certain subsets of ECMWF data are also published via ESGF
Data structures
This pilot use case will make use of NetCDF data setso binary, header-based formato variable data are stored in multidimensional arrays
Access to relevant datasets is described via THREDDS records on each node’s catalogue http://www.unidata.ucar.edu/software/thredds/current/tds/catalog/
The data are described using the Climate Forecasting (CF) conventions http://cfconventions.org
The implementation of this pilot will make use of WRF - the Weather Research and Forecasting model http://wrf-model.org - to perform the dynamical downscalingo the internal metadata will be expressed in a WRF-compatible format. Appropriate
mappings will be put in place so that the conversion is transparent to the user and that it takes place automatically upon ingestion.
Current technological infrastructure at the Pilot partner
The pilot partner - NCSR Demokritos - currently operates WRF on a daily basis to produce weather forecasts on 3 nested computational grids covering Europe, Greece and Attica peninsula
Initial and boundary conditions are provided by GFS data (from US). The GFS data are downloaded automatically and pre-processed to produce the necessary input file for running WRF.
Functional roles involved in this Use Case
Research site/centre (stakeholder): use of the functionality of this pilot either as an external service/product or by deploying an internal instance of the BDE platformo Benefit: increase the potential for its internal processes with minimal disruption and
expansion requirements to its infrastructure and internal policies. Researcher (primary user): effectively search for data, potentially across
different providers, perform processing tasks on the BDE platform or on their departmental resourceso gain data provenance informationo increase the efficiency of experimental runs, o increase the value of their experiments (as they will be archivable and retrievable for
potential future reference, publication and scientific replication)o increase the potential for primary users to perform multiple downscaling computations
and inter-compare their results
IT infrastructure
Potential configurations for a research institute to use the Pilot:o As an external set of services: through agreement with the instance provider
provides the pilot services are accessible and integratable to current research procedures.
o As an internal infrastructure component: the research institute provides the necessary hardware to be used for hosting the BDE instance.
In showcasing the pilot we aim to follow the first approach, taking advantage of the common administration within NCSR-D.
Impacts
This pilot aims to aid climate research, and therefore encourage consequent impactful synergies, by providingo a user-friendly, interactive, way to query and fetch data from resources such as ESGF and
ECMWFo a framework to support the incremental, data-oriented carrying out of climate-related
experiments, while storing intermediate data products and associated data lineage. In particular this pilot aims at:
o improving the productivity of researchersEasier management, ingestion and transformation of external dataoverseeing the execution of model runs over the data, making use of existing infrastructure and
procedures Improve efficiency and reusability in downscaling computational experiments,
o Create opportunities for pilots across communities within the BDE platformClimate change impact assessment studies on sectors such as energy, food and agriculture are
potential future pilot-use cases across societal challenges in the BDE platform.
Climate: Pilot Architecture
Thank you for your attention
Questions?
top related