dsd-int 2015 - symposium 'earth observation & data science' - reflections on the big...
TRANSCRIPT
1 Data Science Symposium, 26th October 2015, Delft, [email protected]
ABSTRACT
The “Big Data” topic is one of the most heard buzz words within today's IT and Data community. This presentation reflects on some aspects of the big data hype and addresses some key issues as they are perceived by the data providers and the users. An important aspect in this discussion is the notion that users often need data in order to get answers to their questions – which is not a new issue as such however, in the big data era the generation of such answers is due to the exponential growth of the base data representing a significant challenge. The question is on how this could be supported and to what extend by the Data providers and others. An overview is presented on how EUMETSAT considers those aspects within the context of its existing operational data services but also their big data related evolutions. In this context the different models of data delivery (push – pull & nrt – online/offline) are discussed and the bridge a data provider might build in order to make steps towards the user for easier access to the actual information the user requires.
Data Science Symposium, 26th October 2015, Delft, [email protected] 2
MONITORING WEATHER
AND CLIMATE FROM SPACE
Reflections on the big data thematic Data Science Symposium 26.10.2015 - Delft
Lothar Wolf Competence Area Manager for Data Services
3 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT OVERVIEW
4 Data Science Symposium, 26th October 2015, Delft, [email protected]
HUNGARY
Member States
BULGARIA
ICELAND
SERBIA AUSTRIA BELGIUM
DENMARK FINLAND
FRANCE GERMANY GREECE
IRELAND ITALY
UNITED KINGDOM TURKEY
SWEDEN SPAIN
PORTUGAL
NORWAY THE NETHERLANDS LUXEMBOURG
CROATIA
POLAND
LATVIA
SLOVENIA
ROMANIA
CZECH REPUBLIC
LITHUANIA
ESTONIA
SLOVAK
REPUBLIC
SWITZERLAND
EUMETSAT is an intergovernmental organisation with
30 Member States and 1 Cooperating State
5 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT headquarters
6 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT Mission
To establish, maintain and exploit
European operational meteorological
satellite systems, while considering the
recommendations of WMO as much
as possible
A further objective is to contribute to
operational climate monitoring and
detection of global climatic changes
By fulfilling these objectives, contribute
to environmental monitoring, where
interactions with the ocean and the
atmosphere are involved
Deliver cost-effective
operational
satellite data and products
that satisfy
the meteorological and
climate data requirements of
its Member States
Encourage more users to
benefit from
the increasing range of
EUMETSAT
data and products
7 Data Science Symposium, 26th October 2015, Delft, [email protected]
Safety of life, property
and infrastructure
Transport Climate policy and
environment protection
….... ......
Areas benefitting from weather forecasting
Energy, agriculture,
tourism
8 Data Science Symposium, 26th October 2015, Delft, [email protected]
Current EUMETSAT satellites
METOP-B JASON-2
METEOSAT-10 METEOSAT-9
METEOSAT-7
METOP-A (98.7° incl.)
EUMETSAT POLAR SYSTEM
In nominal mid-morning sun synchronous orbit at 817km altitude ,as part of the EUMETSAT Polar System (EPS) .
JASON-2 (66° incl.)
OCEAN SURFACE TOPOGRAPHY
In nominal non-synchronous low Earth orbit at 1,336km altitude, in support of the Ocean Surface Topography Mission.
METEOSAT-7 (57.5° EAST)
INDIAN OCEAN DATA COVERAGE
Operated in support of the Indian Ocean Data Coverage (IODC) mission, bridging an observational gap in this region.
METEOSAT-9 (9.5° EAST)
RAPID SCANNING SERVICE (RSS)
Provides the Rapid Scanning Service (RSS) delivering more frequent images every five minutes over parts of Europe, Africa and the adjacent seas.
METEOSAT-8 (3.5° EAST)
BACKUP SERVICE
Serves as a back-up to both the Meteosat-9 and -10 spacecraft for full disc imagery and rapid scanning.
METOP-A
METEOSAT-8
METOP-B (98.7° incl.)
EUMETSAT POLAR SYSTEM
In orbit at 817 km altitude, the primary operational satellite of the EUMETSAT Polar System (EPS).
METEOSAT-10 (0°)
METEOSAT FULL DISC IMAGERY
Positioned at 0° supporting the prime Meteosat full disc imagery service over the European continent, Africa and parts of the Atlantic and Indian oceans.
9 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT’s geostationary and polar-orbiting
satellite programmes
Geostationary
Primary mission: “Now casting” of rapidly
developing, high-impact weather up to six
hours ahead.
One of the most challenging tasks of the forecasters, vital
for the safety of life, property and infrastructure
Requires informative images of the atmosphere at a high
frequency (some minutes) that can only be achieved from
the geostationary orbit (36,000 km)
10 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT’s geostationary and polar orbiting
satellite programmes
Polar-orbiting
A second system in lower orbit is needed
to complement the data from geostationary
orbit and provide global coverage.
Flying at a much lower altitude (817 km), Metop satellites
deliver a wealth of less frequent but global and quantitative
observations which are the most critical inputs to the
Numerical Weather Prediction models used to forecast
weather up to 10 days and for climate monitoring
11 Data Science Symposium, 26th October 2015, Delft, [email protected]
MTG – Overall system configuration
Satellite Control Centre
Mission Control Centre
Product Processing Facilities
Data Centre Archive
Data Dissemination via EUMETCast
Direct reception by users
in all countries
EUMETSAT Network of
Satellite Application
Facilities (SAFs)
External data sources
Search and
Rescue Beacons
TELEMETRY, TELECOMMAND
AND CONTROL
GROUND STATION SITES
EUMETSAT CORE GROUND SEGMENT
MTG-1 RAPID SCAN SERVICE
MTG-S MTG-1 FULL SCAN SERVICE
MISSION DATA
ACQUISITION
GROUND STATION SITES
Search and Rescue
Mission Control Centres
Data Collection
Platforms
12 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT mission planning
03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 YEAR...
METEOSAT FIRST GENERATION
METEOSAT SECOND GENERATION
METEOSAT THIRD GENERATION (MTG)
EUMETSAT POLAR SYSTEM (EPS)
METEOSAT-7
METEOSAT-8
METEOSAT-9
METEOSAT-10
MSG-4/METEOSAT-11*
MTG-I-1
MTG-S-1
MTG-I-2
MTG-I-3
MTG-S-2
MTG-I-4
METOP-A
METOP-B
METOP-C
EPS-SECOND GENERATION (EPS-SG)
METOP-SG SOUNDING & IMAGERY SATELLITES
METOP-SG MICROWAVE SATELLITES
JASON
JASON-2
JASON-3
JASON CONTINUITY OF SERVICES (JASON-CS)
Only the full operational phase of each mission is represented, excluding commissioning.
* MSG-4/Meteosat-11 will be stored in orbit, before replacing Meteosat-10
13 Data Science Symposium, 26th October 2015, Delft, [email protected]
Copernicus is an EC programme
aiming to achieve an autonomous,
multi-level operational Earth
observation capacity
COPERNICUS
14 Data Science Symposium, 26th October 2015, Delft, [email protected]
Copernicus
• PART OF THE COPERNICUS MANDATE IS TO USE DATA FROM
METEOSAT, METOP AND JASON-2 OPERATIONAL SATELLITE SYSTEMS
ESTABLISHED, MAINTAINED AND OPERATED BY EUMETSAT.
• OPERATION OF COPERNICUS-DEDICATED MISSIONS ON
ATMOSPHERE AND OCEANS FALLING WITHIN EUMETSAT’S MANDATE
I.E. THE SENTINEL-3 MARINE MISSION AND THE JASON-3 AND JASON-
CS MISSIONS OF THE COPERNICUS HIGH-PRECISION OCEAN
ALTIMETRY (HPOA) ACTIVITY.
• PLANNING, DEVELOPMENT AND INTEGRATION INTO FUTURE
EUMETSAT SYSTEMS OF COPERNICUS MISSIONS DEDICATED TO
ATMOSPHERIC CHEMISTRY (SENTINEL-4 AND MTG; SENTINEL-5 AND
EPS-SG), AND THEIR EXPLOITATION IN FULL SYNERGY WITH
EUMETSAT’S OWN MISSIONS
15 Data Science Symposium, 26th October 2015, Delft, [email protected]
Copernicus
• MAKING OPTIMAL USE OF THE EUMETSAT MULTI-MISSION
INFRASTRUCTURE IN THE COPERNICUS FRAMEWORK, INCLUDING
THE EUMETCAST REAL-TIME DATA DISSEMINATION SYSTEM,
EUMETSAT EARTH OBSERVATION PORTAL AND ITS ARCHIVES.
• DELIVERY TO THE COPERNICUS PROGRAMME OF DATA AND
PRODUCTS AVAILABLE FROM AND AGREED WITH EUMETSAT
PARTNERS IN THE UNITED STATES, CHINA, INDIA AND JAPAN.
16 Data Science Symposium, 26th October 2015, Delft, [email protected]
METEOSAT JASON-2 INITIAL JOINT POLAR SYSTEM METOP NOAA SATELLITES
CONTROL AND
DATA ACQUISITION
DATA CENTRE EUMETSAT HEADQUARTERS
SATELLITE APPLICATION
FACILITIES WITHIN EUMETSAT MEMBER STATES
THIRD-PARTY
DATA SOURCES
REAL TIME DISSEMINATION OF DATA AND PRODUCTS VIA EUMETCAST
APPLICATIONS GROUND SEGMENT
Central processing and
generation of products
Distributed processing and
generation of products
USERS
FLIGHT
OPERATIONS
EUMETSAT ground segment overview
METEOROLOGICAL PRODUCT
EXTRACTION EUMETSAT HEADQUARTERS
PRE-PROCESSING EUMETSAT HEADQUARTERS
17 Data Science Symposium, 26th October 2015, Delft, [email protected]
REFLECTIONS ...
Big Data
Hype cycle
Use cases
Push
Pull
Standards
Interoperability
Discovery
Search
Registration
Fees
Cost
Resources
Information
Data
User
Provider
Added value
Question
Result
Infrastructure
Evolution
Access
Subscription
Archive
Replication
DOI
Processing
Data mining
Non discriminative
User algorithms
Data Policies
Public Service
Catalogues
Operations
Research
Delivery
Retrieval
Diversity
Scalability
Reliability
Mission
Bridge
Operational
Amazon/Google
18 Data Science Symposium, 26th October 2015, Delft, [email protected]
Gartner’s Hype Cycle 2015
19 Data Science Symposium, 26th October 2015, Delft, [email protected]
Evolution of data rates
20 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT Data Centre growth
Yearly retrieval
YEAR 2012
VO
LU
ME
IN
TE
RA
BY
TE
S
2011 2010 2009 2008 2007 2006 2005 2004 2003
1800
1600
1400
1200
1000
800
600
400
200
0
Yearly ingestion
2013
21 Data Science Symposium, 26th October 2015, Delft, [email protected]
IS THE SITUATION A “NEW PROBLEM”?
? ... Data ... Algorithm ... Information ... Answer
-Distributed data sources -Heterogeneous data access functions & services -Data delivery as well as data retrievals -Large data volumes -Information generation at the user end
22 Data Science Symposium, 26th October 2015, Delft, [email protected]
DAR – DISCOVERY ACCESS RETRIEVAL
Is it the main “job” of a user to search for data? Search and Discovery are just the first step ... One-stop shop user experience is key
Exploitation of the individual strengths of each data provider by systematic use of interoperability standards for Meta data, Search & Discovery
Internal Services & Functions
Individual Strengths & Contributions
Interoperable Service Layer (based on standards)
DAR Portals
Data Provider 1
Processing ...
Data delivery ...
Archive ...
Data Provider N
Processing ...
Archive ... Data
delivery ...
23 Data Science Symposium, 26th October 2015, Delft, [email protected]
BRINGING DATA TO THE USER ...
Push model Data driven To the doorstep Continuous delivery of new data Typically guaranteed service levels
Pull model Off- & On-line, interactive, time series Bespoke and/or user defined data areas of interest Typically without SLA User infrastructure dependent
24 Data Science Symposium, 26th October 2015, Delft, [email protected]
BRINGING DATA TO THE USER ...
NRT world Push model, Fast, current, to the doorstep Operational users Continuous delivery of new data & products & data sets Typically guaranteed service levels
Offline world Pull model, Orders, time series Specific formats Non-time critical and typically without SLA Media delivery
The “grey zone” in between Pull model Faster and easier than offline interactive Added value & bespoke functions/processing
25 Data Science Symposium, 26th October 2015, Delft, [email protected]
BRINGING USERS TO THE DATA ...
Question Data Algorithm Information Answer
Data providers awareness of the information potential and science that is perceived to be obtained from the base data Provision of added services on the “base” data to allow easier information extraction Interconnect Data access across different Providers via Interoperability Manage and interact with the user communities Develop services in the “grey zone”
26 Data Science Symposium, 26th October 2015, Delft, [email protected]
BUILDING THE “BRIDGE” ...
Data Provider builds a “Bridge” for the User by: Highly Interactive User communities engagement process Enabling easy and homogeneous access to the data Pre-paring the data Diversity of delivery mechanisms Harmonised access to push & pull models Efficient use of supporting technologies Understanding and translation of a variety of data policies
27 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT DATA SERVICES
RECOGNISING THE SCOPE ...
One fits all is not the right approach - push & pull is needed as well as “bringing data to the users” & “bringing users to the data”
Fully service oriented approach required
Big Data and associated technologies raise also questions on: Privacy, IPRs, User credentials, IT Security
“Safe harbor” concepts are booming
Fast developing technology supporting the big data concepts
“Free to the user” however represents for data providers potentially substantial cost
New business models are possible (public services, downstream services)
28 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT DATA SERVICES DATA SERVICES VIDEO
EUMETSAT made the link and takes the related steps Re-use of existing operational capabilities Evolution of the service portfolio in full user interaction
•Sequence of pilots with direct benefits to the users
Formalized user interaction processes Service based interfaces Interoperable Data Discovery, Access & Retrieval Diverse Delivery mechanisms addressing diverse use cases (NRT, Online, Offline) with equal service delivery to users Considerations on how can data serve as information
External Data
Providers
SAFs
Partner Agencies
(NOAA, CMA,
ECMWF, JMA,
CNES, NMSs –
MF, DWD, ...)
Data
Data
Data
Data
EU
ME
TS
AT
Gro
un
d S
eg
me
nt
Data
Mu
lti-M
issio
n D
isse
min
atio
n E
lem
en
ts
Internet Data Service
&
ODA
EO
PO
RT
AL
DA
TA
CE
NT
RE
(arc
hiv
e)
EUMETSATs ground
stations network
EUMETCast
Uplink
DVB-S2 Broadcast
Order retrievals
User requests (search, order...) /registration
Push, no SLA,
unicast, multicast
Download, pull, no
SLA
Push, guaranteed SLA,
unicast, multicast
Direct Dissemination Data
Dristributed search
Order Subscription Request & Responses
GEONETCast catalogue & Metadata
Exchange
Product search,
order request,
order status
Push, guaranteed
SLA, multicast
Data
Multicast
Multicast
+ unicast
unicast
Multicast
+ unicastGEANT
Internet
RMDCN/WIS
Internet
Overall EUMETSAT Ground Segment including Multi-Mission Elements
Direct disseminated data
Offline/Archive data provision
Common service (NWC)
High volume service (NWP)
Data for WMO global exchange, bi-lateral
data exchange with required service levels
RGB imagery, ...
Potentially:
High volume pre-operational or test data
without agreed service levels, bi-lateral data
exchange without agreed services levels
29 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT Search & Discovery
Key Features:
Product Navigator provides metadata descriptions, data provider and data access information.
Includes all EUMETSAT products and third-party products on EUMETCast.
Uses open standards (e.g. OGC, Inspire) for catalogue interoperability with other organisations.
30 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETCast – NRT dissemination
Key Features:
Off-the-shelf, commercially available DVB reception components
One-stop-shop - many data services via one station
Secure delivery – multicast to a specific user, or group of users supporting any Data Policy implementation
Handling many file formats, high and low volume data and supporting high-timeliness delivery requirements
Worldwide geographic coverage through GEONETCast partnership
EUMETCast Terrestrial Demonstration Service for high volume data provision to bi-lateral partners
31 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT Data Centre – offline dissemination
Archive dating back to 1981
1.1 Petabytes stored
1.5 Petabytes
retrieved annually
Raw and reprocessed data,
centrally and decentralised produced
Networked with Satellite
Application Facilities (SAFs)
Access online via
Product Navigator
32 Data Science Symposium, 26th October 2015, Delft, [email protected]
Climate data record project – information generation
SATELLITE
DATA
FUNDAMENTAL
CLIMATE DATA
RECORDS*
THEMATICAL
CLIMATE DATA
RECORDS**
MAJOR
MODEL-BASED
REANALYSIS
CLIMATE
INFORMATION
RECORDS
ADAPTION &
MITIGATION POLICY
AND PLANNING
(DECISION MAKING)
OPERATIONAL
MONITORING
OF WEATHER
AND THE
ENVIRONMENT
Short and medium latency
Sustained applications
Long-term information preservation
Climate Data Records
ARCHIVED
SATELLITE DATA
AND RECORDS
Climate services
OPERATIONAL
CLIMATE
MONITORING
LONGER TERM
CLIMATE
VARIABILITY
& CLIMATE
CHANGE ANALYSIS
* Fundamental
Climate Data Record
(FCDR):
a long-term data record
of calibrated and
quality-controlled
sensor data designed
to allow the generation
of homogeneous
products that are
accurate and stable
enough for climate
monitoring
** Thematic Climate
Data Record (TCDR):
a long-term data record
of validated and
quality-controlled
geophysical variables
derived from FCDRs.
ENVIRONMENTAL
DATA RECORDS
INTERIM CLIMATE
DATA RECORDS
Data conversion User service System performance monitoring
and automated corrections
Re-calibration / inter-calibration / reprocessing
33 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT Data Centre and Inter-Calibration –
information generation
Archive of raw and re-processed
data dating back to 1981
Data base for inter-calibration
of sensors over time and
CDR generation.
Fig: Illustration of time series of radiance
correction coefficients and their uncertainties.
t
∆x
Time series of bias estimated from:
– GSICS Near Real-Time Correction
– GSICS Re-Analysis Correction
34 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT SAF network across Europe
EUMETSAT NETWORK OF
SATELLITE APPLICATION
FACILITIES
35 Data Science Symposium, 26th October 2015, Delft, [email protected]
Supporting and expanding the user base
Established End-User requirements process
User involving process for the evolution of the data service portfolio
Training
Fellowships
Capacity building
Infusing science to meet evolving user requirements
Engagement with also new user communities
International partnerships
Cooperation with other operators & data providers
36 Data Science Symposium, 26th October 2015, Delft, [email protected]
THANK YOU
37 Data Science Symposium, 26th October 2015, Delft, [email protected]
ANNEX SLIDES
38 Data Science Symposium, 26th October 2015, Delft, [email protected]
The data provided by Meteosat satellites make a vital
contribution to daily weather forecasting, in particular for
nowcasting and very short range forecasting of high impact
weather like thunderstorms and fog over Europe, Africa and
adjacent seas.
Operated as a two-satellite system since 2006:
One satellite (currently Meteosat-10) provides
“full disc” images every 15 minutes over Europe,
Africa and part of the Atlantic and Indian oceans.
A second (currently Meteosat-9) provides “rapid scan”
images every five minutes over European
continent only.
A further satellite (currently Meteosat-8) serves
as a back-up for both the full disc and rapid
scanning services.
Meteosat Second Generation
39 Data Science Symposium, 26th October 2015, Delft, [email protected]
EUMETSAT Polar System (EPS)
The EUMETSAT Polar System is comprised of two
polar-orbiting satellites (Metop-A and -B) operating
in parallel and their respective ground segments.
Delivers continuous global observations for
meteorological applications and climate monitoring.
Constitutes the European contribution to the Initial
Joint Polar System (IJPS) with the US National
Oceanic and Atmospheric Administration (NOAA).
40 Data Science Symposium, 26th October 2015, Delft, [email protected]
Flying at an altitude of 817 km, each Metop satellite carries
the same dedicated, sophisticated suite of instruments.
They provide fine-scale global data, which can only
be gathered in the low Earth orbit, such as:
vertical profiles of atmospheric temperature
and moisture;
wind speed and direction at the ocean surface;
some atmospheric trace gases.
The satellites deliver data for NWP – the basis of
modern weather forecasting – and climate and
environmental monitoring.
The three Metop satellites, launched sequentially
(2006, 2012, 2018), will provide continuous data until 2020.
EUMETSAT Polar System (EPS)
41 Data Science Symposium, 26th October 2015, Delft, [email protected]
The Jason series: The reference for monitoring
ocean circulation and sea level
Launched on 20 June 2008, Jason-2 conducts
the Ocean Surface Topography Mission.
Continuing US-European high-precision altimetry
cooperation involving EUMETSAT, NOAA, CNES, NASA.
Aimed at expanding the series of high-precision data
initiated by Topex/Poseidon and continued by Jason-1.
Provides most accurate ocean surface topography
measurements available (a few centimetres) for
operational oceanography and sea level monitoring.
Jason-2 provides an indispensable reference against
which measurements of all other altimeter missions,
including Copernicus Sentinel-3, are cross-calibrated.
42 Data Science Symposium, 26th October 2015, Delft, [email protected]
The Jason series: The reference for monitoring
ocean circulation and sea level
Starting in 2015, EUMETSAT will support operations
of Jason-3, a recurrent Jason-2 satellite with the
same ground system.
Jason-3 will provide measurements of sea surface
height to an accuracy better than four centimetres,
every 10 days for a nominal lifetime of five years.
Jason-3 will be the first mission of the high-
precision ocean altimetry activity from the EU’s
Copernicus Programme.
The proposed two Jason-CS (Continuity of Service)
satellites will provide observations with higher
resolution along track and greater accuracy,
and ensure consistency with previous Jason
measurements and cross-calibration with Sentinel-3
and other altimetry data.
43 Data Science Symposium, 26th October 2015, Delft, [email protected]
Imagery mission implemented by
a two-satellite MTG-I system:
Full disc imagery every 10 minutes in 16 spectral
bands
Fast imaging of European weather every 2.5
minutes
new Lightning Imager (LI)
Hyper spectral infrared (IRS)
sounding mission:
3D mapping of water vapour, temperature,
O3 every 1 hour
Air quality monitoring and atmospheric chemistry in
synergy with GMES Sentinel-4 Ultraviolet Visible
Meteosat Third Generation
44 Data Science Symposium, 26th October 2015, Delft, [email protected]
4 imaging (MTG-I) and 2 sounding (MTG-S)
satellites,
start of operations in 2018 and 2019
operational exploitation: 2018 – 2038
full MTG mission implemented by two MTG-I
satellites and MTG-S satellite in orbit
Meteosat Third Generation
45 Data Science Symposium, 26th October 2015, Delft, [email protected]
EPS Second Generation
Primary mission: further improvement of
observational inputs to Numerical Weather
Prediction models
Significant improvements of other
applications
Now casting at high latitudes
Marine meteorology and operational
oceanography
Operational hydrology
Air quality monitoring
Climate monitoring
46 Data Science Symposium, 26th October 2015, Delft, [email protected]
EPS Second Generation
New two-satellite system in polar orbit:
Metop-SG A: optical imagery and
sounding mission
Metop-SG B: microwave imaging mission
Continuation and enhancement of polar-
orbiting service to meet growing requirements
in the years 2021 – 2040
Decision for the full EPS-SG programme
proposal still pending, at least three
Metop-SG A satellites, one or two
Metop-SG B satellites.