well-founded decisions based on climate trend analysis
DESCRIPTION
Various use cases across industries require credible knowledge about long-term weather and climate trends. Historical weather data is available which can be used to analyze trends. However, this data exists in a gigantic number of files, which are not directly usable for analytics. Using traditional techniques, transforming the data will take weeks. Fujitsu has built a showcase which allows you to transform historical weather data, depending on the available resources, within almost any given time slot. The succeeding retrieval of information needed for a certain use case and its visualization happens in a matter of seconds. This enables faster and better decisions, and minimizes the risks associated with these decisions. Speakers: Mr. Gernot Fels (Fujitsu)TRANSCRIPT
0 Copyright 2014 FUJITSU
Human CentricInnovation
Fujitsu Forum2014
19th – 20th November
1 Copyright 2014 FUJITSU
Informed Decisions Based on Climate Trend Analysis
Gernot FelsGlobal Services & Solutions Marketing, Fujitsu
Manager for Cloud Infrastructures and Big Data Innovations, Fujitsu
Dr. Fritz Schinkel
2 Copyright 2014 FUJITSU
The Promise of Big Data
Discover hidden secrets
Predict opportunities
Identify and minimize unknown risks
Take better and faster decisions
Accelerate business processes
Increase performance and productivity
Improve efficiency and effectiveness
Profitability and competitive advantage
Better utilize our planet‘s resources
A convincing value proposition.
3 Copyright 2014 FUJITSU
Manufacturing
Energy
Maintenance
Agriculture
Big Data matters to every industry
Big Data
Healthcare Transportation
New opportunities, new values for enterprises and society.
Retail Finance
Public Sector …
4 Copyright 2014 FUJITSU
Weather and climate trend predictions
Who needs to know credible long-term weather and climate trends?
Renewable power generation (solar, wind)
Power plant operation
Agricultural planning (flood, pest control)
Ski resort planning
Communities, counties, government
Transport, air-traffic control, shipping, sailing
Insurance
Manufacturing, retail, services
TV channels
…
Historical weather data required for trend analysis.
5 Copyright 2014 FUJITSU
Historical weather data for trend analysis
European Centre for Medium-Range Weather Forecasts (ECMWF)
Analysis of weather development
Global weather data since 1979
Time series of weather maps
Usable for climate research and local trend analysis
Weather model ERA Interim
ECMWF Re-Analysis; Interim (highest resolution)
Model resolution
Time interval 6h
Measurement time: 0:00, 6:00, 12:00, 18:00 GMT
Grid of 0,25° (4 grid points per degree)
128 meteorological indicators
Time series of weather maps
6 Copyright 2014 FUJITSU
Meteorological indicators
10 metre U wind component Large-scale snowfall Surface net thermal radiation Vertical integral of eastward cloud liquid water flux10 metre V wind component Logarithm of surface roughness length for heat Surface net thermal radiation, clear sky Vertical integral of eastward geopotential flux10 metre wind gust since previous post-processing Low cloud cover Surface pressure Vertical integral of eastward heat flux2 metre dewpoint temperature Maximum temperature at 2 metres since previous post-processing Surface roughness Vertical integral of eastward kinetic energy flux2 metre temperature Mean sea level pressure Surface sensible heat flux Vertical integral of eastward mass fluxAlbedo Mean wave direction Surface solar radiation downwards Vertical integral of eastward ozone fluxBoundary layer dissipation Mean wave period Surface thermal radiation downwards Vertical integral of eastward total energy fluxBoundary layer height Medium cloud cover Temperature of snow layer Vertical integral of eastward water vapour fluxCharnock Minimum temperature at 2 metres since previous post-processing TOA incident solar radiation Vertical integral of energy conversionClear sky surface photosynthetically active radiation Northward gravity wave surface stress Top net solar radiation Vertical integral of kinetic energyConvective available potential energy Northward turbulent surface stress Top net solar radiation, clear sky Vertical integral of mass of atmosphereConvective precipitation Photosynthetically active radiation at the surface Top net thermal radiation Vertical integral of mass tendencyConvective snowfall Runoff Top net thermal radiation, clear sky Vertical integral of northward cloud frozen water fluxDownward UV radiation at the surface Sea surface temperature Total cloud cover Vertical integral of northward cloud liquid water fluxEastward gravity wave surface stress Sea-ice cover Total column ice water Vertical integral of northward geopotential fluxEastward turbulent surface stress Significant height of combined wind waves and swell Total column liquid water Vertical integral of northward heat fluxEvaporation Skin reservoir content Total column ozone Vertical integral of northward kinetic energy fluxForecast albedo Skin temperature Total column water Vertical integral of northward mass fluxForecast logarithm of surface roughness for heat Snow albedo Total column water vapour Vertical integral of northward ozone fluxForecast surface roughness Snow density Total precipitation Vertical integral of northward total energy fluxGravity wave dissipation Snow depth Vertical integral of cloud frozen water Vertical integral of northward water vapour fluxHigh cloud cover Snow evaporation Vertical integral of cloud liquid water Vertical integral of ozoneIce temperature layer 1 Snowfall Vertical integral of divergence of cloud frozen water flux Vertical integral of potential+internal energyIce temperature layer 2 Snowmelt Vertical integral of divergence of cloud liquid water flux Vertical integral of potential+internal+latent energyIce temperature layer 3 Soil temperature level 1 Vertical integral of divergence of geopotential flux Vertical integral of temperatureIce temperature layer 4 Soil temperature level 2 Vertical integral of divergence of kinetic energy flux Vertical integral of thermal energyInstantaneous eastward turbulent surface stress Soil temperature level 3 Vertical integral of divergence of mass flux Vertical integral of total energyInstantaneous moisture flux Soil temperature level 4 Vertical integral of divergence of moisture flux Vertical integral of water vapourInstantaneous northward turbulent surface stress Sunshine duration Vertical integral of divergence of ozone flux Volumetric soil water layer 1Instantaneous surface sensible heat flux Surface latent heat flux Vertical integral of divergence of thermal energy flux Volumetric soil water layer 2Large-scale precipitation Surface net solar radiation Vertical integral of divergence of total energy flux Volumetric soil water layer 3Large-scale precipitation fraction Surface net solar radiation, clear sky Vertical integral of eastward cloud frozen water flux Volumetric soil water layer 4
7 Copyright 2014 FUJITSU
GRIB for historical and forecast weather data
GRIdded Binary
Compressed binary format
Standard defined by WMO (World Meteorological Organization)
Used to store weather data
Based on rectangular grid
Geographic coordinates as grid points
8 Copyright 2014 FUJITSU
Challenges and objectives
Challenges
Time series of global weather maps do not give immediate insight for certain location
Difficult and long-lasting evaluation(e.g. for wind probability estimation)
Late decisions
Objectives
Retrieve time series for certain location
Create puncture for relevant grid points over relevant period of time
Data transformation needed
How many files and which data volumes?
9 Copyright 2014 FUJITSU
File quantities and data volumes
1 GRIB file per snapshot
4 snapshots per day
51,100 input files (time-related) over 35 years
360 degrees of longitude
180 degrees of latitude
4 grid points per degree of longitude and latitude
Add 1 grid point (north and south pole)
360 x 4 x (180 x 4 +1) =1,038,240 grid points =output files (location-related)
128 meteorological indicators, 4 bytes each
25 TB of historical weather data
Which solution concept will help?
10 Copyright 2014 FUJITSU
Distributed Parallel Processing
DataNode
TaskTracker
DataNode
TaskTracker
DataNode
TaskTracker
NameNode
JobTracker
Clie
nt
Master
Slaves DFSConcept
Distribute data and I/O to server cluster nodes
Local server storage
Move computing to where data resides
Shared nothing architecture
Data replication to several nodes
Benefits
High performance, fast results
Unlimited scalability
Fault-tolerance
Cost-effective (standard servers with OSS)
De-facto standard
11 Copyright 2014 FUJITSU
platform meets the challenges
ECMWF data as input
Load ECMWF data into HDFS
Use MR to invert data
From time series of world-wide weather maps
Into grid point based time series of weather data
Arrange as (sorted) KV pairs
Key = grid point and time combined
Value = Meteorological data
Java apps
Retrieve proximate time series, determine local weather development
Visualize results
Incremental update by short MR jobs
Import weather history
Invert time series
Retrieve proximate time series,determine local weather development
Visualizeresult
12 Copyright 2014 FUJITSU
Example: Wind park planning
Time period: 14 years (2000-2013)
4 snapshots per day
20,456 input files (time-related) from ECMWF
21,221,625,600 records
1,038,240 grid points = output files (location-related)
13 Copyright 2014 FUJITSU
Solution approach effects time to value
PERL
~100 min for processing data from 1 month
12 x 14 x 100 min ~ 12 days(over 14 years)
12 x 35 x 100 min ~ 30 days(over 35 years)
Not acceptable in practice
14 Copyright 2014 FUJITSU
Solution approach effects time to value
PERL
~100 min for processing data from 1 month
12 x 14 x 100 min ~ 12 days(over 14 years)
12 x 35 x 100 min ~ 30 days(over 35 years)
Not acceptable in practice
MapReduce
30 min for import to HDFS
141 min processing time
Read HDFS files (historical data)
Data transformation
Write results to HDFS files
~120 x faster than script approach
8 Slave Nodes (2-socket, 6C/12T)
Servers not fully utilized
Potential for improvement by removing other workload
Speed advantage by parallelization.
15 Copyright 2014 FUJITSU
Solution approach effects time to value
PERL
~100 min for processing data from 1 month
12 x 14 x 100 min ~ 12 days(over 14 years)
12 x 35 x 100 min ~ 30 days(over 35 years)
Not acceptable in practice
MapReduce
30 min for import to HDFS
141 min processing time
Read HDFS files (historical data)
Data transformation
Write results to HDFS files
~120 x faster than script approach
8 Slave Nodes (2-socket, 6C/12T)
Servers not fully utilized
Potential for improvement by removing other workload
Options for further acceleration?
16 Copyright 2014 FUJITSU
In-memory platform as option
For single users Hadoop platform is sufficient
Retrieval and visualization within seconds
Increasing response times
Increasing number of users
Increasing number of queries
Complex queries, e.g. where in certain geographic area are most favorite locations for certain plans
Solution: IMDG
Accelerate retrieval and visualization
Pre-defined queries memory-resident
In-memory platforms help cope with any level of complexity.
Import weather history
Invert time seriesRetrieve proximate time series,
determine local weather development
Visualizeresult
17 Copyright 2014 FUJITSU
Data transformation – Just 1x or more often?
Depending on use case no one-time act
New inversion of weather maps with new questions
Deduction from meteorological indicators in global weather maps
Example: Max. wind speed
Peak speeds of crossing weather front occur only shortly at one location
Often fail at 6 hrs grid
Determine front on weather map timely before and thereafter
What is the effort to realize new questions?
18 Copyright 2014 FUJITSU
Dreamlike Big Data
Display weather data as table –Spreadsheet like Excel
Apply meteorological formulas directly to sample data
Check partial results at once in spreadsheet
Fast test run on significant (filtered) test data set
Simple expansion to total data set and visualization
Big Data for business users.
Process large data volumes, but avoid programming
MR jobs?
19 Copyright 2014 FUJITSU
Dreamlike Big Data
Display weather data as table –Spreadsheet like Excel
Apply meteorological formulas directly to sample data
Check partial results at once in spreadsheet
Fast test run on significant (filtered) test data set
Simple expansion to total data set and visualization
Big Data for business users.
20 Copyright 2014 FUJITSU
Is this a Big Data project?
1 data source
Structured data
Data is not generated at high speed
Analysis not always time-critical
25 TB x 2 is a considerable volume
Traditional technologies do not help
Big Data technologies solve customer problem
Affordable
Scalable with growth
Expected processing time can be controlled
Indeed no day-to-day Big Data project, but a very interesting one.
Volume Variety VelocityVersatility Value
21 Copyright 2014 FUJITSU
Un- / Semi-/Poly-
structureddata
Big Data infrastructure
IMDB
Data Sources Analytics Platform Access
Extract, Collect Clean, Transform Decide, ActAnalyze, Visualize
Consolidated data Distilled essence Applied knowledgeVarious data
IMDB
AppsServicesQueries
….
VisualizationReporting
Notification
DistributedParallel
Processing
IMDG
Dat
a at
res
tD
ata
in m
otio
n
FS
IMDG
DB / DW
DB / DWNoSQL NoSQL
IMDG
CEP
22 Copyright 2014 FUJITSU
IMDG
Un- / Semi-/Poly-
structureddata
Big Data infrastructure
IMDB
Data Sources Analytics Platform Access
Extract, Collect Clean, Transform Decide, ActAnalyze, Visualize
Consolidated data Distilled essence Applied knowledgeVarious data
IMDB
AppsServicesQueries
….
VisualizationReporting
NotificationIMDG
Dat
a at
res
tD
ata
in m
otio
n
FS DB / DW
DB / DWNoSQL NoSQL
IMDG
CEP
23 Copyright 2014 FUJITSU
Un- / Semi-/Poly-
structureddata
Big Data infrastructure
IMDB
Data Sources Analytics Platform Access
Extract, Collect Clean, Transform Decide, ActAnalyze, Visualize
Consolidated data Distilled essence Applied knowledgeVarious data
IMDB
AppsServicesQueries
….
VisualizationReporting
NotificationIMDG
Dat
a at
res
tD
ata
in m
otio
n
FS DB / DW
DB / DWNoSQL NoSQL
IMDG
CEP
24 Copyright 2014 FUJITSU
Un- / Semi-/Poly-
structureddata
Big Data infrastructure
IMDB
Data Sources Analytics Platform Access
Extract, Collect Clean, Transform Decide, ActAnalyze, Visualize
Consolidated data Distilled essence Applied knowledgeVarious data
IMDB
AppsServicesQueries
….
VisualizationReporting
NotificationIMDG
Dat
a at
res
tD
ata
in m
otio
n
FS DB / DW
DB / DWNoSQL NoSQL
IMDG
CEP
25 Copyright 2014 FUJITSU
How can help
Complete analytics platform
Infrastructure and services
Consulting, introduction, operation, maintenance
Apps for analysis and visualization
Integrated Systems for fast deployment
Location-based time series as cloud service
Weather prediction expertise
Everything from a single source: Simple, fast, without risk.
26 Copyright 2014 FUJITSU
FUJITSU Showcase
27 Copyright 2014 FUJITSU
Fujitsu showcase
What are the wind trends in a certain area?
Is weather better during weekends or on working days?
In which areas which differences?
Questions to be answered
28 Copyright 2014 FUJITSU
Summary
29 Copyright 2014 FUJITSU
Summary
Big Data – one of today’s megatrends
Promising value proposition
Exciting use cases across industries
Knowledge about future weather and climate is valuable for many target groups
Historical data available
Transformation needed to get desired insight and recognize trends
End-to-end solutions from Fujitsu
Integrated systems for fast-time to production
Supplementing services
You’d like to look into the future? Have a word with Fujitsu.
Thank you for listening
31 Copyright 2014 FUJITSU
Appendix
32 Copyright 2014 FUJITSU
Fujitsu Showcase: Wind Trends (1)
Select location in map or satellite view by click …
… or select previously saved location
33 Copyright 2014 FUJITSU
Fujitsu Showcase: Wind Trends (2)
Wait a second …
34 Copyright 2014 FUJITSU
Fujitsu Showcase: Wind Trends (3)
See the wind, temperature and
air pressure
Move over the charts and see individual
values
35 Copyright 2014 FUJITSU
Fujitsu Showcase: Wind Trends (4)
Select zoom windows to see details in
wind speed, temperature and
air pressure
36 Copyright 2014 FUJITSU
Fujitsu Showcase: Wind Trends (5)
Distribution of wind speed over time,
and distribution of wind frequency
and speed along wind direction
is displayed and …
37 Copyright 2014 FUJITSU
Fujitsu Showcase: Wind Trends (6)
… month to be taken into account can be
restricted and animated
Distribution of wind speed over time,
and distribution of wind frequency
and speed along wind direction
is displayed and …
38 Copyright 2014 FUJITSU
Fujitsu Showcase: Wind Trends (7)
Select year or span of years
for long term trends
39 Copyright 2014 FUJITSU
Fujitsu Showcase: Wind Trends (8)
MunichMecklenburg-Vorpommern
PaderbornBorkum
(off-shore)
Somalia
Cape Horn
40 Copyright 2014 FUJITSU
Fujitsu Showcase: Weekdays and Weather (1)
Get complete page automatically published
Locations with most significant span between warmest and coldest weekday average
as map and as list
Number of grid points with maximum / minimum temperature on certain weekday
Locations with most significant span between warmest and coldest weekday average
and warmest day on a certain weekday
41 Copyright 2014 FUJITSU
Fujitsu Showcase: Weekdays and Weather (2)
Visualization GUI to study the span of weekday mean
temperature at certain places and to look for possible reasons
Map colored for high span of weekday mean temperature
42 Copyright 2014 FUJITSU
Fujitsu Showcase: Weekdays and Weather (3)
Sliders for span threshold,
contrast and opacity of coloring
43 Copyright 2014 FUJITSU
Fujitsu Showcase: Weekdays and Weather (4)
And an adjustment for grid points with low temperature
span over the complete observation time
44 Copyright 2014 FUJITSU
Fujitsu Showcase: Weekdays and Weather (5)
Using the color settings and the zooming into the map
we can find areas with significant differences of
weekday mean values in the observed timeframe
45 Copyright 2014 FUJITSU
Fujitsu Showcase: Weekdays and Weather (6)
Click to a certain position shows the curve of
average temperature for the weekdays,
the coordinates and the total min/max temperature
of the point
46 Copyright 2014 FUJITSU
Fujitsu Showcase: Weekdays and Weather (7)
Map and satellite can be used to find
possible reasons for mean temperature
related to weekdays
47 Copyright 2014 FUJITSU
Fujitsu Showcase: Weekdays and Weather (8)
Zoom into the source of the color cloud
Industrial complex isshut down on Sunday?
48 Copyright 2014 FUJITSU
Fujitsu Showcase: Weekdays and Weather (9)
US east cost is cooler on Sunday / Monday
Is traffic system heating
the atmosphere over the week?
49 Copyright 2014 FUJITSU
Fujitsu Showcase: Weekdays and Weather (10)
South of Hudson Bay is an area with Wednesday
mean temperature approx. 1C higher than on Saturday
Does wood industry influence the temperature
in the rhythm of the week?
50 Copyright 2014 FUJITSU