from science to operational information: earth observation and … · 2017-09-11 · from science...
TRANSCRIPT
![Page 1: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/1.jpg)
From Science to Operational Information: Earth
Observation and Software Development Trends
Arnold Hougham
Vice President, Sales and Marketing
PCI Geomatics August 24, 2017
![Page 2: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/2.jpg)
Important trends:
Earth Observation Imagery
![Page 3: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/3.jpg)
More EO Missions
Source: EC, JRC, Ispra (2014): http://www.sciencedirect.com/science/article/pii/S0924271614000720
197 missions since 1972
Launches per year/per decade:
1970-80: 2
1980-90: 3
1990-00: 5.5
2000-10: 8
2010-14: 12 (6 x first decade)
![Page 4: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/4.jpg)
Source: EC, JRC, Ispra (2014):
http://www.sciencedirect.com/science/article/pii/S0924271614000720
![Page 5: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/5.jpg)
Worldwide commercial satellite launches
Chart pre-dates
latest launch by
Planet (88
satellites on
February 14 2017)
5
Private & Confidential
![Page 6: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/6.jpg)
“Imagery for
Everyone”
More Accessibility (“free”)
![Page 7: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/7.jpg)
More usage of EO imagery Global Land Cover Facility, at the
University of Maryland was the key catalyst
for open data policy (copying / sharing data
once purchased)
www.glovis.usgs.gov
EROS Data Centre provided approximately
25,000 Landsat images in 2001 (total) vs
250,000 images (per month) in 2010!
25,000 vs 3,000,000 / year
Increased by factor of
120!!
Source: Opening the archive: How free data has enabled the science
and monitoring promise of Landsat, Wulder et al, 2012
![Page 8: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/8.jpg)
More Accessibility (“free”)
![Page 9: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/9.jpg)
Price Pressure on imagery (more options)
Source: https://droneapps.co/price-wars-the-cost-of-drones-planes-and-satellites/
![Page 10: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/10.jpg)
Price Pressure on imagery (resolution)
10
Private & Confidential
![Page 11: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/11.jpg)
Frequency of Acquisition
11
Private & Confidential
![Page 12: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/12.jpg)
Important trends:
Funding
![Page 13: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/13.jpg)
Private funding for commercial
satellite programs
• The funding model for
new space based
startups is changing
• Not only about cost
recovery for big
programs; it’s also about
profit opportunities that
come with constellation /
privately funded
missions / companies Source: https://venturebeat.com/2016/08/14/earth-observation-data-multibillion-dollar-opportunity-or-dud/
![Page 14: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/14.jpg)
Important trends:
More data
![Page 15: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/15.jpg)
Sentinel Data Volumes
15
Private & Confidential
“By the end of 2017,
the operational
Sentinel-1, -2 and -3
satellites alone will
continuously collect a
volume of 20
Terabytes per day.”
https://medium.com/google-earth/a-golden-age-for-earth-observation-f8b281cec4b7
![Page 16: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/16.jpg)
EO Data Trends:
Consequences and Opportunities
![Page 17: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/17.jpg)
Multi-temporal methods possible
Change detection is not new…
…but has not been practical due to data
costs and access… until now. 17
Private & Confidential
![Page 18: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/18.jpg)
Australia Data Cube
18
Private & Confidential
![Page 19: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/19.jpg)
Alignment and Calibration
![Page 20: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/20.jpg)
The need for pixel-perfect alignment
Reference Image Un-aligned Image
![Page 21: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/21.jpg)
The need for pixel-perfect alignment
Reference Image Misaligned Image
Site 1
Site 2
Site 3
![Page 22: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/22.jpg)
Multi-temporal not possible Site 1 Site 2 Site 3
![Page 23: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/23.jpg)
Multi-temporal possible Site 1 Site 2 Site 3
![Page 24: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/24.jpg)
Cross Sensor Calibration PlanetScope
Dec 01/2016
3:05 PM
Local time
![Page 25: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/25.jpg)
Cross Sensor Calibration RapidEye
Dec 03/2016
11:36 AM
Local time
![Page 26: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/26.jpg)
No Calibration NDVI PlanetScope
Dec 01/2016
3:05 PM
Local time
![Page 27: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/27.jpg)
No Calibration NDVI RapidEye
Dec 03/2016
11:36 AM
Local time
![Page 28: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/28.jpg)
Calibrated NDVI PlanetScope
Dec 01/2016
3:05 PM
Local time
![Page 29: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/29.jpg)
RapidEye
Dec 03/2016
11:36 AM
Local time
Calibrated NDVI
![Page 30: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/30.jpg)
Calibration Makes Big Multi-temporal
Analysis Possible
30
Private & Confidential
Dec 01/2016 (PS), Dec 03/2016 (RE), Dec 11/2016 (PS), Dec 20/2016 (RE), Dec 29/2016 (PS)
![Page 31: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/31.jpg)
Big Data Analytics
![Page 32: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/32.jpg)
Software needs to be scalable,
and easily integrated Esri Cloud processing project
![Page 33: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/33.jpg)
Massive Cloud Scalability (PCI/Esri 2011)
Selected
Scenes
IKONOS
Source
S3 ArcGIS
Online
Image Processing Farm Image Processing Farm
Cache Processing Farm
Instances + Lustre + GXL Instances + Lustre + GXL
Instances + Lustre + AGS
Multiple farms
& Lustre file-
systems
isolate I/O and
provide
horizontal
scalability
![Page 34: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/34.jpg)
Node Configuration • m1.xlarge
– 15 GB memory
– 8 EC2 compute units
• 4 virtual cores, each with 2 x EC2
– 1,690 GB ephemeral storage included “free”
– 8 GB EC2 volume
– 64-bit Linux OS
– I/O performance level: High
• Designed to use minimal EC2 resources
Page
34 Copyright © 2014 PCI Geomatics
“Standard”
compute node
![Page 35: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/35.jpg)
Typical Task Provisioning • Per farm, node requirements are typically as follows:
– Ingest 25+ nodes
– Tie-point collection 20+ nodes
– Ortho w/Pansharpen 20+ nodes
– Colour balancing 1 node
– Mosaic Tiling 25+ nodes
– Caching 8 nodes
• Tasks are pipelined on each farm
• Operational nodes per farm is 25+
• For 9 operational farms, potential is 225+ nodes
Page
35 Copyright © 2014 PCI Geomatics
![Page 36: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/36.jpg)
Cloud Geo-Processing Engine
Page
36 Copyright © 2014 PCI Geomatics
Ingest Tie Point & Bundle Ortho
Pan Sharp
Colour Balance
Seam Line
Tile Cache
and deploy
PCI GXL /
Geomatica ArcGIS
Components for 10.0 & 10.1b GP Workflows
![Page 37: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/37.jpg)
Production Throughput • Average based on current cloud compute configuration
• Component km²/node-hr
– Ingest 7007 km²
– Tie-point collection 2685 km²
– Ortho w/Pansharpen 1773 km²
– Colour balancing 39,925 km²
– Mosaic Tiling 1899 km²
– Caching 13,926 km²
• Total processing per area-of-interest ~681 node-hours
• Processing varies with sensor and size of area
Page
37 Copyright © 2014 PCI Geomatics
![Page 38: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/38.jpg)
VC Funding EO / Proliferation
![Page 39: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/39.jpg)
Effect of VC Funding
Verticalization • Examples: “8 Satellite Data Startups Doing Geospatial
Analysis” (nanalyze.com article)
39
Private & Confidential http://www.nanalyze.com/2017/08/8-satellite-data-startups-geospatial-analysis/?lipi=urn%3Ali%3Apage%3Ad_flagship3_feed%3BeM5f1gArTd%2B4lZiZWEDa6Q%3D%3D
“Under Cover of Darkness”
![Page 40: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/40.jpg)
Effect of VC Funding
Proliferation and verticalization • Examples: “8 Satellite Data Startups Doing Geospatial
Analysis” (nanalyze.com article)
40
Private & Confidential http://www.nanalyze.com/2017/08/8-satellite-data-startups-geospatial-analysis/?lipi=urn%3Ali%3Apage%3Ad_flagship3_feed%3BeM5f1gArTd%2B4lZiZWEDa6Q%3D%3D
“Reliable Economic Information”
![Page 41: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/41.jpg)
Effect of VC Funding
Proliferation and verticalization • Examples: “8 Satellite Data Startups Doing Geospatial
Analysis” (nanalyze.com article)
41
Private & Confidential http://www.nanalyze.com/2017/08/8-satellite-data-startups-geospatial-analysis/?lipi=urn%3Ali%3Apage%3Ad_flagship3_feed%3BeM5f1gArTd%2B4lZiZWEDa6Q%3D%3D
“Parking Lot Economics”
![Page 42: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/42.jpg)
Effect of VC Funding
Proliferation and verticalization • Examples: “8 Satellite Data Startups Doing Geospatial
Analysis” (nanalyze.com article)
42
Private & Confidential http://www.nanalyze.com/2017/08/8-satellite-data-startups-geospatial-analysis/?lipi=urn%3Ali%3Apage%3Ad_flagship3_feed%3BeM5f1gArTd%2B4lZiZWEDa6Q%3D%3D
“Old McDonald Had a Farm”
![Page 43: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/43.jpg)
Effect of VC Funding
Proliferation and verticalization • Examples: “8 Satellite Data Startups Doing Geospatial
Analysis” (nanalyze.com article)
43
Private & Confidential http://www.nanalyze.com/2017/08/8-satellite-data-startups-geospatial-analysis/?lipi=urn%3Ali%3Apage%3Ad_flagship3_feed%3BeM5f1gArTd%2B4lZiZWEDa6Q%3D%3D
“Data access for DIY Intelligence”
![Page 44: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/44.jpg)
Big Data Effects
“I can’t wait for download delivery of data. I want to prototype
my complex workflow on the desktop and migrate it to where
the data are: in the “cloud.””
44
Private & Confidential
![Page 45: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/45.jpg)
Build Custom
Applications
Local System
Run Locally
Package Geomatica
Local Deployment
![Page 46: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/46.jpg)
Build Custom
Applications
Local System
Run Remote
Package Geomatica
Remote Deployment
AWS Cluster
![Page 47: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/47.jpg)
Machine Learning • Important technique given large datasets
• Support Vector Machine (SVM implemented in PCI OBIA s/w,
Python integration2018
47
Private & Confidential
![Page 48: From Science to Operational Information: Earth Observation and … · 2017-09-11 · From Science to Operational Information: Earth Observation and Software Development Trends Arnold](https://reader034.vdocuments.us/reader034/viewer/2022042105/5e8327614cea11345513ad27/html5/thumbnails/48.jpg)
Summary • EO proliferation = Big data
• Big data changes everything
– Need for speed
– Need for new deployment models and business models
– Machine learning
• Science matters
– Get the pixels in the right place (this is not easy)
– Calibration and validation (this is not easy)
• Solutions are proliferating
– Consumer beware
– Experience matters