earth observation in the cloud
TRANSCRIPT
![Page 1: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/1.jpg)
© 2015, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Earth Observation in the Cloud
Jed Sundwall, AWS Open Data Global Lead
10 November 2015
Demo Day
![Page 2: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/2.jpg)
2
Agenda
• Welcome! 🌍🌎🌏• Open data on AWS• NEXRAD on AWS• Landsat on AWS
![Page 3: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/3.jpg)
Welcome! 🌍🌎🌏
3
![Page 4: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/4.jpg)
Welcome! 🌏🌍🌎
4
![Page 5: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/5.jpg)
Welcome! 🌎🌏🌍
5
![Page 6: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/6.jpg)
Sponsor Sponsor Sponsor
Venue Sponsor
Thank you to our sponsors!
![Page 7: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/7.jpg)
Our goals for this event
• Show off amazing work being done by our customers• Provide opportunities for you to network• Highlight the diversity of work made possible by Earth
observation data• Learn about your priorities and needs
![Page 8: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/8.jpg)
New whitepaper
We have just published a new AWS Whitepaper on Minimizing Variable Costs for Shared Data.
Download it at:
http://bit.ly/s3-requester-pays-open-data
![Page 9: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/9.jpg)
Open data on AWS
9
![Page 10: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/10.jpg)
10
Why does AWS care about open data?
Open data is data that can be used by anyone for any purpose for free.
Many of our customers rely on quality open data as much as they rely on our computing, storage, and other web services.
![Page 11: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/11.jpg)
Data on AWS
Amazon Web Services provides a comprehensive toolkit for gathering, storing, analyzing, and working with data at any scale.
Amazon Elastic MapReduce (Amazon EMR) provides the Apache Hadoop analytics framework as an easy-to-use managed service.
Amazon S3 lets you store and retrieve any amount of data, at any time, from anywhere on the web.
Amazon DynamoDB is a fully-managed NoSQL database service that makes it cost-effective to store and retrieve any amount of data.
11
![Page 12: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/12.jpg)
1-click deployment to launch, on multiple regions around the world
Pay-as-you-go pricing
Advanced AnalyticsData Integration Analysis & Visualization
http://bit.ly/awsAnalytics12
![Page 13: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/13.jpg)
The power of open data in the cloud
Making data open on AWS enables more innovation by making data available for rapid access to our flexible and low-cost computing resources.
13
![Page 14: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/14.jpg)
Making data open on AWS enables more innovation by making data available for rapid access to our flexible and low-cost computing resources.
The power of open data in the cloud
14
![Page 15: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/15.jpg)
AWS Partners Focused on Public Sector
15
![Page 16: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/16.jpg)
History of InnovationAWS has been continually expanding its services to support virtually any cloud workload, now offering more than 40 services.
Amazon S3
Amazon SQS
Amazon EC2
Amazon SimpleDB
Amazon EBS
Amazon CloudFront
Elastic LoadBalancingAuto ScalingAmazon VPCAmazon RDS
Amazon SNSAWS Identity and Access Management
Amazon Route 53
Amazon SESAWS Elastic BeanstalkAWS CloudFormationAmazon ElastiCacheAWS Direct ConnectAWS GovCloud
AWS Storage Gateway
Amazon DynamoDB
Amazon CloudSearch
Amazon SWF
Amazon Glacier
Amazon Redshift
AWS Data Pipeline
Amazon Elastic TranscoderAWS OpsWorksAWS CloudHSMAmazon AppStreamAWS CloudTrailAmazon WorkSpacesAmazon Kinesis
Amazon ECS
Amazon Lambda
AWS Config
AWS CodeDeploy
Amazon RDS for Aurora
AWS KMS
Amazon Cognito
Amazon WorkDocs
AWS Directory Service
Amazon Mobile Analytics
2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Amazon EFSAmazon WorkMailAmazon Machine Learning
16
![Page 17: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/17.jpg)
AWS has announced price reductions 49* times since our inception in 2006. Recent price drops included…
Amazon ElastiCache reduces prices for cache nodes by an average of 34%
March 26, 2014
34%Amazon S3 reduces prices for Standard and Reduced Redundancy Storage, by an average of 51%
March 26, 2014
51%Amazon Route 53 lowers prices for both standard queries and latency-based routing queriesby 20%
July 31, 2014
20%
17
* As of June 2015
![Page 18: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/18.jpg)
18
Open data as a platform
![Page 19: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/19.jpg)
19
Open data as a platform
![Page 20: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/20.jpg)
20
An Amazonian approach to open data
Two ideas that inform how we approach public data sets:• Work backwards from the customer• Eliminate undifferentiated heavy lifting
![Page 21: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/21.jpg)
21
Working Backwards
• Think of data sets as products• Seek out valuable data by listening to customer needs• Consider real-world use cases for the data• Consider the size of the user community or market
opportunity
![Page 22: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/22.jpg)
22
Undifferentiated heavy lifting
“…data must be organized, well-documented, consistently formatted, and error free. Cleaning the data is often the most taxing part of data science, and is frequently 80% of the work.”— Data Driven by DJ Patil and Hilary Mason
![Page 23: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/23.jpg)
23
Undifferentiated heavy lifting
“…data must be organized, well-documented, consistently formatted, and error free. Cleaning the data is often the most taxing part of data science, and is frequently 80% of the work.”— Data Driven by DJ Patil and Hilary Mason
We ask: How can we get rid of that 80%?
![Page 24: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/24.jpg)
24
Public datasets on AWS
To enable more innovation, AWS hosts a selection of datasets that anyone can access for free. Data in our public datasets is available for rapid access to our flexible and low-cost computing resources.
Earth ScienceLandsat on AWS
Life Sciences1000 Genomes Project
Internet ScienceCommon Crawl Corpus
![Page 25: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/25.jpg)
NEXRAD on AWS
25
![Page 26: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/26.jpg)
NEXRAD on AWS
The Next Generation Weather Radar (NEXRAD) is a network of 160 high-resolution Doppler radar sites that detects precipitation and atmospheric movement and disseminates data in 5 minute intervals from each site.
It has traditionally been time consuming and expensive to acquire, store, and analyze NEXRAD data. Accessing the full historical archive has been impossible.
26
![Page 27: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/27.jpg)
NEXRAD on AWS
NEXRADSites
Public Amazon S3 Bucket
AmazonEC2
Public Amazon S3 Bucket
Real-time data chunks
Volume scan file assembly
Continuously updated archive
With NEXRAD on AWS, we provide an archive of individual volume scan files and real-time chunks as objects in Amazon S3.
This allows the data to be accessed programmatically via a RESTful interface and quickly deployed to any of our products for analysis and processing.
27
![Page 28: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/28.jpg)
NEXRAD on AWS
Our collaborators, including Unidata, The Weather Company, NOAA, Climate Corporation, and CartoDB, have provided early use cases and tutorials on how to use this data in the cloud.
A wide range of users are interested in using NEXRAD on AWS for longitudinal analysis, to study and visualize specific weather events, and develop new products.
More info at http://aws.amazon.com/public-data-sets/nexrad
28
![Page 29: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/29.jpg)
Landsat on AWS
29
![Page 30: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/30.jpg)
30
Landsat on AWS
We have committed to make up to 1 petabyte of Landsat imagery readily available as objects on Amazon S3.
All Landsat 8 scenes from 2015 are available, along with a selection of cloud-free scenes from 2013 and 2014. All new Landsat 8 scenes are made available each day (~700 per day), often within hours of production.
![Page 31: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/31.jpg)
31
The Traditional Approach
Data is most commonly accessed via a web interface and downloaded on premises before being loaded into a web server.
All bands are downloaded in a .tar archive, even if you only need a few bands.
Data acquisition is time consuming and inherently redundant. Analysis is limited by user’s access to bandwidth, storage, memory, and processing power.
![Page 32: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/32.jpg)
32
Landsat on AWS
Landsat on AWS makes each band of each scene readily available as objects on Amazon S3. Data can be accessed programmatically via HTTP and quickly deployed to any of our products for analysis and processing.
Users do not need to worry about local storage and have access to virtually unlimited computing power on demand.
AmazonEC2
s3://landsat-pds
.tarUSGS
.tiff
![Page 33: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/33.jpg)
33
Undifferentiated heavy lifting
We use GDAL to add “internal tiling” on each Landsat on AWS tiff, which allows developers to use HTTP range gets to access specific portions of each scene.
This allows people to only access the data they need when they need it. Standard tiff
objectInternal tiled tiff
object
![Page 34: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/34.jpg)
RGBVisible light
InfraredVegetation
Shortwave infraredUrban areas
Wellington, New Zealand – Made on Snapsat.orghttps://landsat-pds.s3.amazonaws.com/L8/072/089/
![Page 35: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/35.jpg)
RGBVisible light
InfraredVegetation
Shortwave infraredUrban areas
SHARE DATA VIA URLS, NOT COPIES
Wellington, New Zealand – Made on Snapsat.orghttps://landsat-pds.s3.amazonaws.com/L8/072/089/
![Page 36: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/36.jpg)
36
Landsat on AWSIn the first 150 days (19 Mar – 16 Aug 2015)
• Over 200,000 scenes available
• Over 500 million hits globally
Image shows frequency of scene requests by path/row.
White: ~100 requestsOrange: >300k requests
Visualization by Drew BollingerDevelopment Seed
![Page 37: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/37.jpg)
37
Landsat on AWS as a platform
![Page 38: Earth Observation in the Cloud](https://reader036.vdocuments.us/reader036/viewer/2022070515/587aef4a1a28ab7f378b626b/html5/thumbnails/38.jpg)
38
New SNS topic for Landsat on AWS
arn:aws:sns:us-west-2:274514004127:OpenObjectAddL8
You can now subscribe to a publicly available Amazon Simple Notification Service (Amazon SNS) topic to be notified whenever a new batch of Landsat scenes are available at s3://landsat-pds.