#earthonaws: how the cloud is transforming earth observation | aws public sector summit 2016
TRANSCRIPT
![Page 1: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/1.jpg)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Jed Sundwall, Global Open Data Lead
June 20, 2016
#EarthOnAWS: How the Cloud Is Transforming Earth Observation
![Page 2: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/2.jpg)
Why does AWS care about open data?
Many of our commercial sector customers rely on quality open data as much as they rely on our cloud infrastructure services.
Many of our public sector customers use AWS to make their data available to a global community of researchers, entrepreneurs, students, and fellow government agencies.
Sharing data on AWS makes it accessible to a large and growing community of researchers, entrepreneurs, and enterprises who use the AWS Cloud.
![Page 3: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/3.jpg)
Traditional data acquisition
“…data must be organized, well-documented, consistently formatted, and error free. Cleaning the data is often the most taxing part of data science, and is frequently 80% of the work.”— Data Driven by DJ Patil and Hilary Mason
Tape Data center Disk Server Client
The big data challengeTraditionally, it has been time-consuming and expensive to acquire, store, and analyze large data sets.
![Page 4: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/4.jpg)
Data acquisition in the cloudOur solution – shared open data on AWSWhen data is staged for analysis in the cloud, anyone can analyze it without needing to download it or store it themselves.
“Ordinarily, hitting ‘copy’ on a 4 gigabyte file is an opportunity to stand up and get a fresh cup of coffee, browse the sports section for a little while, but moving data between servers in an Amazon data center barely affords time to touch your toes a couple times.”
— Paul Ramsey Source: http://s3.cleverelephant.ca.s3.amazonaws.com/2015-ccog.pdf
![Page 5: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/5.jpg)
Landsat on AWS: usageIn the first year: Over 400,000 scenes available
Over 1 billion hits globally
Used for new product development by:
Colin ReillySenior Director GISNYC Department of IT & Telecom
![Page 6: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/6.jpg)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Christa Hasenkopf, Co-Founder of OpenAQ
June 20, 2016
The OpenAQ Community: Fighting Air Inequality Through Open Data
![Page 7: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/7.jpg)
Photo Credit: Lauren Knapp
UB Air Booth
Joe
Ulaanbaatar, Mongolia
![Page 8: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/8.jpg)
Global air inequality
1 out of 8 deaths are due to air pollution
~90% in developing countries(WHO,
2014)
(WHO, 2014)
![Page 9: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/9.jpg)
Global air inequality
![Page 10: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/10.jpg)
Photo Credit: Lauren Knapp
![Page 11: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/11.jpg)
Thousands of “hidden” data sources around the world
![Page 12: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/12.jpg)
Opening up AQ empowers the public
Low-cost sensors/satellites
Public health policy + research
Apps + local activism
Media
Climate + AQ research
Open, transparentdata layer
Real-time air quality data on
public sites
![Page 13: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/13.jpg)
Scaling up in the first year
• Data: PM10, PM2.5, BC, O3, CO, NO2, SO2
• ~13 million data points at 2000+ sites in 20 countries
Utilizing Amazon S3, Amazon EC2, Amazon ECS, Amazon ElastiCache, Amazon RDS, Amazon CloudFront, Amazon
Route 53, AWS Lambda, and SSL via AGU and AWS
![Page 14: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/14.jpg)
OpenAQ’s global grassroots community
• 10 core contributors on 4 continents• ~500,000 API requests/month• Platform accessed by ~100 research orgs
Nov. 2015: Ulaanbaatar workshop
Upcoming in Fall 2016: Jakarta
workshop
![Page 15: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/15.jpg)
Thank you to our partners and sponsors:• American Geophysical Union’s Thriving Earth Exchange providing
AWS credit awards• Development Seed • Echoing Green• Open Science Prize: HHMI, NIH and Welcome Trust• Internews + Earth Journalism Network• Keen.io• And the ENTIRE OpenAQ community!
![Page 16: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/16.jpg)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dr. Adam Pasch, CCMData Strategy and Operations Lead, Weather Science
June 20, 2016
NEXRAD Data on AWSNOAA Big Data Project & The Climate Corporation
![Page 17: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/17.jpg)
Weather is both a feature we provide, and an input to our agronomic models
http://www.climate.com/
The Climate Corporation (TCC) provides decision-making tools for farmers
![Page 18: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/18.jpg)
NOAA Big Data Project
Transform the Department of Commerce data capacity to enhance the value, accessibility and usability of Commerce data for government, business, and the public.
![Page 19: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/19.jpg)
Large-scale analysis was prone to errors
Request & Wait& Download
TCC Amazon S3 Process
Research: Analyses & Evaluations
![Page 20: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/20.jpg)
Data and processing in AWS, reduces errorsProcess
Research: Analyses & Evaluations
AWS NEXRAD S3
![Page 21: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/21.jpg)
Everyone wins
TCC projects are several weeks shorter.TCC evaluations of new methods happen on larger datasets.
We don’t pay Amazon for the S3 bucket to store NEXRAD data.Instead, we pay Amazon for the EC2 instances to process the larger dataset.
NOAA data is used more widely, but without overwhelming NCEI.TCC/AWS found a long-standing problem in NOAA archive, improving data quality.
21
![Page 22: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/22.jpg)
© 2016, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Dr. Bruno Sánchez-Andrade Nuño
Innovation Labs World Bank Group
Open Data Use Case:Monitoring Electrification from Space
nighlights.io
![Page 23: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/23.jpg)
![Page 24: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/24.jpg)
![Page 25: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/25.jpg)
![Page 26: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/26.jpg)
![Page 27: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/27.jpg)
![Page 28: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/28.jpg)
![Page 29: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/29.jpg)
![Page 30: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/30.jpg)
![Page 31: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/31.jpg)
Section Title
![Page 32: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/32.jpg)
Section Title
![Page 33: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/33.jpg)
Section Title
![Page 34: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/34.jpg)
Section Title
![Page 35: #EarthOnAWS: How the Cloud Is Transforming Earth Observation | AWS Public Sector Summit 2016](https://reader033.vdocuments.us/reader033/viewer/2022042907/587aef581a28ab7f378b62bf/html5/thumbnails/35.jpg)
Thank you!