the noaa big data project: public-private partnerships at scale

AWS Government, Education, and Nonprofit Symposium Washington, DC I June 25-26, 2015


The NOAA Big Data Project: Public-Private Partnerships at Scale

Ariel Gold, AWSMaia Hansen, NOAA

David McClure, NOAA

©2015, Amazon Web Services, Inc. or its affiliates. All rights reserved.


Agenda1. Overview of the NOAA Big Data Project and the AWS Data Alliance2. Fireside Chat with NOAA’s Maia Hansen and David McClure3. Audience Q&A

Goals1. Increase understanding of the NOAA Big Data Project and AWS approach2. Identify lessons learned from the early days of the initiative3. Identify ways to engage with the AWS Data Alliance now and in the future

Audience• Anyone who is interested in NOAA data. Individuals and organizations in the public and private sectors. Direct

end users as well as value added service providers. • We assume that you have little or no technical expertise and knowledge of the NOAA Big Data Project and AWS.


The Value of NOAA Data


NOAA Big Data ProjectNOAA’s Goal• Identify sustainable ways to increase output of open NOAA data from about 2TB/day to 20TB/day to drive new

economic activity.

NOAA’s Approach• Tap into private sector incentives to increase availability of NOAA data. • Push data to commercial cloud platforms using open standards and with no net new cost to the government.• Assess the feasibility of this model through Cooperative Research and Development Agreements (CRADAs) with

IaaS providers.• Encourage IaaS providers to form data alliances with end users and value added service providers.

Why CRADA• Under the CRADA, participating IaaS providers (and their collaborators) can work closely with NOAA, access

much more data and information, and engage in collaborative experimentation.• This is necessary to assess potential data sets, identify opportunities for the development of new data

extraction interfaces and value added services, and ultimately, determine whether a viable and sustainable business case exists.


NOAA Big Data Project – Announced April 21, 2015

https://data-alliance.noaa.gov/

http://www.commerce.gov/news/press-releases/2015/04/us-secretary-commerce-penny-pritzker-announces-new-collaboration-unleash







NOAA Big Data Project – Announced April 21, 2015

https://aws.amazon.com/noaa-big-data/

http://https://aws.amazon.com/blogs/aws/announcing-the-noaa-big-data-project/


http://https/aws.amazon.com/blogs/aws/announcing-the-noaa-big-data-project/



Open Data on AWS

Open data is data that can be used by anyone for any purpose for free.

Many of our customers, such as Esri, the Weather Company, and Planet OS, rely on quality open data as much as they rely on our computing, storage, and other web services.

Driving Innovation with Open Data session is today at 3:30 PM!


Open Data on AWS

Making data open on AWS enables more innovation by making data available for rapid access to our flexible and low-cost computing resources.

AmazonEC2

AmazonEMR

AmazonRedshift

AmazonDynamoDB

AWSLambda


Open Data on AWS

Making data open on AWS enables more innovation by making data available for rapid access to our flexible and low-cost computing resources and ecosystem.

AmazonEC2

AmazonEMR

AmazonRedshift

AmazonDynamoDB

AWSLambda


AWS Public Data SetsTo enable more innovation, AWS hosts a selection of data sets that anyone can access for free. Data in our Public Data Sets program is available for rapid access to our flexible and low-cost computing resources.

Earth ScienceOpen NASA Earth Exchange

(OpenNEX)

Life Sciences1000 Genomes Project

Internet ScienceCommon Crawl Corpus


Landsat on AWS

• The Landsat program is a joint effort of the U.S. Geological Survey and NASA. It is the longest running program to gather Earth imagery from space and is considered the gold standard for natural resources satellite imagery.

• It has traditionally been time consuming and expensive to acquire, store, and analyze Landsat data. Landsat is big open data.

https://aws.amazon.com/public-data-sets/landsat/




Landsat on AWS

• We have committed to making up to 1 petabyte of Landsat images readily available as objects on Amazon S3.

• This allows the data to be accessed programmatically via a RESTful interface and quickly deployed to any of our products for analysis and processing.

• Now anyone can analyze Landsat data at web scale with no significant up-front investment of time or capital expense.

• Development Seed, Esri, MathWorks, Mapbox, and Planet Labs jumpstarted the ecosystem of value-added services around the data.

• Less than 48 hours after release, the data was used to produce a POC at a mining industry hackathon in Australia. https://aws.amazon.com/public-data-sets/landsat/




AWS Data Alliance – Announced April 21, 2015


http://https://aws.amazon.com/blogs/aws/announcing-the-noaa-big-data-project/





AWS Data Alliance – Roles

End Users• Help prioritize which data we investigate • Help define format requirements • Serve as beta users for the data• Participate in launch activities • Use the data

NOAA

Public data

Value-added Service Providers• Create paid or free value-added

services using the data• Other “end user” activities

Data Extraction Collaborators• Design data extraction

interfaces for data• Manage public data on AWS• Other “end user” activities

Anyone can become a member of the AWS Data Alliance by signing up to receive project updates:http://aws.amazon.com/noaa-big-data/

http://aws.amazon.com/noaa-big-data/




AWS Data Alliance – Phased Approach

Sample survey language“We are assessing potential end user interest in gaining programmatic access to the data listed below. Are you interested in any of these data sets? If yes, can we set up a call to discuss with relevant members of your team? We’d like to understand potential use cases and requirements for making them widely accessible.”

Upfront Research Period • Scope out potential Proof of Concept (POC) data projects• Call to action: Respond to surveys, tell us what data you want

Proof of Concept• Launch data publicly and start to build out ecosystem• Example: https://aws.amazon.com/public-data-sets/landsat/

Scale• Launch more data publicly and build out ecosystem



AWS Data Alliance – FAQsQ: How will you select the data that is made available?A: We will prioritize data based on end user input.

Q: How can I request data to be made available?A: Tell us what data you want via http://aws.amazon.com/noaa-big-data/.

Q: When will you make data available? How much data will you make available? What data…?A: We are taking a phased, iterative approach to this research and moving quickly to scope out potential data projects. Sign up to receive updates at http://aws.amazon.com/noaa-big-data/.

Q: How does AWS make money through this model?A: As with our public data sets, we will make data available at no cost to end users. When customers choose to use our services to process the data, they will pay for that usage.

Q: How do value added service providers make money through this model?A: NOAA data is complex, and many end users will choose to pay for services that help them use the data, or that incorporate the data into consumer products.





Fireside Chat


Audience Q&A


Thank You.This presentation will be loaded to SlideShare the week following the Symposium.

http://www.slideshare.net/AmazonWebServices


the noaa big data project: public-private partnerships at scale

Technology

aws open data

value of noaa data

aws data alliance

availability of noaa

data http

data alliances

push data

aws government