climate data lake: empowering citizen scientists in acadia national park
TRANSCRIPT
1© Copyright 2013 Pivotal. All rights reserved. 1© Copyright 2013 Pivotal. All rights reserved.
Climate Data Lake: Empowering Citizen Scientists in Acadia National Park
Srivatsan RamanujamPrincipal Data Scientist, Pivotal
@being_bayesian
John CardenteDistinguished Engineer/Lead Data Scientist, EMC
@johncardente
19-Feb-2015
2© Copyright 2013 Pivotal. All rights reserved. 2© Copyright 2013 Pivotal. All rights reserved.
Climate Change is happening
Source: NASA
3© Copyright 2013 Pivotal. All rights reserved.Source: National Geographic
At the Ice Caps…
From Chasing Ice
by James Balog
4© Copyright 2013 Pivotal. All rights reserved.
And in the
Oceans…
5© Copyright 2013 Pivotal. All rights reserved. 5© Copyright 2013 Pivotal. All rights reserved.
Weather is Nature’s clock
6© Copyright 2013 Pivotal. All rights reserved. 6© Copyright 2013 Pivotal. All rights reserved.
And the clock is shifting
Source: EPA
7© Copyright 2013 Pivotal. All rights reserved. 7© Copyright 2013 Pivotal. All rights reserved.
English Oak Winter Moth Pied Flycatcher
Both et al. 2006 Nature
EARLIER SAME TIME
EACH YEAR
Disrupting species interactions
8© Copyright 2013 Pivotal. All rights reserved. 8© Copyright 2013 Pivotal. All rights reserved.
…the study of the timing of recurring plant and animal life-
cycle stages, or phenophases, and their relationship to
environmental conditions.
Phenology
Source: USA National Phenology Network
9© Copyright 2013 Pivotal. All rights reserved. 9© Copyright 2013 Pivotal. All rights reserved.
Acadia National Park
Source: Wikipedia
A key phenology
observation point
10© Copyright 2013 Pivotal. All rights reserved. 10© Copyright 2013 Pivotal. All rights reserved.
Citizen Science, the “Hadoop” of Phenology
11© Copyright 2013 Pivotal. All rights reserved.
Challenges impeding scientific progress
A multitude of disparate data sources
Different approaches to data conditioning
Different analysis platforms and scripts
Inconsistent data treatment distracts the
scientific debate from the findings!
12© Copyright 2013 Pivotal. All rights reserved.
Challenges to encouraging citizen science
Different infrastructures for data collection
Unable to see how contributions help science
Inability to analyze the data and collaborate with scientists
Missing a feedback loop to encourage and
engage citizen scientists
13© Copyright 2013 Pivotal. All rights reserved.
EMC Federation teams up with Earthwatch
14© Copyright 2013 Pivotal. All rights reserved.
Objectives
• Authoritative, curated data repository
• Shared analysis platform and software libraries
• Web portals to
– Show citizen scientists how contributions benefit science
– Present data visualization and exploration tools
– Provide access to data sets for volunteer analysis
– Offer common data collection APIs and interfaces
– Host forums to interact with leading scientists
15© Copyright 2013 Pivotal. All rights reserved.
Climate Data Lake
PL/X
Hadoop++
Data
Data Lake Foundation
Explore
CloudFoundry
Collect
Isilon ECS
16© Copyright 2013 Pivotal. All rights reserved.
The kick-off expedition at Acadia
17© Copyright 2013 Pivotal. All rights reserved.
Post kick-off activities
• Established partnerships with data providers
• Stood up initial Data Lake platform
• Active collaboration across all participants on data
conditioning, analysis, feature selection, and visualization
• Launching an initial web presence in March
18© Copyright 2013 Pivotal. All rights reserved.
What’s next
• Pursuing the “full vision” using Federation Technologies
• Talking with other potential corporate sponsors
• Exploring ways to expand support to other climate and
academic projects
19© Copyright 2013 Pivotal. All rights reserved. 19© Copyright 2013 Pivotal. All rights reserved.
20© Copyright 2013 Pivotal. All rights reserved.
Learn More
• White House Press Release:
Harnessing Climate Data to Boost
Ecosystem & Water Resilience
• Pivotal Blog: Pivotal Data Science
Looks at Climate Change @ Acadia
NP
• EMC Reflections Blog: Big Data vs.
Climate Change
BlogsEMC TV video
21© Copyright 2013 Pivotal. All rights reserved.
The Team
@alyssacaddle1
@stevendozen
@johncardente
@lilewite
@being_bayesian
@maryanngarwood
@emctv
@TheMatthewNolan
@johnnysuire
@sriauggie31
@earthwatch_org
@schoodicinst
@hwschoodic
+ many more (not on twitter ) !