neon: from scientific strategy to long-term operationoptimise.dcs.aber.ac.uk › wp-content ›...
TRANSCRIPT
National Ecological Observatory NetworkA project sponsored by the National Science Foundation and operated under cooperative agreement by Battelle.
1
NEON: from scientific strategy to long-term operation
Stefan Metzger1, Ankur R. Desai2, David Durden1, Jörg Hartmann3, Jiahong Li4, Hongyan Luo1, Natchaya Pingintha-Durden1, Torsten Sachs5, Andrei Serafimovich5, Cove Sturtevant1, Ke Xu2
[1]: Battelle Ecology, National Ecological Observatory Network Project, Boulder, CO, USA[2]: University of Wisconsin-Madison, Dept. of Atmospheric and Oceanic Sciences, Madison, WI, USA[3]: Alfred Wegener Institute - Helmholtz Centre for Polar and Marine Research, Bremerhaven, Germany[4]: LI-COR Biosciences, Lincoln, NE, USA[5]: GFZ German Research Centre for Geosciences, Potsdam, Germany
• incorporate lessons-learned through collaborations with bottom-up networks like AmeriFlux, ICOS, LTER, TERN…
• NEON's centralized approach lends itself to explore novel systemic solutions
• starting to give back:
• boots on the ground
• software
• data
• educational resources
2
the spirit
3
overview
• why bridging observational scales?
• NEON: designed for scaling
• centralized data operations and monitoring
• scalable scientific computing
• combining ground-based, airborne and space-borne data
• summary and outlook
4
overview
• why bridging observational scales?
• NEON: designed for scaling
• centralized data operations and monitoring
• scalable scientific computing
• combining ground-based, airborne and space-borne data
• summary and outlook
5
How does spatial structure influence ecosystem function and how do we integrate within and between spatial scales to assess function?
“…data still remain too sparse spatially to test mechanisms of change using models…”
“…provide the required data at high spatial and temporal resolution with the necessary continuity…”
6
7
“... scaling down from global and continental measurements and scaling up from stand-level measurements both are critical for an integrative program of C research.”
“Each approach has different spatial and temporal domains ..., and they have the potential to constrain the next level up or down”
“One of the main barriers to rapid improvement of models is the lack of ground data for validation purposes.”
8
overview
• why bridging observational scales?
• NEON: designed for scaling
• centralized data operations and monitoring
• scalable scientific computing
• combining ground-based, airborne and space-borne data
• summary and outlook
9
The National Ecological Observatory Network
10
NEON multi-scale observing system
11
state of the NEON “fairytale”
12
state of the NEON “fairytale”
instrumented sites
instrumented sites
instrumented sites
instrumented sites
instrumented sites
instrumented sites
instrumented sites
instrumented sites
instrumented sites
instrumented sites
instrumented sites
instrumented sites
25
in-situ sampling, proximal and remote sensing
26
in-situ sampling, proximal and remote sensing
27
in-situ sampling, proximal and remote sensing
28
in-situ sampling, proximal and remote sensing
29
in-situ sampling, proximal and remote sensing
30
in-situ sampling, proximal and remote sensing
31
32
the National Ecological Observatory Network
33
0 10 20 30Exp
lain
ed
eco
-
clim
ati
c va
ria
bil
ity
Number of Domains
Insufficient Explanatory Power Unbounded Expense
why 20 domains?
2/3 detection rate
34
data: science implications
Mahecha et al., 2017
detecting extreme events
128 AF sites
39 NEON sitesAmeriFlux
sites across eco-climatic regions
Hargrove and Hoffman, 1999 & 2004
NEON
35
overview
• why bridging observational scales?
• NEON: designed for scaling
• centralized data operations and monitoring
• scalable scientific computing
• combining ground-based, airborne and space-borne data
• summary and outlook
36
data operations architecture
37
science operations management
• problem tracking and resolution along the entire chain
• training
• sensor preventive maintenance
• sensor calibration
• sensor health status monitoring, incident tracking and resolution
• data processing
• continuous data quality monitoring
• data revisioning
38
• problem tracking and resolution along the entire chain
science operations management
39
• problem tracking and resolution along the entire chain
science operations management
40
data products and access
41
data products and access
42
data products and access
43
data products and access
44
data products and access
45
data: interoperability
46
data: interoperability
47
data: interoperability
48
data: interoperability
NEON airborne remote sensing
49
data: interoperability
collocated1. Bartlett Experimental Forest (US-Bar) / BART2. Harvard Forest (US-Ha1) / HOPB3. Konza Prairie (US-Kon) / KONZ-KING-KONA cluster4. Kansas Field Station (US-KSF) / KFS5. Niwot Ridge (US-NR1) / NIWO-COMO6. Santa Rita Creosote (US-SRC) / SRER7. Eight Mile Lake Permafrost (US-EML) / HEAL8. Barrow (US-NGB & US-Bes) / BARR
adjacent9. Poker Flat Research Range (US-Prr) / CARI-BONA
in vicinity (assignable assets)10. Slashpine Austin Cary (US-SP1) / BARC-OSBS-SUGG cluster11. Sylvania (US-Syv) / UNDE- CRAM cluster
NEON airborne remote sensing
50
educational resources
51
educational resources
52
educational resources
53
overview
• why bridging observational scales?
• NEON: designed for scaling
• centralized data operations and monitoring
• scalable scientific computing
• combining ground-based, airborne and space-borne data
• summary and outlook
54
data operations architecture
• eddy4R family of R-packages (raw data → 30 min)
55
software: eddy4R eddy-covariance R-packages
• eddy4R family of R-packages (raw data → 30 min)
• NEON's eddy4R, nneo, metscanner + MPI's REddyProc R-packages: end-to-end, modularly adjustable and extensible workflows in single R-environment
56
software: eddy4R eddy-covariance R-packages
R workflow
tur ulen e
processing
parameters
L1 – L4
HDF5 files
loop around planar-fit/Wavelet period
collect around planar-fit/Wavelet period
ingest remote-sensing data
collect: turbulence data
transform SONIC planar-fit
field-validate IRGA
data-parallel: turbulence data read L0p
transform AMRS
calculate Wavelet
data-parallel: processing core correct frequency response
calculate L1-L4 DPs, Wavelet stats
collect: processing core
model footprint
perform, summarize QA/QC
quantify, summarize uncertainty
lag-correction
• Docker shipping container system for code
• eddy4R-Docker: turn-key, reproducible, extensible and portable data processing + analysis environment
• DevOps community development framework
57
software: community access and extensibility
Computer
58
data: eddy-covariance
processing pipeline characteristics
• near-real-time (1 week → 1 month)
• extensible: e.g. operational data fusion
59
overview
• why bridging observational scales?
• NEON: designed for scaling
• centralized data operations and monitoring
• scalable scientific computing
• combining ground-based, airborne and space-borne data
• summary and outlook
60
scope of the taskTe
mpo
ral s
cale
Spatial scale [km2]
Second
Minute
Hour
DayWeekMonth
Year
Decade
Century
Airborne
Space-borne
10−6 100 106 1012
Ground-based
Airborne
Space-borne
Ground-based
Resolution
Coverage
Model-data fusion
Earth systemmodel
61
environmental response function virtual control volume
Footprint modeling over coordinated observations of drivers and responses
Metzger (2017)
62
environmental response function virtual control volumegrid-projected turbulent heat flux
r CpdT / dt
[W /m3]
2011-08-13 CST
volume-projected heat storage change
Xu et al. (2017)
63
environmental response function virtual control volumestorage flux turbulent flux grids surface-atmosphere
exchange grids
[W/m2]
r CpdT / dt
[W /m3]
aaadfweferggdgfa
Xu et al. (2017)
application examples
• near-real-time capability (1 week → 1 month, e.g. AmeriFlux)
• operational data fusion (e.g., NASA ECOSTRESS)
64
overview
• why bridging observational scales?
• NEON: designed for scaling
• centralized data operations and monitoring
• scalable scientific computing
• combining ground-based, airborne and space-borne data
• summary and outlook
• bridging scales one of the fundamental challenges in ecology
• NEON’s observation hierarchy is designed for scaling (continuity)
• centralized data operations efficiently manage data quality and standardization
• coordinated, interoperable NEON flux, in-situ, proximal and remote sensing data
• public NEON “eddy4R” + GitHub + Docker flux software and usability tools
• combining data across scales with environmental response functions
• joint proposal writing for operational data fusion
65
conclusions and outlook
66
• NEON data products span large spatio-temporal scales, e.g. airborne remote sensing and automated tower measurements
• we aim to bridge these spatio-temporal gaps in NEON data products using a unmanned aerial vehicle remote-sensing platform
airborne remote-sensing observation node (ARGON)
• objectives
develop less expensive and more agile remotely sensed data products.
enabling target-of-opportunity measurement campaigns for extreme ecological events
67
overview
• why bridging observational scales?
• NEON: designed for scaling
• centralized data operations and monitoring
• scalable scientific computing
• combining ground-based, airborne and space-borne data
• summary and outlook