1. Scaling from seeds to ecosystems Computer Science and engineering challenges for NextGen ecosystem monitoring and genomics Tim Brown, PhD Research Fellow, Borevitz Lab ARC Centre for Plant Energy Biology Australian National University 2. In the next 100 years we will face challenges of unprecedented scale and complexity Massive Biodiversity loss (since 1500): ~30% decline in land animals; ~70% of invertebrates (insects, etc.) show >45% declines4 > 90% of the worlds fish stocks are fully exploited or overexploited 6 > 7 billion people on the planet for at least the next century 11 billion people on the planet by 2100 1 We must grow more food by 2100 than all the food produced in human history2 Global climate will warm by 4-6C by 2100 No one under the age of 24 has lived in a year when the earth was cooler than any time in the last 2,000 years 53% of Eucalyptus species will be out of their native ranges 21003 Sea levels will rise by 0.5 1.1m4 By 2100, extreme sea level events that happened every few years now, are likely to occur every few days 1) Gerland, et al, 2014, [DOI:10.1126/science.1257469]; 2) Seeds of Doubt New Yorker, Aug 25, 2014 3) Hughes et al. Global Ecology and Biogeography Letters (1996): 23-29. 4) Climate Change Risks to Australias Coast, 2009 5) Dirzo, Rodolfo, et al. 2014. DOI: 10.1126/science.1251817 6) http://worldoceanreview.com/en/wor-2/fisheries/state-of-fisheries-worldwide/ 2/2 0 3. (re) Terraforming the earth Terraforming: To alter the environment of (a celestial body) in order to make it capable of supporting terrestrial life forms. We are currently un-terraforming the earth at an exceptional rate. To meet the challenges of the coming century we need to think about how to restore and re-engineer the environment to support >7 billion people for the next 100 years in the face of climate change 4. The grand challenge of the 21st century Sustain, manage and create ecosystems that optimize benefit to other living things provide sustainable ecosystem services for humans in the face of over-population, resource scarcity, climate change and other major negative humans impacts We no longer have the luxury of trusting low resolution ecological models 4/2 0 5. The current resolution of field ecology is very limited Low spatial/time resolution data Limited sensors other than weather Sampling is manual; subjective Observations not-interoperable little or no data sharing often proprietary Repeat experiments to verify results are often at different site by different observers Paradigm for sample resolution is Forest or field not Tree or Plant Very little data from the last century of ecology is available for reuse 6. Research focus: How can we use new technology to capture what is going on in the environment, quantify it and provide it to scientists (and the public) in a format that they can use? (1) Lab phenomics Phenotype (plant behaviors) + Genomics (plant genomes) Identifying the genetic basis of plant growth and development (2) Field ecology High resolution monitoring of ecosystems How do we continuously monitor thousands of plants at high resolution and low cost? 7. Why Phenomics matters Maximize carrying capacity Improve crop yields and land use efficiency Maintain and restore ecosystem services Requires advanced environmental monitoring and high- accuracy ecosystem models Maintain and restore natural areas and biodiversity Requires accounting for significant projected climate change impacts Climate ready seed selections (yield stability) Climate ready forests Inform conservation and management decisions for how to spend scarce conservation dollars (i.e. identify most at-risk ecosystems) 8. Genotype x Environment = Phenotype Phenotype (and ecosystem function) emerges as a cross-scale interaction between genotypes and the environment they experience. The degree to which we can measure all three components is the degree to which we can understand ecosystem function 9. Borevitz Lab Approach Measure more precisely in the Field Create more natural Lab conditions Suzanne Marselis enviro-net.org 10. Lab vs field phenotyping Field: Realistic environment but low precision measurements In the field we have real environments but the complexity (and bad lighting!) reduces our ability to measure things with precision youtu.be/gFnXXT1d_7s 11. Lab vs field phenotyping Lab: High precision measurement and control but low realism youtu.be/d3vUwCbpDk0 12. Lab phenotyping Normal lab growth conditions arent very natural Kulheim, Agren, and Jansson 2002 Growth Chamber Real World 13. SpectralPhenoClimatron Dynamic semi-realistic environmental & lighting conditions Plant growth cabinets with 8 & 10-band controllable LEDs Control light spectra and intensity, Temperature and Humidity at 1-min intervals JPG + RAW imaging every 10 minutes Tools for the lab L4A s20: 10-band L4A s10: 8-band LED lights Spectral response of Heliospectra LEDs 14. Climatron Grow plants in simulated regional and seasonal conditions E.g. coastal vs inland | Late Spring or Early Fall Simulate full or short season, climate shifts, drought conditions, etc. Expose cryptic phenotypes Repeat environmental conditions Between studies and collaborators Field sites Real-time monitoring, analysis and data visualization Image analysis pipeline extracts growth data from 120,000 pot images a day Corrected Segmented Original 15. High Throughput Phenotyping for the masses Off the shelf low-cost cameras ( Phenomics 20 years of technical advances have turned genetics into genomics into phenomics and yielded the ability to address fundamental and very complex questions State of the art phenomics is pretty high resolution Precision environmental controls 3D time-lapse models of every plant growing with each pixel in all spectra mapped to the 3D data cube model of each plant Genome sequence for every plant Automated bioinformatics pipeline for trait extraction But ecosystems are more complex than plants in growth chambers We need much higher resolution capacity in the field 18. The real world is way more complex than plants in the lab Dont we need equally complex datasets and models to understand real world ecosystems? This hasnt traditionally been possible The questions we ask are often defined more by what data we can get than by what the best question would be What questions could we ask if we had a 10-year high-resolution dataset of our field sites? 18/20 19. The challenge Measure everything all the time Persistent 3D, time-series multi data-layer ecosystem models tracking every tree 19/ 20 20. 30 years ago was ~1980 (everything was analog) The first space shuttle (1981) 1MB of RAM1 (your phone has 2000x this) 4G is the same bandwidth that MODIS uses to download a full satellite image of the earth every day (MODIS: 6.1Mbps avg; Verizon 4G: 5-12Mbps 2,3) Even 20 years ago: no Web, WiFi, cell phones, Google anything Facebook is barely 10yrs old with 1.32 bn active monthly users (more people than any country but China) The first iPhone was only released in 2007 New technology is changing everything 1.http://www.popsci.com/node/31716 2.http://modis.gsfc.nasa.gov/about/specifications.php 3.https://www.lte.vzw.com/AboutLTE/VerizonWirelessLTENetwork/tabid/6003/Default.aspx 1980: 60 min of music 2010: 9,000 min of music 21. New technology is changing everything 1.8 billion (mostly geolocated) images are uploaded to social media every day1 Consider: 75% of cars may be self-driving by 20402 continuously imaging, laser scanning and 3D modelling their immediate environment: 6.2 billion miles3 of roadside environments in US, imaged in 3D daily! 1.Meeker, 2014 2.http://www.ieee.org/about/news/2012/5september_2_2012.html 3.http://www.fhwa.dot.gov/policyinformation/travel_monitoring/13dectvt/figure1.cfm 21/ 20 22. 40 Gadgets replaced by the smartphone wired.com 23. On demand super-computing in our pockets Cloud computing backed by ubiquitous high speed bandwidth changes everything 24. National Arboretum Phenomic & Environmental Sensor Array ANU Major Equipment Grant, 2014 Collaboration with Cris Brack, Albert Van Dijk (Fenner school) and Borevitz Lab 24/ 20 25. NextGen monitoring Within ten years we will have similar data to lab phenomics Automated time-series (weekly/daily) aerial (UAV) scans measure: RGB; Hyperspectral; Thermal; LiDAR Centimeter resolution 3D model of every tree on field site Gigapixel imaging to track phenology in every tree/plant Automated computer visions analysis for change detection LIDAR scanning DWEL / Zebedee high resolution 3D scans (ground based) Dense point clouds of 3D structure Microclimate sensor networks PAR Temp, Humidity Soil moisture @ multiple depths mm resolution dendrometers Full genome for every tree on site ( 1,000 trees Campbell weather stations (baseline data for verification) All data live online in realtime LIDAR: DWEL / Zebedee UAV overflights (bi-weekly/monthly) Georectified Google Earth/GIS image layers High resolution DEM 3D point cloud of site in time-series Total Cost ~$200K 27/ 20 28. 28/ 20 29. Tools for the field - Golfer, 7km distant Gigapixel Imaging 30. (Single 15MP image) Area: ~7ha Area: ~1m2 The Gigapan and Gigavision systems allow you to capture hundreds or thousands of zoomed-in images in a panorama. Images are then Stitched into a seamless panorama. The super-high resolution of the final panorama lets you monitor huge landscape areas in great detail. Gigapixel Imaging How it works 31. Australian time-series monitoring Canberra, Telstra Tower National Arboretum Gigapixel imaging lets us monitor seasonal change for huge areas 32. 20 gigapixel image of Canberra, Australia from the Black Mountain Telstra Tower Zoom in to the National Arboretum 33. Midsummer Zoom in to the Each forest at the Arboretum 34. Low cost sequencing lets us genotype every individual tree and identify genetic loci that correlate with observed phenotypic differences between trees. We can do this for all trees at the arboretum within view of the camera. Fall Color change shows differing rates of fall senescence in trees Late fall 35. Gigavision Control Software (Chuong) Need to automate stitching Need automated change detection and phenotyping for every tree (Much bigger project) 35/ 20 36. UAVs for monitoring $4-5K airframe (aeronavics or similar) w/ 20MP digital camera (~1kg payload). $1,800 AUD Pix4D Software (non-commercial license) 3D models of field site (cm resolutions are possible) Orthorectified image and map layers LAS / point cloud data Automated pipeline: Flight -> tree data Tree Height; Volume, foliage density GPS location DEM of site 36/ 20 37. Other data from the site: 37/ 20 NextGen Ground based LiDAR: Zebedee 38. Merge aerial and understory scans 38/ 20 39. DWEL (CSIRO NextGen laser Multiband Lidar with full point returns ~30 millions point in a 50m2 area (vs 5-10 pts/m for aerial) 39/ 20 40. So what do we do with all this data? The challenge is no longer to gather the data, the challenge is how we do science with the data once we have it A sample is no longer a data point Example: Gigavision data From Sample: Camera hardware: 1200 images (per hour) Automate stitching into panoramas (5,000 tiles images / pano /hr) Need to align time-series images to each other and to the real world (despite hardware failures, camera upgrades) How to visualize a time-series of 22 million images/year? Computer vision analysis Automated feature detection and phenophase detections To Data: Phenophase transition and growth data from 1,000 trees And then how do you even analyze that?! 40/ 20 41. Virtual 3D Arboretum Project Goal: Explore new methods for visualizing time-series environmental data Historic and real-time data layers integrated into persistent 3D model of the national arboretum in the Unreal gaming engine Collaboration with CS/TechLauncher students Stuart Ramsden, ANU VISlab 42. Classified point cloud data to 3D model Image: Stuart Ramsden (w/Houdini); Data: Suzanne Marselis 43. Unreal Arboretum project 44. Unreal Arboretum project Time slider for watching data change over time Temperature shown as dynamically colored particle cloud, positioning deformed by prevailing wind data Tree color can be set to tree height and growth data for instrumented trees All trees on site adjust to measured height, volume and foliage density Ground color scales to match moisture User can fly to specific locations with higher density sampling to see additional layers (gigapixel timelapse; DWEL Lidar sites, etc) 45. Conclusions What we do in the next 50-100 years will have a critical impact on the earth and its inhabitants for centuries to come Scaling environmental monitoring to the prevision required for meeting the challenges of the next century is in large part an engineering and computer science problem. Better optimizing transfer of engineering and CS research from lab to real world applications can have a major impact on how well we address these challenges. 46. Thanks and Contacts Justin Borevitz Lab Leader Lab web page: http://borevitzlab.anu.edu.au Funding: Arboretum ANU Major Equipment Grant ARC: Center of Excellence in Planet Energy Biology | Linkage 20014 Arboretum http://bit.ly/PESA2014 Cris Brack, Albert VanDijk, Justin Borevitz (PESA Project PIs) UAV data: Darrell Burkey, ProUAV 3D site modelling: Pix4D.com / Zac Hatfield Dodds / ANUVR team Dendrometers & site infrastructure Darius Culvenor: Environmental Sensing Systems Mesh sensors: EnviroStatus, Alberta, CA TraitCapture: Chuong Nguyen; Joel Granados; Kevin Murray; Gareth Dunstone; Jiri Fajkus Pip Wilson; Keng Rugrat; Borevitz Lab Contact me: tim.brown@anu.edu.au / http://bit.ly/Tim_ANU 46/ 20