virtual geophysics laboratory (vgl) vgl v1.1 launch ryan fraser, terry rankine, joshua vote, lesley...

Virtual Geophysics Laboratory (VGL)

VGL v1.1 LaunchRyan Fraser, Terry Rankine, Joshua Vote, Lesley Wyborn, Ben Evans, Robert Woodcock

February 2013

CSIRO | MINERALS DOWN UNDER FLAGSHIP

Gather data, process it, publish resultsSimple, isn’t it?

bedrock

surficial

mineral

geochemical

geochronologic

hyrdrogeological

Geo-information

geophysical

knowledgedata

Virtual Geophysics Laboratory v1.1 | Ryan Fraser2 |

Hardcopy of dataHardcopy of data

SSH ClientSSH Client

Maths PackageMaths PackageProprietary PackageProprietary Package

Let’s map it out for Geophysics …

Transform to a regular grid


Crop data to area of interest


Reformat data for processingReformat data for processing

Upload data to HPC

resource

Upload data to HPC

resource

Configure job and start

processing


processing

Download results

Download results

Get handed field data

Get handed field data Visualise dataVisualise data


The Past• Compile raw data using proprietary FORTRAN

• Also use other proprietary software • Transform to a regular grid using more software

• MATLAB, Intrepid, ER Mapper, ESRI ArcGIS, QGIS, etc• Crop data spatially

• eg: everything in Victoria• Transform data into a file format that can be read by proprietary

scientific code• This is usually done with some handwritten python or c• There is no version control, code is often rewritten / redone

• Upload data to HPC• Manually enter input parameters/start job• Where did my results go, how do I visualise?


There seems to be a problem…

• Reproducibility – there is none• What was the input of your model?• What transformations occurred?

• It’s a manual process• Time consuming• Error prone

• Expensive • Licensing costs• Effort/time


Scientific workflow – Virtual Geophysics Laboratory (VGL)• Automated - Simplify access and integration• Reproducible process• Open access (less expense)• Allow scientists to focus on the science and not the tech• Exploiting the Cloud and HPC• Operational or education tool

• Automates and massively expands geophysics computational capacity via the Cloud


V(what)GL

• VEGL – Virtual Exploration Geophysics Laboratory• One primary science collaboration• One primary workflow• One collection of geophysical data sets

• VGL - Virtual Geophysics Laboratory• NeCTAR funded activity• Collaboration with multiple partners (CSIRO, NCI, GA, UQ, Monash, ANU)• Supporting multiple workflows• New data types and collections• New Use – Not just exploration.

DoneDone


Hardcopy of dataHardcopy of data

SSH ClientSSH Client

Maths PackageMaths PackageProprietary PackageProprietary Package

Let’s map it out…





Reformat data for processingReformat data for processing

Upload data to HPC resource

Upload data to HPC resource


processing


processing

Download results

Download results

Get handed field data

Get handed field data Visualise dataVisualise data


…to thisVirtual Geophysics LaboratoryVirtual Geophysics Laboratory

Build “science” from existing

libraries

Build “science” from existing

libraries

Run jobRun jobCollect and

publish resultsCollect and

publish results

Discover raw data

Discover raw data

Select spatial bounds

Select spatial bounds



Introducing The Virtual Geophysics Laboratory

Powered by the Spatial Information Services StackCommon Platform

MarineEnvironment, Water

Groundwater GeologyGeophysics


VGL: Opportunities

• Free for research use• Supporting multiple workflows• Model Registry (3D) – researcher contributed models• New Scientific Codes – Underworld, eScript, UBC, Airborne EM inversion

codes + others (relatively “easy” to add)• Data sets from GA: National Airborne Geophysical DB including

– Gravity, Radiometric, AEM, Magnetics– Data sets from State Surveys

• Broad application and generic implementation• Scalable Infrastructure (Cloud)


What’s left?

• More data, more codes, more workflows!• BYO Resource (cloud allocation)• Confidential Data• Pawsey Centre, NCI, commercial clouds

14 | Virtual Geophysics Laboratory v1.1 | Ryan Fraser

Thank you

Ryan FraserResearch Manager

t +61 8 6436 8760e [email protected] siss.auscope.org

VGL is Power by:

VGL acknowledges funding from the NeCTAR project http://www.nectar.org.au NeCTAR is an Australian Government project conducted as part of the Super Science initiative and financed by the Education Investment Fund.

VGL is a collaboration between:

VGL is supported by: University of Queensland, Monash University and the Australian National University

VGL Launch Event is sponsored by:

Virtual Geophysics Laboratory | Ryan Fraser / Josh Vote16 |

Introducing The Virtual Geophysics Laboratory

No Installation Necessary!


Don’t Compute on your Laptop


Flexibility

What I want to do ≠


Where I want to do itHow I want to do it ≠

Provenance


Don’t be this guy...

To Quickly Recap

• VGL is a different kind of workflow engine...

• No installation necessary• Doesn’t process on your machine• Very flexible• Records results and the process


Time for a live demo...


VGL is Power by:





http://vgl.auscope.org

Data discovery


Layers discovered via remote registries

Layers consist of numerous remote data services

Data discovery


Some data services support subsetting

Some data services support reformattinge.g. CSV, NetCDF, GeoTIFF

Data discovery


Some data is only registered with flat files

Data processing


A variety of different scientific codes are already available in the form of “Toolboxes”

Data processing


Further input files can be uploaded.

Input files are passed directly into the cloud

Data processing


The steps so far have been building an environment to run a processing script

...or build from existing templates

Either write your own...

Managing results - provenance

Presentation title | Presenter name30 |

All of a job’s outputs are also accessible Each job has a lifecycle that

can be managed

Successful jobs can have their entire process captured in a ISO 19115 ‘provenance record’

VGL is Power by:





http://vgl.auscope.org

virtual geophysics laboratory (vgl) vgl v1.1 launch ryan fraser, terry rankine, joshua vote, lesley...

Documents

processing upload data

handed field data

redone upload data

regular grid crop data

launch ryan fraser

past compile raw data

flagship slide

processing download