the new iagos data portal · •new software (mongodb) •better performances •better for...

54
IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th The new IAGOS data portal Damien Boulanger CNRS – Observatoire Midi-Pyrénées IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

Upload: others

Post on 01-Aug-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

The new IAGOS data portal

Damien Boulanger

CNRS – Observatoire Midi-Pyrénées

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

Page 2: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

IAGOS data workflow

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

Page 3: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

New database

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

• New software (MongoDB)• Better performances

• Better for heterogeneous data management

• Simplified database management (backup, etc.)

• Data processing• A lot of processes : validation, formatting, merging, added-value products

• Use of the tool Apache Camel (rule-based routing and mediation engine)

• Better management of the data flow

• Content• IAGOS-MOZAIC and IAGOS-CORE data

• IAGOS-CARIBIC (Aerosols and cloud data on going)

• BCP data on going

• Flags and uncertainties (on going for CARIBIC)

• Added-value products: PV, tropopause height, CO representation error, etc.

Page 4: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

Live demo

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

http://iagos.sedoo.fr

Page 5: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

Future improvements

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

• Data download• Save user requests

• Generation of a single file containing only the observations matching the request

• Download any levels of data for the IAGOS team

• Web service for scripts

• Data plotting• Plot two parameters at a time for comparison

• Tools for the PIs• Manage instruments

• Manage comparisons

• Data• New parameters : Boundary layer height, potential temperature, etc.

• Metadata• Available through AERIS portal, GAW and ENVRI+ metadata portals

• Implementation of DOI (common work with AERIS)

Page 6: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

Released next week.

http://www.iagos-data.fr

Any comments are welcome!

Page 7: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

Screenshots

Page 8: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

Home

Page 9: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 10: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 11: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

Registration

Page 12: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 13: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 14: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 15: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

Data status

Page 16: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 17: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 18: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 19: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 20: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 21: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 22: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 23: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

Metadata

Page 24: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 25: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 26: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 27: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 28: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 29: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 30: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 31: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 32: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 33: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 34: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 35: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 36: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 37: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

Search and download

Page 38: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 39: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 40: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 41: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 42: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 43: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 44: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 45: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 46: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data

IAGOS Scientific Symposium on Atmospheric Composition, Manchester - October 19th

Administration

Page 47: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 48: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 49: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 50: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 51: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 52: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 53: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data
Page 54: The new IAGOS data portal · •New software (MongoDB) •Better performances •Better for heterogeneous data management •Simplified database management (backup, etc.) • Data