imos profiling ctd dataimos.org.au/fileadmin/user_upload/shared/imos...

IMOS PROFILING CTD DATA IMPLEMENTATION OF AN AUTOMATED

POST-PROCESSING METHOD

Cristian Munoz Mas, Craig Steinberg, Greg Coleman, Felicity McAllister, Paul Rigby, Guillaume Galibert, John Luetchford, Chris Bartlett, Juergen Zier, Mederic Mainson

IMOS QC Summit Profiling CTD Processing Workshop Hobart, 11th and 12th December 2013

CONTENTS

1. OBJECTIVE

2. BACKGROUND

3. PROFILING CTD DATA POST-PROCESSING

4. CURRENT APPLICATIONS

5. CONCLUSIONS

6. NEXT STEPS

7. DISCUSSION TOPICS

1. OBJECTIVE

Automate the CTD post-processing method

RAW DATA *.hex format

PROCESSED DATA *.cnv format

netCDF DATA *.nc format

2. BACKGROUND

NRS-BGC ANMN Standardised Profiling CTD Data Post-Processing Procedures v.1.5

SIMS, CSIRO, SARDI and AIMS. 08 March 2012.

Instrument output CTD Binary File *.hex

(e.g. YON201202080112.hex)

ASCII file *c.cnv

(e.g. YON201202080112c.cnv)

SBE Data Processing-Win32

Converts binary *.hex to *.cnv ASCII

SBE Data Processing-Win32

Post-processing *.cnv ASCII

ASCII file *cf.cnv (e.g. YON201202080112cf.cnv)

ASCII file *cfa.cnv (e.g. YON201202080112cfa.cnv)

ASCII file *cfal.cnv (e.g. YON201202080112cfal.cnv)

ASCII file *cfald.cnv (e.g. YON201202080112cfald.cnv)

ASCII file *cfaldb.cnv (e.g. YON201202080112cfaldb.cnv)

SBE SeatermV2

Download CTD Binary File *.hex

Matlab / imosToolbox

Converts cfaldb.cnv file to NetCDF format

NetCDf File *.nc (e.g. IMOS_ANMN-

NRS_CDEKSTZ_20120208T011200Z_NRSYON_

FV00_005554-NRS-BGC-CTD-SBE19plus-29_C-

20130819T234329Z)

3. PROFILING CTD DATA POST-PROCESSING Scope of the process

RAW data *.hex

Processed data *.cnv FV00

netCDF data *.nc FV00

SBE Post-processing Auto-ctd

IMOS Toolbox

Metadata

Metadata Local deployment

database

Config. file *.con / *.xmlcon

IMOS – eMII Data

center

3. SBE POST-PROCESSING PROCEDURE MANUAL METHOD

SEMI-AUTOMATED METHOD

AUTOMATED METHOD

3. SBE POST-PROCESSING Manual method

3. BATCH RUN SBE POST-PROCESSING Semi-Automated method

3. BATCH RUN SBE POST-PROCESSING Automated method

3. PROFILING CTD DATA POST-PROCESSING Scope of the process

RAW data *.hex

Processed data *.cnv FV00

netCDF data *.nc FV00

SBE Post-processing Auto-ctd

IMOS Toolbox

Metadata

Metadata Local deployment

database

Config. file *.con / *.xmlcon

IMOS – eMII Data

center

3. PROFILING CTD DATA POST-PROCESSING IMOS DATABASE

3. PROFILING CTD DATA POST-PROCESSING Scope of the process

RAW data *.hex

Processed data *.cnv FV00

netCDF data *.nc FV00

SBE Post-processing Auto-ctd

IMOS Toolbox

Metadata

Metadata Local deployment

database

Config. file *.con / *.xmlcon

IMOS – eMII Data

center

3. PROFILING CTD DATA POST-PROCESSING IMOS MATLAB TOOLBOX

4. CURRENT APPLICATIONS National Reference Station Biogeochemical Sampling

4. CURRENT APPLICATIONS Australian National Mooring Network

4. CURRENT APPLICATIONS Australian Institute of Marine Science CTD transects

• Importance of an Automated process:

saving time when processing data.

avoiding mistakes from the user.

• Multiple applications using the same standardized method.

• Multiple users using the same standardized method.

• SBE Post-processing procedure can be understood as a QC process in itself in some steps.

5. CONCLUSIONS

• Keep improving automated process.

Solve errors.

Add new variables for different applications.

6. NEXT STEPS

• Keep improving automated process.

• Possible inclusion as a pre-processing routine in the IMOS toolbox and/or QC routines.

6. NEXT STEPS

IMOS Toolbox

RAW data *.hex

SBE Post-processing Pre-process routine

Local deployment

database

Metadata

Metadata

Config file *.con/*.xmlcon

Display data

Metadata

Converted data

*.cnv FV00

netCDF data

*.nc FV01

netCDF data

*.nc FV00

IMOS – eMII Data center

Export data Interactive display

Quality Control Interactive display

Yes No

RAW format netCDF format

QC routines

IMOS Toolbox

RAW data *.hex

SBE Post-processing Pre-process routine

Local deployment

database

Metadata

Config file *.con/*.xmlcon

Metadata

Display data

Metadata

Converted data

*.cnv FV00

netCDF data *.nc FV01

netCDF data *.nc FV00

IMOS – eMII Data center

Export data Interactive display

Quality Control Interactive display

Yes

No

Mode Profile SBE Post-processing

netCDF format RAW format

QC routines

• Keep improving automated process.

• Possible inclusion as a pre-processing tool in the IMOS toolbox and/or QC routines.

• QC routines applicable to CTD profile data.

6. NEXT STEPS

• Keep improving automated process.

• Possible inclusion as a pre-processing tool in the IMOS toolbox and/or QC routines.

• QC routines applicable to CTD profile data.

• Need to archive and make available raw data sampled & processed.

6. NEXT STEPS

Instrument Output RAW data *.hex

Imos-toolbox

Converted data

*c.cnv

Processed data

*cfaldb.cnv

Processed data

*cfaldb.cnv +QC

eMII – Data Center

Converted data

*FV00.cnv

netCDF data

*FV00.nc

netCDF data

*FV01.nc

Sampled

QC

1. Include auto-ctd process as pre-processing routine within imos-toolbox.

2. Case YES: divide auto-ctd method as pre-processing routine and QC within imos-toolbox.

3. Current imos-QC routines applicable to CTD profile data.

4. Archive and make available raw data sampled & processed in different files.

6. DISCUSSION TOPICS