ANMN IMOS-ToolboxUpdate on QC tests

Guillaume Galibert, 30th of August 2012

PP QC

Context

Convert data files from ANMN sensors to IMOS NetCDF QC’d files

Data file

Local deployment

database

Data fileData files

IMOS - eMIIdata center

IMOS NetCDF

file

Data fileData file

Data files

Basic QC policy-Qualitative QC.

-Every data point is checked procedure byprocedure.

-For each procedure, only a higher flagcan/must override the current flag of a point.

-As an overall result, only the number of pointswhich failed is recorded for each test.

-For tests involving neighbours, only pointswith flags lower or equal to 3 are considered.

IOC quality flag scale

Procedures for CTD / WQM on mooringsQC procedure Variables checked

(* can be tuned by user)Parameterized

Pass flag

Fail flag

Impossible date TIME Yes Good Bad potentially correctable

Impossible location LATITUDE, LONGITUDE Yes, and per site

Good Bad potentially correctable

In/out water All No Non QC’d

Bad potentially correctable

Global range TEMP, PSAL, PRES/DEPTH, DO, CHL-A * Yes Good Bad

Regional range TEMP, PSAL, DO * Yes, and per site

Good Bad

Impossible depth PRES/DEPTH Yes Good Bad

Spike TEMP, PSAL, PRES/DEPTH, DO, CHL-A * Yes Good Bad

Gradient / Rate of change

TEMP, PSAL, PRES/DEPTH, DO, CHL-A * Yes Good Bad potentially correctable

Stationarity TEMP, PSAL, PRES/DEPTH, DO, CHL-A * No Good Bad

Climatology TEMP, PSAL, DO (when possible?) Yes Good Bad

1 - Impossible dateThe recorded time in a file takes place after a minimum valid date and after a maximum valid date. TIME is flagged, not the PARAMETERS. Default values are :

01 Jan. 2007 <= TIME <= today

2 - Impossible LocationActual position of a mooring is in a box or circle centred on the site nominal location :

Site Nominal longitude

Nominal latitude

Box +/- in Longitude

Box +/- in Latitude

Circle radius (km)

NRSMAI 148.2333 -42.59667 0.2 0.25 2.5

NRSNSI 153.58 -27.389 0.2 0.25 2.5

NRSKAI 136.4473 -35.83217 0.2 0.25 2.5

NRSESP 121.85 -33.93333 0.2 0.25 2.5

NRSROT 115.4167 -32 0.2 0.25 2.5

NRSNIN 113.947 -21.86652 0.2 0.25 2.5

NRSDAR 130.7681 -12.4 0.2 0.25 2.5

NRSYON 147.6184 -19.30847 0.2 0.25 2.5

NRSPHB 151.25 -34.08333 0.2 0.25 2.5

... ... ... ... ... ...

3 – In/out water testAnything outside a specified time period fails the test.

In water date/time is taken from the ddb by order of preference :1. Time first in position2. Time first wet3. Time switched on

Out water date/time is taken from the ddb by order of preference :1. Time last in position2. Time on deck3. Time switched off

4 – Global rangeEach listed parameter has its values compared to global upper and lower valid values :

IMOS parameter

CF title Unit Valid min

Valid max

Source

TEMP Sea water temperature Celsius -2.5 40 ARGO Quality control manual v2.7 (Jan. 2012)

PSAL Sea water salinity PSU 2.0 41 ARGO Quality control manual v2.7 (Jan. 2012)

PRES Sea water pressure dbar 5 12 010 ARGO Quality control manual v2.7 (Jan. 2012)

PRES_REL Sea water relative pressure

dbar -5 12 000 ARGO Quality control manual v2.7 (Jan. 2012)

DEPTH Depth m -5 12 000 ARGO Quality control manual v2.7 (Jan. 2012)

DOX1 Mole concentration of dissolved molecular oxygen in sea water

umol/l 0 900 000 MyOcean - Real Time Quality Control of biogeochemical measurements v1.0 (Jan. 2011)

DOX2 Moles of oxygen per unit mass in sea water

umol/kg 0 880 000 MyOcean - Real Time Quality Control of biogeochemical measurements v1.0 (Jan. 2011)

CPHL, CHLU, CHLF

Mass concentration of chlorophyll in sea water

mg/m3 0 100 MyOcean - Real Time Quality Control of biogeochemical measurements v1.0 (Jan. 2011)

... ... ... ... ...

5 – Regional rangeEach listed Site and parameter has its values compared to regional upper and lower valid values :

Site IMOS parameter

CF title Unit Valid min

Valid max

Source

NRSPHB TEMP Sea water temperature Celsius 10 28 Port Hacking historical water samples.

NRSPHB PSAL Sea water salinity PSU 31 37 Port Hacking historical water samples.

NRSPHB DOX2 Moles of oxygen per unit mass in sea water

umol/kg 50 400 Port Hacking historical water samples.

NRSMAI TEMP Sea water temperature Celsius 8 21 Maria Island historical water samples.

NRSMAI PSAL Sea water salinity PSU 33 37 Maria Island historical water samples.

NRSMAI DOX2 Moles of oxygen per unit mass in sea water

umol/kg 100 400 Maria Island historical water samples.

... ... ... ... ... ... ...

6 – Impossible depthCompares the actual depth of the instruments to its nominal depth.

Acceptable ranges are derived from this formula :

upperBound = instNomZ + upperCoeff * (siteNomZ / instNomZ)

lowerBound = instNomZ - lowerCoeff * (siteNomZ / instNomZ)

Default value for upperCoeff = 3.Default value for lowerCoeff = 7.

7 – Spike testEach listed parameter has its own threshold value to be used in the ARGO spike test :

|Vn-(Vn+1 + Vn-1)/2| - |(Vn+1 - Vn-1)/2| <= threshold

PABIM white book v1.3 (Feb. 2010) threshold :

|median(Vn-2,Vn-1,Vn,Vn+1,Vn+2)| + |stDev(Vn-2,Vn-1,Vn,Vn+1,Vn+2)|

IMOS parameter

CF title Unit Threshold

Source

TEMP Sea water temperature Celsius 6 ARGO Quality control manual v2.7 (Jan. 2012)

PSAL Sea water salinity PSU 0.9 ARGO Quality control manual v2.7 (Jan. 2012)

PRES, PRES_REL, DEPTH

Sea water pressure, Sea water relative pressure, Depth

dbar, m 3

DOX1, DOX2

Mole concentration of dissolved molecular oxygen in sea water, Moles of oxygen per unit mass in sea water

umol/l, umol/kg

PABIM MyOcean - Real Time Quality Control of biogeochemical measurements v1.0 (Jan. 2011)

CPHL, CHLU, CHLF

Mass concentration of chlorophyll in sea water

mg/m3 PABIM MyOcean - Real Time Quality Control of biogeochemical measurements v1.0 (Jan. 2011)

... ... ... ...

8 – Gradient Vs Rate of changeGradientEach listed parameter has its ownthreshold value to be used in the ARGOgradient test :

|Vn-(Vn+1 + Vn-1)/2| <= threshold

IMOS parameter

CF title Unit Gradientthreshold

Rate of changethreshold

TEMP Sea water temperature

Celsius 9 2*stDev(first month)

PSAL Sea water salinity PSU 1.5 2*stDev(first month)

PRES, PRES_REL, DEPTH

Sea water pressure, Sea water relative pressure, Depth

dbar, m

3 2*stDev(first month)

DOX1, DOX2

Mole concentration of dissolved molecular oxygen in sea water, Moles of oxygen per unit mass in sea water

umol/l, umol/kg

90 2*stDev(first month)

CPHL, CHLU, CHLF

Mass concentration of chlorophyll in sea water

mg/m3 3 2*stDev(first month)

... ... ... ... ...

Rate of changeEach listed parameter has its ownthreshold value to be used in theMyOcean rate of change test :

|Vn – Vn-1| + |Vn – Vn+1| <= 2*threshold

-Points with more than 1h apartare not considered asneighbours and current point isflagged as “Non QC’d”.

-Tests only flag current point Vn.

8 – Gradient Vs Rate of changeGradient

|Vn-(Vn+1 + Vn-1)/2| <= constant

Rate of change

|Vn – Vn-1| + |Vn – Vn+1| <=2* 2*stDev(first month)

9 - StationarityThis IOC test finds and flags any consecutive equal values of the below listedparameters when :

number of consecutive points of equal values > 24*(60/delta_t)

with delta_t the sampling interval in minutes.

IMOS parameter CF title Unit

TEMP Sea water temperature Celsius

PSAL Sea water salinity PSU

PRES Sea water pressure dbar

PRES_REL Sea water relative pressure dbar

DEPTH Depth m

DOX1 Mole concentration of dissolved molecular oxygen in sea water umol/l

DOX2 Moles of oxygen per unit mass in sea water umol/kg

CPHL, CHLU, CHLF Mass concentration of chlorophyll in sea water mg/m3

... ... ...

10 – Climatology testIf a climatology exist for the studied site then the deployment data set will be checkedagainst it using pre-defined minimum and maximum valid values.

The default range is defined as follow :

validMin = min(mean - 6*stdDev, mini-2*stdDev)

validMax = max(mean + 6*stdDev, maxi+2*stdDev)

Procedures for ADCP on mooringsQC procedure Variables checked

(* can be tuned by user)Parameterized

Pass flag

Fail flag

Impossible date TIME Yes Good Bad potentially correctable

Impossible location LATITUDE, LONGITUDE Yes, and per site

Good Bad potentially correctable

In/out water All No Non QC’d

Bad potentially correctable

Global range TEMP, PSAL, PRES/DEPTH, UCUR, VCUR, WCUR ? *

Yes Good Bad

Regional range TEMP, PSAL * Yes, and per site

Good Bad

Impossible depth PRES/DEPTH Yes Good Bad

Spike TEMP, PSAL, PRES/DEPTH * Yes Good Bad

Rate of change TEMP, PSAL, PRES/DEPTH * Yes Good Bad potentially correctable

Stationarity TEMP, PSAL, PRES/DEPTH * No Good Bad

Climatology TEMP, PSAL Yes Good Bad

Teledyne UCUR, VCUR, WCUR Yes Good Bad

Side lobe contamination

UCUR, VCUR, WCUR No Good Bad

11 – Teledyne-For Workhorse ADCPs manufactured by Teledyne RD Instruments.

12 – Side lobe contaminationFlags all the bins above the last one which measurements are contaminated by the sidelobes of the antenna diagram :

contaminated depth = depth*(1-cos(beam angle))

Future work / ideas-Choose Gradient or Rate of Change test with best threshold values.-Define and describe final strategy for climatology building.-Define best valid ranges for climatology test.-Introduce overall QC flag per deployment like ARGO.-Introduce visual QC to validate automated QC.

-Introduce quantitative QC (Daniel Smith and Monique Breslin presentations).

Thank you+ special thanks to QC working groupAnd Esmee Van Wijk (ARGO DM QC)