development of quality monitoring system on satellite sea surface salinity products yongsheng zhang...
TRANSCRIPT
![Page 1: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/1.jpg)
Development of Quality Monitoring System on satellite Sea Surface
Salinity Products
Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS
![Page 2: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/2.jpg)
• In the past four years, NODC has developed a quality monitoring system known as the data Rich Inventory (RI) for Jason-2 level-2 products.
• The concept of RI is originally developed by NGDC/NESDIS for data management in CLASS. The basic idea is to extract existing metadata and calculate the QA statistics in a granule, store them in a database, and make them available as part of a data discovery system.
• NODC’s Jason-2 data quality monitoring system provides:
1) real-time original data process and visualization, statistical value calculation and QA information packing, and conversion of the outputs in CF compliant NetCDF format.
2) Provided data achiever and user the data quality information in numerical presentation via various data access tools (http, ftp and OPeNDAP) and graphical one via web interface.
Development of Quality Monitoring System at NODC
http://www.nodc.noaa.gov/SatelliteData/Jason2/qa.html
![Page 3: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/3.jpg)
1. To develop automation processing and visualizing tools for near real-time level-2 SMOS data in swath, daily, 3-day and monthly mean time frames;
2. To develop tools for calculation of QA statistical values in a SMOS level-2 swath data file and visualization (valid observation number, mean, standard deviation, minimum & maximum) ;
3. To perform and develop tools and methods for histogram analyses for comparing the monthly mean SSS among SMOS, Aquarius and NODC objective analysis from In-situ observations;
4. To Conduct investigation on SMOS QA flags choices
Current tasks and goals of QA monitoring on SMOS/Aquarius Sea surface salinity products
![Page 4: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/4.jpg)
• 1.0x1.0 longitude/latitude box average;
• Applied Quality Flags: 1. Dg_quality_sss_# (model #):
Quality index for sss, lower=better, values over 300 are filtered
2. Control_Flags_# (model #):
Total = 32, but only contains 27 meaningful control flags, all applied except
mask(tag # 1), roughness correction (tag # 16), availability of ECMWF(tag # 19)
and grid point measurement discrimination test (tag # 19).
• Software: FORTRAN,C-shell, GrADS and Linux image
convert;
• Outputs: EPS and PNG format.
Development visualizations of level-2 SMOS data in swath, daily, 3-day and monthly mean
![Page 5: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/5.jpg)
![Page 6: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/6.jpg)
![Page 7: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/7.jpg)
![Page 8: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/8.jpg)
Development of tools for calculation of quality statistics a level-2 granule
• Six QA statistics:– Valid number of observations– Number of observations over 3-sigma– Mean– Standard deviation– Minimum & Maximum
• Provide visualizations (EPS and PNG);
• Convert the data into CF compliant NetCDF format and provide data access via LAS/OPeNDAP/THREDDS servers (future work)
![Page 9: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/9.jpg)
Data filters: SSS1=-999; Dg_quality_SSS_1=999, Control flags not applied
Zero Obs NO
![Page 10: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/10.jpg)
![Page 11: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/11.jpg)
![Page 12: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/12.jpg)
![Page 13: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/13.jpg)
Data Sets and Histogram Analysis
Data sets: monthly 1x1 degree, April 20120• Aquarius: JPL PO.DDAC Level-3 mapped and smoothed version 1.3.• SMOS: gridded by NODC from level-2 swath data with Dq_quality and control flag filters.• NODC monthly objective analysis.
Histogram analysisReferences: “The SST quality Monitor (SQUAM)”, P. Dash et al. 2010. Median and Robust Standard Deviation (RSD) are used to construct thresholds to remove extreme values, compared to conventional mean and Standard Deviation (SD) :
SD= RSD=IOR/S
IOR is the interquartile range: difference of the values of 75th and 25th percentile in an ordered dataset.S=1.348 for an ideal normal distribution
Outliers: median ± 4xRSD
![Page 14: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/14.jpg)
Difference of monthly SSS between SMOS and Aquarius
![Page 15: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/15.jpg)
![Page 16: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/16.jpg)
![Page 17: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/17.jpg)
Histograms of SSS difference between SMOS and AquariusApril 2012
Num
ber D
ensi
ty (%
) N
umbe
r Den
sity
(%)
SMOS Model 1 SMOS Model 2 SMOS Model 3
Left outlier: Value < median-4xRSD; Right outlier: Value > median+4xRSD
![Page 18: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/18.jpg)
Histograms of SSS difference between NODC OA and SMOSApril 2012
SMOS Model 1 SMOS Model 2 SMOS Model 3
Num
ber D
ensi
ty (%
) N
umbe
r Den
sity
(%)
![Page 19: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/19.jpg)
Num
ber D
ensi
ty (%
)
Histogram of SSS difference between NODC OA and Aquarius April 2012
![Page 20: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/20.jpg)
Histogram statistics for monthly difference among NODC OA, SMOS, and Aquarius SSS, April 2012
Observation N
mean medianStandard Deviation
Robust Standard Deviation
N: left outlier
N: right outlier
SMOS(M1)-Aqu 32856 -0.09 -0.01 1.42 0.42 2018 863
SMOS(M2)-Aqu 31981 -0.19 0.03 1.38 0.42 2219 730
SMOS(M3)-Aqu 32889 -0.12 0.0 1.47 0.45 2159 805
NODC OA-SMOS (M1) 25850 0.31 0.27 1.64 1.20 138 182
NODC OA-SMOS (M2) 24941 0.36 0.25 1.72 1.26 69 277
NODC OA-SMOS (M3) 25927 0.34 0.16 1.85 1.43 72 324
NODC OA-Aqu 25883 0.34 0.31 3.70 1.15 209 358
![Page 21: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/21.jpg)
SMOS level-2 data quality flags choices and impact in monthly mean results
Test and Applied Quality Flags:
• Dg_quality_sss_#
– Quality index for sss, lower=better, values over 300 are filtered
• Control_Flags_#
– Total 32, but only contains 27 meaningful control flags, all
applied except:
– Mask (tag # 1)
– Roughness correction (tag # 16)
– Availability of ECMWF(tag # 19)
– Grid point measurement discrimination test (tag # 19).
![Page 22: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/22.jpg)
Distributions of number density of Dg_quality_sss_# in April 2012 (849 swath files)
num
ber d
ensi
ty (1
0**6
)
Values of Dg_quality_sss_# flags
![Page 23: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/23.jpg)
Dg_quality_SSS_1
Good Bad
![Page 24: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/24.jpg)
Data filtered: Dg_quality_SSS_1 <= 300
![Page 25: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/25.jpg)
Data filtered: Dg_quality_SSS_1 >= 300
![Page 26: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/26.jpg)
SMOS Filter: Dg_quality_SSS_1 =999
SMOS Filter: Dg_quality_SSS_1 <300
SMOS Filters: Dg_quality_SSS_1 =999; Control_flags_1, except tag# 1,6,16,18,19
SMOS Filters: Dg_quality_SSS_1 <300; Control_flags_1, except tag# 1,6,16,18,19
![Page 27: Development of Quality Monitoring System on satellite Sea Surface Salinity Products Yongsheng Zhang NESDIS/NODC – UMD College Park/ESSIC/CICS](https://reader035.vdocuments.us/reader035/viewer/2022062713/56649cf35503460f949c1270/html5/thumbnails/27.jpg)
Future Work• Application of more flags in data quality analysis in SMOS level-2:
Control_Flags_#, Science_Flags_#;
• Compare difference among SMOS Models 1, 2 and 3.
• Develop NetCDF format for the QA statistics and data visualization tools;
• Histogram analysis and development NetCDF format for the histogram statistical data and visualizations;
• Development of software for automation processing and user service interfaces