L2OS RFI status
Nicolas Lamquin, Jean-Luc VergelyJacqueline Boutin
Paul Spurgeon
ICM, Barcelona, 16/17 May 2013
RFI flags in v600 processingVersion of products:
• v600 source: sources-L2OS-core-v6.00.tar.gz
• job orbit: descending SM_TEST_MIR_SCSF1C_20110505T040402_20110505T045804_600_001_0
• L1c flag description : SO-TN-IDR-GS-0005 (24/10/2012)
• CNFOSF file: SM_TEST_AUX_CNFOSF_20050101T000000_20500101T000000_001_023_8
• RFI region: arctic
L1c RFI flags (measurement)
• Fm_rfi_mitigation_tot (bit 7) (v600) rfi_xx– analysis of NIR BT in X polarization
• Fm_rfi_strong_tot (bit 15) (v600) rfi_yy– analysis of NIR BT in Y polarization
• Fm_rfi_tails_tot (bit 12) never raised – affected by tails identified in the AUX_RFI list
• Fm_rfi_point_tot (bit 16)– affected by point source identified in the AUX_RFI
list
L1c L2 (DAP: measurements)
• L2 flag « L1c_rfi »: True if any of L1c RFI flag True
snapshot 34 (id:79011306): Fm_rfi_point_tot flag in L1c L2 « L1c_rfi » is True
No RFI detection from NIR
Tails in Fm_rfi_point_tot ?
Y
green = RFI_XX True
red = RFI_XX True and RFI_YY True
snap. 632id:79012029
snap. 633id: 79012031
snap. 634id: 79012032
?
Y
X
L2 RFI flags (measurements)
• Fm_rfi_L1• Fm_l2_rfi• Fm_l2_rfi_outlier• Fm_l2_rfi_snapshot_out_of_range• Fm_l2_rfi_high_snapshot_std• Fm_l2_rfi_high_snapshot_std_st3• Fm_l2_rfi_high_snapshot_std_st4
L2 RFI flags (measurements)
• Fm_l1c_rfi• Fm_l2_rfi• Fm_l2_rfi_outlier• Fm_l2_rfi_snapshot_out_of_range• Fm_l2_rfi_high_snapshot_std• Fm_l2_rfi_high_snapshot_std_st3• Fm_l2_rfi_high_snapshot_std_st4
gridpoint analysis (risk of RFI + RFI_nsig test) local information per measurement
snapshotanalysis raised for complete snapshot
from L1c
True if Fm_l2_rfi_outlier or Fm_out_of_range, nominally includes snapshot analysis below but not in current configuration
Fm_l2_rfi = Fm_l2_rfi_outlierOR Fm_out_of_range
• Fm_out_of_range: threshold test on residual per measurement (Tm_out_of_range=50 K)
• Fm_l2_rfi_outlier: analysis of all residuals per gridpoint (« risk of RFI ») + threshold test on residuals (comparison to std)
snapshot 34 (id:79011306)
• extra points in Fm_l2_rfi = Fm_out_of_range• Fm_l2_rfi_outlier is not raised although
Fm_out_of_range is raised bRiskOfRFI seems not raised
• sensitivity to examine (threshold 50 K)
snapshot 649 (id: 79012050)Sun tails ?
• flags all snapshot if any Fm_out_of_range
• does not proceed if some coast/ice nearby: set to True only if %(coast/ice) < Ts_snapshot_out_of_rangeTs_snapshot_out_of_range = 20%
test threshold
Fm_l2_rfi_snapshot_out_of_range
Fm_l2_rfi_snapshot_out_of_rangeexample 1 snapshot 34 (id:79011306):
example 2 snapshot 649 (id: 79012050):
Land contamination :
Snapshot flag not raised
Fm_l2_rfi_high_snapshot_stdFm_l2_rfi_high_snapshot_std_st3
and Fm_l2_rfi_high_snapshot_std_st4
• snapshot standard deviation tests• per measurement in snapshot (« Fs_high_std... »)• propagates through complete snapshot
• test 1: amount of data for statistics must be > Ts_meas_min (default: 35%)
• test 2: sqrt(std / nMeas) > Ts_std (default: 2.5)
• same threshold for « std », « std_st3 », and « std_st4 »
refinement of threshold(s) ?
Test definition
• for each configuration : process 1 month of data ? NO ! (too much thresholds to be tested)
• tests on one orbits and do relative comparison : relative SSS, number of valid TBs …etc.
• tests on several orbits in a (lat,lon) window in order to check the salinity estimation (comparison with ISAS maps).
• after a choice of configuration : monthly test
Remarks
• would be useful to have snapshot_ID for all measurements in L2 DAP
• need « Fs_high_std... » in DAP ?• DPM not coherent in 8_2-1 and 8_3-1 (OK in
the code)• Some variables not defined in the DPM:
RFI_c1, RFI_c2• Fm_RFI_L1_tot (type ‘s’) in DPM ? (OK in the
code : all L1c flags are used)
Introduction
• Difference between outlier and RFI ?
RFIs : unexpected physical source + Gibbs and tailsOutliers : reconstruction problem ?
- we expect snapshot signature for RFI and outlier- long time period contamination for RFI : seen by all the orbits which overpass the source : AUX_DGG_RFI information
Flags in the decision treeif Fm_Resol(im,igp) or (Fm_outlier(im,igp) and not Fg_ctrl_suspect_RFI) or (Fm_L2_RFI(im,igp) and Fg_ctrl_suspect_RFI) or Fm_out_of_range(im,igp) or Fm_border_fov(im,igp) or Fm_Moon_SpecDir(im,igp) or Fm_gal_noise_error(im,igp) or Fm_sun_limit(im,igp) or Fm_L1c_sun(im,igp) thenFm_valid = falseendif
Fg_ctrl_suspect_RFI is raised if (number of Fm_L2_RFI==1 )/Na > Tg_num_RFI_max
Fm_outlier : comes from comparison between TB_smos and TB_model corrected from a global shift applied at the dwell line level. nsigma comparison.
Fm_L2_RFI : Fm_L2_RFI_outlier or Fm_L2_RFI_snapshot_out_of_range or Fm_L2_RFI_high_snapshot_std
Fm_L2_RFI_outlier : comes from comparison between TB_smos and TB_model. RFI_nsigma comparison.
Fm_outlier and Fm_L2_RFI_outlier very close: NO REASON TO USE DIFFERENT nsigma criteria (?).
Proposition : suppress Fm_L2_RFI_outlier and keep only snapshot information for Fm_L2_RFI.
INSTEAD OF :
if Fm_Resol(im,igp) or (Fm_outlier(im,igp) and not Fg_ctrl_suspect_RFI) or (Fm_L2_RFI(im,igp) and Fg_ctrl_suspect_RFI) or Fm_out_of_range(im,igp) or Fm_border_fov(im,igp) or Fm_Moon_SpecDir(im,igp) or Fm_gal_noise_error(im,igp) or Fm_sun_limit(im,igp) or Fm_L1c_sun(im,igp) thenFm_valid = falseEndif
PUT :
if Fm_Resol(im,igp) or Fm_outlier(im,igp) or (Fm_L2_RFI(im,igp) and Fg_ctrl_suspect_RFI) or Fm_out_of_range(im,igp) or Fm_border_fov(im,igp) or Fm_Moon_SpecDir(im,igp) or Fm_gal_noise_error(im,igp) or Fm_sun_limit(im,igp) or Fm_L1c_sun(im,igp) thenFm_valid = falseEndif
WITH
Fm_L2_RFI(im,igp) : Fm_L2_RFI_snapshot_out_of_range or Fm_L2_RFI_high_snapshot_std
Remarks
• Fm_l2_RFI and Fg_ctrl_suspect_RFI do not integrate L1c RFI information
• Fm_l2_RFI and Fg_ctrl_suspect_RFI do not integrate AUX_DGG_RFI information
-> AUX_DGG_RFI and L1c information not used in the decision tree
Snapshot out-of-range TB detection (PRP_8_2)
-> Fs_out_of_range
-> Fm_l2_rfi_snapshot_out_of_range
Snapshot standard deviation tests (PRP_8_3)
-> Fs_high_std
-> Fm_l2_rfi_high_snapshot_std
Measurement outlier detection (PRP_8_4)
-> Fm_l2_RFI_outlier
-> Fm_outlier
OTT correction
RFI AUX detection (PRP_8_5)
-> Fm_l2_RFI (from Fm_l2_RFI_outlier, Fm_l2_rfi_snapshot_out_of_range,
Fm_l2_rfi_high_snapshot_std)
-> Fg_ctrl_suspect_RFI
Snapshot level
Snapshot level
Dwell line level
Grid point level