radial-by-radial noise power estimation
DESCRIPTION
the University of Oklahoma. Radial-by-Radial Noise Power Estimation. Igor Ivić and Christopher Curtis CIMMS/University of Oklahoma and NSSL/National Oceanic and Atmospheric Administration. NEXRAD TAC Norman, OK August 29, 2012. Motivation. - PowerPoint PPT PresentationTRANSCRIPT
Radial-by-RadialNoise Power
Estimation
Igor Ivić and Christopher CurtisCIMMS/University of Oklahoma
and NSSL/National Oceanic and Atmospheric Administration
NEXRAD TACNorman, OKAugust 29, 2012
2
• Incorrect noise power measurements can lead to:– Reduction of coverage when noise power is overestimated
• Case in most radar sites
• Radar product images cluttered by noise speckles if the noise power is underestimated
– Usually occurs in cases of strong interference• Biased meteorological variables at low to moderate SNR
• Blue-sky noise used to produce system noise power– Adjusted for lower elevations using correction factors– For each elevation, the same value is used at all azimuths
• However, noise drifts with time and varies with antenna position in azimuth and elevation
Motivation
+ the system gain can change within minutes …
3
Solution• Estimate receiver noise power at each
antenna position– Noise power needs to be computed in real-
time (i.e., from data containing mixed signals and noise)
• BUT HOW DO YOU DO THAT?
• Radial-by-radial noise power estimation technique provides the solution
4
Radial-by-Radial Noise Estimation
• History– First Version presented at ERAD in September 2010
• required rough initial noise value– Second version presented at AMS in January 2011
• no rough initial noise value required– Algorithm description of the latest version
delivered to the ROC in May 2012• includes high gradient signal removal• simplified so it operates only on the measured powers
• The technique has been in use on the NWRT since June 2011
5
Technique Description• Expert noise power Ne determined
visually
• Used to assess the technique accuracy
200 400 600 800 1000 1200 1400 1600 1800
0
10
20
30
40
50
60
70
801
0 lo
g10
( P/ Ne)
(d
B)
Samples no.
M = 17
Expert noise Ne (gates 1300 to 1836)
6
Power Profile of Test Data
Received power as a function of range at the elevation angle of 0.5 deg. The expert noise power (Ne) is indicated with a grey line.
200 400 600 800 1000 1200 1400 1600 1800
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 15
Expert noise Ne (gates 1440 to 1490)
200 400 600 800 1000 1200 1400 1600 1800
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 28
Expert noise Ne (gates 1600 to 1700)
7
Step 1: Strong Point Clutter Rejection
• Gate at location k is considered to contain point clutter if its power is much larger than the power at surrounding gates
200 400 600 800 1000 1200 1400 1600 1800
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 28
200 400 600 800 1000 1200 1400 1600
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 28
200 400 600 800 1000 1200 1400 1600 1800
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 15
200 400 600 800 1000 1200 1400 1600
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 15
BEFORE POINT CLUTTER REJECTIONAFTER POINT CLUTTER REJECTION
8
Step 2: Detect Flat Sections• The flat sections of the power profile are
identified as this is an indication of the potential signal-free regions
– This is done by estimating local variance along range
– (local range variance at k < threshold) => range gate at k considered potentially signal-free
200 400 600 800 1000 1200 1400 1600
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 28
Local range variance
Notice that the local range variance is smaller in noise regions
9
Step 2: Flat Sections Detected
• The mean power is computed for each group of contiguous range gates classified as signal-free
• Out of those, the smallest one is taken to be the intermediate noise power estimate (Nint)
200 400 600 800 1000 1200 1400 1600
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 28
200 400 600 800 1000 1200 1400 1600
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 28
10 log10
(Nint
/Ne) = -0.04 dB
200 400 600 800 1000 1200 1400 1600
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 15
200 400 600 800 1000 1200 1400 1600
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 15
10 log10
(Nint
/Ne) = -0.4508 dB
10
Step 3: SNR Censoring Using Nint
• Discard all samples at range locations for which the power estimate is larger than the threshold (THR(M)xNint)
200 400 600 800 1000 1200 1400 1600
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 15
SNR thresholdNint
200 400 600 800 1000 1200 1400 1600
0
10
20
30
40
50
60
70
80
10
log
10( P
/ Ne)
(dB
)
Samples no.
M = 28
SNR thresholdNint
11
100 200 300 400 500 600 700
-3
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 28
200 400 600 800 1000 1200 1400
-4
-3
-2
-1
0
1
2
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 15
Step 3: Output after Nint SNR Censoring
SNR profile of data after discarding samples at range gates where power estimate is larger than the threshold.
Mean power/expert noise = -0.216 dB
Mean power/expert noise = 0.243 dB
Potential signal regions
12
Step 4: Apply "range persistence" Filter
• Detects larger sample powers that exhibit continuity in range– Finds 10 or more consecutive power values larger than
the median power
100 200 300 400 500 600 700
-3
-2.5
-2
-1.5
-1
-0.5
0
0.5
1
1.5
2
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 28
Gates labeled by the "persistence filter"Median power
200 400 600 800 1000 1200 1400
-4
-3
-2
-1
0
1
2
10
log
10( P
/ Ne)
(d
B)
Samples no.
M = 15
Gates labeled by the "persistence filter"Median power
13
Steps 5&6: Update Active Range Gates and Censor using SNR Threshold
Mean power/expert noise = -0.2376 dB
Mean power/expert noise = 0.1586 dB
• Discard the samples marked by the range persistence filter
• Compute the mean power of the remaining samples
• Perform SNR censoring
100 200 300 400 500 600 700
-3
-2
-1
0
1
2
10
log
10( P
/ Ne)
(dB
)
Samples no.
M = 28
Mean powerThreshold
200 400 600 800 1000 1200 1400
-4
-3
-2
-1
0
1
2
3
10
log
10( P
/ Ne)
(dB
)
Samples no.
M = 15
Mean powerThreshold
14
Step 7: Apply Running Sum Filter
• A running sum is performed over the array of remaining powers
– makes regions with weak signals visible
After 1 iterationmean power/expert noise = -
0.0043 dB
After 1 iterationmean power/expert noise = -0.3 dB
200 400 600 800 1000 1200 1400
28
30
32
34
36
38
Samples no.
Ru
nn
ing
su
m
M = 15
Sum of 33 samplesMean powerThreshold
100 200 300 400 500 600 700
17
18
19
20
21
22
23
Samples no.
Ru
nn
ing
su
m
M = 28
Sum of 18 samplesMean powerThreshold
15
Accuracy Assessment
Algorithm failed to produce noise estimate 43 times out of 155164 (0.03%)
BIAS = -4×10-3 dB or -0.086% of the true noiseSTD = 5.3×10-2 dB or 1.22% around the mean
• To assess the accuracy‒ estimated noise is subtracted from all powers‒ simulated noise is artificially added to the real data
-0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.40
1
2
3
4
5
6
7
8
9
10
Estimated noise/True noise (dB)
%
16
SUMMARY
Radial-by-radial noise power estimation technique accurately produces noise power measurements in real-time! GREAT
SUCCESS!
1750 100 150 200 250 300 350
-80.5
-80
-79.5
-79
-78.5
-78
-77.5
-77
-76.5
Az (deg)
10
log
10(N
) (d
Bm
)
Measured and legacy noise (El = 0.87 deg)
Measured HLegacy HMeasured VLegacy V
Benefits of Radial-by-Radial Noise Estimation
• Comparison between the measured noise power and legacy noise power
– Can see effects from man-made (KVNX) and mountain (KPDT) clutter
– Will use data from these cases to illustrate benefits
KVNX: Vance AFB, OK KPDT: Portland, OR
50 100 150 200 250 300 350-81.4
-81.2
-81
-80.8
-80.6
-80.4
-80.2
-80
-79.8
X: 284.4Y: -80.6
Az (deg)
10
log
10(N
) (d
Bm
)
Measured and legacy noise (El = 0.53 deg)
Measured HLegacy HMeasured VLegacy V
0.9 dB
1 dB
4 dB
1 dB
18
Reflectivity CoverageKPDT Data with Legacy Noise (0.9°, VCP 12, Cut 3)
19
Coverage increased by 18%
Reflectivity CoverageKPDT Data with Measured Noise (0.9°, VCP 12, Cut
3)
20
Invalid Spectrum Width Values
• 69.6% of v estimates with legacy noise are valid (i.e., v > 0)
• 89.0% of v estimates with measured noise are valid
1
2ˆ2ˆ ln
ˆ2 1a h h
v
h
v P N
R
KPDT Data with Legacy vs. Measured Noise (0.9°, VCP 12, Cuts 1&2)
49.4% IMPROVEMENT IN THE NUMBER OF VALID ESTIMATES
21
• 78% of |hv(0)| estimates with legacy noise are less than one• 84.48% of hv(0) estimates with measured noise are less than one
14.9% IMPROVEMENT IN THE NUMBER OF VALID ESTIMATES
ˆ 0
ˆ 0ˆ ˆ
hv
hv
h h v v
R
P N P N
Invalid Correlation Coefficient Values
KVNX Data with Legacy vs. Measured Noise (0.5°, VCP 11, Cut 1)
22
Effects on ZDR
• Differential reflectivity is estimated as
where Nh and Nv are errors in noise floors
• Using perturbation analysis, we get
– If errors in H and V are of different sign they add up, otherwise they may cancel
10
ˆˆ 10log
ˆh h h
DR
v v v
P N NZ
P N N
10 1 1_
ˆ ˆln10DR v h
v v h h
BIAS Z NOISE N NP N P N
23
Contribution of Noise Error to ZDR Bias
KPDTNh = -80.6 dBm & Nv = -80.3 dBm
Nh = -4 dB & Nv = -1 dB
KVNXNh = -80.7 dBm & Nv = -81.2
dBmNh = -0.9 dB & Nv = -1 dB
• In the current system, noise errors are significant contributors to the ZDR bias for low to moderate SNR
• Real-time noise estimates eliminate this contribution because of accurate measurements (i.e., Nh 0, Nv 0)
5 10 15 20-0.5
-0.45
-0.4
-0.35
-0.3
-0.25
-0.2
-0.15
-0.1
-0.05
SNR (dB)
BIA
S_Z
DR
(NO
ISE
) (d
B)
M = 17, v = 2 m s-1, Z
DR = 4 dB, |
hv(0)| = 0.99
SimulationTheoretical calculation
5 10 15 20
0.1
0.2
0.3
0.4
0.5
0.6
SNR (dB)
BIA
S_Z
DR
(NO
ISE
) (d
B)
M = 15, v = 2 m s-1, Z
DR = 0 dB, |
hv(0)| = 0.99
SimulationTheoretical calculation
24
Summary• Improved data quality with measured noise:– Increased coverage for all moments and dual-
polarization variables– Reflectivity – decreased bias– Spectrum Width (v) – decreased bias and
increased number of valid estimates– Correlation coefficient (|hv(0)|) – decreased bias
and increased number of valid estimates– Differential reflectivity (ZDR) – decreased bias
• Accurate noise measurement is crucial for keeping measurement biases within acceptable levels
25
Conclusions• Accurate noise estimation accounts for noise
variations due to:– Changes in system gain (Melnikov et al.)– Radiation caused by clutter, storms and man made objects– These noise variations can lead to:
• Reduction of coverage when noise power is overestimated• Radar product images cluttered by noise speckles if the noise
power is underestimated• Biased meteorological variables at low to moderate SNR (e.g.,
differential reflectivity and correlation coefficient)
• The current system calibration does NOT address these issues!
• Radial-by-Radial Noise Estimation eliminates or greatly mitigates all of these issues stemming from incorrect noise power measurements.
26
Backup Slides
27
KVNX Data with Legacy Noise (0.5°, VCP 11, Cut 1)
Reflectivity Coverage
28
Coverage increased by 6.18%
Reflectivity CoverageKVNX Data with Measured Noise (0.5°, VCP 11, Cut
1)
29
KPDT Data with Legacy Noise (0.5°, VCP 12, Cuts 1&2)
Invalid Spectrum Width Values
30
Invalid Spectrum Width Values
KPDT Data with Measured Noise (0.9°, VCP 12, Cuts 1&2)