march 29, 2010 rfi mitigation workshop, groningen the netherlands 1 statistics of the spectral...

March 29, 2010 RFI Mitigation Workshop, Groningen The Netherlands

1

Statistics of the Statistics of the Spectral Kurtosis Spectral Kurtosis

EstimatorEstimatorGelu M. Nita and Dale E. Gelu M. Nita and Dale E.

GaryGary

New Jersey Institute of New Jersey Institute of TechnologyTechnology

March 29, 2010 2RFI Mitigation Workshop, Groningen The Netherlands

Population Spectral Population Spectral KurtosisKurtosis

The PSD estimate obtained by an N-points FFT

of a Gaussian time domain sig

1

nal h

as an expo

nential dist

; 1 1

ributio

2

n

kP

k

Np P e k

2

2

Population Spectral Kurtosis: statistical RFI discrim

in tor

1

a

k

k

SK


Spectral Kurtosis Spectral Kurtosis EstimatorEstimator

22

2 21

Spectral Kurtosis biased estimat

1

r

1

o

k

k

MSMSK

M S

2

1 21 1

Power and squared power sums

M M

k ki i

S P S P

22 2 1

1

1

Mean and variance unbiased estimat

(

rs

1

o

)k k

MS SS

M M M


The SK SpectrometerThe SK Spectrometer

Key features:

Conceptual simplicity

Frequency channel independence

Straightforward FPGA implementation


Statistical thresholds for Statistical thresholds for the rejection of RFI outliers the rejection of RFI outliers

(M>>1)(M>>1)(Nita et. al, 2007 PASP, 119, 805:827)(Nita et. al, 2007 PASP, 119, 805:827)

Assumed negligible bias

E 1SK

Variance (theoretically proven)

4 Var SK

M

RFI symmetrical thresholds (1 3 )

(assumed SK normalit

6

y)

1M

Hardware implementation of the SK excision algorithm and initial tests revealed that, although it performs generally well,

•the lower threshold level is set too low, failing to reject some RFI contaminated channels

•the upper threshold level is set too low, rejecting more RFI free channels than statistically expected

CONCLUSION: More accurate RFI rejection levels are needed for improved and reliable performance.


Characteristics of the SK Characteristics of the SK EstimatorEstimator

(Monte Carlo Simulations)(Monte Carlo Simulations)

1Skewness: 0

1

M

Bias: 1

1E SKM

2Kurtosis Excess: 50

2

M

The probability distribution of the SK estimator remains asymmetric even for a fairly large accumulation number M, approaching normality at a slower pace than needed for practical applications.

Main goals of this study:

Redefine the SK estimator to remove bias

Find an analytical expression for probability distribution of the SK estimator to allow accurate calculation of its tail probabilities


Starting PointStarting Point

22

1

Spectral Kurtosis estimator

11

MSMSK

M S

2 2

22 22

1 1

SK statistics is determined by the statistics of the

mean of squares

to square of mean ra

tio

k

k

SPMS M

S S PM


Joint Distribution (Monte Joint Distribution (Monte Carlo): Carlo):

Mean of Squares-Square of Mean of Squares-Square of MeanMean SS22 and and SS1122 are are

strongly correlated strongly correlated random variablesrandom variables

Their (unknown) joint Their (unknown) joint distribution would be distribution would be needed to derive the needed to derive the distribution of their distribution of their ratioratio

Is there any work-Is there any work-around approach?around approach?


Joint Distribution (Monte Carlo): Joint Distribution (Monte Carlo): Mean of Squares to Square of Mean Ratio - Mean of Squares to Square of Mean Ratio -

Square of MeanSquare of Mean

Monte Carlo simulations suggest:Monte Carlo simulations suggest:

SS22 / /SS112 2 and and SS112 2 are are uncorrelated random variablesuncorrelated random variables

Their individual distributions are Their individual distributions are independentindependent

This is the fundamental property that This is the fundamental property that makes the whole SK concept work makes the whole SK concept work by allowing SK to have a unity by allowing SK to have a unity value independently of the power value independently of the power levellevel

This property was analytically This property was analytically proven for the exponential proven for the exponential distribution based on first distribution based on first principles (Nita & Gary, PASP, principles (Nita & Gary, PASP, in press)in press)


Raw statistical moments of the Raw statistical moments of the Mean of Squares to Square of Mean of Squares to Square of

Mean RatioMean Ratio

2 2

2 1 2 2 12 2

1 1

, 0MS S S MS S

Cov E E ES M M S M

,Cov x y E xy E x E y

2

22 21 1

SE

MS ME

S SE

M

2 1 1, ,n n n nCov f g n f g Cov f g

12 2( 1) 222 1 2 1 2 1

2 2 21 1 1

, , 0n nn n

MS S MS S MS SCov n Cov

S M S M S M

2

1

2

221

n

SM

nSEn MMS

ES

E


Analytical Results Analytical Results (Nita & Gary, PASP, in press)(Nita & Gary, PASP, in press)

2 2

1

Raw moments of the square of mea

1 2 !

1 !

nn nM nS

M M ME

2

2

00

Raw moments of the mean of square

2

s

!

!

Mnn n nr

nr

t

rSt

M t rME

0

0

Raw moments of the mean of squares to square of mean

1 ! 2 !

1 2 !

ratio

!2

21

Mn n nr

nr

t

M M rt

M n t r

nMS

ES


Redefinition of the SK Redefinition of the SK Estimator Estimator

2 22 2

1 1

Bias of the originally propose

11 1

d SK e

11 1 1

stimator

MS MSM M ME

MS

M SK

S M ME E

22

1

The unbiased SK esti

11

1

m

1

ator

SKMSM

M

E K

S

S


/1

2

2 2

223

1 2 232

3 2

42 22

2

1 1 3/2

2 2 2

1

4 4 1

1 2 3

4 2 3 5 7

1 4 5

3 2 3 98 185 78

1 4 5 ( 6)( 7)

10 1

256 13

E SK

MO

M M M M M

M M M

M M M

M M M M M

M M M M M

OMM

OM M

Standard Moments of the Standard Moments of the SK EstimatorSK Estimator


Moment based Moment based approximation of the SK approximation of the SK estimator distribution estimator distribution

using Pearson Probability using Pearson Probability CurvesCurves

2

1 2

2 1 2 1

3

4 4 3 2 3 6


Pearson Type IV PDFPearson Type IV PDF

21 2 2

( )

Pearso

11

n 1895; Nagahara 1999

2

m i m ix x

p x Exp ArcTana aa m m

1

21

22 12 1

2 1

/1 2 1

( 2)

16( 1) ( 2)

6( 1) 16 1 2

2 3 6 4

2 1 ( 2)

H

einrich 20

2 4

04r r

r r

r a r r

rm r


Pearson IV CF and CCFPearson IV CF and CCF

( ) ( ) ; 1 ( ) ( )x

x

P x p x dx P x p x dx

To compute the tail probabilities of the SK estimator, one To compute the tail probabilities of the SK estimator, one needs to evaluate the cumulative function (CF) and needs to evaluate the cumulative function (CF) and complementary cumulative function (CCF) of the Pearson complementary cumulative function (CCF) of the Pearson IV probability curveIV probability curve

Knowing the analytical expression for the Pearson IV Knowing the analytical expression for the Pearson IV PDF, the CF and CCF can be computed analytically, by PDF, the CF and CCF can be computed analytically, by using the closed form expressions involving using the closed form expressions involving Hypergeometric series provided Heinrich(2004) or Hypergeometric series provided Heinrich(2004) or Willink(2008). Alternatively, CF and CCF can be Willink(2008). Alternatively, CF and CCF can be computed by a simple numerical integration.computed by a simple numerical integration.

The asymmetrical RFI thresholds are then chosen so as The asymmetrical RFI thresholds are then chosen so as to provide symmetric tails probabilities of rejecting true to provide symmetric tails probabilities of rejecting true Gaussian signals of 0.13499%, which are equivalent to Gaussian signals of 0.13499%, which are equivalent to the ±3the ±3 thresholds of a normal distribution. thresholds of a normal distribution.


RFI Threshold Computation RFI Threshold Computation ExampleExample


Pearson IV PDF vs. Monte Carlo Pearson IV PDF vs. Monte Carlo SimulationsSimulations


Time Domain Kurtosis Time Domain Kurtosis EstimatorEstimator

The DC (k=0) and Nyquist (k=N/2) frequency channels of a

FFT Spectrometer obey a different statistics.

The same statistics is obeyed by the power corresponding

to a narrow band time domain signal (FIR

1/21

filter s

pectrometer)

kP

kkp P P e

22

1

Following similar steps, an unbiased Time Domain Kurtosis (TDK) estimator

may be defined in

2

this case to detect RFI

conta

1

mina .

1

tion

MSMK

M S


Standard Moments of the Standard Moments of the time domain Kurtosis time domain Kurtosis

EstimatorEstimator

/1

2

2 2

223

1 2 232

3 2

42 2

2

1 1 3/2

2 2 2

2

24 24 1

1 4 6

216 4 6 2

1 8 10

3 4 6 213 474 368

1 8 10 ( 12)( 14)

6 6 1

540 13

E K

MO

M M M M M

M M M

M M M

M M M M M

M M M M M

OMM

OM M


Kurtosis estimator Pearson Kurtosis estimator Pearson IV PDF IV PDF

and RFI thresholds (M>45)and RFI thresholds (M>45)


SummarySummary

The Spectral Kurtosis RFI excision algorithm recommends itself by

Conceptual simplicity

Frequency channel independence

Straightforward FPGA implementation (See Dale Gary’s following presentation)

Theoretically determined RFI thresholds for arbitrary integration time


Main ReferencesMain References Gelu M. Nita and dale E. Gary, “Statistics of The Spectral

Kurtosis Estimator”, 2010, PASP, in press Gelu M. Nita, Dale E. Gary, Zhiwei Liu, Gordon J. Hurford,

& Stephen M. White, 2007, "Radio Frequency Interference Excision Using Spectral Domain Statistics," Publications Of The Astronomical Society Of The Pacific, 119, 805.

Yuichi Nagahara, 1999, "The PDF and CF of Pearson type IV distributions and the ML estimation of the parameters," Statistics & Probability Letters, 43, (1999), page 251.

Joel Heinrich, 2004, "A Guide to the Pearson Type IV Distribution," Collider Detector at Fermilab internal note 6820, 2004, http://www-cdf.fnal.gov/publications/cdf6820 pearson4.pdf

Willink, R., 2008, "A Closed-Form Expression For The Pearson Type IV Distribution Function,“ Aust. N. Z. J. Stat. 50(2), 199, 205

march 29, 2010 rfi mitigation workshop, groningen the netherlands 1 statistics of the spectral...

Documents

rfi mitigation workshop

sk estimator slide

press slide

point slide

netherlands characteristics

netherlands redefinition

square of mean ratio

rfi free channels