rfi mitigation for the parkes galactic all sky survey (gass)

Peter M.W. Kalberla Argelander-Institut für Astronomie Bonn

RFI mitigation for the Parkes Galactic All Sky Survey

(GASS)

2

Galactic All Sky Survey (GASS)

McClure-Griffiths et al. (2009)Kalberla et al. (2010)

N. M. McClure-Griffiths, D. J. Pisano, M. R. Calabretta, H. Alyson Ford, Felix J. Lockman, L. Staveley-Smith, P. M. W. Kalberla, J. Bailin, L. Dedes, S. Janowiecki, B. K. Gibson, T. Murphy, H. Nakanishi, K. Newton-McGee, J. Kerp, B. Winkel

3

GASS data final version (-0.12 to 50 K, log scale)

4

GASS: survey parameters• 13 beam receiver• 21-cm line survey of the Galactic HI emission

– Declinations δ < 1 deg – (-500) < -468 < vLSR < +468 < (+500) km/s– Δv = 1 km/s– In-band frequency switching, Δv = 660 km/s

• Beam FWHM 14.4 arcmin • OTF mapping in RA and DEC, two coverages • 2.8·107 spectra, 5 sec dumps, noise ~0.4 K• 10 observing sessions between 2005 and 2006

• FITS maps: noise at full resolution (15.6 arcmin): 60 mK

5

Every Thing You Always Wanted to Know About..

6

Problems

• RFI at fixed frequency without significant variation in time– Causing in many cases negative signals (ghosts)

• Broad lines (Δv ~ 15 km/s) in March 2006

• Bandpass ghosts from HVC gas due to folding

• Footprints: strong RFI signals for short time intervals

• Ringing (Gibbs phenomenon) from correlator

7

First step: Use livedata flags

• LAB data are used for fitting the instrumental baseline

• At that stage it is easy to replace channels flagged by livedata during first stage of reduction with LAB data

• Alternatively flagged data can be interpolated from neighboring channels of Parkes data

• The replacement using LAB is far better!!

• 0.1% of all data affected

8

Remaining RFI: „footprints“

9

Clean

10

Median filter (at any observed position)

• Determine median, mean and rms fluctuations within a radius of 6 arcmin (consistent with HIPASS)

• Find channels that have– High rms scatter (> 3 σrms) and

– Large differences between median and mean (>σm)

• Replace data that deviate > 2 σm from median by median– Do not filter for T > 0.5 K (T > 2 K at b > 10 deg)– Do not filter at positions with continuum > 200 mJy

• 0.07% of all data affected

11

Clean data

12

Observed, flagged RFI replaced

13

Peirce criterion (1852) AJ 2, 161

• Criterion for the rejection of doubtful observations

• Cutoff limit for exclusion of outliers depends on number of available data points

• For 40 profiles (typically) a 2 σrms limit is adequate if about 10% of the data are suspect

• A 1.6 σrms limit would be adequate if about 20% of the data are suspect

• We use a fixed 2 σrms limit with deviations from the median

14

Extra treatment:• Eliminate spectra with high noise (>3 times average)

and with more than 30 flagged channels (0.3% affected)

• Bandpass ghosts can be minimized by median filtering

• RFI in March 2006 (broad Gaussian lines)– Fit parameters– Flag data accordingly – Median filtering as usual RFI

• Emission lines > 2 K– No automatic filtering– Inspect data and filter only those regions that are

affected

15

Reorganize database for computational reasons

• 300 GB sdfits files with 2.8·107 spectra are hard to handle

• Generate compressed random access database– 135 GB in single file, pointer information – fast access of individual profiles

• Benefit of new data format:– Allows fast filtering

– Very fast on-the-fly processing of FITS cubes

16

Stray radiation (the reason for the second data

release)

17

21cm line work and Darwinism• Correction for stray radiation suffers from detailed

observations of the antenna diagram • Antenna parameters:• Model parameters need to be self-consistent• ~60 different runs• Baseline correction:• Code and parameters need to survive • ~50 different versions necessary• RFI mitigation • Comparison of all profiles at any position within 6 arcmin (109 cases)• >2 CPU years in total Hornet magazine, 1871

?

How are the Profiles today?

Does the solution survive?

18

Summary

• RFI post-processing needs redundancy – Typically no more that 25% of the data are bad – Limit: 50%

• Fast data access necessary for filtering– New data format needed (random access)– Advantages: generation of FITS cubes very fast

• Replace bad data by LAB data or by median – Surprisingly simple recipe to use other data

19

This all was about…

RFI in the protected band

But <0.5% of data affected

20

Dirty stuff you don’t want to see….