data reduction techniques - science website€“ data transfer over internet ... vla data reduction...

VLA data reduction – part 1: Post-observing, pre-calibration

Loránt Sjouwerman, NRAO

VLA data reduction - Part 1: observing to calibration – Socorro, October 2017 2

Outline

• After the observations:– Obtaining your data from the archive

• Which CPU processes the data? (Home or NRAO)

– Examine your data• Structure and potential issues: all expected data

present, RFI, calibrators, reference antenna…

– Prepare for calibration steps• Use examination to flag bad data upfront

Better preparation eases the process!


Assumptions (for all these lectures)This presentation assumes that you are familiar with

the basics of:• radio interferometry• flux density calibration, antenna-based calibration (complex gain,

bandpass), and self-calibration• imaging and deconvolutionFor references on the above, please check:• The lectures of the 2014 or 2016 synthesis imaging workshop

https://science.nrao.edu/science/meetings/2014/14th-synthesis-imaging-workshop/https://science.nrao.edu/science/meetings/2016/15th-synthesis-imaging-workshop/lectures

• Synthesis Imaging for Radio Astronomy II (eds. Taylor, Carilli, and Perley).

• Interferometry and Synthesis in Radio Astronomy (by Thompson, Moran, and Swenson).

https://science.nrao.edu/science/meetings/2014/14th-synthesis-imaging-workshop/

https://science.nrao.edu/science/meetings/2016/15th-synthesis-imaging-workshop/lectures


NRAO versus Local/home computing

• Note that NRAO offers computing facilities for demanding projects upon request– Registered user– Limited capacity, compete with others, no guarantee– See computing policy page

https://science.nrao.edu/facilities/vla/docs/manuals/computing-resources

• Here assume processing at home institute– Data transfer over internet (up to ~ 100 GB)– Data shipped on disk (purchase, up to 1.8 TB/disk)

https://science.nrao.edu/facilities/vla/docs/manuals/computing-resources


Obtaining data from the NRAO archivePeek at using the new archive tool

(still a work in progress)Here: using the current archive tool

https://archive-new.nrao.edu


The NRAO Data Archive Toolhttps://science.nrao.edu/ Facilities VLA, Data Archive (left menu), VLA/VLBA Archive

https://science.nrao.edu/


The Archive ToolLog in for proprietary data here

Also https://archive.nrao.edu/

https://archive.nrao.edu/


The Archive Tool


Basic Search: simpler interface


Query return

• For each match, the archive query return presents per observation (i.e. per row):– The observing run identifier (i.e., the SB name)

– Any data quality issues (highlighted in yellow/red)– The SDM-BDF set (content of the SDM directory)

– The individual scans with their details – see next– The operator log (usually, also sent by email)


Scan listing:Scan details (source, date, setup, etc)


Scan listing

FYI: reference pointing and OTF have subscans


• Data formats: – SDM-BDF

– CASA measurement set, i.e., CASA MS (default)

– SDM tables only• Flagging and averaging

options only apply to CASA MS format

Download options: data format

• If CASA MS is requested, the native SDM-BDF is converted to MS

using CASA’s importevla task (which allows flagging and averaging)


• If the “apply flags” option is not checked, the flags are written to a FLAG_CMD MS table. They can later be applied by using the CASA task flagcmd

• If checked, flags are applied to the data in the MS conversion

Download options: flagging

‘Telescope flags’• Online flags, e.g., antenna not on source, sub-reflector error

• Shadowing flags, and• Zero flags (pure zero’s)


• Possible to average MS data in time and/or in frequency• Selection of scan numbers (use scan listing mentioned before)• For these, the archive tool uses the CASA task split

Download options: averaging


Averaging decreases data size which helps in the transfer and data reduction speed

When averaging:• Apply the flags!• Frequency averaging may cause

coherence lossCheck that delays are small before frequency averaging

• Amount of allowable time averaging depends on the science goal The VLA Observational Status Summary discusses amplitude

loss due to time averaging.

Notes on averaging


• The SDM-BDF and MS are data directories!– For downloading over internet, “tar” is recommended

(but requires twice the disk space)

– Alternatively, use “wget”

Transfer of SDM and MS directories:


• The native SDM-BDF data is always good:• May take a while to convert to MS at home• Should be usable for any CASA version available• Can also be used for AIPS

• Archive processed (averaged/flagged MS) data may need the same CASA version to proceed

• Version used should be listed in a file in the download directory in *__asdm2MS.log or *__casalog.log

• Pipeline processed (MS) data and/or products may need the same CASA version to proceed

• Calibration tables specific to CASA version

Some final archive notes


• NRAO can ship data on hard disks upon request, e.g.:– when the size of the data is large (over a few 100 GB)– when the internet connection cannot handle the request

• This disk-ordering process is done through the archive tool.• Data is shipped on a 2 TB disk (which holds1.8 TB of data) • Cost: USD 125 per disk, potentially plus shipping cost

• Disk shipment information and policies are posted athttps://science.nrao.edu/facilities/vla/archive/shipment

Requesting data on a hard disk

https://science.nrao.edu/facilities/vla/archive/shipment


• Upcoming VLA CASA pipeline talks…

• Note that VLA CASA calibration pipeline products are not yet available through the archive

(work is in progress)

• Request pipelined data products through the VLA Pipeline department of the NRAO help desk (https://help.nrao.edu/)

Download through the internet or ask for a hard disk (purchase)

Getting CASA Pipeline Calibrated Data

https://help.nrao.edu/


• Conversion from the native SDM into UV FITS format is no longer supported through the archive

Download the native SDM-BDF from the archive. Use OBIT to load into AIPS using task ’bdf2aips’.

http://www.cv.nrao.edu/~bcotton/Obit.html

For more details on the VLA data archive, see https://science.nrao.edu/facilities/vla/archive/index

Loading data into AIPS

http://www.cv.nrao.edu/%7Ebcotton/Obit.html

https://science.nrao.edu/facilities/vla/archive/index


Examine the visibility data (in CASA)


CASA

• Web site: http://casa.nrao.edu/• Available for both Linux and Mac OS

• Make sure to subscribe to the CASA mailing lists:– casa-announce: announcements of new releases,

workshops, etc…– casa-users: critical bugs and code updates

http://casa.nrao.edu/ Getting Help Mailing lists

http://casa.nrao.edu/



CASA

• Documentation is available athttp://casa.nrao.edu/ ‘Using CASA’

• Training material is available athttp://casaguides.nrao.edu

• For help, use the NRAO help desk at: http://help.nrao.edu

CASA 5.1.0 will be used at this workshop


http://casaguides.nrao.edu/

http://help.nrao.edu


CASA• All CASA tasks can be listed by tasklist• The tasks are grouped as:

– Import/export– Information– Editing– Manipulation– Calibration– Modeling

• AIPS(/MIRIAD/CLIC) to CASA dictionary in the CASA cookbook:http://casa.nrao.edu/ ‘Using CASA’ ‘User Reference and Cookbook’(see Appendix I)

– Imaging– Analysis– Visualization– Simulation– Single dish– Utility



Loading The Data: importevla

If one chooses to download the SDM-BDF (not CASA MS)

• Task importevla converts the SDM-BDF to MS

• importevla understands VLA online flags:– It converts the data into a MS while applying various

types of flagging (online flags, pure zeros, shadowing).

> default importevla> inp> asdm = 'archive_sdm_directory_name'> vis = 'output MS name'> ocorr_mode = 'co' (or load ca, ao)> scans = ''


Loading The Data: importevlaFlags:

online = True tbuff = 0.0

flagzero = True flagpol = True

shadow = True tolerance = 0.0

applyflags = False

• If applyflags = False (default) => the flags are written to a FLAG_CMD MS table. They can be examined (listed, plotted) and applied by using the task flagcmd [recommended]

• If applyflags = True => the flags are applied to the data


Examining Your Data• Operator observing log (email, posted on web)

• Observing summary : listobs(sources, scans, spectral windows, antennas, etc…)

• Plotting the antenna positions: plotants

• Plotting/displaying data: plotms, and msview or viewer

Examine your data carefully before flagging:That is, know your data content


Observing Summary: listobsvis = ‘my.ms’

verbose = True (or False)

listfile = ‘file_with_listobs_output’


Plotting the antennas: plotants

vis = ‘my.ms’

• Reference antenna:• Pick a few, need

baselines to all other antennas (to be checked)

• Keep in mind when examining data (use the one with in the end least data flagged)


Data Review: plotms (unix command line casaplotms)Top Tabs

Graphics Panel

Control Panel

Tools Panel

Side

Tab

s


Data Review: plotms

Control Panel: Data

Check the ‘Reload’ box if the MS has been modified through another task.

Use the ‘Options’ to divide the screen into multiple panels, and ‘Add plot’ to be able make plots of multiple data sets (or one data set but using different axes) onto the graphic panel.


Data Review: plotmsMS Ids and other meta info:

'scan' (number)'field' (index)'time','interval'='timeint'='timeinterval'='time_interval''spw' (index)'chan'='channel' (index)'freq'='frequency' (GHz)'vel'='velocity' (km/s)'corr'='correlation' (index)'ant1'='antenna1' (index)'ant2'='antenna2' (index)'baseline' (a baseline index)'row' (absoute row Id from the MS)

Visibility values, flags:'amp'='amplitude''phase' (deg)'real''imag'='imaginary''wt'='weight''flag''flagrow'

Axe

s


Data Review: plotmsObservational geometry:

'uvdist' (meters)'uvwave'='uvdistl'='uvdist_l' (wavelengths, per

channel)'u' (meters)'v' (meters)'w' (meters)'azimuth' (at array reference; degrees)'elevation' (at array reference; degrees)'hourang'='hourangle' (at array reference; hours)'parang'='parangle'='parallacticangle' (at array

reference; degrees)Antenna-based (only works vs. data Ids):

'ant'='antenna''ant-azimuth''ant-elevation''ant-parang'='ant-parangle'

Axe

s


Data Review: plotms

ScanFieldSpwBaselineAntennaTime

Page: to iterate on

Tool panel


Data Review: plotms

Frame: TOPO, GEO, BARY, LSRK, LSRD, etc..

(While examining your data you probably want to keep the data in channel or frequency)

Transformations


Data Review: plotms

Colorize by:ScanFieldSpwAntenna1Antenna2BaselineChannelCorrelationTime

Display


What are we looking for?

• A feel of the overall structure of the data (see also the OPT schedule): – Calibrators and target visibilities, frequency setup– Observing conditions, instrumental response

• Where to expect bad data– Specific ill-performing antennas/baseline(boards)– In time

• Start of scans• Bad weather/pointing (observing conditions)

– In frequency• Bandpass, subband edges• RFI – not your line!


Data Review: plotms

Example: xaxis=‘time’, yaxis=‘amp,’ coloraxis=‘field’Page: iterating on spw (with all channels averaged)


Radio Frequency Interference (RFI)

1. VLA observations, particularly at the lower frequency bands, will be severely affected by RFI.

2. VLA RFI information is available at:

https://science.nrao.edu/ Facilities VLA Observing Guide to VLA Observing Radio Frequency Interference

• RFI listings per frequency band.

• Spectra of various RFI sweeps between1-50 GHz.

https://science.nrao.edu/


RFI is present at lower frequency bands

S band

L band

C band X band


RFI/birdies at the higher frequency bands

Q band

Ku band K band

Ka band


Data Review: plotms

Example: xaxis=‘frequency’, yaxis=‘amp’,coloraxis=‘spw’Iterating on scan


RFI: feedback from observers

• The VLA has opened the full 1 to 50 GHz frequency range.• Also the 230-470 MHz range.

• This exposed us to all types of RFI.• RFI is direction dependent.• User feedback is critical for our ongoing RFI identification and

monitoring efforts.• Observers are asked to email [email protected] and provide:

– Observation/project code– Frequency and time of the observations– The characteristics of the RFI signal (e.g., continuous, intermittent)– A spectrum

mailto:[email protected]


RFI: spectral (Gibbs) ringing

• Strong RFI will introduce disturbing spectral ringing.

• Hanning-smoothing should be applied on such data sets before attempting any spectral flagging, or calibration.

• In CASA, the task to use is hanningsmooth.

• Probably want to flag this affected data after HS (bad antennas, etc., you probably want to flag before smoothing)


Preparing for calibration: flagging


Flagging (or unflagging) Data

1. flagdata: All purpose flagging task based on selection.• Includes RFI flagging capabilities (RFLAG, TFCROP).

2. flagcmd: All purpose flagging task based on commands (alternative to flagdata for certain types of flagging).

3. plotms: Interactive flagging4. msview/viewer: Interactive flagging

Review the VLA operator’s log carefully.Certain issues (e.g., antennas without receivers), do not end up in the online flags, and may need to be flagged manually.


Flagging (or unflagging) DataA few important notes

1. Data in CASA are either flagged or not flagged. • Every MS has a flag column.• Every bit of data has its own flag (set either to true or false).

• Applying flags means setting the flag column entries of the selected bits of data to true.

2. Most flagging tasks have the option of creating a flag backup.

3. A flag backup is an MS table made by a given flagging task and contains the state of the flags before running the flagging task.

4. With flagmanager flag back-ups can be restored (and made)


Flagging Data: flagdata - Modes

• list = apply a list of flagging commands• manual = flagging based on specific selection parameters• clip = clip data according to values• quack = remove/keep specific time range at scan beginning/end• shadow = remove antenna-shadowed data• elevation = remove data below/above given elevations• tfcrop = auto identification of outliers on the time-freq plane• rflag = auto detection of outliers based on sliding-window RMS filters• extend = extend and/or grow flags • Also summary (per antenna, correlation, field, scan, total), and unflag.• Can also flag calibration tables.


Flagging Data: flagcmd

• It allows listing, plotting, saving, applying, or un-applying flags.• Flagging modes (inpmode) are:

• table: uses the FLAG_CMD MS table (initially created by importevla)

• list: uses an ASCII file that contains a set of flagging commands.

• xml: uses the online flags from Flag.xml in the MS.• It allows the user to save the flag records in the FLAG_CMD

MS table or a file.


Examining the flags with flagcmd

list


Examining the flags with flagcmd

plot


Flagging Data: flagdata vs. flagcmd

• Complementary flagging tasks.• Have several common features.• Some of the important differences:

Flagdata Flagcmd

RFI flagging (tfcrop, rflag)* Access to the Flag.xml

Runtime displays*(before and after flagging)

Apply the online (and other) flags in FLAG_CMD MS table

Plot Flags

* More details on Tuesday (RFI talk)


Flagging Data: plotms



To flag

To select regions



To select regions

To locate



The output of “locate” in the casalog – look for common lines


Flagging Data: plotmsA few important notes

• Use plotms carefully for flagging data.• Keep in mind that flagging data with plotms often

requires extending the flags (through the Flag tab).• plotms does not produce a flag backup (flagmanager has

to be used).• Use plotms to identify bad data (through the locate

option). Then flag the bad data using flagcmd or flagdata.


Flagging Data: msview

• Shows gray scale (or colored) waterfall, plots.

• Plots Time vs. Baseline, or Time vs. Channel for

– Amplitude (or amplitude diff or amplitude rms)

– Phase (or phase diff or phase rms)

– Real

– Imaginary

• Provides interactive flagging tools (comparable to TVFLG and SPFLG in AIPS).



http://casaguides.nrao.edu/ CASA Tips Data flagging with viewer

http://casaguides.nrao.edu/



Use the Flagging Options• to expand the flags.• to apply the flags.


Ready to calibrate the data?

The data structure is understood, reference antenna picked Calibrators (flux density, bandpass, gain) are identified Bad antennas and bad basebands are flagged RFI is removed (as much as possible), hanning smooth? Bad individual visibilities/baselines/times are flagged

• Maybe inspect (some parts of) the data again to make sure Likely more flagging may need to be done during/after calibration steps

• Ready to start with data calibrationNext lecture…

data reduction techniques - science website€“ data transfer over internet ... vla data reduction...

Documents