18 april 2007 second generation vlt instruments 1 vircam & cpl: lessons learned jim lewis and...

18 April 200718 April 2007 Second Generation VLT InstrumentsSecond Generation VLT Instruments 11

VIRCAM & CPL: Lessons VIRCAM & CPL: Lessons LearnedLearned

Jim Lewis and Peter BunclarkJim Lewis and Peter Bunclark

Cambridge Astronomy Survey UnitCambridge Astronomy Survey Unit

18 April 2007 Second Generation VLT Instruments 2

Introduction to CASUIntroduction to CASU

► Small group within the Institute of Astronomy, specialising Small group within the Institute of Astronomy, specialising in survey astronomy.in survey astronomy.

► Pipeline reduction of imaging dataPipeline reduction of imaging data APM (Schmidt Plates 1 x 40k x 40k)APM (Schmidt Plates 1 x 40k x 40k) INT Wide Field Camera (4 x 2k x 4k)INT Wide Field Camera (4 x 2k x 4k) ESO WFI on 2.2m at La Silla (8 x 2k x 4k)ESO WFI on 2.2m at La Silla (8 x 2k x 4k) MOSAIC-1 on KPNO 4m (8 x 2k x 4k)MOSAIC-1 on KPNO 4m (8 x 2k x 4k) MOSAIC-2 on Blanco 4m at CTIO (8 x 2k x 4k)MOSAIC-2 on Blanco 4m at CTIO (8 x 2k x 4k) AAO WFI on AAT (8 x 2k x 4k)AAO WFI on AAT (8 x 2k x 4k) CIRSI on INT (4 x 1k x 1k)CIRSI on INT (4 x 1k x 1k) INGRID on WHT (1k x 1k)INGRID on WHT (1k x 1k) UFTI on UKIRT (1k x 1k)UFTI on UKIRT (1k x 1k) WFCAM on UKIRT (4 x 2k x 2k)WFCAM on UKIRT (4 x 2k x 2k)


Introduction to VISTA and Introduction to VISTA and VDFSVDFS

► VISTA: 4m survey telescope on ParanalVISTA: 4m survey telescope on Paranal► VIRCAM: The IR camera for VISTAVIRCAM: The IR camera for VISTA

16 Non-buttable 2k x 2k detectors16 Non-buttable 2k x 2k detectors► VDFS: PPARC funded facility to provide an end-to-VDFS: PPARC funded facility to provide an end-to-

end data-flow system for VISTA and WFCAM.end data-flow system for VISTA and WFCAM.► Quality control and calibration pipelinesQuality control and calibration pipelines

Paranal and GarchingParanal and Garching► Science pipeline for full calibration of science data.Science pipeline for full calibration of science data.

CambridgeCambridge► Science archive acts as the point of access of the Science archive acts as the point of access of the

reduced data. Plus some further processing.reduced data. Plus some further processing. WFAU, EdinburghWFAU, Edinburgh


Data FlowData Flow

►Raw telescope data is assessed by the Raw telescope data is assessed by the summit pipeline (QC1)summit pipeline (QC1)

►Shipped to Garching (discs)Shipped to Garching (discs)►Shipped to CASU for science reduction Shipped to CASU for science reduction

and calibration. (discs)and calibration. (discs)►Calibrated data shipped to Edinburgh Calibrated data shipped to Edinburgh

for archiving (ftp)for archiving (ftp)


IR Data Reduction WorriesIR Data Reduction Worries

► IR detectors are currently inherently more IR detectors are currently inherently more unstable than optical CCDs. unstable than optical CCDs. Some odd electronic effectsSome odd electronic effects Much poorer cosmetic quality than CCDsMuch poorer cosmetic quality than CCDs

► Sky emission > 100x brighter than most Sky emission > 100x brighter than most objectsobjects And it’s variable both spatially and temporally!And it’s variable both spatially and temporally!

► Exposure times are short, so data rates are Exposure times are short, so data rates are very high.very high. 200-500 Gb/night expected for VISTA public surveys200-500 Gb/night expected for VISTA public surveys Rice tile compression can save factors of 3-4 in 32 bit Rice tile compression can save factors of 3-4 in 32 bit

integer datainteger data


VDFS Pipeline RecipesVDFS Pipeline Recipes

►Create master calibration frames (dark, Create master calibration frames (dark, twilight flats, confidence maps, etc)twilight flats, confidence maps, etc)

►Linearity analysisLinearity analysis►Detector noise & dark current propertiesDetector noise & dark current properties►Persistence and crosstalk analysisPersistence and crosstalk analysis► Illumination correction analysisIllumination correction analysis►Full reduction recipes for standard star Full reduction recipes for standard star

and programme fieldsand programme fields


Summit & Garching PipelinesSummit & Garching Pipelines

►QC1 parametersQC1 parameters e.g. photometric zeropoints, astrometric fit e.g. photometric zeropoints, astrometric fit

qualityquality

►Written using ESO qfits/CPL infrastructureWritten using ESO qfits/CPL infrastructure Both use the same software modulesBoth use the same software modules

►Amount of processing can be scaled downAmount of processing can be scaled down►Calibration images (flats etc) from a Calibration images (flats etc) from a

master library.master library.


Cambridge PipelineCambridge Pipeline

►Full reductionFull reduction e.g. better sky correction e.g. better sky correction

►Results shipped to WFAUResults shipped to WFAU►Some of the same CPL based modules Some of the same CPL based modules

reused. Some additional modules reused. Some additional modules required.required.


Current Status of VISTA Current Status of VISTA PipelinesPipelines

►Version 0.5 released to ESO in MarchVersion 0.5 released to ESO in March►Full compliment of recipes (almost!)Full compliment of recipes (almost!)►PAEPAE


The Decision To Use CPL (1)The Decision To Use CPL (1)

►We were given the choice.We were given the choice.►We already had advanced pipeline We already had advanced pipeline

software in our own environment for software in our own environment for WFCAM and other IR instruments.WFCAM and other IR instruments.

►First view of CPL and qfits was not First view of CPL and qfits was not favourable.favourable.


The Decision To Use CPL (2)The Decision To Use CPL (2)

►CPL v2.0 was a great improvementCPL v2.0 was a great improvement►CASU has an ‘ESO workstation’ to CASU has an ‘ESO workstation’ to

generate ‘data’ using the VIRCAM generate ‘data’ using the VIRCAM acquisition softwareacquisition software

► Integration into Paranal and Garching Integration into Paranal and Garching systems greatly simplified by using systems greatly simplified by using CPLCPL

18 April 200718 April 2007 Second Generation VLT InstrumentsSecond Generation VLT Instruments 1212

Things I wish I had Things I wish I had known…known…


My First Attempt At A RecipeMy First Attempt At A Recipe

► INPUT:INPUT: FITS image of WFCAM dataFITS image of WFCAM data Binary FITS table of astronomical objectsBinary FITS table of astronomical objects Binary FITS tables containing the 2MASS catalogueBinary FITS tables containing the 2MASS catalogue

► TASK:TASK: Work out RA, Dec coverage of image using header Work out RA, Dec coverage of image using header

FITS WCS keysFITS WCS keys Extract 2MASS standardsExtract 2MASS standards Cross match standards to object catalogueCross match standards to object catalogue Fit a WCSFit a WCS Write resulting WCS to FITS imageWrite resulting WCS to FITS image


World Coordinate SystemsWorld Coordinate Systems

►CPL has no WCS facilityCPL has no WCS facility But it soon willBut it soon will


Input File AccessInput File Access

►CPL doesn’t give write access to input CPL doesn’t give write access to input files.files. cpl_frame, cpl_image cpl_frame, cpl_image Data abstraction is not perfectData abstraction is not perfect cpl recipes can’t be written to update a cpl recipes can’t be written to update a

filefile


CPL vs FITS tablesCPL vs FITS tables

► CPL does not allow for direct manipulation of CPL does not allow for direct manipulation of FITS tablesFITS tables cpl_table structures can be created from a FITS table. cpl_table structures can be created from a FITS table. But one must read the WHOLE table even though you But one must read the WHOLE table even though you

might only be interested in a few rows or columns.might only be interested in a few rows or columns.► Tables with many rows take too long to readTables with many rows take too long to read

Break them upBreak them up► Tables with many character type entries cause Tables with many character type entries cause

the system to run out of memory pointersthe system to run out of memory pointers Remove unnecessary character columnsRemove unnecessary character columns


User DocumentationUser Documentation

►Reference manual, users guide, Reference manual, users guide, software developers kitsoftware developers kit

►User’s guide and SDK are helpful but User’s guide and SDK are helpful but limited in scopelimited in scope Very few examplesVery few examples Get a copy of a working CPL pipelineGet a copy of a working CPL pipeline


QCQC

►Garching and Summit pipelines generate QC Garching and Summit pipelines generate QC parametersparameters Provides data for QC trending to be done both on Provides data for QC trending to be done both on

site and in Garchingsite and in Garching The ‘ESO System’ is really two systemsThe ‘ESO System’ is really two systems

►QC passed through FITS headers (Garching)QC passed through FITS headers (Garching)►QC passed through PAF files (Chile)QC passed through PAF files (Chile)►Neither system appears to be documented.Neither system appears to be documented.►PAF files contents are not documented in CPL users or PAF files contents are not documented in CPL users or

reference manuals.reference manuals.►CPL has no standard class for passing QC parametersCPL has no standard class for passing QC parameters


File TagsFile Tags

►DPR TYPEDPR TYPE OBSTYPEOBSTYPE

► PRO CATGPRO CATG Product typeProduct type No two recipes can produce the same valueNo two recipes can produce the same value

►DO CATGDO CATG File type for files used as input to recipesFile type for files used as input to recipes Raw frame DO CATG values must be unique to Raw frame DO CATG values must be unique to

each recipe.each recipe. There are some rules on the values here. There are some rules on the values here.


Output File NamesOutput File Names

►Recipe creates and names output files.Recipe creates and names output files. File names must be hard coded!File names must be hard coded! Several files of the same PRO CATG must Several files of the same PRO CATG must

be indexed in a predictable way.be indexed in a predictable way.

►esorex can change the file nameesorex can change the file name


HeadersHeaders

►CPL cannot directly manipulate FITS CPL cannot directly manipulate FITS headersheaders These must be read into a cpl_propertylist These must be read into a cpl_propertylist

structure.structure.

18 april 2007 second generation vlt instruments 1 vircam & cpl: lessons learned jim lewis and...

Documents

x brighter

generation vlt instruments10

integer data slide

calibration of science

edinburgh slide

data rates

reduced data

pae slide