nhsc/pacs(web(tutorials running(pacs(photometer(pipelines...
Post on 09-Aug-2020
33 Views
Preview:
TRANSCRIPT
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
NHSC/PACS Web TutorialsRunning PACS Photometer pipelines
[PACS-‐401]
Level 2.5 Map-‐Making with MADmap
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
IntroducMon
• This tutorial provides will walk you through Level 1 to 2.5 processing using the MADmap branch of the PACS photometer pipeline.
• The tutorial follows the ipipe script: L25_scanMapMadMap.py
• At the end of the tutorial, you will have created a PACS map from calibrated bolometer readouts at Level 1 using the opMmal map-‐mapping algorithm MADmap.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
What is MADmap?
• MADmap paper: Cantalupo et al. 2010, ApJS, 187, 212.
• It implements a least square soluMon of the projecMon inversion problem.
• It makes use of noise properMes to separate sky signal from 1/f noise.– It assumes the covariance matrix of the noise is diagonal in the Fourier
space.
– Actual noise does not really comply with this, pre-‐processing will be required.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Pipeline secMon covered here
Level 0 container
PACS data, House Keeping
PoinMng
Assign poinMng
Masking: bad/dead pixels, glitches, saturaMon
Data Reorganiza<onLogical blocksMme stampsUnit conversion
Chopper ProcessingAssign offsets on skyFlag “moving” frames
Distor<on correcMon
Level 0.5Minimally processed uncalibrated data cubes
IniMal Processing
Chopped AOTs
Combine & Subtract ON-‐OFF beams
Correct Flat Field
Flux Calibrate
Mapping AOTs
High Pass Filter
Correct Flat Field
Calibrate
Project on sky
Subtract offsets
Correct Flat Field
Calibrate
OpMmal Map Making Algorithm
Combine individual beams
(a) (b)
Level 1 cube
Level 2 images
Level 1 cubes
Level 2 images
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
DocumentaMon Reference
• PACS data reducMon guide, chapter 7
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Processing Overview
Step 1Check script and so2ware pre-‐requisites
Step 2Loading ipipe script L25_scanMapMadMap.py
Step 3Pre-‐amble and script parameters
Step 4IdenFfy the data to process
Step 5Making sense of the main processing loop
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Processing (Cont.)
Step 6MADmap pre-‐processing (post Level 1)
Step 7Remove correlated signal dri2s
Step 8Re-‐visit Deglitching
Step 9Select which frames to use for mapping
Step 10 Create MADmap ToD product
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Processing (Cont.)
Step 11Create the “naive” and opFmal maps
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 1Check your software version
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Check # 1: HIPE 6.0 build ....
From the “Help” menu, select “About”
Which shows the HIPE version
If your Build number does not start with 6.0, stop and upgrade.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 2Load ipipe script
“L25_scanMapMadMap.py”
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 2: Load ipipe script
From the pipeline menu, make the selections as shown to get to “L25_scanMapMadMap”
If you successfully loaded the script, it’ll appear as a folder tab under the Editor window.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 2: Warnings
• You should always save the template ipipe script under a new name before making changes to prevent accidentally overwriMng and destroying the original templates.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 3Pre-amble and script parameters
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
The preambleHighlight and execute the block of import and definition statements with the single green arrow.
The ini?al lines (57 in the case of example shown) are comments and may be skipped.
The import statement define java classes and func?onal blocks to be used later.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Preamble (cont.)
Replace these lines with this one, which selects the red (long-wavelength) PACS channel.
Execute this statement by positioning the marker next to the statement and clicking on the single green arrow.
The next set of lines define the PACS bolometer channel to work on.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Processing Parameters
The next segment sets the run-time parameter values. These are discussed briefly in the next slide.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
ParametersParameter Descrip<on Recommend Value
Verbose Print processing step details. True
doplot For signal drii correcMon, show the plot of best-‐fit model.
True
Outdir Where to save the output from ‘doplot’ and final maps and reduced data.
A valid directory on your system.
globalDriiModel OpMon for drii correcMon. See step 6. 1
ModelOrder Polynomial order. See step 6. 2
ignoreFirst Mask and remove this many readouts from the start of the observaMon
460
noDeglitching do not deglitch the data? False
unmaskSources Obsolete parameter. Ignore. False
useIIndLevelDeglitching Which deglitcher to use? MMT or IInd level. True
envSize Envelope size for IInd level deglitcher. 10
nsigma Glitch rejecMon criteria 30
scales Parameter for MMT deglitching 1
mmt_mode Parameter for MMT deglitching ‘mulMply’
Incr_fact Parameter for MMT deglitching. 2
unMaskRad Obsolete parameter. Ignore 20
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
The parameters aier the recommended edits.
Highlight and execute this stanza after editing the values.
WARNING: The directory specified in ‘outdir’ must already exist on your system.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Check # 2: Preamble okay?
Execute the following line either by typing in console or editing your script.
print verbose doplot ignoreFirst noDeglitching
The output should contain the values set in Step 3.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 4Identify the data (scan and cross-scan) for
processing.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 4
Add, highlight and execute the following lines in your script.
The scan and cross-scan OBSID pairs are identified in a jython vector. The values are long integers.
pooldir is the full path name to where the data pools are stored. It can be ignored if the data has been placed in your default local store, as defined in the HIPE properties.
The nested getObservation commands in the last line read and populate a vector of observation contexts.This identifies your data to HIPE.
To explore an observation context, try printing it!print obsList[0]
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Check # 3: Data is properly loaded
Highlight and execute the following lines in your script.
If there was an error, you (likely) already got a notification. This step ensures that there are no problems with the pools themselves.The output should be the programmed values for the right ascension and declination of the object in the first OBSID.
Shows the intended pointing as found in the metadata.The metadata holds a wealth of information. Try print obsList[0].meta
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 5Making sense of the main processing loop
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
The Processing Loop
A. For each observaMon in your scan and cross-‐scan pair, the following processing steps are executed:• Step 6: Post level 1, MADmap pre-‐processing.
• Step 7: Remove correlated signal drii.
• Step 8: Re-‐deglitch the Level 1 product.
• Aier the processing steps, on the first pass through the loop a super-‐frames object is created.
• On the next pass the cross-‐scan data is appended to the super-‐frames object.
• At the end of the loop, we have a frames object that contain all the observaMon data.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
The Processing Loop
A. Step 6
A. Step 7
B. First pass
C. Second pass
Is this the first time through?
Next pass is not the first one.
See previous slide for explanation of A. B. C.
A. Step 8
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
ExecuMng the loop
Highlight the section of the code shown and execute with the single green arrow.
You are encouraged to read about the steps performed in the loop in the next few slides before executing the loop.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 6Post level 1 MADmap pre-processing
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
MADmap preprocessingExecuting the loop will automatically execute these steps for both OBSIDs.
These commands pull level 1 data out of the observation context and the associated calibration files.
Cleanup unstable frames at the start of observation.
Assign pointing to each and every pixel and readout
Apply pixel-to-pixel electronic offset correction.
Which Blue filter?
In the data distributed for the workshop, the calibration data is not present. Replace this line with:calTree = getCalTree()
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Your output will likely look slightly different but you should NOT get an error message and the important “ra” and “dec” datasets exist in your “frames” object.
Issue this command in the console window
Look for “dataset”s Ra and Dec
Check # 4: PosiMon cubes exist
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Check # 5: Offsets are removed
Display( frames.signal[:,:,100] )
Expected Output:With proper offset removal, the image shows a rela?vely constant signal. Note: extremely bright sources may also be observed on a single image.
Type this command
An example of improper or no pixel-‐to-‐pixel offset correc?on.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 7Remove Correlated Signal Drifts
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
PACS’ correlated signal driT.
Background
This Figure illustrates what is meant by both correlated and driT for PACS signal. The Figure shows the median value of the bolometer array as a func?on of readout index. The monotonic signal showing a decay in intensity is commonly observed in PACS’ image cubes, and is thought to be related to focal plan temperature driTs.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Background
Mi?ga?ng the signal driT.
Divide the array median in bins of N readouts. N is typically 1000 readouts.
Take the minimum value in each N readout bins.
Fit the resul?ng curve with a polynomial.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Background
If the sources are weak (i.e. do not produce significant signal in a single image) it may be sufficient to fit the median values directly. However, for strong sources, the minimum approach becomes necessary.
An observa?on with strong sources. The minimum values s?ll manage to trace the overall driT fairly accurately.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
The drift correction is automatically applied to the data when the main loop is executed.
Step 7
The photGlobalDri-Correc0on module allows several op?ons for fi[ng and removing the driT. See: PDRG chapter 7 Or type Print photGlobalDriTCorrec?onIn the HIPE window.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
The most important parameters
model=1This is the default and uses the minimum of the bins as discussed above.
polyOrder=3Sets the order of the fiZed polynomial
binSize=1000Sets the size of the bin from which the minimum value is determined.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 8Re-visit Deglitching
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Re-‐Deglitch
Why redo the glitch removal?
By default, the wavelet based “MMT deglitcher” is used by the standard pipeline. One issue noted for this deglitcher has been that the cores of bright point sources, or indeed any PSF-‐like structure is incorrectly iden?fied as a glitch and (usually) re-‐interpolated by MADmap using surrounding values in the ?me-‐line. This leads to “shadow” ar?facts in the final maps.
Instead, we introduce a 2nd level deglitcher which uses sigma-‐clipping algorithms to remove glitches.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 8: details
Was MMTdeglitching done?
By serng noDeglitching to ‘True’, you can unmask all glitches. This parameter in Step 3.
Also idenMfied in Step 3 is whether to use 2nd level sigma-‐clipper deglitcher.
Execute the 2nd level deglitcher. The following condiMons have to be met:nframes < the limit shown. The default limit of 50000 appears here because larger data require very large amount of RAM. Before serng this to a larger value, make sure your HIPE session has adequate memory allocaMon.useIIndLevelDeglitching = True (Step 3)noDeglitching = False (Step 3)
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 9Select which frames to use for mapping
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 9
Select and execute one of these options.
Including the turn-‐around frames provides more data at the edges of the map; however, the coverage is also sparse and may result in non-‐op?mal solu?on with MADmap.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Check # 6: On-‐Target flag is
Type this command
PlotXY( Int1d( frames.getStatus(“OnTarget”) ) )
Expected output. Either all values of OnTarget flag are True (for op?on #1), OR
The edges of each scan legs are set to false (op?on #2)
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 10Create MADmap ToD product
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 10
Add the parameters shown, Select and execute this block of commands
The ToD stands for Time-‐ordered-‐Data and is the internal format used by MADmap.In fact, makeTodArray will create a binary file in your temporary area that has the rearranged PACS signal in the proper format.• The scale parameter selects the size of the output sky grid rela?ve to the ideal PACS pixel sizes. E.g. scale=0.5 for PACS blue channel will result in final pixel sampling of 1.6”/pixel.• Crota2 is the FITS keyword crota2 of the final map.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Check # 7: TesMng the ToD
Print the WCS associated with this time-ordered-data (ToD) product.
Expected output: You should see some familiar FITS keywords. You can also check to make sure that crval1 and crval2 values are consistent with your object.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 11Create Naive and Optimal Maps
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Step 11
Documenta?on Reference:PDRG Chapter 7
Both the naive and op?mal maps are created with the same call. The last parameter is set to ‘True’ for naive map and ‘False’ for op?mal map.MADmap uses maximum likelihood and conjugate gradient solvers to find the op?mal solu?on. The parameters maxRelError and maxItera?ons control both the convergence tolerance and the number of itera?ons in finding the op?mal solu?on. See the above reference for details.
Select and execute this block of commands
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Check # 8: Output map
print madmap
Issue this command in the console window
Expected output: The output from madmap or naivemap making is a simpleImage product class with several datasets.
NHSC PACS Web Tutorial
http://nhsc.ipac.caltech.edu/helpdesk/index.php
Check # 9: Display the final map
Issue this command in the console window
Display(madmap)
Expected output: A mosiac of all images in your PACS data cube.
top related