CRUSH

IMAGES

OPTIONS

@350 um Atmospheric Background is 107 times brighter than faint galaxies.Analogous to Observing a ~16 Magnitude star at daytime!

The Submillimeter Challenge

Turnover to white noise regime at ~ 10 Hz

The Atmospheric Power Spectrum (5 January 2003)

Strong 1/f characteristic

Need Fast Sampling of Background(SHARC-2 has samples every 36ms)

Differencing of Signals

100 Jy source

Lissajous Sweep

Chopped Image has Limiting noise typically 2-3 times higher than 'state of the art' reduction.Not including deconvolution noise!!!

Simulated 4 Hz chopper with 40” throw under better than average conditions

Vesta ~5 Jy(2 min)

Blank Sky(10 min)

Chopped

Residual Sky Noise

The Need for Speed (Sweeping)

Moving Several pixels onto the same sky position within the limiting instrumental time scale 'calibrates' pixels against one another.

The more pixels that can be thus related, the more robust the 'calibration' measurement.

'Calibrated' pixels can observe several positions on the sky within the limiting time scale, leading to high fidelity maps.

Faster Better pixel-to-pixel calibration

Larger and higher fidelity maps

Smarter and Better Crossing Sweep Patterns

Non-periodic source crossing

Move Primary to avoid changing illumination pattern

Scanning Strategies without a Chopping Secondary

For compact and point sourcesMaximizes time coverage over a small area.

For large map making. Obtains uniform coverage over an area much larger than the array

From Chopped Data to Discreet Modeling

Singular Value Decomposition

Difficulties with SVD

Computationally costly. (Large Matrices to invert)

Non-linearities. (Gain fitting).

Degeneracies, Singularities and Constraints

Time dependent noise

Mathematically Rigorous Maximum Entropy Solution.

Parallel SVD Effort at Goddard Space Flight Centre

Model 1

Final SourceModel

Model N

Model 2

Iterative Reduction

Series of Maximum Likelihood Estimators.

Brightest First

Convergence via Iterating

Model 3

Advantages

Intuitive

Fast. (Linear with Data Size)

Able to Deal with Non-Linearities

Easily Configurable / Changeable

Instrument Specific Models

=

Static Residual Pixel Offsets~ 2000 Jy

Static Residual Pixel Offsets~ 2000 Jy

Acceleration Response~ 0.2 Jy

Detector 1/f Drift Model~ 1 Jy

Electronic RowDrifts ~ 1 Jy

Source Model~ 5 Jy

Vesta 82935 January 2003Excellent Conditions

The Format of the SHARC-2 Array

32 x 12 pixels. Nearly Nyquist Sampled.

Maximum Likelihood Estimators

Alternatively, in terms of the Residuals:

As if residuals only contained given model...

CRUSH Models with Maximum Likelihood Estimators

Model Name

Residual Pixel Offsets

Correlated Background Noise

Gain Modelling

Pixel Weights

Chopping Residuals.

Temperature Gradients.

Electronic Row Drifts

Detector 1/f Drifts

Time Weights

Residuals Spikes

Regional Correlations

Acceleration Response

Temporal Features

Spectral Features

Typical Flux

1,000 – 10,000 Jy

10-1000 Jy

~ 10 Jy

1-10 Jy

~1 Jy

~1 Jy

~10 mJy

~10 mJy

Typical Time

Scale

scan

10 min

scan

scan

1 chop*

1 frame

1 second

~30 seconds

~1 second

0.1-1 second

scan

all time scales

all time scales

Typical Spacial

Scale

pixel

array

pixel

pixel

pixel

array

row

pixel

array

4x4 to 8x6

pixel

pixel

pixel

Reduction Goal

Simulated source with only white noise – this is the best any analysis could achieve

Simulated Raw Data (1/f correlated)

Partial Cleaning (50 iterations)

Deep Cleaning (200 iterations)

SHARC 1.5 Uranus Raw (Dowell)

SHARC 1.5 Uranus Reduced (Dowell)

Preliminary Simulations for the Iterative Reduction Method (Feb 2001)

SHARC 1.5 – single row of bolometers. Edge pixels used for estimating correlated noise.

Simulations with CRUSH

Source Fluxes Recovered within 1%

100 mJy Ring surrounding compact star in 1 hour10' x 10' Billiard Ball Scan.

Imperfect cleaning offaint large scale structures

100 mJy CompactLissajous Sweepin 1 hour

Clean background on small scales

Tom Tyranowski

Correlated Noise And Gain Fitting

Non-Linear Response

where,

Small signal gain

Large signal gain

Weighting

Weights Separated into a product of pixel-only weights and time-only weights.

Where B is the number of Active Bolometers, and P is the number of Parameters fitted in the time interval T'

Where P is the number of fitted parameters in the time interval T

Identifying Anomalous Pixel Behaviour

One of the Most challenging aspects of deep reductions

Removal of Residual Spikes

Flagging of Pixels with Unreasonable Gain Fits

Looking for Statistically Significant Temporal Features...

... and Spectral Features

About 300 of 384 pixels are good enough

The Lockman Hole z ~ 2-3

Follow-up of SCUBA galaxies

4 Fields, 3-4 Hours of Grade I Each

1-sigma of ~ 5mJy depth.

3 of 4 SCUBA galaxies detected

Additional 7 objects detected above noise

SHARC-2

Map Noise Characteristics & Detection Confidence

Perfectly Gaussian Noise

~2 Times wider than expected statistically from independent pixel noise! Detectors are NOT independent!

Positive tail Clearly indicating the presence of source flux.

Residual Pixel-to-Pixel Covariances(Learning from the Data Themself...)

Gaussian with ca.35” FWHM

Principal Options to CRUSH

Reduction Options

Scan Specific Options

Model Specific Options

Source Map

Gain Modelling

Frame and Pixel Weighting

Row Model

Detector Drift Model

2-D Gradient Model

Residual Spike Removal

Temporal and Spectral Feature Identification

Chopping Residual Model

Acceleration Response Model

Options to CRUSH

crush [options] -help

http://www.submm.caltech.edu/~sharc/crush/glossary.html

A concise listing of ALL available options to crush, including their current settings.

A detailed explanation of the options, with corresponding configuration keys, and valid arguments. Cross-referenced and easily searchable. The true “CRUSH User's Guide”.

Principal Reduction Options

Brightness Related

Miscalleneous

< 1 Jy

< 100 mJy. Similar to -faint -compact, but more aggressive.

-faint

-deep

EXTENDED_STRUCTURE = false-compact

LOAD_CONFIG = faint.cfg

LOAD_CONFIG = deep.cfg

Do not worry about filtering extended structures on the scale of the array. Instead clean aggressively...

Size Related

-rounds=

-point

-config=xxx

-altaz

ITERATIONS

LOAD_CONFIG = point.cfg

LOAD_CONFIG = xxx

ALTAZ = true

The number of iterations before final solution.

Perform pointing after reduction.

Load setting for confguration file xxx.

Reduce maps in Alt/Az coordinatesinstead of the default RA/DEC.

Scan Specific Options

Preconditioning of Scan data

Location

The number of 36 millisecond frames to integrate data over. E.g. -average=3.

If a chopping secondary was used.

-average=

-chopped=

-scale=

-FAZO=-FZAO=

-tau=id:value

AVERAGE_FRAMES

CHOPPED_OBSERVATION = true

Scale the scan data by this factor. E.g. -scale=1.23 13852 13853

Adjust the pointing. E.g. -FAZO=-103.0 -FZAO=12.7 9712

Change the tau value to use. E.g -tau=225GHz:0.046 15224

Adjustments

-path= RAW_DATA_PATH Specify the directory where the raw data resides. E.g. -path=/home/nobody/data/SHARC2

Options affect all scans that are subsequently listed.

Generic Options

Activation Iteration

Specify when a given model is to be activated. Several models have 'auto' setting available. When specified each model will evaluate a number of possible criteria, based on which it activates or delays. One can also explicitly activate a model at the given iteration.

-Ixxx=

XXX_T =

-xxxT=

XXX_TURN

Characteristic Time Constant

Several of the models have time constants assigned to them. Most commonly they control the time resolution of the given model. The longer the time constant, the less often a parameter is fitted for the given model.

Large ~ RobustSmall ~ Aggressive

Options to Source Map

Pipeline Auto Configure Related

Do not allow self-activating models to activate until the given iteration. Also, if negative clipping is enabled, clip images until the specified iteration. E.g. -Ifidel=3

If EXTENDED_STRUCTURE is set true, do not self activate models that may remove extended structure until the specified number of source generations is obtained. E.g. -Iextended=4

-Ifidel=

-Iextended=

MAP_RELATIVE_EXPOSURE =

CONVOLVING_BEAM_FWHM =

-minExp=

-convolve=

MAP_FAITHFUL_TURN

EXTENDED_STRUCTURE_TURN

Map Appearance Related

Clip noisy map edges. Use exposure time relative to peak exposure to define clipping criterion. E.g. -minExp=0.1

Convolve map to beam size to get rid of unwanted high spacial frequency structure. E.g. -convolve=4.0

Gain Fitting and Gain Adjustment Options

Gain Related

The number of initial gain iterations. Like the full pipeline iterations, except only gain and models preceding gain are solved for.

The desired gain convergence if GAIN_ITERATIONS is set to 'auto'. The fainter the source, the better the convergence that is required.

-gainRounds=

-gainGoal=

TAU_ADJUST=true/false-tauAdjust-notauAdjust

GAIN_ITERATIONS

GAIN_CONVERGENCE_GOAL

Fast Line-of-Sight Opacity Adjustment

Allow/Disallow for fast (real-time) adjustment of the extinction around the mean extinction value (explicitly defined or obtained from Mai-Tau) base on the background power variations.

Weighting and Flagging

Weight Related

Set the time scales over which pixel weights are to be determined. Max should be set such that it is smaller than beam crossing time.

-pWeightT=min-max

MIN_DEGREES_OF_FREEDOM

DESPIKE_LEVEL

REJECT_SPIKE_FRACTION

MAX_TEMPORAL_FEATURE

MAX_SPECTRAL_FEATURE

-minDOF=

-spikeLevel=

-spikeRatio=

-maxTemporal=

-maxSpectral=

PIXEL_WEIGHT_T

Flagging Related

The minimu degrees of freedom that must remain per pixel for the pixel to remain active.

Remove residual spikes above the specified significance level.

Flag pixels that have a higher fraction of spikes to data point.

Flag pixels with temporal features above the spec'd significance level.

Flag pixels with spectral features above the spec'd significance level.

Output Related Options

Gain Related

-outpath=

-name=

-precess=

-resolution=

REDUCED_MAP_PATH

REDUCED_MAP_EPOCH

MAP_RESOLUTION

The output directory in which to place files that are automatically names as [SourceName].[Scan1#]. ... .[ScanN#].fits.E.g. -outpath=/home/nobody/reduced

The name of the output file including absolute path. E.g. -name=/home/nobody/reduced/mysource.fits

The coordinate epoch in which the reduced map is written. CRUSH does precession!!! E.g. -epoch=1950.0

The map resolution. The setting 'auto' defaults to 1/3 Cassegrain detector pixel size, ~1”.62, but one can set this to any desired value really.

CRUSH Tips

Reduce as much data at once as you possibly can. Coadd later. Increase the memory allocated to the Java VM.

Reduce Data together that were taken with different Position Angles and Scanning Angles.

If default reduction is 'funny'.

Try different brightness. (-bright, -faint, -deep)

Adjust gain convergence criterion.

Try -compact if emission is not on the scale of the array.

Adjust specific model time-scales.

Try to identify problematic scans. Act on them!

Trade Extended flux for Flatter Baselines.

Check on Pointing.

Use Mai-Tau.

Observing Tips

Choose Appropriate Scanning Pattern

Size (smaller patterns yield cleaner data)

Coverage (Lissajous vs Billiard Ball)

Speed (Faster = Better)

Orientation (not along rows!!!)

Feasibility.

Chopping / Not Chopping...

Point Often on Nearby object (esp. in ZA)

Calibrate Often on known calibrator

Check on Beam Quality every night

Use DSOS (Dish Surface Optimization System)

Keep Detailed Logs

The Data Structure

* Free FITS viewing software FV

Primary Image: Flux Distribution

“Measurement Flux”As would be seen by detector if there was no atmospheric absorbtion Natural units are Voltage (nV, V)

Pseudo flux units(Jy/beam, Jy/sr, Jy/arcsec**2)

Response to 1 Jy source

Aperture Flux

For some Aperture

With default crush pixelization(1/3 Cassegrain Pixel Size)

Peak Flux – (Flux Inside a Beam)

Where, the typical 350um valuesfor SHARC-2 are,

Smoothing with a Beam

Define Smoothing as

Where, the a typical smoothing beam will be a Gaussian of the form:

Second Image: RMS

“Measurement Uncertainty”Uncertainty of the Detector Measurement

“Map Uncertainty”The Uncertainty of the Flux value on the map.

For default pixelization...

...and Gaussian Beam

Flux Uncertainty inside an Aperture

With default crush pixelization(1/3 Cassegrain Pixel Size)

Excess Noise and Non-Independent Pixels

Correlated Pixel Noise

Some Linear Quantity A

The Uncertainty on A

Excess Noise Determination

From Map Chi-Squared (No source)

From Map NoiseDistribution Gaussian Fit upper limit

Crush Suite of Utilities

imagetoolAiming to be a comprehensive image manipulation tool. Scaling, rms scaling, map units, regridding, smoothing, filtering and clipping. Under development...

show (via imagetool)Display tool that allows quick view of images with some limited capabilities such as PSF fitting, image toggle, units on the fly etc.

coaddProduce maps out of multiple images.

jiggle Shift maps on the fly to determine alignment

covarseeA visualization tool for pixel-to-pixel covariance matrixes.

histogramMap signal-to-noise Histrogram generator. Useful for determining excess noise + quick diagnostic

deconvolveObtain super-resolution. Undocumented...