development status and priorities jeff kern nrao

Development Statusand Priorities

Jeff KernNRAO

CASAInterferometry Progress• Re-architecting of Clean task implementation to tClean

– MT-MFS with mosaics, AW-Projections, parallelization– Available for use, many “nice” features still to be added– Used in ALMA imaging pipeline

• OTF calibration (cal library)– Supported and well tested in plotMS– Support in tasks implemented, needs to be

commissioned– Not yet hooked up to tClean

• MPI Based parallelization– Fully available and documented for calibration (see talk

tomorrow).– Easier integration with most scheduling systems

2015 CASA Users Committee Face to Face 2

CASAInterferometry Progress (2)• MS Transform

– Single location for many similar tasks (deprecating old versions)

– Fully MMS aware (parallel where available)• Robust statistics algorithms in imstat and ia.statistics

– Several algorithms• Channel based weights, weight handling defined, and tools

for weight manipulation.

• Linear mosaic (lm) tool to mosaic weighted images– Important for VLASS

• msuvbin – UV plane accumulation – Designed for CHILES project, allows managing very large

data sets


CASASingle Dish Progress• MS transition (since 2014 Q4)

– All highest priority capabilities, i.e., ones used in standard manual and pipeline data processing of ALMA-TP, have already been implemented. (But the standard data processing of ALMA-TP is still based on ASAP)

– 21 out of 58 items are implemented in CASA 4.5 release

• Support ALMA-TP data processing: manually processed data started to be delivered in Aug. 2015Bug fixes and minor enhancements in ASAP to process Cycle 1 and 2 observation data

• Support of New multi-beam receiver, FOREST, on Nobeyama 45-m

– Bug fixes in ASAP to support multi-beam data and enhancements for FOREST

– Currently, a dataset should be split by beam for proper imaging.


CASAPipeline Progress• Successfully supported ALMA Cycle 2 operations.• ALMA SD Pipeline acceptance• ALMA Imaging Pipeline

– Continuum Image– Per SPW continuum subtracted cubes

• VLA Calibration Pipeline in operations• Parallelization

– Implemented in pipeline, not yet deployed– See talk tomorrow

• ALMA-QA0 Flagging Implemented– Additional online flags applied for ALMA


Cycle 2 Pipeline Processing

Success w/ Additional Flagging

Tweaked

SuccessManual

CASABuild and Test Progress • Support for OSX 10.9 and 10.10• Daily automated test versions with automated tests on all

four platforms– Stables (those that pass tests) are made available to the

community. 89 stable versions available during 4.5 development cycle

• Integration with github::CASACore project– Unified casacore with rest of community

• Normalization and standardization of build process across OSX and Linux

• libsakura integration– High performance vector processor library developed at

NAOJ• C++11 Support in compilers

– Allows removal of custom solutions: container classes, smartpointers

• MPI Integration


CASAMisses (Things we didn’t do).• Updates to FFTW (Done, but to late for 4.5 release)

• Complete switch of calibration to new IO infrastructure layer– Full scratchless operation in calibration (done, but needs

validation)

• Rewrite of visstat– Scope increase (support Single Dish) and new developer. 4.6

• tClean:– Transition to new IO infrastructure (scratchless operation)– Improved image weighting (fixes interpolation issues)– Interactive interface improvements (descoped early)– automasking– Parallelization (Done, but needs validation)


CASAMisses (Things we didn’t do).• SetJy

– Full reuse of imaging infrasturucture

• StatWt improvements

• Enhanced RFI mitigation in RFLAG (requirement mismatch)


CASAInterferometric Priorities

• Reliability and Stability Initiatives– Testing

• Enabling and requiring tests at the C++ module level– Technical Debt

• Continued migration to standard IO routines• Namespace cleanup, removal of bespoke code

– Obsolete Code• Removal of unused or unsupported options• Consolidation of routines to use a single pattern for

accomplishing the same thing.


CASAInterferometric Priorities• Close out of ongoing initiatives

– tClean– Scratchless operation in Calibration, testing and

perfecting• Some “behind the scenes” changes for

maintainability– VisStat

• Speed, parallelization, and support for large data• Robust estimation

– Scratchless operation extended to tClean• No more applycal?

• New initiatives:– Performance effort: Particularly looking at low level IO

access patterns


CASAPipeline Priorities• ALMA:

– Implement antenna outlier amplitude vs time flagging heuristic

– ALMA imaging acceptance (and improvement)– HPC in production– Sessions (?) Multiple contiguous executions of the same SB

• Sharing calibration across exectuions• VLA

– VLASS specializations (OTF Mosaic support)– Spectral calibration support

• Both– Low signal to noise ratio heuristics– Polarization calibration– Science Ready Data Products


CASASingle Dish Priorities• Complete the MS transition project

Aim finishing migration of most of the items by the end of 2016• Support ALMA to start using the new single dish module

based on MSesThe new single dish module is ready for use for ALMA-TP. Need to update manual reduction scripts and pipeline

• Support ALMA Cycle 4 TP observationsMajor new capabilities are fast-scanning observations of the Sun and observations of multiple science targets by an SB

• Support Nobeyama start using CASA as the primary data reduction softwareNobeyama decided to adopt CASA for new backendsPreliminary schedule: 2016 Q4 - the observatory test use of CASA, 2017 Q4 - observatory start supporting reduction by CASA


CASABuild and Test Priorities• Add support for more Oss

– OSX 10.11 (10.9 going away!)– RHEL 7– Ubuntu support (MeerKAT?) Resources permitting

• Transition to Git as version control system for all of CASA– Investigating more integrated tool chain

• Automation of new tests under development by development team– Introduce and track test coverage metrics

• Addition of tools to enforce Software Quality initiatives• Implementation of framework to support user

documentation effort


CASA2014 CUC Report RecommendationsSince speed and ease-of-use are the main laments of CASA users, we recommend to emphasize improvements in performance and parallelization, and to improve the overall robustness of basic operation.The CASA HPC framework is now in place and available for general use. Parallel imaging is implemented and tested in pipeline context but not yet undergone full validation. Scratchless operation is available in calibration, plotMS, and MSTransform, but not yet in imaging.

Additional attention to setting user expectations may also help ameliorate these issues, such as (crude) time estimators for critical tasks. This has not yet been addressed, it is on the list of remaining imaging re-architecting tasks.


CASA2014 CUC Report RecommendationsComplete the CLEAN refactor with high priority, as this will bring important capabilities as well as retire some speed and usability issues. The first version of this is now available in CASA 4.5

Improve and curate better CASA documentation; a small number of well-maintained documents is preferable to many that are not maintained and become out of date. See discussion in talk on documentation

Provide updates to CASA users about ongoing and planned developments, as well as prioritized fixes, in a regular and easily digestible form. The first CASA newsletter was published in April, the second is in prepartion and will be out this week or next.

• Improve feedback to users on helpdesk/ticket disposition, especially where challenging problems may take a long time to address, or will not be addressed.

• User surveys can uncover problematic issues and help to guide future development; we would like to work with the CASA team to develop and deploy more user surveys.

• As there always will be a longer wish list of CASA improvements than available resources can support, we recommend to engage the external community by stabilizing and documenting interfaces and import/export formats to better enable users to contribute and to maintain CASA tasks and scripts.

• Continue to promote CASA for telescopes other than ALMA and the VLA on a best effort basis; major initiatives formulated by external groups should be undertaken only if the additional resources required are provided.


CASA2014 CUC Report RecommendationsImprove feedback to users on helpdesk/ticket disposition, especially where challenging problems may take a long time to address, or will not be addressed. We have created a web page to address this, some of the CASA management consultant recommendations are focused on improving determinism for development.

User surveys can uncover problematic issues and help to guide future development; we would like to work with the CASA team to develop and deploy more user surveys. First user survey was conducted and a response has been prepared (see upcoming newsletter). Further details in discussion on user survey.


CASA2014 CUC Report RecommendationsAs there always will be a longer wish list of CASA improvements than available resources can support, we recommend to engage the external community by stabilizing and documenting interfaces and import/export formats to better enable users to contribute and to maintain CASA tasks and scripts. We have addressed a number of known issues with the uvfits import/export tasks. Documentation was discussed earlier. The transition to an enterprise Git implementation will allow more dynamic interaction with external developers.

Continue to promote CASA for telescopes other than ALMA and the VLA on a best effort basis; major initiatives formulated by external groups should be undertaken only if the additional resources required are provided. Under these auspices we have entered into a collaboration to support the ACA reduction with CASA. We are working to incorporate resources from other observatories in the NRAO helpdesk to assist in the support of CASA.


development status and priorities jeff kern nrao

Documents

alma cycle

cubesvla calibration

multibeam data

development cycle integration

test progress support

large data sets

support single dish

smartpointersmpi integration