SKA/KAT SPIN Presentation

Software Engineering (!?)

2007-01-17

Robert Crida

Overview

Software Configuration Management (SCM) Automated Build System External Dependencies Design Study – Antenna Control Software Module

(ACSM)

SCM Challenge

KAT is expected to have a life span in excess of 20 years (project will outlive the people)

Maintenance is inevitable during that period Continuous development is also likely Importance over that period of being able to freeze

build environment What about hardware failure that far into the

future?

SCM

Use subversion – potentially multiple repositories (collaborations, different subsections)

Store multiple projects in a tree Standard svn hierarchy for each project

trunk (primary development) tags (releases) branches (qa, users, etc)

Try to include all dependencies as projects within the hierarchy

Subversion Layout Tools (used by the build process)

doxygen tags

doxygen-1.4.7 doxygen-1.5.1

etc (typically only store tags for each tool) Code (ultimately linked into products)

ThirdPartyLibraries (typically only tags) InternalProjectsHierarchy

Each layer can only depend on other projects in the same or lower layers, ideally wanted to only depend on projects in lower levels but deemed too difficult for users, instead have to work out build order within each layer.

Build system based on distutils Actually we use from recursivebuild import setup recursivebuild.setup delegates to setuptools.setup and

provides a convenient place to centrally add new commands (eg doc, pylint)

Use setuptools because of egg support Allows multiple builds of the same library to be present

simultaneously from pkg_resources import require require(“numpy==1.0”) import numpy

Most distutils projects easily converted by changing import of setup function

Python projects

Build System

Each directory of the hierarchy has a script to build its children script determines order passes arguments from parent arguments modeled after distutils.setup

Leaves are the projects themselves, must provide script understanding build targets Python projects use setup.py Other projects provide eg build.py to wrap make or

scons etc

Build system contd 1

All scripts must understand standard build targets install (more like build)

puts python eggs into suitable directory should compile and link other libs but not deploy

test (runs unit tests on the project) doc (runs doxygen on project) pylint (runs pylint on project – rename to lint!)

Build system contd 2

Build controller machine stores svn repository mirror plus overall build script

Build environment to be configured on a virtual machine, currently requires only gcc and python

Virtual machine is cloned prior to each build svn repository mirror is updated and copied to the

build vm Multiple build steps are executed remotely on the

vm by the controller

Build system contd 3

This allows multiple vm's with different compiler versions and even OS versions

Each can be built using (probably) the same control script

Testing

Integrate unit testing based on xUnit into the auto-build

Investigating Fitnesse based testing for automated ATP type testing

Looked at web testing technologies such as Twill and selenium to assist with automated web app testing

Dependencies

Python projects can depend on other python projects using the egg mechanism, hence can just find suitable egg in standard PYTHONPATH

Depend on other projects by specifying paths within the hierarchy multiple versions of a lib can coexist requires extensive include and lib paths use $PROJECT_ROOT environment variable instead of

relative paths due to svn moves (different nesting levels)

External Dependencies Try to minimize external dependencies

all need to go into SCM increases code-base (potential maintenance problem) traded off against leveraging existing code

AIPS++ many dependencies hard to build – difficult to configure tools constantly changing

Use each one's own build system must configure appropriate include and lib paths

External Python Dependencies

Eggs help a lot – in theory Despite distutils, many python packages have non-

standard builds which makes life difficult Haven't yet found a python project which defaults

to using setuptools