1

SM

S M

anag

emen

t & T

echn

olog

y

Reviewing the ATW Wizard’s Past and Considering its Future

Adrian Dormer14th October 2009

2

SM

S M

anag

emen

t & T

echn

olog

y

Abstract

This presentation’s purpose is to review the track record of developing and maintaining Stanwell Corporation’s ATW Wizard IT system from a business perspective, and to consider the variables and scenarios that may lead to rebuilding ATW at some stage in the future

3

SM

S M

anag

emen

t & T

echn

olog

y

Agenda

1. “ATW Wizard Review” – what was the point?2. A review of the ATW development “journey”3. Total cost of the ATW Wizard so far4. What has been learnt? Considered?5. Should the Wizard be rebuilt?6. Wizard Cost Scenarios7. Where does the “ATW Wizard Review” fit in?8. Justifying the expense of a rebuild

4

SM

S M

anag

emen

t & T

echn

olog

y

“ATW Wizard Review” – what was the point?

SMS was asked to look at the ATW Wizard from a technological perspective, give a health-check and report any problem areas

The “ATW Wizard Review” found the system poorly designed – it evolved from the ground up as opposed to being designed from the top-down – and highlighted numerous problem areas

What value was this review document to Stanwell as a business? What business value was gained? What was the point?

5

SM

S M

anag

emen

t & T

echn

olog

y

A review of the ATW development “journey” Development of ATW began in November 2004 after roughly 6 months

of general drafting and planning by Nathan Liebke working full-time and Greg Carlon and Peter Eckersley acting as consultants

During development, requirements grew. And grew. And grew. Resources were added & more funding given to the project

A pilot release was adopted by Kareeya on September 25th 2006 After nearly 3 years of development, a significant blow-out beyond initial

scope, a version of the Wizard was rolled-out (Production) on April 17th 2007. Stanwell’s 2 hydro plants began use of the Wizard

Roughly 1 year later, staff dissatisfaction with the ATW Wizard prompted a new project to fix defects and enhance functionality. This project took roughly 6 months to complete, rolling-out on September 25th 2008

7 months later SMS were hired to fix further defects. This new project, like all others, blew-out beyond expected scope. It is expected that this version of the ATW Wizard will roll-out in November of this year (2009) and that Stanwell Power Station (SPS) will begin using the Wizard

Source: Greg Carlon, O&M Safe Work Systems Support

6

SM

S M

anag

emen

t & T

echn

olog

y

The journey begins

Development BeginsNov 2004

(James Smith)

Another Developer AddedJan 2005

(James Smith & Greg Behrendt)

Another Developer AddedMar 2005

(James Smith & Greg Behrendt & Linda Colpoys)

Another Developer AddedAug 2005

(James Smith & Greg Behrendt & Linda Colpoys & Martin Wallace)

Additional 2 Months development with 1 resource (EC): $16,000 + $32,000 = $48,000

Costing Assumption

All resources, contractors, developers, managers are conservatively estimated at $800/day (inc GST). A month is conservatively estimated To average 20 working days. 1 resource costs $16,000/month

Additional 2 month dev with 2 resources(EC): $48,000 + $64,000 = $112,000

Additional 3 month dev with 3 resources (EC): $112,000 + $144,000 = $256,000

1 Month Initial DevelopmentMid-2004

(Nathan Liebke)

1 Month development with 1 resource Estimated Cost (EC): $16,000

Developer LeavesNov 2005

(James Smith & Greg Behrendt & Linda Colpoys & Martin Wallace)

Additional 3 month dev with 4 resources (EC): $256,000 + $192,000 = $448,000

7

SM

S M

anag

emen

t & T

echn

olog

y

The journey continues

Project Manager AddedDeveloper Added

Feb 2006(James Smith & Martin Wallace & Peter Downes & Edwin Ashdown)

Management declares ATW Project finished

April 17th 2007“Mission Accomplished”

Developer LeavesJan 2006

(James Smith & Linda Colpoys & Martin Wallace)

Additional 2 month dev with 3 resources (EC): $448,000 + $96,000 = $544,000

Additional 1 month dev with 2 resources (EC): $544,000 + $32,000 = $576,000

Developer AddedOct 2006

(James Smith & Martin Wallace & Peter Downes & Edwin Ashdown &

Kerry De Baar)

Developer Added for 3 Months in late 2006

(Stephen Walsh)

Tester Added for 1.5 Months in late 2006

(Martin Bansey)

3 months = $48,000

1.5 months = $24,000

Additional 8 months dev with 4 resources + Total Cost of the 2 temporary resources

(EC): $576,000 + $512,000 + $72,000 = $1,160,000

Additional 6 month dev with 5 resources (EC): $1,160,000 + $480,000 = $1,640,000

8

SM

S M

anag

emen

t & T

echn

olog

y

The journey continuesStaff dissatisfaction with ATW prompts further enhancements & resolution of defects, after almost a year hiatus in

developmentMar 2008

(Kerry De Baar & Mark Dowwell & Lee Milburne)

SMS brought in to fix further defects and do minor enhancements

Apr 2009(Kerry De Baar & Qianfu Ni

& Adrian Dormer)

Completion of “ATW Version 2 Wrap-Up”

Oct 2009

Costing Assumption

All resources, contractors, developers, managers are conservatively estimated at $800/day (inc GST). A month is conservatively estimated To average 20 working days. 1 resource costs $16,000/month

Completion of “Priority 1 Defects and Enhancements Project”

25th Sept 2008

Additional 7 month (rounded up very slightly) dev with 3 resources

(EC): $1,640,000 + $336,000 = $1,976,000

SMS Leaves Sept 2009

(Kerry De Baar & Qianfu Ni & Adrian Dormer)

Additional 5 month dev with 3 resources (EC): $1,976,000 + $240,000 = $2,216,000

1 SMS Developer Re-instated for a month

Sept/Oct 2009(Kerry De Baar & Adrian Dormer)

Time with just Kerry De Baar maintaining ATW will not be added to the total as a symbol for all hidden costs not

accounted for in this rough estimate

Additional 1 month dev with 2 resources (EC): $2,216,000 + $32,000 = $2,248,000

9

SM

S M

anag

emen

t & T

echn

olog

y

Total cost of the ATW Wizard so far Rough estimate of $2,248,000 This estimate is considered conservative for the following

reasons:1. A total cost of $800/day for all resources is conservative. Most contractors,

testers and managers cost more than that, especially in later years2. A month is estimated to average 20 working days. This is rounded down from

a more accurate average of 21-22 working days.3. The estimate does not include the time of many part-time internal Stanwell

managers4. The estimate does not include the virtually full-time resource of Greg Carlon,

O&M Safe Work Systems Support.5. It does not include any pre-planning costs associated with the 3 separate

stages of development6. It does not include any additional fees including contractor hire charges,

flights, accommodation, food, transport etc.

10

SM

S M

anag

emen

t & T

echn

olog

y

What has been learnt? Considered? The ATW Wizard project was not planned very well The cost of a project growing from the bottom-up due to ad-

hoc requirements that are left to grow is generally more expensive than a top-down, thoroughly pre-planned project where the majority of business requirements have been considered before development begins

What value has the Wizard provided to the 2 hyrdo plants currently using it? Has the value attempted to be quantified? (see next slide which discusses quantifying value from the ATW Wizard)

If so, has this value been contrasted with the cost of the wizard (the cost up until production release for the 2 hydro plants)?

11

SM

S M

anag

emen

t & T

echn

olog

y

Should the Wizard be rebuilt? Only if it saves Stanwell money in the long-run. It is

assumed that the ATW Wizard is expected to be used (in one form of another) for the long-haul

What is the annual value of the Wizard to Stanwell? Since the primary objective of the Wizard, ensuring the safety of human lives, is immeasurable, and the expected cost reduction due to increased efficiency and reduced staff is not expected to accelerate or decelerate when it is fully adopted, we assume the annual value to be fairly constant in the long-run

What is, and will be, the ongoing annual cost of ownership of the Wizard? Ownership costs of a typical IT system include enhancements, defects, electricity, hardware or hardware upgrades, staff monitoring the system, spending time responding to queries from other staff or training new staff to use the system

12

SM

S M

anag

emen

t & T

echn

olog

y

Scenarios$

Time in Years (the long-haul)

$

Time in Years (the long-haul)

Scenario 1: Annual Cost of Ownership is decreasing.

The Trend is expected to continue

Scenario 2: Annual Cost of Ownership is increasing

The Trend is expected to continue

Assumptions: 1. The annual ($) value added by ATW Wizard is constant once fully adopted2. ATW Wizard has no end-date.

Decision: Maintain current system. A break-even point isinevitable. After that, profitable.

Scenario Indicators:1. Defect numbers declining2. Enhancement numbers declining3. Staff system maintenance/monitoring costs not expected to increase beyond normal4. Hardware costs of system not expected to increase beyond normal5. Staff training costs not expected to increase beyond normal6. Electricity prices not expected to increase beyond normal

Decision: One of the following:1. Bear the cost in the name of safety2. Find somewhere (else) to cut costs to stop the haemorrhage3. Rebuild ATW to attempt to ensure continual cost of ownership will decrease

Scenario Indicators:1. Defect numbers continually increasing2. Enhancement numbers continually increasing3. Increasing time and cost of development due to bad design and patchwork code being continually added by different contractors over time4. Staff system maintenance, hardware, training and/or electricity costs expected to increase beyond normal

Value to Business

Cost to Business

13

SM

S M

anag

emen

t & T

echn

olog

y

Scenarios (Cont.)

$

Time in Years (the long-haul)

$

Time in Years (the long-haul)

Scenario 3: Annual Cost of Ownership is decreasing

However, costs are expected to rise or accelerate

Scenario 4: Annual Cost of Ownership is increasing

And, costs are expected to rise or accelerate

Value to Business

Cost to Business

Assumptions: 1. The annual ($) value added by ATW Wizard is constant once fully adopted2. ATW Wizard has no end-date.

Decision: One of the following:1. Bear the cost in the name of safety2. Find somewhere (else) to cut costs to stop haemorrhage3. Rebuild ATW to attempt to ensure continual cost of ownership will decrease. Possible disinvestment of obsolete technologies

Scenario Indicators:1. Same as Scenario 2 with blow-outs in any or all indicators2. Increased cost of human resources due to technology obsolescence (less supply of skills on market)3. Increased costs associated with any technology becoming officially unsupported (less skill supply, future IT environment change problems/bugs, unforseen technology obsolescence events)

Decision: Unsure. Depends on the scale of the expectedrise and its trend (Minor or significant rises? Temporary or long-term trend?) Note that the expected trend opposes the existing track record.

Scenario Indicators:1. Same as Scenario 2 with blow-outs in any or all indicators2. Increased cost of human resources due to technology obsolescence (less supply of skills on market)3. Increased costs associated with any technology becoming officially unsupported (less skill supply, future IT environment change problems/bugs, unforseen technology obsolescence events)

14

SM

S M

anag

emen

t & T

echn

olog

y

Scenarios (Cont.)

$

Time in Years (the long-haul)

$

Time in Years (the long-haul)

Scenario 5: Annual Cost of Ownership is decreasing

And, costs are expected to decrease or decelerate

Scenario 6: Annual Cost of Ownership Is increasing

However, costs expected to Decrease or decelerate

Value to Business

Cost to Business

Assumptions: 1. The annual ($) value added by ATW Wizard is constant once fully adopted2. ATW Wizard has no end-date.

Decision: Maintain current system. A break-even point isinevitable. After that, profitable.

Scenario Indicators:1. Same as Scenario 1 with cost decreases in any or all indicators2. No technological obsolescence cost increases expected to increase more than total savings in other areas

Scenario Indicators:1. Same as Scenario 1 with cost decreases in any or all indicators2. No technological obsolescence cost increases expected to increase more than total savings in other areas

Decision: Unsure. Depends on the scale of the expecteddecrease and its trend (Minor or significant decreases? Temporary or long-term trend?) Note that the expected trend opposes the existing track record.

15

SM

S M

anag

emen

t & T

echn

olog

y

Where does the “ATW Wizard Review” fit in?

The “ATW Wizard Review” document discusses a number of the scenario indicators in the previous slides. These are highlighted in blue.

The shortage of support for technologies employed by ATW was discussed in the document and pinpointed as a well-known IT system obsolescence problem. Of particular risk were ADO.NET and several crucial custom-made and third-party monolithic software modules. Will VB.NET developers be around in 10 years? What problems can occur when a future environment variable clashes with our legacy ATW system? Who do we pay to help if ADO.NET is officially unsupported?

The ATW Wizard Review may help the business make decisions on the scenario indicators

16

SM

S M

anag

emen

t & T

echn

olog

y

Justifying the expense of a rebuild Proper software architecture results in far fewer software

defects. This translates directly to a reduction in time and money spent maintaining the system

The reduction in time and money spent on software maintenance becomes significantly larger and more justifiable if the system has been operational for many years and is still yielding many defects, many of which are costly and require outsourcing to address

ATW is currently written in VB.NET which is becoming less favoured among Microsoft developers. C# is Microsoft’s flagship programming language. If the system is to be operational well into the future then considering this preference is important for the future recruitment of skilled developers. Rebuilding ATW would allow for the introduction of C# and for some sections of the system, and perhaps all to be re-written in C#

17

SM

S M

anag

emen

t & T

echn

olog

y

Justifying the expense of a rebuild (Cont.)

If rebuilding ATW is a serious consideration then a comparison should be done against the maintenance costs of a similar-scale application that utilises best-practice architectural design and design patterns. A good example would be to look at the maintenance costs of banking applications due to banks usually practising a high level of software engineering

A comparison of systems would try to make use of metrics such as lines of code, bugs per line of code (total defects divided by lines of code), average man-hours spent on maintenance and the average cost over time.

If there are savings to be made over the long run by having a better system, then these savings would be contrasted against the expected cost of the rebuild project