19/04/23 Dr Andy Brooks 1

Fyrirlestrar 23 & 24

Management Strategies

I am going to have move the release deadline.

Or maybe I can contract a few more developers to help with this release.

MSc Software MaintenanceMS Viðhald hugbúnaðar

19/04/23 Dr Andy Brooks 2

ReferenceSoftware Maintenance Management Strategies: Observations from the Field, George Stark (MITRE) and Paul Ohman (Univ of Idaho), Technical Report.http://members.aol.com/geshome/ibelieve/strategies.PDF

Case StudyDæmisaga

This report is based on data collected over two years (November 1994 – December 1996) concerning software maintenance management strategies at six organisations.

“The key task for a manager is defining the attribute that they would most like to optimize and choosing a strategy that supports that goal.”

19/04/23 Dr Andy Brooks 3

Remote Maintenance

The software maintenance organisations were all several hundred miles from their users.

Advantages: better documentation, formal release procedures, and better defined processes.

Disadvantages: more training is required and user support may not be as it should be.

19/04/23 Dr Andy Brooks 4

Measured MaintenanceHaving a strong measurement program was observed to help managers communicate with customers on cost estimates, requirements volatility, schedules, and product quality.

19/04/23 Dr Andy Brooks 5

Software Maintenance 2 main activities

1. Requirements Validation• Is a change request clear, complete,

feasible to implement, and consistent with the backlog of change requests?

2. Maintenance Release• Involves a group of change requests taking

account of user priorities, the subsystems affected, and various other factors.

• Involves choosing an implementation strategy.

19/04/23 Dr Andy Brooks 6

Implementation Strategies

• fixed staff / variable schedule

• fixed schedule / variable staff

• variable schedule / variable staff

• fixed staff / fixed schedule– not implemented by any of the six

organisations studied

19/04/23 Dr Andy Brooks 7

fixed staff / variable schedule

VCN Version Content NoticeDT&E Developmental Testing & EvaluationCM Configuration Management

19/04/23 Dr Andy Brooks 8

fixed staff / variable schedule• fixed pool of maintenance engineers• software engineers work as many changes as possible

through unit test until management declare a release freeze

• integration testing (DT&E) scheduled for one month after the release freeze

• release content is 100% unit-tested and should always be delivered on time

• advantages:– flexibility over release content– relatively easy to assign work

19/04/23 Dr Andy Brooks 9

variable staff / fixed schedule

6-18 months

19/04/23 Dr Andy Brooks 10

variable staff / fixed schedule

• pool of maintenance engineers available • release date established between customer and supplier• high priority changes agreed and analysts write a

preliminary VCN and release plan• software engineers begin work on priority changes • content and staffing can be renegotiated

– if VCN is too ambitious, for example

• a final VCN is issued prior to completion of unit testing• advantages:

– clearly defined need dates– prioritized changes

19/04/23 Dr Andy Brooks 11

variable staff / variable schedule

19/04/23 Dr Andy Brooks 12

variable staff / variable schedule• users submit a version content notice (VCN) to the

maintenance team• maintenance team reviews the request and estimates

the cost, schedule and risks• in negotiations with the users, content may be altered

(added or subtracted)• maintenance team then negotiates a release cost and

schedule with a software maintenance contractor• milestone reviews are held at which changes are made

to the plan as necessary• advantages:

– milestone reviews identify issues and help ensure quality of the release

19/04/23 Dr Andy Brooks 13

Process Performance

1. throughput• average number of changes delivered to the

user per year

2. priority cycle time• 75th percentile of the number of days

required to deliver a priority changeEmergency (highest) Urgent (High) Routine (Low)

• cycle time is the time between a requirement being validated and the time the change is delivered

measurementsmælingar

19/04/23 Dr Andy Brooks 14

Cumulative Distribution Of Priority Changes

• 75% of the time, high priority changes are delivered in less than 135 days

Mean 129 daysSD 148 days

onestrategy

19/04/23 Dr Andy Brooks 15

Process Economics

• Cost per change delivered (average)

• Associated costs are closely tracked:– Engineering– Management– Travel– Configuration Management– Hardware Maintenance

measurements

deliveredchangesTotal

spentdollarsTotalchangeperCost

__

____

mælingar

19/04/23 Dr Andy Brooks 16

Process Efficiency

• The percent of schedules met:

measurements

100*____

______#

madedeliveriesofnumbertotal

dateplannedbeforeoronaccepteddeliveries

mælingar

19/04/23 Dr Andy Brooks 17

Schedule Performance

• Only 24% of planned delivery dates were met.

onestrategy

19/04/23 Dr Andy Brooks 18

Process Efficiency

• The percent of releases that experienced no content changes after the release plan was established.

• “... customers changing their minds about priorities as well as schedule and quality pressure...”

measurements

100*____

___#

madedeliveriesofnumbertotal

changescontentwithoutdeliveries

mælingar

19/04/23 Dr Andy Brooks 19

Requirements Volatility

• 8 out of 13 deliveries experienced content change

19/04/23 Dr Andy Brooks 20

Process Quality

• Percent of changes without customer found defects during operational acceptance testing.

• Customers typically use different test scenarios than the developers and find defects.

measurements

100*____

____#

deliveredchangesofnumbertotal

founddefectswithoutchanges

mælingar

19/04/23 Dr Andy Brooks 21

Release Quality

• Defects found during customer testing are typically written up as new change requests.

19/04/23 Dr Andy Brooks 22

Process Comparison Table

19/04/23 Dr Andy Brooks 23

Process Comparisons

• Variable staff /variable schedule delivered the fewest changes but was the most cost effective and was of the highest quality.– The higher quality is thought to be as a result

of the milestone reviews (which are not present in the other two processes).

19/04/23 Dr Andy Brooks 24

Conclusions Stark and Oman

• All three processes had useful attributes.• The fixed staff /variable schedule strategy made

it easy for managers to allocate work based on individual skills.

• The variable staff / fixed schedule strategy delivered a high number of changes and dealt with requirement volatility.

• The variable staff /variable schedule strategy had more formal management (reviews) but had the lowest cost per change and highest quality.

19/04/23 Dr Andy Brooks 25

Validity Threats Stark and Oman

• Individual changes vary greatly, from a simple database update to 10 pages of specification for a 1,000 lines of code.– Statistical averaging should be OK because of

the large number of changes and assuming similar distributions of change difficulty across the programs.

19/04/23 Dr Andy Brooks 26

Validity Threats Stark and Oman

• User participation varied considerably.

• In one organization the customer rarely partipicated in the version release process.

• Other organizations had user teams involved in the planning and execution of releases.

19/04/23 Dr Andy Brooks 27

Validity Threats Stark and Oman

• Program managers could have influenced the results by placing emphasis on quality or cost or throughput or...

19/04/23 Dr Andy Brooks 28

Reviews• Variable staff /variable schedule had the

best quality but the lowest cost.– Conflicting with the cost/accuracy trade-off.

• Milestone reviews help find defects and they are relatively cheap to perform.

• Or was customer acceptance testing particularly weak for the single variable staff / variable schedule organisation studied?

19/04/23 Dr Andy Brooks 29

Software Quality Assurance

• It is stated that code inspections or peer reviews were carried out by some engineers, but that these activities were not formalised.– These activities or their absence can have a

considerable bearing on quality.

• Testing effort and coverage is unknown for the unit, integration, and acceptance testing that took place.

19/04/23 Dr Andy Brooks 30

Which Strategy?

• The results are not unambiguously intepretable and are not generalisable.

• You cannot decide on a variable staff /variable schedule strategy, for example, as a guarantee of low throughput, low cost, high quality software production.

• The study does, however, help identify useful measurements that can be of help to control the maintenance process.

19/04/23 Dr Andy Brooks 31

ReferenceEvaluating the Release, Delivery, and Deployment Processes of Eight Large Product Software Vendors applying the Customer Configuration Update Model, Slinger Jansen and Sjaak Brinkkemper. WISER´06 ©ACM (2006) 4pp.

Case StudyDæmisaga

6 descriptive case studies of software vendors using: interviews, studying the software, document study, and direct observations.

2 open source organisations for whom all on-line material was used.

Between 65 an 1500 employees in each company.Between 20 and 1 million customers per company.

“For software vendors the processes of release, delivery, and deployment to customers are inherently complex.”

Figure 1: CUU Model

19/04/23 Dr Andy Brooks 33

Table 1: Release Key PracticesRelease Key Practice V1 V2 V3 V4 V5 V6 V7 V8

Pilot customers test early releases √ √ √ √

Release planning is published internally √ M √ M M √ M

Configuration restrictions managed (int.) M √ √ M M √ √ √

Configuration restrictions managed (ext.) M M √ M M √ √ √

Tools for rel., deliv., deploy. are managed. √ √ M M M √ √ √

Formalized release procedure. √ √ √ √ √ √ √ √

Releases are stored in respositories √ √ √ √ √ √

Release planning adjusted to customers. √ √ √

Kepping customers regularly informed √ M M M M √ M M

All possible information channels used M √ M √

M: manual steps are required which would be easy to automate.

19/04/23 Dr Andy Brooks 34

Table 2: Delivery Key PracticesPractice V1 V2 V3 V4 V5 V6 V7 V8

Automatic pull is available √ NA √

Any medium (Internet, DVD, floppy) √ M M M NA √ √

Download site abstraction √ M √ M √ √

Kepping customers regularly informed √ M M M M √ M M

All possible information channels used M √ M √

M: manual steps required which would be easy to automate.NA: Not Applicable

Pull release from vendor.Abstract details of customer site.

19/04/23 Dr Andy Brooks 35

Discussion points

• In the commercial case studies, 50%-70% of yearly revenue came from service contracts from existing customers.

• Sometimes it was found that up to 15% of deployments failed at the customer side.

• License management is “not such a large issue in open source software”.

19/04/23 Dr Andy Brooks 36

Discussion points

• Software vendors “must integrate their CRM, PDM, and SCM systems to automate the processes related to CCU.– Customer Relationship Management– Software Configuration Management– Product Data Management– Customer Configuration Updating

• Using feedback reports “deserves more attention”.