Survey of experience of using COSMIC in Agile projects

- interim findings

Presented at UK COSMIC Special Interest Group

10th April 2014Charles Symons

The Guideline on using COSMIC in Agile projects makes several claimsStory Points are a subjective size measure

COSMIC sizes are an objective measure that:• can be measured at any level of an Agile

project• may be aggregated from User Stories

upwards to any level• hence can be used to estimate at any level• hence can be used to track progress in line

with Agile processes, e.g. of the backlog, of earned value, etc.

10/04/2014 2

User Story

Project

ComponentRelease

Sprint/Iteration

(Sub-) Task

(N. B. The number of

levels in the hierarchy, and terminology

vary)

3

Feedback from a brief survey of how COSMIC is used in Agile

projects in practice

• Canada• UQAM Masters student study of four projects• UQAM Masters student studying 15 projects: aim to build an estimation model

(WIP)

• India, Italy• General industry experience

• Netherlands• Fairly extensive experience in Sogeti Nederland

• Russia• SRG (small software house) experience

10/04/2014

4

Use COSMIC to measure at the Sprint level?

Difficult:• ‘Wrong decomposition of requirements’ (Canada)• Some User Stories may have part of a FP, others map to 4 x FP’s (Canada)• Some US may comprise a new FUR plus a CR and be implemented over

multiple sprints. Overall, # US > # FP’s (Canada)• Hard to use for small sprints. We measure only large ‘tasks’ (= several

sprints) and the whole application. We don’t measure every sprint. Small tasks are measured with ‘proxies’, e.g. ‘add simple web form = n CFP’ (Russia)• We measure at sprint level for progress control (Netherlands)

10/04/2014

5

How are Agile projects estimated? At what level?

• The strength of using USP is in the team discussions and use of ‘Poker Planning’ – it’s not the same as traditional expert judgement• Good correlation of CFP and effort expected at the Project level• Now researching correlation of USP and CFP (Canada)• We are mostly required to estimate whole projects before the

requirements are known in detail. We estimate using NESMA approximate sizing. Most projects use Agile processes (Netherlands)• We have a satisfactory correlation of size with effort for the whole

application and for large tasks (Russia)

10/04/2014

6

COSMIC can help with QC of requirements

Study 1 (and a 2011 study)• Objective measurement of the quality of User Stories showed improvement with each

iteration, as a result of CFP measurement

Study 2 (UQAM)• No data models, poor naming consistency, etc leads to poor US that can’t easily be

measured• having US that are measurable objectively, using methods such as COSMIC, could

increase the accuracy of future estimations10/04/2014

Total projets Itération 1 Itération 2 Itération 3

% a 0 7 63

% b 0 72 37

% c 18 21 0

% d 82 0 0

% e 0 0 0

7

COSMIC measurements are used to control progress

UQAM Study 1

Sogeti experience: ‘We also use QSM Control to control the project progress and for this COSMIC is much more powerful than NESMA (≡IFPUG FPA), as it gives a more accurate picture of the actual functionality being ready at a certain point in time, resulting in more accurate forecasts.’10/04/2014

8

Some general industry observations

• Companies are starting to use Agile, but there is not yet much published information (Italy)• Lack of standardization of ‘story points’ is a concern when carrying

out performance measurement across projects. COSMIC is being studied/piloted. (India)• Use of traditional estimation methods at bid/proposal stage, and use

of User Story or Work Breakdown Structure at the iteration stage, seem most common (Indian BSPIN report, 2012)• UK Government Digital Services experience is very positive for small

web projects, but no real performance measurements (?)

10/04/2014

9

Use of COSMIC with Agile projects: Conclusions

• The claims for the applicability of COSMIC are theoretically true• In practice, given current Agile process maturity, it is difficult to use COSMIC at

the US level for sizing and estimating, due to:• the variable quality of requirements• some US or tasks include re-factoring of earlier US• difficulties of dealing with NFR, etc

• Agile Project teams would benefit from being trained in COSMIC sizing. The ‘Planning Poker’ process is also important• COSMIC can be used with advantage for Agile projects for:• whole project sizing & estimating• QC of requirements• progress control

10/04/2014