PAIRED COMPARISON: A USER PERSPECTIVEShigeru Sasao

Master of Software Engineering

Carnegie Mellon University

AGENDA

Introduction to Paired Comparison Experiment comparing Ad-hoc, Planning

Poker, and Paired Comparison Observation of Usage in a Project Points of Improvement New Version Using Incomplete Cyclic Design Conclusion

2

INTRODUCTION TO PAIRED COMPARISON

We want to estimate the size of the following objects.

We know that size of C is 10 units big.

A B

CD 3

4

Estimate the relative sizes for all pairs: I think A is 3 times as large as D. I think A is half the size of B. I think B is 6 times as large as D....

A B

CD

5

  A B C D

A 1  0.5 2 3

B  2  1 0.5 6

C  0.5  2  1 2

D  0.3  0.17  0.5  1

This is the “Judgment Matrix”. Principal diagonal is always 1. The other shaded regions are reciprocals of

the un-shaded region. Does not have to be perfectly consistent,

since it is an estimate.

I think A is 3 times

bigger than D…

From the Judgment Matrix, we can calculate the estimates of the object size:

1. Calculate the geometric mean of each row.

2. Obtain the “ratio scale”.

3. Calculate the estimated size.

6

7

For our example,1. Calculate the geometric mean of each row.

2. Obtain the “ratio scale”.• Sum the geometric means

1.32 + 1.57 + 1.19 + 0.41 = 4.49• Divide geometric mean of the row by the sum

of the geometric mean.

A 1.32B 1.57C 1.19D 0.41

A 0.29B 0.35C 0.26D 0.09

8

3. Calculate the estimated size.• We know from the beginning that C has size

of 10 units. • So, the reference value is C with ratio scale

entry of 0.26.

A 0.29/0.26 * 10 = 11.15 units

B 0.35/0.26 * 10 = 13.46 units

C 0.26/0.26 * 10 = 10.00 units

D 0.09/0.26 * 10 = 3.46 units

INTRODUCTION TO PAIRED COMPARISON

What did we do? We knew the size of 1 object (reference size). We compared pairs of relative sizes, and entered

it into a judgment matrix. Calculated the estimates for the objects.

The method reduces individual judgment errors by requiring multiple pair-wise comparisons of relative values.

9

EXPERIMENT COMPARING AD-HOC, PLANNING POKER, AND PAIRED COMPARISON

Comparison was made between ad-hoc methods, planning poker and paired comparison.

Study conducted among students from the Master of Software Engineering program at Carnegie Mellon.

Students have strong technical background with two to three years of industry experience.

10

EXPERIMENT SETUP

Students divided into three groups, with each group using either ad-hoc, planning poker, or paired comparison.

Estimations conducted in pairs. Five pairs per estimation technique. Students were asked to estimate the size

(Lines of Code) of different data structures (stack, queue, binary tree, etc).

Students were told that the size of “linked list (a)” was 40 LOC.

11

EXPERIMENT RESULT

12

EXPERIMENT RESULT

Planning poker and paired comparison show a vast improvement in precision over ad-hoc methods.

This result is consistent with previous reports by Miranda.

13

Miranda, Eduardo, An Evaluation of the Paired Comparisons Method for Software Sizing, Proceedings of the 22nd International Conference on Software Engineering, 2000.

EXPERIMENT RESULT

Comparison of standard deviation between planning poker and paired comparison:

14

Data Structure Std. Dev.(Planning Poker)

Std. Dev. (Paired Comparison)

Stack 9.8 2.4Queue 11.4 4.9Binary Tree 25.5 13.3String Manipulation 31.6 8.6Linked List (b) 42.4 5.5Balanced Tree 56.0 21.4Hash Table 33.1 31.5

EXPERIMENT RESULT

Paired comparison produced more consistent estimations among estimators as compared to ad-hoc and planning poker.

Earlier studies by both Miranda and Shepperd show that paired comparison produces more reliable estimations than ad-hoc estimation methods.

15

OBSERVATION OF USAGE IN A PROJECT In the Master of Software Engineering

program, a team of four to five members work for an external customer.

16 months with a total resource of 4608 to 5760 person hours.

Observed the usage of paired comparison by a MSE team.

The project was to produce an integrated development environment to support global distributed teams.

Estimation of effort for architectural components prior to the detailed design/implementation stage. 16

OBSERVATION OF USAGE IN A PROJECT

Five team members and observers gathered in a conference for the estimation session.

A list of 12 components was prepared beforehand.

Pair-wise comparison between all 12 components, for a total of 66 comparisons.

For each comparison, team discussed size, complexity, number of state transitions to agree on a relative size.

Values entered into a spreadsheet tool.

17

OBSERVATION OF USAGE IN A PROJECT

Snapshot of the resulting judgment matrix:

Total of 66 comparisons in 54 minutes.

18

Reference Value

Artifact Name

Dat

a E

lem

, Mgr

Trac

eabi

lity

cont

rolle

r

Impo

rt U

I

Trac

. Vie

w C

ont

Dat

a U

I

Trac

eabi

lity

UI

Trac

eabi

lity

Ser

ver

AD

T D

ef

Tran

sf. C

hain

Impo

rt M

gr

Trac

eabi

lity

Con

f

RTC

Bri

dge

Ratio Scale

Estimated Value

Data Elem, Mgr 2.0 9.5 4.1 4.5 4.3 5.5 5.6 7.0 9.0 8.0 1.5 0.256 258.7Traceability controller 7.0 1.8 2.5 2.2 2.0 2.2 5.5 6.5 6.0 0.5 0.135 136.5

20 Import UI 0.5 0.2 0.2 0.4 0.4 0.7 0.8 0.9 0.1 0.020 20.0Trac. View Cont 2.0 1.8 1.5 1.6 2.2 3.5 3.0 0.3 0.076 76.7Data UI 0.8 0.5 0.6 0.8 2.8 2.4 0.4 0.050 50.8Traceability UI 0.8 0.7 2.5 3.8 3.5 0.4 0.067 67.4Traceability Server 1.1 2.0 3.5 3.2 0.4 0.067 67.9ADT Def 2.2 3.7 3.4 0.4 0.067 67.5Transf. Chain 2.5 2.8 0.3 0.038 38.4Import Mgr 1.6 0.1 0.021 21.0Traceability Conf 0.1 0.020 20.4RTC Bridge 0.182 183.7

POINTS OF IMPROVEMENT

1. Estimators became noticeably tired during the 54 minute estimation session.

2. As estimators became tired, they started extrapolating the new relative size values from old ones that have already been determined.

“Since A was 3 times the size of C and C was 2 times the size of D, A must be 6 times the size of D…”

19

NEW VERSION USING INCOMPLETE CYCLIC DESIGN To consider the estimators’ stamina, we need

to reduce the number of comparison. Use incomplete cyclic designs to reduce

comparisons.

Maintains balance and connectedness. 20

NEW VERSION USING INCOMPLETE CYCLIC DESIGN

Control the number of comparisons by the replication factor.

Example for a 10 component comparison:

21

Replication Factor Number of Comparisons10 458 406 304 202 10

NEW VERSION USINGINCOMPLETE CYCLIC DESIGN

Full comparison of 8 components (28 comparisons)

Replication factor of 4 (16 comparisons)

22

A B C D E F G HABCDEFGH

A B C D E F G HABCDEFGH

NEW VERSION USINGINCOMPLETE CYCLIC DESIGN

The reduction of comparison is made possible through: Incomplete cyclic design (ICD) Imputing missing values as the geometric mean

of comparisons made Studies have shown that very low replication

factors still produce reliable estimates, although precision may diminish as number of comparisons decrease.

23

NEW VERSION USING CYCLIC DESIGN

Developed a tool which uses the new version with cyclic design:

24

Replication factor

Randomized

comparison order

INTEGRATION WITH COCOMO

Use paired comparison to estimate LOC size as input into COCOMO.

25

CONCLUSION

Paired comparison produces more consistent estimations than ad-hoc methods and planning poker.

Can be used to directly estimate effort, or to estimate LOC size for input into parametric models such as COCOMO.

New version using incomplete cyclic designs reduce the number of comparison.

If you are interested in the tool, please contact me.

26

REFERENCES [Aguaron et al. 2003] Aguaron, Juan, Moreno-Jimenez, Jose Maria, The Geometric

Consistency Index: Approximated Thresholds, European Journal of Operational Research, 147 (2003), Pages 137-145.

[Crawford, Williams 1985] Crawford, Gordon, Williams, Cindy, The Analysis of Subjective Judgment Matrices, Rand Corporation, 1985.

[Miranda 2000] Miranda, Eduardo, An Evaluation of the Paired Comparisons Method for Software Sizing, Proceedings of the 22nd International Conference on Software Engineering, 2000.

[Miranda 2001] Miranda, Eduardo, Improving Subjective Estimates Using Paired Comparison, IEEE Software, 2001.

[Miranda et al. 2009] Miranda, Eduardo, Bourque, Pierre, Abran, Alain, Sizing User Stories Using Paired Comparisons, Information and Software Technology Volume 51, Issue 9, September 2009, Pages 1327-1337, Butterworth-Heinemann, 2009.

[PlanningPoker] Cohn, Mike, Planning Poker, www.planningpoker.com, Mountain Goat Software.

[Shepperd et al. 2001] Shepperd, Martin, Cartwright, Michelle, Predicting with Sparse Data, IEEE Transactions on Software Engineering, VOL. 27, NO. 11, 2001.

[Spencer 1982] Spencer, Ian, Incomplete Experimental Designs for Multidimensional Scaling, Chapter 3. In R.G. Golledge and J.N. Rayner (Eds.), Proximity and Preference: Problems in the Multidimensional Analysis of Large Data Sets, University of Minnesota Press, 1982.

[Spencer 1983] Spencer, Ian, Monte Carlo Simulation Studies, Applied Psychological Measurement Vol. 7, No. 4, 1983.

27