observations from computer-supported even swaps experiments using the smart-swaps software

S ystemsAnalysis LaboratoryHelsinki University of Technology

Observations from computer-

supported Even Swaps experiments

using the Smart-Swaps software

Jyri MustajokiRaimo P. Hämäläinen

Petri LievonenSystems Analysis Laboratory

Helsinki University of Technologywww.sal.hut.fi


Introduction

Even Swaps method• Hammond, Keeney and Raiffa (1998, 1999)• Easy-to-use multi-criteria decision analytical (MCDA)

method based on value tradeoffs

Web-based Smart-Swaps software• Procedural support to carry out even swaps

In this study, observations of students using the method with the help of the software

• Do they like and understand it?• How laborious it is felt to be?• Role and importance of the computer support?


Structure of this presentation

Introduction to the Even Swaps process and the Smart-Swaps software

Observations from computer-supported Even Swaps experiments:

Research questions and experimental procedure

Results and evidence

Conclusions and discussion


PrOACT-modelSmart Choices (1999)

Define your decision problem tosolve the right problem.

Clarify what you’re really trying to achieve with your decision.

Make smarter choices by creating better alternatives to choose from.

Describe how well each alternative meets your objectives.

Make tough compromises when you can’t achieve all your objectives at once.

Problem

Objectives

Alternatives

Consequences

TradeoffsSmart-Swapssoftware -this experiment


+ Uncertainty, Risk profiles, Linked decisions


Even Swaps elimination process

Carry out even swaps that makea) Alternatives dominated (attribute-wise)

≡ There is another alternative, which is equal or better than this in every attribute, and better at least in one attribute

b) Attributes irrelevant≡ Each alternative has the same value on this attribute

» These can be eliminated

Process continues until one alternative, i.e. the best one, remains



Smart-Swaps softwarewww.smart-swaps.hut.fi



Example

Office selection problem (Hammond et al. 1999)

Dominatedby

Lombard

Practicallydominated

byMontana

(Slightly better in Monthly Cost, but equal or worse in all other attributes)

78

25

An even swap

Commute time removed as irrelevant



Supporting Even Swaps with Preference Programming

Support for1. Finding candidates for the next even swap

2. Identifying practically (i.e. almost) dominated alternatives

Both tasks need comprehensive technical screening

Idea: supporting the process – not automating it



Decision support in Smart-Swaps

More than oneremaining alternative

The most preferred alternative is found

Trade-off information

Even swap suggestions

Practical dominance candidates

Initial statements about the attributesProblem initialization

Updating of

the model

Make an even swap

Eliminate irrelevant attributes

Eliminate dominated alternatives

Yes

No

Even Swaps Preference Programming



Previous studies of Even SwapsComparison of Even Swaps and MAVT

• Belton et al. (2005): “MCDA in E-democracy. Why weight? Comparing Even Swaps and MAVT.”, TED Workshop, May 19-22, Helsinki

Environmental planning• Gregory et al. (2001): “Bringing stakeholder values into

environmental policy choices: a community-based estuary case study.” Ecological Economics 39, 37-52.

Strategy selection in a rural enterprise• Kajanus et al. (2001): “Application of even swaps for

strategy selection in a rural enterprise.” Management Decision 39(5), 394-402.




Thoughts about Even Swaps process

From a cognitive point of view, Even Swaps process has several characteristics that are of a special interest, e.g.

• What kind of swaps the DMs tend to carry out?• How the DMs understand the alternatives with the

revised consequences?• Does the DM end up with the same result following

different paths of even swaps?

Smart-Swaps software available» Easy to study how the DMs carry out the process in

practice




Questions of interest in our experiment

In this study, the focus is on:

1) Issues related to carrying out the process with the Smart-Swaps software or (manually) with Microsoft Excel?

2) Issues related to the size of the problem?

3) What are the benefits of using the Preference Programming functionality of Smart-Swaps software, if any?






Material and methods

• Subjects consisting of engineering studentsas DMs

• Controlled experiments included in courses1) Assignments in applied mathematics

2) Decision making and problem solving

• Questionnaires with open and scaled questions

• Decision process logs




Experimental procedure After a brief introduction to the Even Swaps process each DM carried out two decision analytical assignments:

1) The Even Swaps process on a small introductory problem. On this assignment,

A. half of the DMs used Excel manually

B. the other half conducted the process with the Smart-Swaps software

2) The Even Swaps process on a much larger problem. Every DM used the Smart-Swaps software but

A. half of the DMs were instructed to ignore the Preference Programming functionality (‘recommender’)

B. the other half was instructed to utilize it




1st: a small introductory problem

You are choosing an apartment. After preliminary elimination there are three alternatives: Lombard, Baranov and Montana. You are in a happy position, as all want you as a tenant. It all boils down to what do you want. You have ended up with four criteria to base your decision on. You gather up a consequences table:

Lombard Baranov Montana

Distance to school

25 min 20 min 45 min

Condition Poor Good Excellent

Size 35 m2 42 m2 23 m2

Rent 350 €/month 450 €/month 250 €/month




2nd: a large primary problem

Your company is about to choose the facility for a new office.

After serious thinking you consider eight attributes relevant: a1: size of the officea2: rental costsa3: renovation need a4: car park opportunitiesa5: means of commuting a6: distance to city center a7: other facilities in the neighborhood a8: habitability

After searching for a while you find 12 alternatives to choose from (X1-X12). The consequences are given in a table.


1st

study2nd

studyboth

1. Smallproblem

A (Excel) 10 DMs 6 DMs 16 pers.

B (Smart-Swaps) 10 DMs 14 DMs 24 pers.

2. Large problem

A (without recomm.) 10 DMs 9 DMs 19 pers.

B (with recommender) 10 DMs 11 DMs 21 pers.



Overall sample sizes

Small but sufficient for our purposes to get preliminary results.


Hypotheses on the experiment

H1.1: The DMs using the Smart-Swaps software evaluate the results more positively than the DMs using Excel.

H1.2: The DMs using the Smart-Swaps require less time to get the result than the DMs using Excel.

H2.1: The support by Smart-Swaps is evaluated to be more useful in large problems than in small ones.

H2.2: The DMs tend to eliminate rather dominated alternatives than irrelevant attributes.

H3.1: The even swap recommender is evaluated as useful.H3.2: Utilizing the recommender results in fewer swaps.H3.3: Practical dominance propositions are seldom

neglected.




DMs arrived to various decisions


Decisions in the 1st (small) and 2nd (large) problems. Areas indicatefrequencies of subjects that chose certain alternative.

1 2 7 2 9 1 3 5 1

X0 X1 X2 X3 X4 X5 X6 X7 X8 X9 X10 X11 X12

All DMs (N=31, NA=9)

5 6 29 All DMs (N=40)

Montana BaranovLombard


H1: Smart-Swaps software vs. Excel?


• The DMs using Smart-Swaps were faster • No significant difference between no. of swaps

• Note that the problem was small

• No significant difference between decisions made

• No significant difference between the opinions on the result


The DMs using Smart-Swaps were faster


Decision time comparison in the 1st (small) problem. Areas indicate frequencies of subjects that completed the task in certain time.

Smart-Swaps software vs. Excel:

4 1 1 2 7 2 5 1 1 7 3 1 1 1 1 1 1

1 3 1 4 2 1 1 1 1 1

3 1 1 2 4 2 5 1 3 1 1

0 min 5 min 10 min 15 min 20 min 25 min 30 min 35 min

Median

All DMs (N=40)

DMs using MS Excel(N=16)

DMs using Smart-Swaps(N=24)


No significant difference between no. of swaps Number of swaps on the 1st problem. Areas indicate frequencies of

subjects that completed the task with certain amount of swaps.


Smart-Swaps software vs. Excel:

1 1 1 14 12 1 8 1 1

4 7 1 2 1 1

1 1 1 10 5 6

0 swaps 1 swaps 2 swaps 3 swaps 4 swaps 5 swaps 6 swaps 7 swaps 8 swaps 9 swapsMedian

All DMs (N=40)

DMs using MS Excel (N=16)

DMs using Smart-Swaps (N=24)


H2: Issues related to the size of the problem

• In large problems, it may be very difficult to manually carry out the even swaps process• Smart-Swaps is regarded as helpful in

applying the Even Swaps process also in large problems

• DMs tend to eliminate more dominated alternatives than irrelevant attributes



The Even Swaps process is seen to apply best to small problems


Areas indicate frequencies of subjects that gave certain answer on Osgood-scale.

Size of the problem:

2 5 4 3 2

6 6 1 1 1 1

2 1 3 3 6

3 1 2 2 4 4

0 1 2 3 4 5 6 7 Median

The apartment selection problem couldbe solved applying Even Swaps processwith pen and paper (weakly-strongly)(N=16, N/A=4)

The office selection problem could besolved applying Even Swaps processwith pen and paper (weakly-strongly)(N=16, N/A=4)

Smart-Swaps helps applying EvenSwaps process to small problems (e.g.apartment selection problem) (weakly-strongly) (N=15, N/A=5)

Smart-Swaps helps applying EvenSwaps process to large problems (e.g.office selection problem) (weakly-strongly) (N=16, N/A=4)


Elimination by dominance was used more than irrelevance


Number of all eliminations in the 2nd problem studies are presented in the table.

Size of the problem:

No proposer With proposer

ALL ALL-% A A-% B B-%

Removal 66 11% 53 21% 13 4%

Dominance 345 59% 130 52% 215 64%

Irrelevance 163 28% 58 23% 105 31%

Practical dom. 11 2% 7 3% 4 1%

585 100% 248 100% 337 100%


H3: Benefits of using Smart-Swaps

• The DMs using the Preference Programming functionality (the recommender) were faster

• The DMs using the recommender made fewer swaps

• The recommender was rated high• The DMs using the recommender gave

more positive opinions on the method



DMs using the proposer were faster


Decision time comparison in the 2nd (large) problem. Areas indicate frequencies of subjects that completed the task in certain time.

Benefits of Smart-Swaps:

2 1 1 3 6 1 3 1 1 1 3 2 1 2 1 1

2 1 1 1 2 1 1 1

2 1 1 3 4 2 1 1 3 1 1

0 min 5 min 10 min 15 min 20 min 25 min 30 min 35 min 40 min 45 min Median

All DMs (N=30, N/A=10)

DMs without proposer(N=10, N/A=9)

DMs using proposer(N=20, N/A=1)


DMs using the proposer made fewer swaps


Number of swaps on the 2nd problem. Areas indicate frequencies of subjects that completed the task with certain amount of swaps.


2 1 1 4 3 4 2 1 1 1 1 1 2 1 1 1 1 1 1 1

1 1 1 1 1 1 1 1 1 1 1

2 1 1 3 3 4 2 1 1 1 1

0 swaps 10 swaps 20 swaps 30 swaps 40 swaps 50 swaps 60 swapsMedian

All DMs (N=31, N/A=9)

DMs without proposer(N=11, N/A=8)

DMs using proposer(N=20, N/A=1)


Preference programming functionality was rated high



1 4 7 9

2 4 8 3 2

1 1 3 5

2 2 7

2 4 5

3 4 3

1 1 1 1 6

0 1 2 3 4 5 6 7

Median

I regard proposer as (cumbersome-convenient) (N=21)

I regard proposer as (unreliable-reliable)(N=19, N/A=2)

I regard proposer as (difficult-easy) (N=10)

I used the proposer (not at all - much)(N=11)

I did the swap the proposer suggested(never - always) (N=11)

I used proposer more for (irrelevance-dominance) (N=10, N/A=1)

I accepted practical dominancesuggestions (never - always) (N=10, N/A=1)

Opinions on the recommender functionality of the Smart-Swaps software.Areas indicate frequencies of subjects that gave certain answer on Osgood-scale.


DMs without proposer vs. DMs using proposer



Differences in opinions between DMs ignoring the recommender and DMs utilizing it.Areas indicate frequencies of subjects that gave certain answer on Osgood-scale.

2 5 3 4 12 7 4

2 5 2 1 5 2

1 3 7 5 4

2 4 3 4 11 7 8

2 3 2 2 6 3

1 1 2 5 4 8

0 1 2 3 4 5 6 7

Median

The method is suitable for problems thathave a large number of attributes - allopinions (weakly-strongly) (N=37, N/A=3)

Opinions of DMs ignoring the proposer(N=17, N/A=2)

Opinions of DMs utilizing the proposer(N=20, N/A=1)

I regard the method as (easy-difficult) touse - all opinions (N=39, N/A=1)


Opinions of DMs utilizing the proposer(N=21)


DMs without proposer vs. DMs using proposer



1 1 3 8 1 4

1 1 3 3

5 1 4

2 1 6 7 12 9 1

2 1 4 4 5 2

2 3 7 7 1

2 5 9 10 10 1 1

2 2 7 3 3

3 2 7 7 1 1

0 1 2 3 4 5 6 7 Median

Software helps justifying decisions toothers - all opinions (weakly-strongly)(N=18, N/A=2)Opinions of DMs ignoring the proposer(N=8, N/A=2)


Method improves my decision making - allopinions (weakly-strongly) (N=38, N/A=2)


Opinions of DMs utilizing the proposer(N=20, N/A=1)

I trust the method (mildly-strongly) - allopinions (N=38, N/A=2)




Conclusions

Our first experiments on the Smart-Swaps software suggest:

• The support provided by the Smart-Swaps software is perceived to be useful

• Preference Programming approach provided considerable help

• A significant difference between the opinions of the group utilizing the recommender and the group instructed to ignore it

» Further studies needed to get a deeper insight of how the DMs perceive the approach in practice

» In addition, we got a lot of usability feedback to help improve the Smart-Swaps software

Conclusions and discussion


ReferencesV. Belton, G. Wright and G. Montibeller (2005). “MCDA in E-democracy. Why weight?

Comparing Even Swaps and MAVT.” Presentation at the TED Workshop on e-Participation in Environmental Decision Making, May 2005, Helsinki. (Downloadable at http://www.ted.tkk.fi/presentations/Belton-TED.ppt)

Hammond, J.S., Keeney, R.L., Raiffa, H., 1998. “Even swaps: A rational method for making trade-offs.” Harvard Business Review 76(2), 137-149.

Hammond, J.S., Keeney, R.L., Raiffa, H., 1999. Smart choices. A practical guide to making better decisions. Harvard Business School Press, Boston.

Mustajoki, J., Hämäläinen, R.P., 2005. “A Preference Programming Approach to Make the Even Swaps Method Even Easier.” Decision Analysis. (to appear) (Downloadable at www.sal.hut.fi/Publications/pdf-files/mmus04.pdf)

Salo, A., Hämäläinen, R.P., 1992. “Preference assessment by imprecise ratio statements.” Operations Research 40(6), 1053-1061.

A. Salo and R.P. Hämäläinen (1995). ”Preference programming through approximate ratio comparisons.” European Journal of Operational Research 82(3), 458-475.

Applications of Even Swaps:Gregory, R., Wellman, K., 2001. “Bringing stakeholder values into environmental policy

choices: a community-based estuary case study.” Ecological Economics 39, 37-52.Kajanus, M., Ahola, J., Kurttila, M., Pesonen, M., 2001. “Application of even swaps for

strategy selection in a rural enterprise.” Management Decision 39(5), 394-402.

observations from computer-supported even swaps experiments using the smart-swaps software

Documents