handson 2 (6/6)

21
Hands-on experiences with incentives and mechanism design www.insemtives.eu 1 Roberta Cuel, University of Trento, IT; Markus Rohde, University of Siegen, DE and Germán Toro del Valle, Telefonica I+D, ES ISWC 2010

Upload: roberta-cuel

Post on 22-Apr-2015

965 views

Category:

Technology


1 download

DESCRIPTION

ISWC 2010: TUTORIAL: Ten Ways to Make your Semantic App Addictive

TRANSCRIPT

Page 1: Handson 2 (6/6)

Hands-on experiences with incentives and mechanism design

www.insemtives.eu 1

Roberta Cuel, University of Trento, IT; Markus Rohde, University of Siegen, DE and

Germán Toro del Valle, Telefonica I+D, ES

ISWC 2010

Page 2: Handson 2 (6/6)

How to design effective incentives/rules

1. Analyze the domain• What?

• Working environment• Job descriptions• Organization (tasks, hierarchy, compensation, social,

communication)

• How? • Qualitative face-to-face interviews and questionnaires• Observations with selected individuals • Quantitative analysis (data collections)

04/11/2023 www.insemtives.eu 2

Page 3: Handson 2 (6/6)

How to design effective incentives/rules (2)

2. Identify the preferences and motivations that drive users

Concentrate on every-day uses for those specific users

3. Formalize the existing reward systemFind yourself in the matrix

4. Design the simplest possible solution that can effectively support those uses

Translate into a small number of alternative testable hypothesis

5. Fine tune the rewarding system

04/11/2023 www.insemtives.eu 3

Page 4: Handson 2 (6/6)

Fine-tuning incentives with mechanism design: a step by step procedure

• Mimic situation in the lab• Set up experiment as close to real life situation as

possible• Run experiment with volunteer subjects, with random

allocation of subjects• Test alternative hypothesis regarding effect of

incentive schemes on behavior• Check differences in outcome

If happy go to next slide, otherwise re-design hypothesis and run a new trial.

Page 5: Handson 2 (6/6)

Fine-tuning part II

• Start adding realism components:– Move to real subjects (field test)– Move to real tasks (with real subjects)– Move to real subjects handling real tasks – Move to real situation (field experiment)

• During the process you:– Lose control over ability to manipulate variables– Gain awareness of interaction between variables

• Let’s look at what we are doing with case studies!

Page 6: Handson 2 (6/6)

Telefonica I+D case study

• Corporate portal• What is the most obvious incentive from

economic point of view?• What can we do with a small budget to be

dedicated to incentivize users?• How do we know which system is the best for

our setting?

04/11/2023 www.insemtives.eu 6

Page 7: Handson 2 (6/6)

Basic experiment

• Test Two rewarding/incentives systems• Pay per click:

– 0,03 € per tag added (up to 3 € maximum). • Winner takes all model:

– The person who adds the higher number of tags/annotation wins 20€

What would you choose?(Participation fee – 5 €)

04/11/2023 www.insemtives.eu 7

Page 8: Handson 2 (6/6)

The experiment (setting)• 36 students

– Random assignment to the two “treatments”• Individual task: annotation of images• Clear set of Instructions• Training (guided) session to give basic

understanding of annotation tool• 8 minutes clocked session (time pressure)• Goal: produce maximum amount of tags in

allotted time on a random set of images04/11/2023 www.insemtives.eu 8

Page 9: Handson 2 (6/6)

The lab

04/11/2023 www.insemtives.eu 9

Page 10: Handson 2 (6/6)

The experiment: screenshots

04/11/2023 www.insemtives.eu 10

Page 11: Handson 2 (6/6)

04/11/2023 www.insemtives.eu 11

Page 12: Handson 2 (6/6)

04/11/2023 www.insemtives.eu 12

Page 13: Handson 2 (6/6)

04/11/2023 www.insemtives.eu 13

Page 14: Handson 2 (6/6)

04/11/2023 www.insemtives.eu 14

Page 15: Handson 2 (6/6)

ID N. Tags Reward Tot. €2 48 1,5 6,53 41 1,5 6,54 78 2,5 7,55 54 2 76 51 2 77 36 1,5 6,58 44 1,5 6,59 54 2 7

10 64 2 711 41 1,5 6,512 63 2 713 58 2 714 60 2 715 25 1 616 43 1,5 6,517 50 1,5 6,518 24 1 619 30 1 620 37 1,5 6,5

tot. 901 31,5 126,5av. 47,42 1,66 6,66 15

ID N. Tags Reward Tot. €2 86 0 53 40 0 54 68 0 55 88 0 56 87 0 57 65 0 58 67 0 59 31 0 5

10 62 0 511 79 0 512 45 0 513 96 20 2514 51 0 515 68 0 517 73 0 518 26 0 519 35 0 5

tot. 1067 20 105av. 62,76 1,18 6,18

Page 16: Handson 2 (6/6)

Number of tagsPay per tag

(N=19)– Total amount of tags: 901– Max n. of tags: 78– N. tags (avg.)= 47.42

– € (avg. per person)= 6.66– € (avg. per tag) =0.1404– € total = 126,5 € (31,5 €

flexible compensation)

Winner takes all model (N=17)

– Total amount of tags: 1067– Max n. of tags: 96– N. tags (avg.)= 62.76 (32%

increase!)– € (avg. per person)= 6.18– € (avg. per tag)=0.098407– € total = 105 € (20 €

flexible compensation)

04/11/2023 www.insemtives.eu 16

Page 17: Handson 2 (6/6)

The results

04/11/2023www.insemtives.eu

17

T-test and F-test are significant

Page 18: Handson 2 (6/6)

Tags distribution: interface matters!

Pay per tag• Tag “nature” 24 times• “snow” 22 times • “green” 20 times • …• 134 tags repeated only 2

times • 437 unique tags

Winner take all• Tag “green” 18 times• “snow” 14 times • “butterfly” 13 times • …• 118 tags repeated only

2 times • 390 unique tags

04/11/2023 www.insemtives.eu 18

Page 19: Handson 2 (6/6)

Some biases

• Students are – Volunteers who are used participating in

experiments– Strong web users and game players – Paid to show up

• Quality of the tags– Quality of tagging has been controlled for: no

obvious ‘mistakes’ or ‘cheating’

04/11/2023 www.insemtives.eu19

Page 20: Handson 2 (6/6)

Summary of results & next lab steps

• Basic hypothesis confirmed• More work needed:

– Effort directed to producing a good (tags) that are not consumed by users (used to achieve other goals) change structure of the game to let users exploit tagging to achieve results (treasure hunt!)

– Re-run experiment with new structure. Now users produce tags to get money and to use tags to perform more tasks)

Page 21: Handson 2 (6/6)

Next steps: Telefonica I+D

04/11/2023 www.insemtives.eu 21

• Replicate experiment with real users– Main change 1: task becomes relevant in terms of

practical usefulness for users – Main change 2: task has social implications– Main change 3: expectations change dramatically

(workers vs. students 5 Euros to participate???)• Add realism

– Mimic social structure in the company:• Run experiment with teammates• Use real tasks• Try alternative pay for performance schemes