measuring the effectiveness of gamesourcing expert oil painting annotations
DESCRIPTION
Presentation at ECIR 2014 in AmsterdamTRANSCRIPT
Myriam C. Traub, Jacco van Ossenbruggen, Jiyin He, Lynda Hardman
Centrum Wiskunde & Informatica
Measuring the Effectiveness of Gamesourcing Expert Oil Painting Annotations
Research Problem
• Expert tasks are hard to crowdsource • too complex for non-
experts• experts difficult to target
and engage• Approach:
• simplify the task by suggesting potential answers
• help non-experts to learn by providing expert feedback
�2
Chosen Task
Subject type annotation of paintings• about 100 different
subject types in Art & Architecture Thesaurus
• subtle differences
Marine?
Seascape?
History painting?
Kacho?�3
�4http://sealincmedia.project.cwi.nl/artgame/
Research Questions
Non-Experts: • How well do they compare with experts, both individually and as a crowd?• Do they memorize the correct subject type?• Can they generalize what they have learned to new paintings?
Task:• How does the presence or absence of a correct answer influence a user's
performance?Data:• Are there features of images or subject types that can predict high or low
agreement?Game / UI / Backend system• None
Experimental Setup - Data
• Subset of Steve Tagger data set: 125 images of paintings with …• … subject type annotations by 4 experts from Rijksmuseum
Amsterdam: 168 expert annotations • Limitation: only one expert per painting
�6
Experimental Setup - Query Image Selection
• first 10: random, no repetition• after 10: random, but repetition probability of 50%
Experimental Setup - Candidate Selection
• total 5 candidates + „other“:• Perfect: 1 correct;
Imperfect: 1 correct (75%) or a related but incorrect (25%),• at most 3 related but incorrect, • at least 1 incorrect
�8
Users - Performance
•Perfect: 21 users, 5640 judgements (range: 26 -1250), correct: 12% - 83%
• Imperfect: 17 users, 1929 judgements (range: 19 - 350), correct: 22% - 54%
�9
020
4060
8010
0
users
perc
enta
ge o
f cor
rect
ann
otat
ions
perfect % imperfect %
random behavior
Users - Performance
�10
perfect # imperfect #
200
400
600
800
1000
1200
num
ber o
f ann
otat
ions
(bar
s)
020
4060
8010
0
users
perc
enta
ge o
f cor
rect
ann
otat
ions
(dot
s)perfect % imperfect %
•Perfect: correlation of .71 and p < 0.001
• Imperfect: correlation of .32 and p=0.215
perfe
ct
�11
48
6
4
8
5
48
4
1
26
6
5
5
5
6
26
38
164
2
27
1
12
39
35
5
1
129
51
34
3
1
1
11
49
2
1
1
29
13
47
1
1
1
3
1
107
3
1
2
2
1
1
286
1
8
16
1
1
2
6
105
2
86
1
2
20
2
203
3
12
1
3
2
53
7
1
1
2
9
6
11
1
1
27
5
1
1
3
1
2
3
846
5
23
8
4
58
1
16
3
1
2
95
2
1
2
77
32
15
15
1
1
2
30
980
4
16
1
27
10
5
9
1
86
6
2
1
9
2
3
6
1
4
20
2
3
136
3
1
6
18
9
3
2
355
18
2
28
4
13
2
5
2
1
86
1
17
6
132
29
86
1
2
3
45
2
21
12
18
1
13
1
5
3
164
1
14
2
7
1
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0
25
50
75
Percent
baseline condition − individual annotations
2
1
7
2
1
1
2
1
3
3
8
3
2
5
3
1
30
1
3
1
1
1
37
1
4
8
12
1
6
7
1
1
8
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0255075100
Percent
baseline condition − aggregated annotations
291
63
8
7
5
52
10
9
6
34
4
29
14
8
13
65
7
3
1
59
2
20
10
9
2
7
2
1
8
3
4
2
13
2
9
5
32
8
2
1
1
60
1
1
1
1
2 6
12
1
1
10
35
1
8
2
2
1
1
1
10
4
1
1
3
3
4
1
6
1
7
5
1
1
1
176
20
1
3
3
30
6
1
6
166
3
7
1
6
1
7
18
6
38
1
4
1
1
3
4
6
3
1
10
4
1
89
1
1
6
1
2
1
62
3
1
7
23
10
4
1
1
1
3 26
3
1
25
2
9
2
5
4
5
31
25
2
1
4
2
othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0
25
50
75
Percent
imperfect condition − individual annotations
imperfect
96
11
4
1
6
3
1
1
6
1
3
1
1
3
9
3
7
1
2
1
2
1
3
2
6
1
2
1
4
2
1
1
1
23
2
3
19
3
3
1
1
12
1
11
5
1
1
5
1
1
4
6othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0255075100
Percent
imperfect condition − aggregated annotations
aggregated
individual
48
6
4
8
5
48
4
1
26
6
5
5
5
6
26
38
164
2
27
1
12
39
35
5
1
129
51
34
3
1
1
11
49
2
1
1
29
13
47
1
1
1
3
1
107
3
1
2
2
1
1
286
1
8
16
1
1
2
6
105
2
86
1
2
20
2
203
3
12
1
3
2
53
7
1
1
2
9
6
11
1
1
27
5
1
1
3
1
2
3
846
5
23
8
4
58
1
16
3
1
2
95
2
1
2
77
32
15
15
1
1
2
30
980
4
16
1
27
10
5
9
1
86
6
2
1
9
2
3
6
1
4
20
2
3
136
3
1
6
18
9
3
2
355
18
2
28
4
13
2
5
2
1
86
1
17
6
132
29
86
1
2
3
45
2
21
12
18
1
13
1
5
3
164
1
14
2
7
1
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0
25
50
75
Percent
baseline condition − individual annotations2
1
7
2
1
1
2
1
3
3
8
3
2
5
3
1
30
1
3
1
1
1
37
1
4
8
12
1
6
7
1
1
8
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0255075100
Percent
baseline condition − aggregated annotations
291
63
8
7
5
52
10
9
6
34
4
29
14
8
13
65
7
3
1
59
2
20
10
9
2
7
2
1
8
3
4
2
13
2
9
5
32
8
2
1
1
60
1
1
1
1
2 6
12
1
1
10
35
1
8
2
2
1
1
1
10
4
1
1
3
3
4
1
6
1
7
5
1
1
1
176
20
1
3
3
30
6
1
6
166
3
7
1
6
1
7
18
6
38
1
4
1
1
3
4
6
3
1
10
4
1
89
1
1
6
1
2
1
62
3
1
7
23
10
4
1
1
1
3 26
3
1
25
2
9
2
5
4
5
31
25
2
1
4
2
othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0
25
50
75
Percent
imperfect condition − individual annotations
96
11
4
1
6
3
1
1
6
1
3
1
1
3
9
3
7
1
2
1
2
1
3
2
6
1
2
1
4
2
1
1
1
23
2
3
19
3
3
1
1
12
1
11
5
1
1
5
1
1
4
6othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0255075100
Percent
imperfect condition − aggregated annotations
Perfect condition, individual: substantial agreement (κ = 0.65)�12
2
1
7
2
1
1
2
1
3
3
8
3
2
5
3
1
30
1
3
1
1
1
37
1
4
8
12
1
6
7
1
1
8
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0255075100
Percent
baseline condition − aggregated annotations
48
6
4
8
5
48
4
1
26
6
5
5
5
6
26
38
164
2
27
1
12
39
35
5
1
129
51
34
3
1
1
11
49
2
1
1
29
13
47
1
1
1
3
1
107
3
1
2
2
1
1
286
1
8
16
1
1
2
6
105
2
86
1
2
20
2
203
3
12
1
3
2
53
7
1
1
2
9
6
11
1
1
27
5
1
1
3
1
2
3
846
5
23
8
4
58
1
16
3
1
2
95
2
1
2
77
32
15
15
1
1
2
30
980
4
16
1
27
10
5
9
1
86
6
2
1
9
2
3
6
1
4
20
2
3
136
3
1
6
18
9
3
2
355
18
2
28
4
13
2
5
2
1
86
1
17
6
132
29
86
1
2
3
45
2
21
12
18
1
13
1
5
3
164
1
14
2
7
1
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0
25
50
75
Percent
baseline condition − individual annotations
291
63
8
7
5
52
10
9
6
34
4
29
14
8
13
65
7
3
1
59
2
20
10
9
2
7
2
1
8
3
4
2
13
2
9
5
32
8
2
1
1
60
1
1
1
1
2 6
12
1
1
10
35
1
8
2
2
1
1
1
10
4
1
1
3
3
4
1
6
1
7
5
1
1
1
176
20
1
3
3
30
6
1
6
166
3
7
1
6
1
7
18
6
38
1
4
1
1
3
4
6
3
1
10
4
1
89
1
1
6
1
2
1
62
3
1
7
23
10
4
1
1
1
3 26
3
1
25
2
9
2
5
4
5
31
25
2
1
4
2
othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0
25
50
75
Percent
imperfect condition − individual annotations
96
11
4
1
6
3
1
1
6
1
3
1
1
3
9
3
7
1
2
1
2
1
3
2
6
1
2
1
4
2
1
1
1
23
2
3
19
3
3
1
1
12
1
11
5
1
1
5
1
1
4
6othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0255075100
Percent
imperfect condition − aggregated annotations
Perfect condition, aggregated: almost perfect agreement (κ = 0.87)�13
“… dramatic bird’s eye view of Broadway and Wall Street…”
Church Street El, 1920 by Charles Sheeler
�14
Townscape
Cityscapes
291
63
8
7
5
52
10
9
6
34
4
29
14
8
13
65
7
3
1
59
2
20
10
9
2
7
2
1
8
3
4
2
13
2
9
5
32
8
2
1
1
60
1
1
1
1
2 6
12
1
1
10
35
1
8
2
2
1
1
1
10
4
1
1
3
3
4
1
6
1
7
5
1
1
1
176
20
1
3
3
30
6
1
6
166
3
7
1
6
1
7
18
6
38
1
4
1
1
3
4
6
3
1
10
4
1
89
1
1
6
1
2
1
62
3
1
7
23
10
4
1
1
1
3 26
3
1
25
2
9
2
5
4
5
31
25
2
1
4
2
othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0
25
50
75
Percent
imperfect condition − individual annotations
48
6
4
8
5
48
4
1
26
6
5
5
5
6
26
38
164
2
27
1
12
39
35
5
1
129
51
34
3
1
1
11
49
2
1
1
29
13
47
1
1
1
3
1
107
3
1
2
2
1
1
286
1
8
16
1
1
2
6
105
2
86
1
2
20
2
203
3
12
1
3
2
53
7
1
1
2
9
6
11
1
1
27
5
1
1
3
1
2
3
846
5
23
8
4
58
1
16
3
1
2
95
2
1
2
77
32
15
15
1
1
2
30
980
4
16
1
27
10
5
9
1
86
6
2
1
9
2
3
6
1
4
20
2
3
136
3
1
6
18
9
3
2
355
18
2
28
4
13
2
5
2
1
86
1
17
6
132
29
86
1
2
3
45
2
21
12
18
1
13
1
5
3
164
1
14
2
7
1
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0
25
50
75
Percent
baseline condition − individual annotations
2
1
7
2
1
1
2
1
3
3
8
3
2
5
3
1
30
1
3
1
1
1
37
1
4
8
12
1
6
7
1
1
8
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0255075100
Percent
baseline condition − aggregated annotations
96
11
4
1
6
3
1
1
6
1
3
1
1
3
9
3
7
1
2
1
2
1
3
2
6
1
2
1
4
2
1
1
1
23
2
3
19
3
3
1
1
12
1
11
5
1
1
5
1
1
4
6othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0255075100
Percent
imperfect condition − aggregated annotations
Imperfect condition, individual: moderate agreement (κ = 0.47)�15
AllegoryOther
Lucretia Calypso
96
11
4
1
6
3
1
1
6
1
3
1
1
3
9
3
7
1
2
1
2
1
3
2
6
1
2
1
4
2
1
1
1
23
2
3
19
3
3
1
1
12
1
11
5
1
1
5
1
1
4
6othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0255075100
Percent
imperfect condition − aggregated annotations
48
6
4
8
5
48
4
1
26
6
5
5
5
6
26
38
164
2
27
1
12
39
35
5
1
129
51
34
3
1
1
11
49
2
1
1
29
13
47
1
1
1
3
1
107
3
1
2
2
1
1
286
1
8
16
1
1
2
6
105
2
86
1
2
20
2
203
3
12
1
3
2
53
7
1
1
2
9
6
11
1
1
27
5
1
1
3
1
2
3
846
5
23
8
4
58
1
16
3
1
2
95
2
1
2
77
32
15
15
1
1
2
30
980
4
16
1
27
10
5
9
1
86
6
2
1
9
2
3
6
1
4
20
2
3
136
3
1
6
18
9
3
2
355
18
2
28
4
13
2
5
2
1
86
1
17
6
132
29
86
1
2
3
45
2
21
12
18
1
13
1
5
3
164
1
14
2
7
1
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0
25
50
75
Percent
baseline condition − individual annotations
2
1
7
2
1
1
2
1
3
3
8
3
2
5
3
1
30
1
3
1
1
1
37
1
4
8
12
1
6
7
1
1
8
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0255075100
Percent
baseline condition − aggregated annotations
291
63
8
7
5
52
10
9
6
34
4
29
14
8
13
65
7
3
1
59
2
20
10
9
2
7
2
1
8
3
4
2
13
2
9
5
32
8
2
1
1
60
1
1
1
1
2 6
12
1
1
10
35
1
8
2
2
1
1
1
10
4
1
1
3
3
4
1
6
1
7
5
1
1
1
176
20
1
3
3
30
6
1
6
166
3
7
1
6
1
7
18
6
38
1
4
1
1
3
4
6
3
1
10
4
1
89
1
1
6
1
2
1
62
3
1
7
23
10
4
1
1
1
3 26
3
1
25
2
9
2
5
4
5
31
25
2
1
4
2
othe
figu
land
full
port
alle
half
genr
hist
kach
city
seas
stil
anim
town
flow
mari
maes
othe figu land full port alle half genr hist kach city seas stil anim town flow mari maesNon−Experts
Experts
0
25
50
75
Percent
imperfect condition − individual annotations
Imperfect condition, aggregated: moderate agreement (κ = 0.55)�17
�18Cityscapes
Other Genre Landscape
Learning Memorizing
�19
2 4 6 8 10
020
4060
8010
0
number of repetitions
perc
enta
ge o
f cor
rect
ann
otat
ions
random behavior
Learning Memorizing
�20
2 4 6 8 10
020
4060
8010
0
number of repetitions
perc
enta
ge o
f cor
rect
ann
otat
ions
perfect % imperfect %
Learning Generalizing
�21
sequence number of new images
perc
enta
ge o
f cor
rect
ann
otat
ions
[1,20] (60,80] (120,140] (200,220] (280,300] (360,380]
020
4060
8010
0
Learning Generalizing
�22
sequence number of new images
perc
enta
ge o
f cor
rect
ann
otat
ions
perfect % imperfect %
[1,20] (60,80] (120,140] (200,220] (280,300] (360,380]
020
4060
8010
0
Conclusions
Agreement between experts and non-experts:
• substantial in perfect, moderate in imperfect condition
• aggregation reduces deviations
Strong disagreement may indicate:
• need for additional metadata
• incorrect or incomplete expert judgements
Learning: • users memorize and
generalize• users need a training phase
on high quality data!Results in line with
He, J., van Ossenbruggen, J., de Vries, A.P.: Do you need experts in the crowd?: a case study in image annotation for marine biology. OAIR 2013
Future Work
- scarce+ high quality data - need training data
+ high quantity data
- lack expertise+ high quality when trained & assisted
�24
Thank you for your attention!
Myriam C. Traub
!!http://sealincmedia.project.cwi.nl/artgame/
On the beach of Trouville Eugène Boudin