Center for Computational Analysis of Social and Organizational Systems

http://www.casos.cs.cmu.edu/

Labeling images with a computer game

By von Ahn & Dabbish

Motivation• ‘Our goal is ambitious: to label the majority of

images on the World Wide Web.’• Machine learning to identify and label image

components is generally ineffective.

Peter Landwehr 2

Past Work

Peter Landwehr 3

• Lempel & Soffer, 2002• Duygulu et al., 2002

Peter Landwehr 4

Implementation notes• Image reintroduction possibility• Age restriction possibility• Play alone possibility• Taboo threshold (X=1)• Image completion• Randomized partners• 350K images from random.bounceme.net• 73K word dictionary• ‘[T]he game doesn’t ask the players to describe

the image: all they are told is that they have to “think like each other” and type the same string.’

Peter Landwehr 5

August 9 – December 10, 2003

• 13,360 Players• 1,271,451 labels for 293,760 images• 80% of players returned at least once, 33 spent

50 hours playing• Mean = 3.89 labels/image/minute of play, std.

dev = 0.69.

Peter Landwehr 6

Peter Landwehr 7

Validation (1)

car dogman woman

stamp Witherspoonsmiling Aliascartoon green

1. ‘Please type the six individual words that you feel best describe the contents of this image. Type one word per line below; words should be less than 13 character’– For all of the images…

• at least 5 (83%) of the 6 labels produced by the game were entered by at least one participant.

• the three most common words entered by participants were contained among the labels produced by the game.

Peter Landwehr 8

15 participants aged 20-25, 20 images with >5 labels

Validation (2)

1. ‘How many of the words above would you use in describing this image to someone who couldn’t see it?’– Mean = 5.105, std. dev = 1.3087 (85% useful)

2. ‘How many of the words have nothing to do with the image (i.e., you don't understand why they are listed with this image)?’– Mean = 0.105, std. dev = 0.2529 (1.7%

useless)

Peter Landwehr 9

15 participants aged 20-25, 20 images with >5 labels

Validation (3)

Broad implications• ‘At this rate, 5,000 people playing the ESP game

24 hours a day would label all images on Google (425,000,000 images) [with one tag] in 31 days.’

• ‘…[O]ur main contribution stems from the way in which we attack the labeling problem. …[W]e have shown that it’s conceivable that a large-scale problem can be solved with a method that uses people playing on the Web. We’ve turned tedious work into something people want to do.’

Peter Landwehr 10

Peter Landwehr 11

‘The ESP game can be used, with minor modifications, to label sound or video clips (i.e., there is nothing inherent about

images).’

Peter Landwehr 12

‘Other problems that could be solved by having people play games include categorizing web pages into topics and monitoring security

cameras.’

More recent work

Peter Landwehr 13

Walsh & Golbeck, 2010

Recent Work• McGonigal, 2003• Law, von Ahn, Dannenberg, and Crawford, 2007• Hacker and von Ahn, 2009• Dong and Fu, 2010• Walsh and Golbeck, 2010

Peter Landwehr 14

Center for Computational Analysis of Social and Organizational Systems

http://www.casos.cs.cmu.edu/

Questions?