kisskissban: a competitive human computation game for image annotation
DESCRIPTION
In this paper, we propose a competitive human computation game, KissKissBan (KKB), for image annotation. KKB is different from other human computation games since it integrates both collaborative and competitive elements in the game design. In a KKB game, one player, the blocker, competes with the other two collaborative players, the couples; while the couples try to find consensual descriptions about an image, the blocker’s mission is to prevent the couples from reaching consensus. Because of its design, KKB possesses two nice properties over the traditional human computation game. First, since the blocker is encouraged to stop the couples from reaching consensual descriptions, he will try to detect and prevent coalition between the couples; therefore, these efforts naturally form a player-level cheating-proof mechanism. Second, to evade the restrictions set by the blocker, the couples would endeavor to bring up a more diverse set of image annotations. Experiments hosted on Amazon Mechanical Turk and a gameplay survey involving 17 participants have shown that KKB is a fun and efficient game for collecting diverse image annotations.TRANSCRIPT
![Page 1: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/1.jpg)
KissKissBanA Competitive Human Computation
Game for Image Annotation
Chien-Ju Ho*, Tao-Hsuan Chang+, Jong-Chuan Lee+, Jane Yun-jen Hsu+, Kuan-Ta Chen*
*Academia Sinica+National Taiwan University
![Page 2: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/2.jpg)
Human Computation Game
Collaborative Mechanisms
![Page 3: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/3.jpg)
Coalition Problem
Limited Data Diversity
![Page 4: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/4.jpg)
The ESP Game
• Coalition problem
• partner with themselves
• unified strategy
• Limited diversity
• easy words first
![Page 5: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/5.jpg)
Competitive Mechanism?
![Page 6: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/6.jpg)
DOGFLOWERFLOOR
CUTEWOODDOG
Player 1 Player 2
![Page 7: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/7.jpg)
DOGFLOWERFLOOR
CUTEWOODDOG
DELIVER EMOTION BY TYPING THE SAME WORD
Player 1 Player 2
![Page 8: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/8.jpg)
DOGFLOWERFLOOR
CUTEWOODDOG
DELIVER EMOTION BY TYPING THE SAME WORD
Couple Couple
![Page 9: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/9.jpg)
DOGFLOWERFLOOR
CUTEWOODDOG
DOGWHITE
Couple CoupleBlocker
![Page 10: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/10.jpg)
DOGFLOWERFLOOR
CUTEWOODDOG
DOGWHITE
Couple CoupleBlocker
KissKissBanA Competitive Human Computation
Game for Image Annotation
![Page 11: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/11.jpg)
Couple CoupleBlocker
1. First, the blocker has 7 seconds to type words
DOGWHITE
The blocked words
2. Then, the couples has 30 seconds to type words
DOGFLOWERFLOOR
CUTEWOODFLOWER
-5s game time -5s game time
Blocked
![Page 12: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/12.jpg)
Couple CoupleBlocker
1. First, the blocker has 7 seconds to type words
DOGWHITE
2. Then, the couples has 30 seconds to type words
DOGFLOWERFLOOR
CUTEWOODFLOWER
Couples win for matching within time limit
![Page 13: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/13.jpg)
Screenshots of KissKissBan
![Page 14: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/14.jpg)
Intuitions Behind the Game
![Page 15: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/15.jpg)
Incentive Structure
• Zero sum game
• strict competitive
• players can not benefit from coalition
• Different decision time periods
• harder words, longer consideration [Barry et al. 2001]
• diverse data contribution
![Page 16: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/16.jpg)
Small-scale Experiment
• Image Source
• 125 images randomly from ESP dataset
• Experiment Setup
• 17-player gameplay survey
• Data collection on Mechanical Turk
![Page 17: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/17.jpg)
Gameplay Survey
• Fun
• 3.76/5 overall rating
• 15/17 players claim that would play again
• over 60% like to play as a blocker
![Page 18: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/18.jpg)
MTurk Experiment
• Collecting data from anonymous users
• Publish two games for comparison
• KissKissBan
• ESP Game without taboo words
• Each game is played for ~5000 rounds
![Page 19: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/19.jpg)
Example of Collected DataESP ML-KKB BL-KKB
man 21beach 10karate 5water 1
beach 3water 3sand 3sea 2
ninja 1kungfu 1ocean 1
sea 9man 8
ocean 3black 1china 1sand 1
ML-KKB: matching between couplesBL-KKB: matching between couple and blocker
![Page 20: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/20.jpg)
MTurk Experiment
• Data property
• For each image
• ESP: 39.96 labels, 6.56 distinct labels
• KKB: 44.17 labels, 11.53 distinct labels
• A more diverse player contribution
![Page 21: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/21.jpg)
Conclusion
• We proposed a competitive human computation game, KissKissBan.
• KissKissBan has two nice properties
• a player-level anti-coalition mechanism
• diverse player input
![Page 22: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/22.jpg)
More in the Future
• Publish the game on Facebook within the next one to two months.
• Study the different tagging behaviors of the blocker and the couples.
• Apply the competitive mechanism to more general human computation game design.
![Page 23: KissKissBan: A Competitive Human Computation Game for Image Annotation](https://reader038.vdocuments.us/reader038/viewer/2022110115/54930d13ac7959132e8b4772/html5/thumbnails/23.jpg)
Q & A