systematic analysis of oag v4
DESCRIPTION
TRANSCRIPT
![Page 1: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/1.jpg)
Systematic Analysis of Output Agreement Games:Effects of Gaming Environment,Social Interaction, and Feedback
Shih-Wen Huang and Wai-Tat Fu
![Page 2: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/2.jpg)
GWAP Today – ESP Game
[Von Ahn and Dabbish, CHI 2004]
![Page 3: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/3.jpg)
GWAP Today - Foldit
[Cooper et al., Nature 2010]
![Page 4: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/4.jpg)
Two Unanswered Questions
1. Can GWAP collect outputs with high quality? Why?
2. Which component of GWAP motivate volunteers to work?
• To answer these questions, a systematic analysis that tests the effects of different components of GWAP needs to be done
![Page 5: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/5.jpg)
Output Agreement Games
![Page 6: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/6.jpg)
Game Interface
Database1. Randomly match multiple strangers and provide them withthe same input
Output Agreement Games
![Page 7: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/7.jpg)
Game Interface
Database1. Randomly match multiple strangers and provide them withthe same input
Output Agreement Games
2. Players are asked to generate outputs that are related to the input
![Page 8: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/8.jpg)
Game Interface
Database1. Randomly match multiple strangers and provide them withthe same input
Output Agreement Games
3. Players will be rewarded if the outputs are matched. The matched outputs would be used to describe the input
2. Players are asked to generate outputs that are related to the input
![Page 9: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/9.jpg)
Three Major Components
• Gaming Environment (G): Encouraging players to generate matched answers using scoring system, leaderboard, etc.
• Social Interaction (S): Players need to cooperate with other players
• Feedback (F): Players can implicitly learn the answers of other players
![Page 10: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/10.jpg)
Five Different Interfaces
• (B)• (F)• (F+S)• (F+G)• (F+S+G)
B: baseline F: feedback G: gaming environment S: social interaction
![Page 11: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/11.jpg)
Five Different Interfaces
• (B)• (F)• (F+S)• (F+G)• (F+S+G)
B: baseline F: feedback G: gaming environment S: social interaction
![Page 12: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/12.jpg)
Five Different Interfaces
• (B)• (F)• (F+S)• (F+G)• (F+S+G)
B: baseline F: feedback G: gaming environment S: social interaction
Teammate searching page Teammate waiting page
![Page 13: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/13.jpg)
Five Different Interfaces
• (B)• (F)• (F+S)• (F+G)• (F+S+G)
B: baseline F: feedback G: gaming environment S: social interaction
![Page 14: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/14.jpg)
Five Different Interfaces
• (B)• (F)• (F+S)• (F+G)• (F+S+G)
B: baseline F: feedback G: gaming environment S: social interaction
![Page 15: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/15.jpg)
Experimental Design
• 150 workers (30 per interface) were recruited from Amazon Mechanical Turk within two weeks.
• Each worker earned $0.05 for generating 20 labels
• A label was considered as a high-quality label if its distance to a gold standard label < 1
![Page 16: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/16.jpg)
Gaming Environment Helps Collect More High-Quality Labels
Interfaces with Gaming Environment collect significantly more high quality labels than other three interfaces
Amount of high-quality labels collected by each interface
![Page 17: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/17.jpg)
Social Interaction Motivates More Volunteers
Interfaces with Social Interaction motivate more volunteers than other three interfaces
Percentage of workers reported that they would love to do the task again voluntarily
![Page 18: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/18.jpg)
Implications for Human Computation
1. Gaming Environment allows output agreement games to collect more high-quality outputs
2. Social Interaction helps motivate more volunteers in output agreement games
![Page 19: Systematic analysis of oag v4](https://reader033.vdocuments.us/reader033/viewer/2022061122/547079b9b4af9fcd0a8b47a4/html5/thumbnails/19.jpg)
Questions?