![Page 1: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/1.jpg)
• Game: https://funlines.co
• Demo: https://youtu.be/5OXJMxDBaLY
• Dataset: https://cs.rochester.edu/u/nhossain/funlines.html
![Page 2: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/2.jpg)
• Competitive game to collect data for a human creativity task
• Generate humorous headlines:
• Edit headlines to make them funny
• Rate funniness of headlines edited by others
FunLines https://funlines.co
ORIG:EDIT:
![Page 3: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/3.jpg)
• Desirable features
• build mechanisms to increase participation & stimulate human creativity
• make it fun, engaging, interactive, educational, collaborative, rewarding
• collect high quality humor data at low cost
• verified as better and cheaper than Mechanical Turk
FunLines https://funlines.co
![Page 5: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/5.jpg)
• Task: Rate headlines on a 0-3 funniness scale
• Problem: Rating agreement
• Rating feedback:
• calibrate (de-bias) individual ratings
• crowd defines rating scale (objective ratings)
• Rating points proportional to agreement with other raters
• rewards players with high rating agreement
Rating
REWARDING
EDUCATIONAL
INTERACTIVE
COLLABORATIVE
![Page 6: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/6.jpg)
• Task: Replace entity/noun/verb to make headline funny
• Information layout makes editing engaging & interactive
• headlines extracted daily, sorted by recency
• Instant funniness estimation using BERT NSP model
• Editing points proportional to funniness ratings received by headline
• rewards players who create very funny headlines
Editing
INTERACTIVE
COLLABORATIVE
ENGAGING
REWARDING
Link to article
[Devlin et al. NAACL 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding]
![Page 7: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/7.jpg)
• Players choose headlines to attempt
• edit headlines they can make funny
• rate headlines they understand
• Attempt headlines by news category
Player Freedom
FLEXIBILITY
ENGAGING
INTERACTIVE
![Page 8: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/8.jpg)
• Helps adjust editing style and rating scale
• makes player come back to FunLines
Performance Feedback
COLLBORATIVE
ENGAGING
EDUCATIONAL
FUN
![Page 9: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/9.jpg)
• 5 week competition
• Leaderboards and prize money
• Task balancing: points for balancing editing and rating submissions
• Qualification: 50-150 edits, 200-500 ratings
Competition
COMPETITIVE
REWARDING
ENGAGING
![Page 10: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/10.jpg)
• 290 Players
• Mechanical Turk seeding
• Social media, TV News Interview
Players
REWARDING
25
ENGAGING
![Page 11: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/11.jpg)
• Humicroedit — similar dataset obtained using Mechanical Turk
• FunLines costs 60% less per datum
• more funny, higher agreement
• more diversity of word usage, editors, raters
FunLines Dataset
[Hossain et al. NAACL 2019. President vows to cut taxes hair: Dataset and Analysis of Creative Text Editing for Humorous Headlines]
Humicroedit and FunLines data comparison
![Page 12: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/12.jpg)
Player Behavior AnalysisPlayers’ editing quality Players’ rating differences
Players get better at editing and rating via practice
• Dataset gets funnier as competition proceeds
Dataset’s Funniness
![Page 13: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/13.jpg)
Player Behavior AnalysisPlayers’ editing quality Players’ rating differences
• Players get better at editing and rating via practice
Dataset’s Funniness
EDUCATIONAL
COLLABORATIVE
![Page 14: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/14.jpg)
Player Behavior AnalysisPlayers’ editing quality Players’ rating differences
• Dataset gets funnierDataset’s Funniness
EDUCATIONAL
COLLABORATIVE• Players get better at editing and rating via practice
COMPETITIVE
ENGAGING
REWARDING
![Page 15: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/15.jpg)
Humor Detection
Deployed BERT model gets accurate as more data becomes available
![Page 16: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/16.jpg)
Humor Detection
Deployed BERT model gets accurate as more data becomes available
FunLines data improves humor detection in Humicroedit
Uses FunLines
![Page 17: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/17.jpg)
• Dataset found useful in humor detection in:
Humor Detection
Deployed BERT model gets accurate as more data becomes available
Hossain et al. SemEval-2020 Task 7: Assessing Humor in Edited News Headlines. In SemEval 2020.
FunLines data improves humor detection in Humicroedit
Uses FunLines
![Page 18: Game: Demo: Dataset · 2020. 6. 19. · ORIG: EDIT: •Desirable features. •build mechanisms to increase participation & stimulate human creativity. •make it fun, engaging, interactive,](https://reader036.vdocuments.us/reader036/viewer/2022071503/6123abadbb9d023db3550c97/html5/thumbnails/18.jpg)
• FunLines: collect highly funny headlines at low very cost by stimulating human creativity
• FunLines is fun: sticky, engaging, interactive, competitive, rewarding, educational
• Fully automatic humor generation
• Extend to crowdsourcing for style transfer
Conclusion and Future Work
Play FunLines: https://funlines.co