ir game: how well do you know information retrieval papers?
TRANSCRIPT
![Page 1: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/1.jpg)
![Page 2: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/2.jpg)
![Page 3: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/3.jpg)
![Page 6: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/6.jpg)
Expertise Profiling
03 / 20
![Page 7: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/7.jpg)
Gamification
04 / 20
๏ question-answering quiz๏ IR publications
(1111 papers from SIGIR, WWW, CIKM, KDD & WSDM)
๏ two difficulty modes ๏ max 3 mistakes; time limit๏ goal: answer as many questions as possible๏ motivation for players: position on the leader board
![Page 8: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/8.jpg)
Research questions
05 / 20
๏ Which level of difficulty is preferred?
๏ Does a competitive element, such as a leader board, increase the level of engagement?
๏ When do users stop playing?
๏ Do users return to play again? After how long?
๏ What types of players can we identify?
๏ Are more cited papers also more easily recognized?
๏ Are more popular authors also more easily recognized?
๏ Do people prefer to play anonymously?
![Page 12: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/12.jpg)
Usage stats
09 / 20
Duration of measurements:
# webpage visitors:
# game players:
# games played:
# games (beginner / advanced):
# avg. #games per player:
5 days (Jan 31 - Feb 4, 2015)
302 (from 33 countries)
116
387
347 / 39
3.34
![Page 13: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/13.jpg)
Analysis of resultsby answers
10 / 20
Time limit (both game modes):
Avg. time to answer (beginner / advanced):
15 s
6.7 s / 8.5 s
![Page 14: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/14.jpg)
Analysis of resultsby answers
11 / 20
Score distribution
a) beginner mode b) advanced mode
![Page 15: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/15.jpg)
Analysis of resultsby players
12 / 20
Returning players
# players who played more than 1x:
# games played within 1 hour:
56
42
Time elapsed between games
![Page 16: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/16.jpg)
Analysis of resultsby players
13 / 20
Player type: Jumper
![Page 17: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/17.jpg)
Analysis of resultsby players
14 / 20
Player type: Give-uper
![Page 18: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/18.jpg)
Analysis of resultsby players
15 / 20
Player type: Fighter
![Page 19: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/19.jpg)
Analysis of resultsby players
16 / 20
Player type: Achiever
![Page 20: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/20.jpg)
Analysis of resultsby papers
17 / 20
Paper’s recognition ratioNumber of times the publication was successfully recognized by playersdivided by number of times it was shown to players.
Citation counts vs. recognition ratio
![Page 21: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/21.jpg)
Analysis of resultsby authors
18 / 20
Author’s recognition ratioNumber of times the author’s publications was successfully recognized by players divided by total number of times her publications were shown to players.
Number of author’s publications vs. recognition ratio.
![Page 22: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/22.jpg)
Observations
19 / 20
๏ Learning
The game was useful for learning about new publications.
๏ Unfair behavior
The best scoring user had the longest response time.
๏ Head-start
A user was restarting the game until he was able to answer the first question correctly.
๏ Engaging users
It is important to keep the user stay in the game as long as possible when she comes for the first time.
๏ Identity
Some people (∼ 10%) opted to use their full civil name as opposed to a nickname.
![Page 23: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/23.jpg)
Conclusions
20 / 20
๏ Could we make an assessment exercise, in the context of expertise profiling, more appealing for users?
Yes.
๏ We gathered valuable data about authors and publications.
Future work ๏ Introduce a controlled bias into selection of alternative answers.
๏ Adjust the difficulty of the questions as the game progresses.
๏ New game modes.
๏ Expand to other research fields.
![Page 24: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/24.jpg)
Questions?
![Page 25: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/25.jpg)
Thank you
![Page 26: IR Game: How well do you know information retrieval papers?](https://reader034.vdocuments.us/reader034/viewer/2022052311/55c31dcdbb61ebaa458b4765/html5/thumbnails/26.jpg)
Credits
© Luboš Volkov
© iconsmind.com
© Cédric Villain
© convoy
© baabullah hasan
© Luis Prado
© Mark Shorter
© Gerhard Meier
Presented by Jenda Rybák at ECIR 2015, Wien