![Page 1: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/1.jpg)
1
A WHOLE NEW ZOONIVERSEGUIDELINES AND TOOLS FOR
CROWDSOURCED SCIENCE
Elena [email protected] @esimperlNovember 16th, 2016
![Page 2: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/2.jpg)
OVERVIEW• Citizen science is a fascinating subject
for Web science research• Our work helps system designers with• Frameworks of motivations and incentives
engineering• Design guidelines and recommendations• Methods to make crowdsourced tasks more
effective• Methods to study engagement and community
health
TUTORIAL@ISWC2013
![Page 3: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/3.jpg)
3
CITIZEN SCIENCE@WAIS
![Page 4: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/4.jpg)
WHAT IS CITIZEN SCIENCE
![Page 5: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/5.jpg)
5
CITIZEN SCIENCE PROJECTS
![Page 6: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/6.jpg)
6
CITIZEN SCIENCE PLATFORMS
![Page 7: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/7.jpg)
STUDYING CITIZEN SCIENCE: HUMAN COMPUTATION & CROWDSOURCING
Task design Task assignment Answer validation and aggregation Contributors’ performance Motivation and incentives
![Page 8: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/8.jpg)
STUDYING CITIZEN SCIENCE: ONLINE COMMUNITY Roles and activities Patterns of participation Community health Motivation and incentives
![Page 9: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/9.jpg)
STUDYING CITIZEN SCIENCE: OPEN SCIENCE Scientific workflows Scientific practice Publishing, citation, and peer-review models
9
![Page 10: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/10.jpg)
STUDYING CITIZEN SCIENCE: EDUCATION AND SCICOMM
Teaching methods and assessment Tutorial design Learning analytics Engagement strategy
10
![Page 11: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/11.jpg)
11
WHEN MAKES CITIZEN SCIENCE SUCCESSFUL
tasks people time
quality science community
learning social media …
![Page 12: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/12.jpg)
12
WHAT MAKES CITIZEN SCIENCE SUCCESSFUL (2)
[Cox et al., 2015]
![Page 13: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/13.jpg)
LEVELS OF ENGAGEMENT
1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 3102468
101214161820
Month since registration
Activ
e us
ers
in %
~1% of participants contribute 72% of Talk & 29% of Task
[Luczak-Roesch et al., 2014]
![Page 14: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/14.jpg)
14
DATA QUALITYExisting quality inference algorithms are limited to binary classificationConceptualise the problem for realistic workflowsDevelop efficient implementations of algorithmsCompare them
![Page 15: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/15.jpg)
15
DATA QUALITY (2) Majority Voting Find the annotation with the top vote as
the true label for each object
Message Passing Use object-specific worker messages to
represent how reliable a worker is in labelling each specific object.
Expectation Maximization Infer the true label for each object, using
annotations from all users, accounting for the error rates of each user;
Estimates the error rates of each user by comparing their annotations with inferred true labels.
Results measured in terms of different accuracy metrics and time
Experiments still ongoing Preview Majority voting performs
exceptionally well for large numbers of annotations
If less data is available, one could explore message passing (possibly in combination with majority voting)
![Page 16: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/16.jpg)
GAMIFICATION
Survey of 27 papers & 31 VCS projects
16
![Page 17: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/17.jpg)
17
SOCIALITY Discussions and engagement with volunteers are integral part of the experience
Leads to serendipitous scientific discoveries
Encourages autonomy and helps with community building
![Page 18: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/18.jpg)
WORK VS TALK
40.5%
Classifications
Talk
con
trib
ution
s
Classifications
[Luczak-Roesch et al., 2014; Tinati et al., 2015]
![Page 19: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/19.jpg)
CHAT AND INSTANT MESSAGING
Microposts
PH SG SW NN GZ CC PF SF AP WS
91%2
0
6
4
10
8
[Luczak-Roesch et al., 2014; Tinati et al., 2015, WebSci]
![Page 20: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/20.jpg)
DISCUSSION PROFILES
Deeply engaged
volunteers, few threads but multiple posts within
them
9 0.1%
Content producers,
posting across many boards and
threads
70.1%
Thread followers
and PM (one-to-
one) talkers
8 0.4%
First to respond and
question answerers
4 1%
Highly active thread
starters and answerers
across a wide range of topics
1 2.8%
Infrequent volunteers,
single thread
posts, no personal messages
5 5.5%
Watcher and starter of
many threads, but not first to
reply
36.5%
Highly active thread
starters and first to reply back
2 14.6%
Long active volunteers
(the core group), posting
sporadically
6 69.0%
[Tinati et al., 2015, WebSci]
![Page 21: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/21.jpg)
21
FROM CROWD TO COMMUNITY Survey of 48 projects and 150 publications
Identifying affordances from online community themes within literature Task visibility Goals Feedback Rewards
Community features found to have greater role than previously considered Encourage task completion, discussions
etc.
Themes align to key success factors of volunteer engagement, task completion and submission accuracy
[Reeves et al., 2017]
![Page 22: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/22.jpg)
FROM PROJECTS TO ECOSYSTEMS
Project A
Project B
Project C
Participant X
Part. Y
[Luczak-Roesch et al., 2014]
![Page 23: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/23.jpg)
DESIGNING PLATFORMSTask
specificity
Community
development
Task design
PR and engagem
ent
Bootstrapping the community
Serendipitous scientific discovery
Engaging with people, supporting profession team
Supporting individuals, finding new scientific discoveries
Obtaining new citizen scientists
Retaining people
Supporting people, improving task completion
Obtaining new citizen scientists
Reinvigorating old users
[Tinati et al., 2015, CHI]
![Page 24: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/24.jpg)
WHAT’S NEXT? Human computation & crowdsourcing Task assignment: what tasks are interesting/relevant for whom? Data quality: scalable and in real-time Peer review, collaborative approaches The role of gamification: is science a game?
Online community Making discussions more effective
Science Citizen science platforms that everyone can use New forms of publishing, citation, reproducibility, and replication
![Page 25: A whole new Zooniverse: guidelines and tools for crowdsourced science](https://reader031.vdocuments.us/reader031/viewer/2022022203/587295971a28ab36118b741b/html5/thumbnails/25.jpg)
05/02/2023 25
[email protected]@ESIMPERL
WWW.SOCIAM.ORGWWW.STARS4ALL.EU
All publications available at http://dblp.uni-trier.de/pers/hd/s/Simperl:Elena_Paslaru_Bontas