1. 1. R. Tinati, M. Van Kleek, E. Simperl, M. Lucsak-Roesch, R. Simpson, N. Shadbolt University of Southampton, SOCIAM Project, Web & Internet Science University of Oxford, Department of Astrophysics CHI 2015 Designing for Citizen Data Analysis: A Cross-Sectional Study of A Multi-Domain Citizen Science Platform
2. 2. background method findings related work conclusions
3. 3. background method findings related work conclusions
4. 4. The Zooniverse (July 2007 - present) 1.31m+ active volunteers today 35+ projects space, climate, humanities nature, health 87+ academic papers
5. 5. Solar Stormwatch GZ Supernova* GZ Hubble* Moon Zoo Old Weather Milky Way Planet Hunters Ancient Lives Ice Hunters NEEMO* Whale FM SETI Live* Galaxy Zoo 4 Seafloor Explorer Cyclone Center Bat Detective Cell Slider Andromeda Project* Snapshot Serengeti Planet Four Notes from Nature Space Warps* Worm Watch Lab Plankton Portal Radio Galaxy Zoo Operation War Diary Disk Detective Condor Watch Sunspotter Asteroid Zoo Whats the Score Floating Forests Chicago Wildlife Watch Galaxy Zoo Quench Snapshot Supernova Penguin Watch Higgs Hunters space (astrophysics) nature & biology humanities & history climatology
6. 6. Citizen-led investigations Serendipitous Discoveries Persistent Cross-Project Community
7. 7. Hannys Voorwerp Galaxy Zoo [2007] Green Pea Galaxies Galaxy Zoo [2007] Yellow Balls Milky Way [2009] Circumbinary Planet Ph1b Planet Hunter [2012] Convict Worm Seafloor Explorer [2012] Spanish Flu Operation War Diaries [2014]
8. 8. factors that most differentiated project outcomes that led to rapid project completion & adoption on some, slower uptake on others that fostered citizen-led investigations & serendipitous discovery? what tacit knowledge gained by the Zooniverse team can be elicited to establish an understanding of: patterns and anti-patterns toward the design of future Citizen Science (CS) Study Motivations
9. 9. background method findings related work conclusions
10. 10. method Structured reflection (based on Kolb, DA. 1984) with 2 core members of the Zooniverse team project founder and lead project manager Explore the Zooniverse with Chris Lintott Setting Zooniverse HQ Oxford Data access to lab notes digital notes code repository logs blog posts historical usage log statistics
11. 11. Thematic analysis Structured reflection sessions w/ Zooniverse Team Recordings transcribed Transcriptions coded and reduced into themes Themes were then re-presented to the Zooniverse team members for discussion and refinement Final themes derived
12. 12. Thematic analysis Structured reflection sessions w/ Zooniverse Team Recordings transcribed Transcriptions coded and reduced into themes Themes were then re-presented to the Zooniverse team members for discussion and refinement Final themes derived
13. 13. Thematic analysis Structured reflection sessions w/ Zooniverse Team Recordings transcribed Transcriptions coded and reduced into themes Themes were then re-presented to the Zooniverse team members for discussion and refinement Final themes derived
14. 14. Thematic analysis Structured reflection sessions w/ Zooniverse Team Recordings transcribed Transcriptions coded and reduced into themes Themes were then re-presented to the Zooniverse team members for discussion and refinement Final themes derived
15. 15. Thematic analysis Structured reflection sessions w/ Zooniverse Team Recordings transcribed Transcriptions coded and reduced into themes Themes were then re-presented to the Zooniverse team members for discussion and refinement Final themes derived
16. 16. method http://mitpress.mit.edu/boo ks/building-successful- online-communities comparison with online communities Design Claims Kraut, Resnick et al.s Building Successful Online Communities Ch. 2. Encouraging Contribution to Online Communities Ch. 3. Encouraging Commitment in Online Communities Ch. 5. The Challenges of Dealing with Newcomers Ch. 6. Starting New Online Communities
17. 17. background method findings related work conclusions
18. 18. findings We organised our Design Claims (DCs) around the following 4 themes:* TASK-COMMUNITY INTEGRATION COMMUNITY DEVELOPMENT TASK DESIGN PUBLIC RELATIONS & ENGAGEMENT *themes were for structuring purposes, rather than theoretically driven
19. 19. TASK-COMMUNITY INTEGRATION a) citizen science participants benefit from discussing many aspects of projects, and discussion compels prolonged engagement & peripheral participation b) standard message boards work well for small online citizen science communities, but tend not to scale as communities grow c) separation between discussion & task interfaces can hinder community growth & discussion by making tasks disconnected & decontextualised from discussions (more on this next!) DC1: let them talk! - Discussion facilitation essential to CS, even from Day 1.
20. 20. TASK-COMMUNITY INTEGRATION a) citizen science participants benefit from discussing many aspects of projects, and discussion compels prolonged engagement & peripheral participation b) standard message boards work well for small online citizen science communities, but tend not to scale as communities grow c) separation between discussion & task interfaces can hinder community growth & discussion by making tasks disconnected & decontextualised from discussions (more on this next!) DC1: let them talk! - Discussion facilitation essential to CS, even from Day 1.
21. 21. DC2: Discussion enables citizen-led discovery Talking about subjects enables people to ask questions and share observations about what they see, outside the task workflows. Such affordances open up pathways to user-initiated, user-directed processes of inquiry that can sometimes lead to discovery. Linking discussions in situ with task workflow offers many advantages: establishes necessary task context framing for discussion, lowers engagement barriers to participate, & enhances overall discoverability of relevant posts. TASK-COMMUNITY INTEGRATION
22. 22. DC2: Discussion enables citizen-led discovery Talking about subjects enables people to ask questions and share observations about what they see, outside the task workflows. Such affordances open up pathways to user-initiated, user-directed processes of inquiry that can sometimes lead to discovery. Linking discussions in situ with task workflow offers many advantages: establishes necessary task context framing for discussion, lowers engagement barriers to participate, & enhances overall discoverability of relevant posts. TASK-COMMUNITY INTEGRATION
23. 23. DC3: (a) Granting roles and privileges to participants with experience can motivate contributors to assume community leadership & essential moderator roles. (b) Volunteer moderators can effectively & quickly solve many kinds of issues without science team intervention. [Adapted from Kraut et al.] 300+ posts/day across the Zooniverse DC4*: Timely support from science team members fosters citizen-led discovery follow up on potential leads by bringing in expertise, external data resources not available to citizens, extra analytical tools & techniques. COMMUNITY DEVELOPMENT
24. 24. Encouraging Contribution DC23: Rewards, whether in the form of status, privileges, or material benefits, will motivate contributions. Starting New Online Communities DC33: Promoting the status or readiness benefits of being early to an online community can attract early adopters to the community. Regulating Behaviour DC26: Reputation systems, which summarize the history of someones online behaviour, help to encourage good behaviour and deter norm violations. Comparing to Kraut et al.s Building Successful Online Communities
25. 25. Any method to skip forward in tasks, (e.g. I dont know / Skip This) discouraged attempts at best guesses and undermined motivation. (Being able to see new subjects served as a form of implicit reward; thus any Skip mechanism devalued such rewards.) Contributions from best guesses had substantial cumulative effect on project completion. DC5*: Design tasks to encourage best guesses to avoid the dont know effort trap TASK DESIGN
26. 26. DC6*: factors that seemed to most differentiate participants perceptions of Zooniverse projects included intrinsic interestingness of domain & subjects, task difficulty (effort & skill required), time required to complete each task, & frequency of targets in rare-item search tasks. However, other factors including: type of task (e.g. classification, transcription), domain of study (humanities, astrophysics, climate science, natural science), research cause (e.g. cancer research), or source of data had no perceived impact. TASK DESIGN
27. 27. DC7*: get new participants contributing straight away Avoiding introductory materials, (videos, tutorials) and allowing participants to get to work straight away on tasks with real data increased engagement, reduced bounce rate, enabled leveraging work from 1-visit wonders. (6,620,423 classifications, ~15% came from transient users who never signed up) Mandatory tutorials slowed down participant uptake, and, in turn, project completion. In-task guidance, interposed gold-standard tasks worked well. TASK DESIGN
28. 28. DC9: Providing context to tasks (e.g. background info about subjects being classified) can provide value by making tasks more interesting, thereby increasing engagement. Initially, it was thought that such background information might serve as a distraction from the primary task. However, in moderation, it led to people getting more interested in tasks, which more often led to citizen-led lines of inquiry, and increased chances of citizen-led discoveries. TASK DESIGN DC8*: performance feedback can motivate continued participation. Interspersing gold standard tasks among the real tasks, offering participants feedback, caused an increase in sustained engagement & improved performance. Participants became more accurate and faster at completing tasks in the presence of feedback.
29. 29. DC9: Providing context to tasks (e.g. background info about subjects being classified) can provide value by making tasks more interesting, thereby increasing engagement. Initially, it was thought that such background information might serve as a distraction from the primary task. However, in moderation, it led to people getting more interested in tasks, which more often led to citizen-led lines of inquiry, and increased chances of citizen-led discoveries. TASK DESIGN DC8*: performance feedback can motivate continued participation. Interspersing gold standard tasks among the real tasks, offering participants feedback, caused an increase in sustained engagement & improved performance. Participants became more accurate and faster at completing tasks in the presence of feedback.
30. 30. First 24 hrs of a project served as an accurate indicator of how the project would continue to perform. Problems could be detected early, PR strategies could be evaluated & tuned DC10*: Project launch performance indicative of future success; pre-launch PR important for recruiting new & existing members. PUBLIC RELATIONS & ENGAGEMENT DC11*: Citizen science communities can be extremely effective at promoting projects through social media campaigns.
31. 31. First 24 hrs of a project served as an accurate indicator of how the project would continue to perform. Problems could be detected early, PR strategies could be evaluated & tuned DC10*: Project launch performance indicative of future success; pre-launch PR important for recruiting new & existing members. PUBLIC RELATIONS & ENGAGEMENT DC11*: Citizen science communities can be extremely effective at promoting projects through social media campaigns.
32. 32. background method findings related work conclusions
33. 33. related work Motivation of citizen scientists (Raddick et al. 2010, 2013) Discussions related to task completion and success (Hassman et al. 2013, Mugar et al. 2014) Cross-project participation bolsters engagement and sustained activity (Luczak-Rsch et al. 2014) Changes in participants Motivations over time (Rotman, 2012)
34. 34. background method findings related work conclusions
35. 35. conclusions Designing successful citizen science projects and platforms can be improved by knowing domain knowledge and expertise, which primarily involves understanding the needs of citizen scientists. Contributing to the existing literature concerning the citizen science design space, we have provided patterns and anti-patterns of design which can improve the likelihood of sustaining an active community of participants Citizen Scientists are not crowd workers, they need to be considered as part of the scientific process. The discussions and extra work that they contribute may be extremely beneficial to the overall project success.
36. 36. thank you r.tinati@soton.ac.uk max@hip.cat sociam.org