1. daniel archambaultmaterials.dagstuhl.de/files/15/15481/15481.swm1.other.pdf · platform...
Post on 30-Apr-2020
2 Views
Preview:
TRANSCRIPT
Running order Daniel Archambault
Benjamin Bach
Kathrin Ballweg
Rita Borgo
Alessandro Bozzon
Sheelagh Carpendale
Remco Chang
Min Chen
Stephan Diehl
Darren Edwards
Sebastian Egger
Sara Fabrikant
Brian Fisher
Ujwal Gadiraju
Neha Gupta
Matthias Hirth
Tobias Hoszligfeld
Jason Jacques
Radu Jianu
Christian Keimel
Andreas Kerren
Stephen Kobourov
Bongshin Lee
David Martin
Andrea Mauri
Fintan McGee
Luana Micallef
Sebastian Moumlller
Babak Naderi
Martin Noumlllenburg
Helen Purchase
Judith Redi
Peter Rodgers
Dietmar Saupe
Ognjen Šćekić
Paolo Simonetto
Tatiana von Landesberger
Ina Wechsung
Michael Wybrow
Michelle Zhou
Daniel Archambault
Daniel Archambault
bull Lecturer Swansea University
bull Research Interests
bull Information Visualisation
bull Graph Drawing and Visualisation
bull Perceptual Issues
bull Scalability Issues
bull Human-Centred Methodology to Evaluate Visualisations
Seminar Interests
bull Human-Centred Evaluation of Visualisations
bull How to Effectively and Correctly Use Crowdsourcing
bull Experimental methodology (between vs within)
bull When not to use crowdsourcing as well as when to use it
Benjamin Bach
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
Temporal Data Dynamic Networks Domain Collaborations
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
How to motivate experts in participating
What kind of feedback can we expect
Which infrastructure do we need
How to find interested domain experts
How to design (micro) tasks
Kathrin Ballweg
Kathrin Ballweg ndash About me
Interests
Computer Science
Cognitive Psychology
User Studies
PhD Focus
Perception and cognition in
network visualization
Visualization for the masses
(journalists and public ndash news readers)
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 9
Kathrin Ballweg
M Sc Computer Science
Technische Universitaumlt
Darmstadt
Design
Evaluation in the Crowd Crowdsourcing and
Human-Centered Experiments
Topics of Interest
Crowdsourcing Platforms vs
The Laboratory
Scientifically Rigorous Methodologies
Crowdsourcing Experiments in Human-
Computer Interaction Visualization and
Applied PerceptionGraphics
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 10
httpwwwtagesnetzwerkde
Rita Borgo
Introduction
Rita Borgo
Visual Computing
Swansea University
Large Data Visualisation HPC
Glyph Based Visualisation
Time Series Analysis
Perception in Visualisation
Visual Computing Group Swansea University
Prof MWJones
D Archamabault R Borgo
R S Laramee B Mora
K W Tam X Xie
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Daniel Archambault
Daniel Archambault
bull Lecturer Swansea University
bull Research Interests
bull Information Visualisation
bull Graph Drawing and Visualisation
bull Perceptual Issues
bull Scalability Issues
bull Human-Centred Methodology to Evaluate Visualisations
Seminar Interests
bull Human-Centred Evaluation of Visualisations
bull How to Effectively and Correctly Use Crowdsourcing
bull Experimental methodology (between vs within)
bull When not to use crowdsourcing as well as when to use it
Benjamin Bach
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
Temporal Data Dynamic Networks Domain Collaborations
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
How to motivate experts in participating
What kind of feedback can we expect
Which infrastructure do we need
How to find interested domain experts
How to design (micro) tasks
Kathrin Ballweg
Kathrin Ballweg ndash About me
Interests
Computer Science
Cognitive Psychology
User Studies
PhD Focus
Perception and cognition in
network visualization
Visualization for the masses
(journalists and public ndash news readers)
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 9
Kathrin Ballweg
M Sc Computer Science
Technische Universitaumlt
Darmstadt
Design
Evaluation in the Crowd Crowdsourcing and
Human-Centered Experiments
Topics of Interest
Crowdsourcing Platforms vs
The Laboratory
Scientifically Rigorous Methodologies
Crowdsourcing Experiments in Human-
Computer Interaction Visualization and
Applied PerceptionGraphics
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 10
httpwwwtagesnetzwerkde
Rita Borgo
Introduction
Rita Borgo
Visual Computing
Swansea University
Large Data Visualisation HPC
Glyph Based Visualisation
Time Series Analysis
Perception in Visualisation
Visual Computing Group Swansea University
Prof MWJones
D Archamabault R Borgo
R S Laramee B Mora
K W Tam X Xie
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Daniel Archambault
bull Lecturer Swansea University
bull Research Interests
bull Information Visualisation
bull Graph Drawing and Visualisation
bull Perceptual Issues
bull Scalability Issues
bull Human-Centred Methodology to Evaluate Visualisations
Seminar Interests
bull Human-Centred Evaluation of Visualisations
bull How to Effectively and Correctly Use Crowdsourcing
bull Experimental methodology (between vs within)
bull When not to use crowdsourcing as well as when to use it
Benjamin Bach
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
Temporal Data Dynamic Networks Domain Collaborations
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
How to motivate experts in participating
What kind of feedback can we expect
Which infrastructure do we need
How to find interested domain experts
How to design (micro) tasks
Kathrin Ballweg
Kathrin Ballweg ndash About me
Interests
Computer Science
Cognitive Psychology
User Studies
PhD Focus
Perception and cognition in
network visualization
Visualization for the masses
(journalists and public ndash news readers)
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 9
Kathrin Ballweg
M Sc Computer Science
Technische Universitaumlt
Darmstadt
Design
Evaluation in the Crowd Crowdsourcing and
Human-Centered Experiments
Topics of Interest
Crowdsourcing Platforms vs
The Laboratory
Scientifically Rigorous Methodologies
Crowdsourcing Experiments in Human-
Computer Interaction Visualization and
Applied PerceptionGraphics
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 10
httpwwwtagesnetzwerkde
Rita Borgo
Introduction
Rita Borgo
Visual Computing
Swansea University
Large Data Visualisation HPC
Glyph Based Visualisation
Time Series Analysis
Perception in Visualisation
Visual Computing Group Swansea University
Prof MWJones
D Archamabault R Borgo
R S Laramee B Mora
K W Tam X Xie
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Seminar Interests
bull Human-Centred Evaluation of Visualisations
bull How to Effectively and Correctly Use Crowdsourcing
bull Experimental methodology (between vs within)
bull When not to use crowdsourcing as well as when to use it
Benjamin Bach
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
Temporal Data Dynamic Networks Domain Collaborations
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
How to motivate experts in participating
What kind of feedback can we expect
Which infrastructure do we need
How to find interested domain experts
How to design (micro) tasks
Kathrin Ballweg
Kathrin Ballweg ndash About me
Interests
Computer Science
Cognitive Psychology
User Studies
PhD Focus
Perception and cognition in
network visualization
Visualization for the masses
(journalists and public ndash news readers)
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 9
Kathrin Ballweg
M Sc Computer Science
Technische Universitaumlt
Darmstadt
Design
Evaluation in the Crowd Crowdsourcing and
Human-Centered Experiments
Topics of Interest
Crowdsourcing Platforms vs
The Laboratory
Scientifically Rigorous Methodologies
Crowdsourcing Experiments in Human-
Computer Interaction Visualization and
Applied PerceptionGraphics
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 10
httpwwwtagesnetzwerkde
Rita Borgo
Introduction
Rita Borgo
Visual Computing
Swansea University
Large Data Visualisation HPC
Glyph Based Visualisation
Time Series Analysis
Perception in Visualisation
Visual Computing Group Swansea University
Prof MWJones
D Archamabault R Borgo
R S Laramee B Mora
K W Tam X Xie
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Benjamin Bach
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
Temporal Data Dynamic Networks Domain Collaborations
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
How to motivate experts in participating
What kind of feedback can we expect
Which infrastructure do we need
How to find interested domain experts
How to design (micro) tasks
Kathrin Ballweg
Kathrin Ballweg ndash About me
Interests
Computer Science
Cognitive Psychology
User Studies
PhD Focus
Perception and cognition in
network visualization
Visualization for the masses
(journalists and public ndash news readers)
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 9
Kathrin Ballweg
M Sc Computer Science
Technische Universitaumlt
Darmstadt
Design
Evaluation in the Crowd Crowdsourcing and
Human-Centered Experiments
Topics of Interest
Crowdsourcing Platforms vs
The Laboratory
Scientifically Rigorous Methodologies
Crowdsourcing Experiments in Human-
Computer Interaction Visualization and
Applied PerceptionGraphics
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 10
httpwwwtagesnetzwerkde
Rita Borgo
Introduction
Rita Borgo
Visual Computing
Swansea University
Large Data Visualisation HPC
Glyph Based Visualisation
Time Series Analysis
Perception in Visualisation
Visual Computing Group Swansea University
Prof MWJones
D Archamabault R Borgo
R S Laramee B Mora
K W Tam X Xie
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
Temporal Data Dynamic Networks Domain Collaborations
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
How to motivate experts in participating
What kind of feedback can we expect
Which infrastructure do we need
How to find interested domain experts
How to design (micro) tasks
Kathrin Ballweg
Kathrin Ballweg ndash About me
Interests
Computer Science
Cognitive Psychology
User Studies
PhD Focus
Perception and cognition in
network visualization
Visualization for the masses
(journalists and public ndash news readers)
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 9
Kathrin Ballweg
M Sc Computer Science
Technische Universitaumlt
Darmstadt
Design
Evaluation in the Crowd Crowdsourcing and
Human-Centered Experiments
Topics of Interest
Crowdsourcing Platforms vs
The Laboratory
Scientifically Rigorous Methodologies
Crowdsourcing Experiments in Human-
Computer Interaction Visualization and
Applied PerceptionGraphics
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 10
httpwwwtagesnetzwerkde
Rita Borgo
Introduction
Rita Borgo
Visual Computing
Swansea University
Large Data Visualisation HPC
Glyph Based Visualisation
Time Series Analysis
Perception in Visualisation
Visual Computing Group Swansea University
Prof MWJones
D Archamabault R Borgo
R S Laramee B Mora
K W Tam X Xie
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Benjamin Bach Microsoft Research ndash Inria Joint Centre Saclay France
How to motivate experts in participating
What kind of feedback can we expect
Which infrastructure do we need
How to find interested domain experts
How to design (micro) tasks
Kathrin Ballweg
Kathrin Ballweg ndash About me
Interests
Computer Science
Cognitive Psychology
User Studies
PhD Focus
Perception and cognition in
network visualization
Visualization for the masses
(journalists and public ndash news readers)
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 9
Kathrin Ballweg
M Sc Computer Science
Technische Universitaumlt
Darmstadt
Design
Evaluation in the Crowd Crowdsourcing and
Human-Centered Experiments
Topics of Interest
Crowdsourcing Platforms vs
The Laboratory
Scientifically Rigorous Methodologies
Crowdsourcing Experiments in Human-
Computer Interaction Visualization and
Applied PerceptionGraphics
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 10
httpwwwtagesnetzwerkde
Rita Borgo
Introduction
Rita Borgo
Visual Computing
Swansea University
Large Data Visualisation HPC
Glyph Based Visualisation
Time Series Analysis
Perception in Visualisation
Visual Computing Group Swansea University
Prof MWJones
D Archamabault R Borgo
R S Laramee B Mora
K W Tam X Xie
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Kathrin Ballweg
Kathrin Ballweg ndash About me
Interests
Computer Science
Cognitive Psychology
User Studies
PhD Focus
Perception and cognition in
network visualization
Visualization for the masses
(journalists and public ndash news readers)
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 9
Kathrin Ballweg
M Sc Computer Science
Technische Universitaumlt
Darmstadt
Design
Evaluation in the Crowd Crowdsourcing and
Human-Centered Experiments
Topics of Interest
Crowdsourcing Platforms vs
The Laboratory
Scientifically Rigorous Methodologies
Crowdsourcing Experiments in Human-
Computer Interaction Visualization and
Applied PerceptionGraphics
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 10
httpwwwtagesnetzwerkde
Rita Borgo
Introduction
Rita Borgo
Visual Computing
Swansea University
Large Data Visualisation HPC
Glyph Based Visualisation
Time Series Analysis
Perception in Visualisation
Visual Computing Group Swansea University
Prof MWJones
D Archamabault R Borgo
R S Laramee B Mora
K W Tam X Xie
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Kathrin Ballweg ndash About me
Interests
Computer Science
Cognitive Psychology
User Studies
PhD Focus
Perception and cognition in
network visualization
Visualization for the masses
(journalists and public ndash news readers)
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 9
Kathrin Ballweg
M Sc Computer Science
Technische Universitaumlt
Darmstadt
Design
Evaluation in the Crowd Crowdsourcing and
Human-Centered Experiments
Topics of Interest
Crowdsourcing Platforms vs
The Laboratory
Scientifically Rigorous Methodologies
Crowdsourcing Experiments in Human-
Computer Interaction Visualization and
Applied PerceptionGraphics
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 10
httpwwwtagesnetzwerkde
Rita Borgo
Introduction
Rita Borgo
Visual Computing
Swansea University
Large Data Visualisation HPC
Glyph Based Visualisation
Time Series Analysis
Perception in Visualisation
Visual Computing Group Swansea University
Prof MWJones
D Archamabault R Borgo
R S Laramee B Mora
K W Tam X Xie
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Evaluation in the Crowd Crowdsourcing and
Human-Centered Experiments
Topics of Interest
Crowdsourcing Platforms vs
The Laboratory
Scientifically Rigorous Methodologies
Crowdsourcing Experiments in Human-
Computer Interaction Visualization and
Applied PerceptionGraphics
231115 | Dagstuhl Seminar ndash Evaluation in the Crowd Crowdsouring and Human-Centered Experiments | K Ballweg | 10
httpwwwtagesnetzwerkde
Rita Borgo
Introduction
Rita Borgo
Visual Computing
Swansea University
Large Data Visualisation HPC
Glyph Based Visualisation
Time Series Analysis
Perception in Visualisation
Visual Computing Group Swansea University
Prof MWJones
D Archamabault R Borgo
R S Laramee B Mora
K W Tam X Xie
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Rita Borgo
Introduction
Rita Borgo
Visual Computing
Swansea University
Large Data Visualisation HPC
Glyph Based Visualisation
Time Series Analysis
Perception in Visualisation
Visual Computing Group Swansea University
Prof MWJones
D Archamabault R Borgo
R S Laramee B Mora
K W Tam X Xie
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Introduction
Rita Borgo
Visual Computing
Swansea University
Large Data Visualisation HPC
Glyph Based Visualisation
Time Series Analysis
Perception in Visualisation
Visual Computing Group Swansea University
Prof MWJones
D Archamabault R Borgo
R S Laramee B Mora
K W Tam X Xie
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
My Interest in the Crowdsourcing Phenomena
bull Crowdsourcing is a precious and very appealing resource
bull Crowdsourcing is a powerful instrument
bull However it is an instrument we do not know thoroughly yet we use ithellip
Therefore
1 Can we help each other to identify ldquoguidelinesrdquo to help us use this instrument in the best way an to its full potentials
2 Is there anything WE can DO to make it even better
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Alessandro Bozzon
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Alessandro Bozzon
Assistant Professor TU Delft Human-Enhanced Data Management Web Information Systems Social Data Science
Faculty Fellow IBM Benelux Inclusive Enterprise
Investigator AMS Social Sensing Smart Citizens
Web Engineering Web Science
Information Retrieval
User Modelling Crowdsourcing
Human Computation
WWW
ICWE
UMAP
HT
VLDBJ
TWEB
IC
SWJ
JWE
JWS
CSCW
ISWC
Publication Venues
wwwalessandrobozzoncom
Topics of Interest
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
How can humans and machines better collaborate in solving
(computational) problems
Machines Social
Data
People
Process
A s
ocio
-tech
nic
al syste
m How can human-generated
Web data be transformed
into a source that informs
Web system design
How to enhance Web-
based systems with
automated large-scale
human interpretation
GO
AL
S Correctness
Task Design
Experiment Methodologies
Efficiency Worker Modeling
Getting to Know the Crowd
Sustainability Incentives
Getting to Know the Crowd
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Sheelagh Carpendale
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Sheelagh Carpendale Canada Research Chair Information Visualization
NSERCAITFSMART Industrial Research Chair Interactive Technologies
InnoVis (Innovations in Visualization)
Interactions Lab
Computer Science University of Calgary
Interactive
Information
Visualization
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
I have come here to learn
bull While I do a lot of empirical research most is qualitative
bull While I think that crowd sourcing is potentially a fantastic new resource I think successful use of crowd sourcing is non-trivial
bull I hope to learn about how to work with this
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Remco Chang
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
21 Remco Chang ndash Dagstuhl 15
From vision science to data science applying perception to problems in big data
Remco Chang
Assistant Professor Computer Science
Tufts University
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
22 Remco Chang ndash Dagstuhl 15
Weberrsquos Law
119889119875 = 119896119889119878
119878
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Min Chen
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Min Chen Professor of Scientific Visualization
University of Oxford (2011 ~ Present) Swansea University (1984 ~ 2011)
Research Interests most aspects of visualization aspects of HCI computer graphics computer vision data mining and information theory
Research Highlight
1 Volume Graphics and Visualization
2 Video Visualization
3 Visual Analytics
4 Perception and Cognition in Visualization
5 Theory of Visualization
1 2
3 4
5
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Research Question in the Context of 15481
Short Version How can we use crowdsourcing to capture perceptual and cognitive measures to facilitate mathematical modelling of visualization and interaction
Long Version Information theory is perhaps the most promising mathematical framework that can underpin the fields of visualization and interaction as well as data processing and communication However to obtain its core measures various types of entropy one needs to capture a probability distribution of all states associated with a concept While such a probability distribution can be captured in a machine-centric data processing or communication pipeline it is not trivial to capture in a human-centric process Crowdsourcing offers an opportunity to conduct ldquobigrdquo observation and measurement of perceptual and cognitive activities It is thus a tool for ldquoBig Sciencerdquo
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Stephan Diehl
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
Stephan Diehl Software Engineering University of Trier Germany
The Java Code Clone
Detection API
Empirical Software Engineering bull [JSS2015 ESEM2015 FSE2014 ICSM 2012
FSE 2011 ESEM 2011 hellip]
Empirical Information Visualization bull [InfoVis 2010 J of Media Psychology 410 hellip]
Analysis and Collaboration SE Tools bull [ASE2014 ASE 2012 ICSE 2011 TSE 2005 ]
Visualizing (Dynamic) Compound Graphs bull [EuroVisSTAR 2014 IV Journal 2012 InfoVis11
SOFTVIS10 EuroVis09 IV09 EuroVis08
CHASE08 AVI08GD04GD02 hellip]
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Dagstuhl Seminar 15481 Evaluation in the Crowd Crowdsourcing and Human-Centred Experiments
bull How to reach your (very specific) target audience without
spamming
bull How to control for resolution animation speed color
bull What crowdsourcing platforms exist and what are their
features
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Darren Edwards
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Darren Edwards Swansea University
Cognitive psychology - categorization
Health and clinical psychology ndash applications of categorization on clinical populations (eg autism)
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Crowdsourcing and psychological interventions
Recent validity studies confirm the use of simple psychological tasks such as decision making
Recent attempts have now explored more complex tasks such as psychological interventions (eg mindfulness)
Present studies now explore more complex forms of mindfulness
What are the limitations of using crowdsourced populations online such as when using psychological interventions or categorization
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Sebastian Egger
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Engineering
Technical high school
bull Telematics
electrotechnical enginieering telecommunications
bull Signal Processing for UWB systems
Sociology
Theory of human societies
Interaction amp collaboration behaviour of groups
Measurement of human behaviour
Currently Human Computer Interaction
Social amp collaborative aspects of technology use
33 26112015
Sebastian Egger
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Motivational aspects of crowdworkers
CS as a collaborative work setting
Optimal Setup of CS experiments
Questions (max Nr of questions max duration)
Gamified reliability checks
Scale designs
CS vs Lab whatlsquos the ground truth
New application domains of CS
34 26112015
Interest in Crowdsourcing
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Sara Fabrikant
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
humanmdashsystem (visuo-spatial) interface
spatio-temporal analytics
spatialization human navigation
multivariate spatial analysis etc)
interface design
large amp small interactive (map) displays
(ie desktop mobile VR etc)
empirical evaluations
(ie eye tracking EDA EEG etc) [httpwwwunitsevenconz]
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
s a r a irina fabrikant geographer | giscientist | cognitive mapepatician
human sensing observatory set up
controlled laboratory | in-situ (messy) real world setting
semi|automatic data collection of various human behavioral data streams
ie psycho-physiological perceptual cognitive and sensory-motor behaviors (through crowdsourcing )
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Brian Fisher
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
My double research life
Brian Fisher
bull PhD in Experimental Psych
bull Postdoc w Cognitive Science
society founder amp president
bull Psychonomics Fellow
bull VIS-related symposia at Cogsci
amp APS papers on cogsci of
interaction
bull Fuzzy-logicBayes models
bull Postdoc funded by Inst for Robotics
and AI
bull VAST SC VEC VACCINE
bull Co-organized Dagstuhl ldquoInteraction
with Information for Visual Reasoningrdquo
Cogsci-based papers at VIS CHI
BELIV
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
My Plan
VIS
Me Psychological Science
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Ujwal Gadiraju
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
p = 0055111
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Crowd Worker
Behavior
Microtask
Design
Crowdsourcing Paradigm Quality amp Effectiveness
UJWAL GADIRAJU
Incentivization Gamification
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Neha Gupta
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Getting to know the crowd
Presented at Dagstuhl by
Dr David Martin amp Neha Gupta
Neha Gupta
PhD Student School of Computer Science University of Nottingham UK
Investigating crowdwork through the platform Amazon Mechanical Turk
psxng1nottinghamacuk
Neha_Gi_ji
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Research amp Research teams
Research Studies
bull Qualitative study of Turkers on online forums like mturkforum and Turker Nation by Dr David Martin XRCE
bull Ethnographic Studies of Turkers in their respective homes and places of work in India by Neha Gupta UoN
Research teams
Xerox
bull David Martin Jacki OrsquoNeill (Now MSR India) Ben Hanrahan (Now PSU)
psxng1nottinghamacuk
Neha_Gi_ji
Nottingham
bull Andy Crabtree
Tom Rodden
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Matthias Hirth
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Matthias Hirth 48 48
About Myself
Matthias Hirth
Research Assistant at University of Wuumlrzburg
httpmatthias-hirthcom
matthiashirthinformatikuni-wuerzburgde
Crowdsourcing
Platform analysis
Platform optimization
Human factors
Use cases
Online Social Networks
Collaboration networks
Structural differences of OSNs
Application of OSN relationships
Interactive applicationsGaming
Challenges of real time interactions
Serious gamingGamification
E-Sports broadcastinglive streaming
Quality of Experience
Monitoring techniques
Quantification techniques
(Crowd-bases) test design
Social interactions
Test methodology Incentive design
Use case
Measurement technique
Identification of
qualification
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Matthias Hirth 49 49
Improving Crowdsourcing Technology and the Way We Use it
Crowdsourcing technology
How to transfer well known lab procedures into an uncontrolled
crowdsourcing environment
Which crowdsourcing platform features are required to conduct
successful experiments but are not present yet
Which technical developments (can) foster the usage of
crowdsourcing instead of lab experiments
How can technological developments be used for novel incentive
mechanisms
Usage of Crowdsourcing
How can we conduct reproducible experiments in diverse
crowdsourcing environment
How can we raise the awareness of crowdsourcing employers for the
individuals behind the crowd
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Tobias Hoszligfeld
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
The crowd community in the lab
Tobias Hoszligfeld wwwmaswiwiuni-duede
bull Background Tobias Hoszligfeld
ndash 2004-2014 Research group ldquoFuture Internet Applicationsrdquo Chair of Communication Networks Univ of Wuumlrzburg
ndash Since Sep 2014 Chair of Modeling of Adaptive Systems (head) University of Duisburg-Essen
bull Interests Application of crowdsourcing
ndash User studies on Quality of Experience (QoE) for Internet apps eg video QoE in the crowd amp in the lab methodology reliability statisticshellip [2]
ndash Mobile crowdsourcing network and context measurements eg Wifi offloading incentives privacy recommendation mechanisms etc [1]
bull Interests The crowd community
ndash Crowd = Human beings Social network wisdom of the crowd community
ndash UnderstandingUtilizing the crowd community for successful human computation For which jobs can social interaction improve the quality of work Classification
ndash How can we deploy such a system where interaction is triggered
ndash System design Task design interaction mechanisms incentives
22 Nov 2015
[1] httpdropsdagstuhldeopusvolltexte20134354
[2] httpswww3informatikuni-wuerzburgdeqoewikiqualinetcrowd
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Challenges Example of Crowdsourced QoE
Tobias Hoszligfeld wwwmaswiwiuni-duede
task design
incentives
Internet
crowd
community
anonymous
remote
subjects
Internet delivery of
test data and user
response
QoE study bull testing methodology bull QoE influence factors
under investigation bull additional QoE factors bull proper subset of items bull hellip
crowdsourcing setting bull short task duration bull instructions amp training bull feedback amp questions bull hidden influence factors bull reliability of users bull hellip
heterogeneous devices
and Internet access
implementation extrinsic vs intrinsic
motivation
Processing of data
Technical aspects bull server implementation bull client application amp
compatibility bull desired test conditions bull context monitoring bull hellip
test items
lab
scientific methodologies
22 Nov 2015
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Jason Jacques
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Jason Jacques supervised by Per Ola Kristensson
T
Crowdsourcing Code Perception
jtj21camacuk peopledscamacukjtj21
jtjacquesgmailcom jsonjcouk
PhD Working Title
science
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Crowdsourcing Interests W
hat
W
ho
Ho
w
Wh
y jtj21camacuk jsonjcouk science
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Radu Jianu
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Radu Jianu Assistant Professor
Florida International University (PhD Brown University lsquo12)
Interests
Applications of eye-tracking in visualization tracking what people look at rather than where they look
bull Data intensive large scale eye-tracking experiments bull Gaze-contingent visualizations bull Gaze-driven data recommendation and big data curation
Automating user studies in data visualizations (on next page)
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Automating user studies
1 Software designs that can (semi)automate a broad range of quantitative evaluations by leveraging
ndash Crowdsourcing ndash Benchmark data and task taxonomies ndash Automated result interpretation
2 Moving user evaluation from the end to within the software design and implementation process as a means to choose between competitive visual designs
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Christian Keimel
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
copy IRT
Where
What
bull Hybrid broadcast broadband technologies
bull Big Data in broadcasting environment and content-creation
bull Audio-visual QoE assessment (with crowdsourcing)
About me
Christian Keimel Slide 60
Lecturer for Digital Broadcast Engineering
Research Engineer for Media Services and Platforms
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
copy IRT
Crowdsourcing in large scale optimisation
The human component of Big Data
Moving crowdsourcing to lean-back devices
Crowdsourcing from the couch
Crowdsourced experiments emulating the laboratory
Is it time for new methodologies ignoring the lab
Questionshellip
Christian Keimel Slide 61
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Andreas Kerren
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Seminar 15481
Andreas Kerren kerrenacmorg
bull Affiliation
ndash Professor in CS at Linnaeus University Sweden
ndash Head of the ISOVIS Research Group
ndash httpcslnuseisovis
bull Current research interests
ndash Multivariate network analysis navigating in networks
ndash Visual text analytics
ndash BCI-supported evaluation
ndash Collaborativeadaptive information visualization
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Seminar 15481
Andreas Kerren kerrenacmorg
bull Irsquom interested in to learn more on or to discuss
ndash Which evaluation tasks for interactive visualizations are
appropriate for crowdsourcing
ndash More concrete is evaluation in the crowd suitable for
testing more complex interactions
bull It seems that this is problematic due to the motivation of
the people
ndash Which platform is the best for my purposes and what are
the pros and cons of each
ndash Practical guidelines to launch such experiments
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Stephen Kobourov
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Background
Stephen Kobourov
Computer Science
University of Arizona
Research Interests
information visualization
graph drawing
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Topics
Point-clouds graphs and maps
Edge crossings in graph drawing
Straight-line and curved edges
Semantic wordclouds
Cartograms
In progress
- memorability
- engagement
- enjoyment
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Bongshin Lee
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Bongshin Lee
Senior Researcher Microsoft Research
PhD University of Maryland College Park 2006
Analyze Share Collect
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Understand and share the experiences in running crowdsourcing experiments
Discuss the strengths and limitations of crowdsourcing experiments
Formulate the future research directions in crowdsourcing experiments
Interests
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
David Martin
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Crowdsourcing and Human-Centred Experiments
bull David Martin
ndash Senior scientist Xerox Research Centre Europe
ndash Sociologist specializing in ethnographic (detailed qualitative observational) studies of social organisation
bull Studies for design and innovation
bull Studies of design and innovation
bull Looking at use and impacts of technology in society
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Crowdsourcing Interests
bull Understanding the use understanding and impact of crowdsourcing on crowdsourcing workers
bull Crowdsourcing as a new form of work ndash ie as a global form of technologically enabled outsourcing and as precarious work
bull Designing technologies to aid and empower crowdsourcing workers
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Andrea Mauri
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 75
bull Third year PhD Student at Politecnico di Milano
bull Thesis ldquoMethodologies for the development of crowd and social-based
applicationsrdquo
Defined a design methodology a specification paradigm deploying and monitoring
crowd-based applications
bull Interests
Crowdsourcing and human computation
Social media content analysis
Model driven engineering
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Dipartimento di Elettronica Informazione e Bioingegneria
Andrea Mauri 76
Crowdsourcing Experiments in Human-Computer Interaction Visualization and
Applied PerceptionGraphics
bull How the crowd can be involved in the development of a modeling language
Crowdsourcing Platforms vs The Laboratory
Scientifically Rigorous Methodologies
bull Best practices
Ethics in Experiments
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Fintan McGee
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
FINTAN MCGEE
Graph amp Information Visualization
bull Lab Based Evaluation
bull Edge Bundling
bull Matrix VS Node link for community visualization
bull Biological Data Visualization
Phd from Trinity College Dublin (2013)
bull Graphics Vision and Visualization Group (GV2)
eScience Unit - Luxembourg Institute of Science and
Technology
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
SEMINAR INTERESTS
Obtaining suitable populations
bull For application domain experts
bull For general visualization approaches
Crowdsourcing best practices
Crowdsourcing evaluation of visualization within
mobile applications
Insights
bull Not available in lab experiments
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Luana Micallef
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Luana Micallef Postdoctoral Researcher
Research Interests
Information Visualization Biological Data Visualization Set Visualization
Diagram Layout Optimization Visual Design and Analytics Human-Computer Interaction
Ubiquitous
Interaction
Group
Probabilistic
Machine
Learning
Group
Giulio Jacucci Samuel Kaski Aalto University and University of Helsinki
Honorary Research Fellow [2014ndash2017]
Research Fellow [2013ndash2014]
PhD [2009ndash2013] Visualizing Set Relations and Cardinalities Using Venn and
Euler Diagrams Advisor Peter Rodgers
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Luana Micallef Interests Related to Crowdsourcing and Human-Centred Experiments
Using Amazon Mechanical Turk (MTurk)
evaluated the effectiveness of different visualizations
eg set visualization techniques for different tasks
Next consider crowdsourcing for the evaluation of
interactive visualizations user interfaces and information retrieval systems
learn more about
other crowdsourcing technologies besides MTurk
characteristics of the crowd and ethics of crowdsourcing in research
Also adopted different spammer-catching techniques and
created templates for crowdsourced web experiments
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Babak Naderi
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Babak Naderi Research Scientist PhD Candidate
Quality and Usability Lab
Telekom Innovation Laboratories
Technische Universitaumlt Berlin
Background Study Bachelorrsquos degree Software Engineering Master lsquos
Geodesy and Geoinformation Science
Activities PhD topic Motivation of Crowd workers on Online Crowdsourcing
Micro-Task Platforms
Focus on Measuring motivation of crowd workers amp evaluating factors influencing it Influence of motivation on performance Design guideline for lsquoproperrsquo crowdsourcing task Difference between Lab and crowdsourcing studies
Crowdee Mobile Crowdsourcing platform provided by QUL TU-Berlin
Turkmotion Initiative that aims to support the crowdworkersrsquo community Share your vote - enjoy your work
Interests Motivation What can change initial Motivation (External -gt
Intrinsic) Look at the motivation in details (not a black box) how does it change
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Martin Noumlllenburg
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Helen Purchase
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Empirical studies in information visualisation and aesthetic design
Sketched graph layout
Graph drawing aesthetics
static graphs dynamic graphs curved edges cascades
Aesthetic interface design and complexity
Helen Purchase
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Interesting questions
It is really lsquobetterrsquo to get crowd-sourced empirical data If so why and under what conditions
Do our empirical design methods need to change when we use crowd-sourced empirical data
If so why and how
Do our analysis methods need to change when we use crowd-sourced empirical data
If so why and how
Who are our participants Why do they do it Does it matter If so why and how does this affect what we do
Helen Purchase
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Judith Redi
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Assistant Professor
Multimedia
Computing Group
Delft
Genoa
Ivrea
Eindhoven
Sophia
Antipolis
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
QOE CONTROL
Configuration of
technology
settings
Quality
restoration
User-centered
technology
design
Low
quality
Q = 03
Quality assessment Quality Preservation
Model of
user preferences
Subjective Testing
93
Usually lab based time consuming
limited ecological
validity
Can crowdsourcing help
Can we obtain similar results in a CS and a contolled
Lab setting for QoE measurements How
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Peter Rodgers
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Peter Rodgers
Reader amp Director of Research
School of Computing
University of Kent
UK
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Research Interests Developing new Set amp Network Visualizations
Empirical Evaluation of Information Visualizations
Currently using MTurk amp Laboratory based techniques
httpwwwcskentacukpeoplestaffpjrlinear With Gem Stapleton and Peter Chapman
httpwwwcskentacukpeoplestaffpjrEulerVennCirclesEulerVennApplethtml With Stirling Chow
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Dietmar Saupe
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Universitaumlt Konstanz
Chair for Multimedia Signal Processing University of Konstanz Germany Prof Dr Dietmar Saupe
Current Research Projects
bull Powerbike Data acquisition analysis modeling optimization of
performance in endurance sports (competitive cycling)
bull BrainCycles Recordinganalysing brain activity of Parkinson
patients during walkingcycling
bull IVQA Objectivesubjective assessment of imagevideo quality
using crowd sourcing eyetracking
bull Automated detection of archaeological sites in high resolution
remotely sensed imagery
Dagstuhl Seminar Crowdsourcing 23-27-112015 98 22112015
CV
Education in (applied) mathematics
Diploma (1979)
Dr rer nat (1982)
Habilitation (1993)
Past research
Numerical continuation methods
Chaos and fractals
Fractal image compression
Computer graphics
Visualization
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Universitaumlt Konstanz
Crowd Sourcing for Image and Video Quality Assessment
Dagstuhl Seminar Crowdsourcing 23-27-112015 99 22112015
Consumer video quality
Quality indices
- Overall visual
- Motion
- Color
- Comfort
Limitations of current subjectiveobjective assessment
bull Video databases have narrow spectrum of content
bull only with controlled artificial distortions (compression blurring )
Can be overcome by crowd sourcing
bull Web harvesting for thousands of videos (diversity)
bull with bdquonaturalldquo degradations (authenticity)
Challenges
bull Only a single version per film
bull No ground truth available (limits to no-reference VQA)
bull The usual limitations of crowd sourcing
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Ognjen Šćekić
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Ognjen Šćekić Research assistant Distributed Systems Group TU Wien Austria httpdsgtuwienacat
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
Research Interests bull Rewarding and incentives in
Social Computing bull Human-Provided Services (HPS) bull Collective Adaptive Systems (CAS) bull Service-Oriented Computing
Currently involved in research of hybrid human-machine socio-technical systems (EU FP7 SmartSociety project)
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Dagstuhl Seminar 15481 November 22ndash27 2015
Evaluation in the Crowd Crowdsourcing and Human-Centered Experiments
bull SmartSociety ndash platform for crowdsourcing complex collaborative tasks to hybrid collectives
bull Expecting to use the platform for running experiments and studies involving human participants negotiations agreement controllability incentives
bull Interested in experiences of other participants and discussing ethical questions
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Paolo Simonetto
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Paolo Simonetto
Studied worked at
bull University of Padua
bull BRC University of Glasgow
bull LaBRI University of Bordeaux
bull GAMA University of Arizona
Topics
bull Visualization of clustered graphs
bull Generation of Euler diagrams
bull Graph drawing with containment constraints
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
How not to misuse crowdsourcing
Crowdsourcing vs Standard
bull Design time equal +
bull Supervision time none +
bull Physical presence nah +
bull Analysis same = _________
Letrsquos make it clear
bull When not to use it
bull How to use it properly
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Tatiana von Landesberger
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Tatiana von Landesberger
Head of Junior Research Group
TU Darmstadt Germany
Research interests Visual Analysis of Networks
Visual Analysis of Spatio-temporal data
Visual Analysis of Medical Data
Applications
Finance
Biology
Transportation
Social Media (News Twitter)
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Interests
Visualization for the masses
Crowdsourcing data
for and in visualization
Evaluation of visualization
for and by the masses
httptagesnetzwerkde
wwwviphyorg
MobilityGraphs
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Ina Wechsung
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Ina Wechsung
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Michael Wybrow
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Michael Wybrow
bull Lecturer ndash Faculty of IT Monash University
bull Research bull Geometric layout constraints bull Diagram amp document layout bull Connector routing bull Information visualisation bull Interaction design and HCI
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Michael Wybrow
bull Some relevant interests to this seminar
bull PCs and tablets are inherently distracting How focused are crowdsourced participants
bull How can we determine a participantrsquos intentions from their interactions with an online system
bull The value of recording interactions for later replay
bull ResearchKit Applersquos framework for building mobile apps for collecting crowdsourced study data
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Michelle Zhou
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
1999 2008 2010 2009
T J Watson
China Research
2014
Almaden Watson Grp
Columbia
Smart Visualization for Conversational Agents
Visual Analytics Concierge
Psychology-based People Analytics
Cognitive Personal Agent Michelle Zhou
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
Getting to Know the Crowd
bull What are they like
ndash What motivates them
ndash How trustworthy are they
bull How to get to know them better
ndash Psychometrics tests
ndash Automated analysis
Motivations
Decision-Making
Style
Personality
top related