an introduction to human computation and games with a purpose - part i

AN INTRODUCTION TO

HUMAN COMPUTATION &

GAMES WITH A PURPOSE

ALESSANDRO BOZZON

DELFT UNIVERSITY OF TECHNOLOGY

LUCA GALLI

POLITECNICO DI MILANO

ICWE 2013 - An Introduction To Human Computation and Games With a Purpose 2

ABOUT THE TUTORIAL• Crowdsourcing, Human Computation, and GWAPs are hot topics

• “Human Computation” => more than 3000 papers• 400 in 2013

• “Crowd Sourcing” => more than 36000 papers• 4800 in 2013

• “Games With A Purpose” => more than 1400 papers• 162 in 2013

• This short tutorial is necessarily shallow, but

• Concrete Examples• Lot of references and links• An outlook on the future

• Slides and additional materials available

• http://hcgwap.blogspot.com

Exact matches queries

performed on Google

Scholar as of July 1 st

2013


ABOUT THE SPEAKERS

ALESSANDRO BOZZON

Assistant Professor - TU Delft

http://www.alessandrobozzon.com

[email protected]

LUCA GALLI

Ph.D. Student - Politecnico di Milano

http://www.lucagalli.me

[email protected]

• RESEARCH BACKGROUND AND INTERESTS

• Web Data Management• Crowdsourcing and Human Computation• Game Design• Web Engineering and Model Driven Development

4ICWE 2013 - An Introduction To Human Computation and Games With a Purpose

AGENDA


AGENDA• PART 1 => CrowdSourcing and Human Computation

• Introduction• Design of Human Computation Tasks• Frameworks And Applications• The Future of Human Computation

• PART 2 => Games With a Purpose


PART 1HUMAN COMPUTATION


INTRODUCTION

PART 1


THE RISE OF CROWDSOURCING

• The “….sourcing” trend, from a business perspective [B_Tibbets2011]

• Outsourcing: Outsource the data center or outsource application development

• Same or better quality, less effort, less money

• Offshoring: Outsourcing to developing countries

(e.g. India, China)

• offshore outsourcing• The same quality software

at a huge discount

• CrowdSourcing: everyday people use their spare cycles to create content, solve problems, etc.

[1990’s] Outsourcing

[2000’s] Offshore outsourcing

[2010’s] CrowdSourcing• Human

Computation

SAVINGS

Jeff Howe [B_Wired2006]


THE AGE OF THE CROWD

• Distributed computing projects: UC Berkeley’s SETI@home?

• Tapping into the unused processing power of millions of individual computers

• “Distributed labor networks”

• Using the Internet (and Web 2.0) to exploit the spare processing power of millions of human brains

• Successful examples?

• Open source software: a network of passionate, geeky volunteers could write code just as well as highly paid developers at Microsoft or Sun Microsystems

• often better

• Wikipedia: creating a sprawling and surprisingly comprehensive online encyclopedia

• Quora, StackExchange: can’t exist without the contributions of users


THE AGE OF THE CROWD

• The productive potential of millions of plugged-in enthusiasts is attracting the attention of old-line business too

• Cheap Labor => Overseas Vs. Connected work forces

• Technological advances (from product design software to digital video cameras) are breaking down the cost barriers that once separated amateurs from professionals

• Smart companies in industries tap the latent talent of the crowd

“The labor isn’t always free, but it costs a lot less than paying traditional employees. It’s not outsourcing: it’s crowdsourcing”


DEFINITION OF HUMAN COMPUTATION

• “…the idea of using human effort to perform tasks that computers cannot yet perform, usually in an enjoyable manner.” [Law2009]

• “…a new research area that studies the process of channeling the vast internet population to perform tasks or provide data towards solving difficult problems that no known efficient computer algorithms can yet solve” [Chandrasekar2010]

• “…systems of computers and large numbers of humans that work together in order to solve problems that could not be solved by either computers or humans alone” (Quinn2009)

SUGGESTED VIEWS http://www.youtube.com/watch?v=tx082gDwGcM http://www.youtube.com/watch?v=Aszl5avDtek


CAPTCHA

http://xkcd.com/233/

“Completely Automated Public Turing test to tell Computers and Humans Apart”Luis von Ahn et al. 2000


THE HUMAN CO-PROCESSING UNITS (HPU) [DAVIS2010]

• Humans are a first class computational platform


A GROWING, MULTIDISCIPLINARY FIELD…


… WITH A BIG MARKET…• estimated future volume

• $454,000,000,000 per year

• 91,000,000,000 hours per year

• 45,000,000 full-time workers


… WITH COMPLEX RELATIONS BETWEEN DISCIPLINES [QUINN2011]

• Crowdsourcing

• Social computing

• Collective intelligence

• Data mining

• A lot of value here!


COLLECTIVE INTELLIGENCE

Large groups of loosely organized people can accomplish great things working together

• Traditional study focused on “decision making capabilities by a large group of people”

• Taxonomical “genome” of collective intelligence

• “… groups of individuals doing things collectively that seem intelligent” [Malone2009]

• Collective intelligence generally encompasses human computation and social computing


CROWDSOURCING AND HUMAN COMPUTATION• “Crowdsourcing is the act of taking a job traditionally performed

by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call.” (Jeff Howe)

• Human computation replaces computers with humans

• Crowdsourcing replaces traditional human workers with members of the public

• Crowdsourcing facilitates human

computation (but they are not equivalent)

• Citizen journalism, sensing, ...


SOCIAL COMPUTING• Social computing is a general term for an area the

intersection of social behavior and computational systems.

• In the broad sense of the term, social computing has to do with supporting any sort of social behavior through computational systems.

• any social software => blogs, email, instant messaging, social network services, wikis, …

• In the narrow sense of the term, social computing has to do with supporting computations that are carried out by groups of people

• collaborative filtering, online auctions, prediction markets, reputation systems, tagging, and verification games


DISTINGUISHING FEATURES OF HUMAN COMPUTATION

• Conscious Effort

• Humans are actively computing something, not merely carrier of sensors and computational devices.

• Explicit Control

• The outcome of the computation is determined by an algorithm, and not the natural dynamics of the crowd.

Although sometimes those constraints can be relaxed

• e.g. human computation on social networks, GWAPS


HISTORY OF HUMAN COMPUTATION

The term “computer” used to refer to

humans who did computation[Grier2005]

1700’sAlexis Claude de Clairaut

1800’sCharles

Babbage

1900’sWorld Wars

1940’sENIAC

Division of labor -- Mass production -- Professional managers


HISTORY OF HUMAN COMPUTATION

Alan Turing wrote in 1950:

“The idea behind digital computers may be explained by saying that these machines are intended to carry out any operations which could be done by a human computer”

[Turing1950]


ELECTRONIC VS. HUMAN COMPUTERS

Electronic• Fast

• Deterministic

• Arithmetic

Human• Slow

• Inconsistent & Noisy

• But… still better at some things


EMULATING HUMAN COMPUTERS

• Computer scientists (in the artificial intelligence field) have been trying to emulate human abilities

• Language• Visual processing• Reasoning• …• Can you think about some other hard-to-imitate human

abilities?

• Now we need humans again for the “AI-complete” tasks


EXAMPLE OF “DIFFICULT” COMPUTATIONAL PROBLEMS

Sorting

Medical Diagnosis

Object recognition

Translation

Editing

Planning

set of objects set of objects sorted

x-ray, lab tests diagnosis

Image Tag

Source sentence sentence corrected

Text Text

Goal, Constraints sequence of actions


THE HUMAN ADVANTAGE• Perception

• Perception/comprehension: reconstructing information that wasn't captured at capture-time (as in a photo or surface scan)

• Constructing/inferring information that was never recorded using knowledge humans naturally possess

• Sketch• Recognizing emotions• Labeling images

• Preference/aesthetic judgments

• evaluate goodness ("beauty") for sorting or optimization• Sims, Electric Sheep, Interactive Genetic Algorithm/Human-

Based Genetic Algorithms, [Little 2009]/[Bernstein 2011]


THE HUMAN ADVANTAGE• Creativity

• search: finding images that go well together• art projects like The Sheep Market [Koblin 2006]• [Little 2009/10] for expanding text/jokes/shirt design• [Yu and Nickerson 2011] for sketching chair designs (“Cooks or

Cobblers”)• [Bernstein 2011] for posing humans• [Kittur 2011] for wikipedia... or wikipedia


MODERN HUMAN COMPUTATION

The Open Mind Initiative (1999)

• “… a web-based collaborative framework for collecting large knowledge bases from non-expert contributors.”

• “an attempt to ... harness some of the distributed human computing power of the Internet, an idea which was then only in its early stages.”

• It accumulated more than a million English facts from 15.000 contributors


MODERN HUMAN COMPUTATION

Luis Von Ahn’s Phd Thesis

• [VonAhn2005] “A paradigm for utilizing human processing power to solve problems that computers cannot yet solve”

• “We treat human brains as processors in a distributed system, each performing a small part of a massive computation.”

• “We argue that humans provide a viable, under-tapped resource that can aid in the solution of several important problems in practice.”


AREAS OF APPLICATIONS

• Data management

• Data analytics

• Training

• Collaboration and knowledge sharing

• Customer loyalty programs

• Ad network optimization

• Virtual goods and currencies.


CROWD-ENHANCED DATA MANAGEMENTRelational

• Information Extraction

• Schema Matching

• Entity Resolution

• Data spaces

• Building structured KBs

• Sorting

• Top-k

• …

Beyond Relational

• Graph Search

• Mining and Classification

• Social Media Analysis

• NLP

• Text Summarization

• Sentiment Analysis

• Search

• …


AMAZON MTURK• Artificial Artificial Intelligence

• Provide a UI and Web Services API to allow developers to easily integrate human intelligence directly into their processing

www.mturk.com


TYPE OF TASKS IN M-TURK


CROWDFLOWER• Labor on-demand

• Less problems with • Worker engagement (see later)

• 27 Channels

• Quality control features


SOME OTHER HUMAN COMPUTATION PLATFORMS

• CloudCrowd

• DoMyStuff

• Livework

• Clickworker

• SmartSheet

• uTest

• Elance

• oDesk

• vWorker (was rent-a-coder)


BUT ALSO SEVERAL OTHERS…

• Social Networks

• Q&A Systems

• Ad-hoc crowds

• …


ETHICS AND OPPORTUNITIES

• Developer Outsourced His Job To China To surf Reddit

• [B_NextWeb2013]• New “Taylorization” era?

• More later

• Duke professor uses crowdsourcing to grade

• [B_Chronicle2009]• We can make some cool

science!

http://www.robcottingham.ca/cartoon/archive/2007-08-07-crowdsourced/


DESIGN

PART 1


COMPUTATIONThe process of mapping an input to an output

• Algorithm: An algorithm is a finite set of rules which gives a sequence of operations for solving a specific type of problem, with five important properties

• Input: quantities that are given to it initially before the algorithm begins, or dynamically as the algorithm runs.

• Output: quantities that have a specified relation to the inputs.• Finiteness: An algorithm must always terminate after a finite number

of steps• Definiteness: Each step of an algorithm must be precisely defined• Effectiveness: its operations must all be sufficiently basic that they

can in principle be done exactly and in a finite length of time by someone using pencil and paper

OUTPUTINPUT


TASK

A crowdsourced data creation/manipulation/analysis activity

typically focused on a single action (although several concurrent actions are allowed)

performed on coherent set of Objects

Also known as HIT (Human Intelligence Task)


EXAMPLES OF TASKS• Recognize and identify the

people contained in a set of image

• Input Objects: images• Output Objects: images + bounding

boxes + names

• Annotate the named entities contained in a book

• Input Objects: text organized in pages• Output Objects: set of named entities

• Crop the silhouette of the models in a set of images

• Input Objects: images• Output Objects: images + polylines

• Create a complete list of the restaurants nearby PoliMI

• Input Objects: none• Output Objects: set of

restaurant names

• Evaluate the courses offered at TUDelft

• Input Objects: set of course names

• Output Objects: course names + vote


PERFORMER

A human being involved in the execution of a Task

• A.k.a. workers, turkers, etc.

• The workforce

• Examples

• Amazon Mechanical Turk Workers• Students of the IR course• ICWE Attendees• My Facebook friends• Javascript Experts on Stack Overflow


MICROTASKAn instance of a Task, operating on a subset of its input objects,

and assigned to one or more performers for execution

• The simplest unit of execution

• Typically rewarded

• Examples

• Locate and identify the faces of the people appearing in the following 5 images

• Order the following papers according to your preference• Find me the email address of the following companies


HUMAN COMPUTATION DESIGN

• How hard is the problem? Is it efficiently solvable?

• Trade-off between human versus machine?

• Is the human computation algorithm correct and efficient?

• How do we aggregate the outputs of many human computers?

• To whom do we route each task, and how?

• How to motivate participation, and incentivize truthful outputs?

What

WhoHow

GOAL => Given a computational problem, design a solution using human computers and automated computers


TRADE-OFF• There is always a tradeoff between how much work the

human does and how much work the computer does.


EXAMPLE: SORT


XKCD CROWD-SORT

http://xkcd.com/1185/

IMPLEMENTATION!!!

http://gkoberger.github.io/stacksort/


PROBLEM TYPES• Simple Problems

• Computational problems solved by using a single human computation task

• Complex Problems

• Computational problems solved by using a set of tasks organized according to a given workflow

• Hybrid Problems

• Computational problems solved by organizing human and automatic computation in one or more workflows

• Human Orchestration

• Tasks are coordinated by humans

• Automatic Orchestration

• Task are automatically coordinated by machines

• Hybrid Orchestration

• Humans and machines coordinate tasks


TYPICAL WORKFLOW

Experiment Design

Task Design

Task Design

Task Design

Task Design

Task Design

Task Execution

Task Control

Output Aggregation and Analysis

Iterate and Improve


Click icon to add picture

SIMPLE PROBLEMS


TASK

TASK DESIGN

μTaskTask UX

Input Objects

Output Objects

Design

Interface

OperationsOutput Aggregation And Quality Control

Task Routing

Incentives

Advertisement

(Requester) Reputation Management


OPERATION TYPES• A possible (non-exhaustive) list of human computation tasks

may include:

• Data creation/modification• Object Recognition/Identification/Detection• Sorting (Clustering/Ordering)• Natural Language Processing • State Space Exploration• Content Generation/Submission• User preference/opinion elicitation


OBJECT RECOGNITION

Recognize one or several pre-specified or learned objects together with their 2D positions in the image or 3D poses in the scene.

Qiang Hao, Rui Cai, Zhiwei Li, Lei Zhang, Yanwei Pang, Feng Wu, and Yong Rui. "Efficient 2D-to-3D Correspondence Filtering for Scalable 3D Object Recognition"


IDENTIFICATION• Recognize an individual instance of an object

• Identification of a specific person's face or fingerprint• Identification of a specific vehicle


DETECTION

An image/text is analyzed to recognize a specific condition or anomalies.


CLUSTERINGTask of grouping a set of objects in such a way that objects in the same group (called cluster) are more similar (in some sense or another) to each other than to those in other groups (clusters).

Task for humans: define a (subjective) similarity measure to compare the input data with and group objects into clusters based on it.


ORDERING• Arranging items of the same kind, class, nature, etc. in some

ordered sequence, based on a particular criteria.

• define a (subjective) evaluation criteria to compare the input data and order the objects based on the chosen criteria.

Altw

aijry, H

.; Be

lon

gie

, S., "R

ela

tive ra

nkin

g o

f facia

l a

ttractive

ne

ss," Ap

plica

tion

s of C

om

pu

ter V

ision

(W

AC

V), 2

01

3 IE

EE

Wo

rksho

p o

n


TASK UI AND INTERACTION• Workers want to maximize their income and their reputation

• The UI is one of the most important aspect of the relationship with workers

• Prepare to iterate

• Ask the right questions

• Keep it short and simple. Brief and concise. • Workers may not be experts: don’t assume the same understanding in

terms of terminology

• Show examples

• Engage with the worker

• Attractiveness (worker’s attention & enjoyment) • Workers also have intrinsic motivations => Avoid boring stuff


OTHER DESIGN PRINCIPLES

• Text alignment

• Legibility

• Reading level: complexity of words and sentences

• Multicultural / multilingual

• Who is the audience (e.g. target worker community)


AGGREGATION

• Challenges:

• Output are noisy (lack of expertise) • Humans are not always reliable (cheating)• Cultural context may bias the answers

• Goal: Automatic procedure to merge Micro-task results

• Assumptions:

• There exists a “True” answer• Redundancy helps

• What to look for?

• Agreement, reliability, validity


WHAT IS TRUTH?

Objective truth

Exists freely or independently from a mind (E.g. ideas, feelings)

• Medical diagnosis, protein structure, number of birds...

Cultural truth

Shared beliefs of a group of people, often involving perceptual judgments.

• Is the music sad? Is this image pornographic? Is this text offending? ...


LATENT CLASS MODEL

• Observed : HIT outputs

• Latent (hidden) : Truth, user experience, task difficulty

• Often, the matrix is incomplete

• Ground truth may never been known

SolutionTasks

Work

ers


MAJORITY VOTE• Ask multiple labelers, keep majority

label as “true” label

• Assumptions• The output that each worker

independently generates depends on the true answer

• There is no prior information about which categories are more or less likely to be the true classification

• Quality is probability of being correct

Onm

Yn

True Answer

Observed Output

M

N


MAJORITY VOTEn computation Task

1-ε probability of the correct answer

j answerm performer


MAJORITY VOTING AND LABEL QUALITY

• Quality is probability of being correct

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

1 3 5 7 9 11 13

Qu

ali

ty fo

r M

ajo

rity

Vo

te

Number of labelers


MEASURING MAJORITY• Some statistics

• Percentage agreement• Cohen’s kappa (2 raters)• Fleiss’ kappa (any number of raters)

• But what if 2 say relevant, 3 say not?

• Use expert to break ties• Collect more judgments as needed to reduce uncertainty

• Can we try and estimate quality?


HIDDEN FACTORS

• Majority vote works best when workers have similar quality

• But workers can make random guesses or make mistakes and still agree by chance

• Worker Characteristics

• Expertise (e.g., bird identification)• Bias (e.g., mother vs college students)• Physical Conditions (e.g., fatigue)

• Task Characteristics

• Quality (e.g., blurry pictures)• Difficulty (e.g., transcription of non-native speech)


INCORPORATING WORKER QUALITY

Objective: Medical diagnosis by doctors

Model: Doctors have different rates and types of errors.

• πjl(k) defines the probability of doctor

k to declare a patient in state l when the true state is j,

• ηil(k) is the number of time the

clinician k gets responses I from patient i.

Onm

Yn

True Answer

Observed Output

M

N

πm

Worker Characteristics


INCORPORATING WORKER QUALITY• Solution: Expectation-Maximization (EM) Algorithm

(Dawid & Skene, 1979)

• Estimate the confusion matrix AND the true state of an object simultaneously, using the Expectation-Maximization (EM) algorithm, which iteratively

• estimates the true states of each object by weighing the votes of the performers according to our current estimates of their quality (as given by the confusion matrix)

• re-estimates the confusion matrices based on the current beliefs about the true states of each patient.


INCORPORATING TASK DIFFICULTY

• EXAMPLE [Welinder 2010]

• HIT: Select images containing at least one “duck”

• Competence varies with bird image

• Worker’s bias toward various mistakes

• Difficulty of the image

Onm

Yn

True Answer

Observed Output

M

N

πm

Worker Characteristics

βn

Task Difficulty


QUALITY CONTROL• An holistic problem

• It is not only about the workers performance• Is the question well expressed?• Is the UI understandable?

• You may think the worker is doing a bad job, but the same worker may think you are a lousy requester (see reputation)

• Strategies

• Beforehand => Qualification test, Screening (by quality/competence), recruiting, training

• During => Assess worker quality “as you go”• After : Accuracy metric, Filter, weight

• Still no success guaranteed!


QUALITY CONTROLSCREENING

• Approval rate

• Typically built-in in human computation platforms• Mechanical Turk recently introduced a Master qualification for workers =>

few, and very “picky”, only for specific tasks• Crowdflower Programmatic Gold

• It can be defeated

• Geographic restrictions / Workers community

• Also built-in• Mechanical Turk: US / India• Crowdflower: Mechanical Turk / Others

• White list / black list of workers

• For known superstars/spammers• To be manually maintained


QUALITY CONTROLQUALIFICATION TEST

• Prescreen workers ability to do the task (accurately)

• AMT: assign qualification to workers

• Advantages

• Great tool for controlling quality

• Disadvantages

• Extra cost to design and implement the test• May turn off workers, hurt completion time• Refresh the test on a regular basis• Hard to verify subjective tasks like judging relevance

• Try creating task-related questions to get worker familiar with task before starting task in earnest


QUALITY CONTROLGOLD TESTING

• Two strategies

• Trap questions with known answers (“honey pots”)• Measure inner-annotator agreement between workers

• An exploration-exploitation scheme:

• Explore: Learn about the quality of the workers• Exploit: Label new examples using the quality

• Assign gold labels when benefit in learning better quality of worker outweighs the loss for labeling a gold (known label) example [Wang et al, WCBI 2011]

• Assign an already labeled example (by other workers) and see if it agrees with majority [Donmez et al., KDD 2009]


GOLD TESTING

No significant advantage under “good conditions” (balanced datasets, good worker quality)

10 labels per example

2 categories, 50/50

Quality range: 0.55:1.0

200 labelers


GOLD TESTING

Advantage under imbalanced datasets


2 categories, 90/10

Quality range: 0.55:0.1.0

200 labelers


GOLD TESTING

Advantage with bad worker quality


2 categories, 50/50


200 labelers


GOLD TESTING


2 categories, 90/10


200 labelers

Significant advantage under “bad conditions” (imbalanced datasets, bad worker quality)


QUALITY CONTROLADDITIONAL HEURISTICS

• Ask workers to rate the difficulty of a task

• Let workers justify answers

• Justification/feedback as quasi-captcha

• Should be optional • Automatically verifying feedback was written by a person may be difficult

(classic spam detection task)

• Broken URL/incorrect object

• Leave an outlier in the data set • Workers will tell you • If somebody answers “excellent” for a broken URL => probably spammer

• Create cross-validating questions

• E.g. workers says that picture does not contain people but tags somebody


QUALITY CONTROLDEALING WITH BAD WORKERS

• Pay for “bad” work instead of rejecting it?

• Pro: preserve reputation, admit if poor design at fault• Con: promote fraud, undermine approval rating system

• Use bonus as incentive

• Pay the minimum $0.01 and $0.01 for bonus • Better than rejecting a $0.02 task

• If spammer “caught”, block from future tasks

• May be easier to always pay, then block as needed


• Work of many non-experts can be aggregated to approximate the answer of an expert

• However the competence and expertise of the workers do matter • E.g. knowledge intensive, domain specific tasks

• Experts are better at [Chi2006]

• generating better, faster and more accurate solutions• detecting features and deeper structures in problems• adding domain-specific and general constraints to problems• self monitoring and judging the difficulty of the task• choosing effective strategies• actively seeking information and resources to solve problems• retrieving domain knowledge with little cognitive effort.

TASK ROUTING


EXPERTISE DIMENSIONS• Knowledge

• Implicit Knowledge (e.g. language, location)• Topical Knowledge (e.g. flowers, fashion)

• Availability Reliability and Trustworthiness

• Response time• Percentage of accepted microtask executions• Masters in AMT

• Soft Skills

• E.g. Attitude


STRATEGIES• Push : System controls the

distribution of tasks

• The worker is passive• Workers have strict

preferences

• Allocation

• Worker’s expertise is known (or estimated).

• A coalition is a group of agents which cooperate in order to achieve a common task.

• (Coalition Problem). Given A, H, T ⟨, the coalition problem is to assign ⟩

tasks t T to coalitions of agents ∈C A such that the total utility is ⊆maximized and the precedence order is respected.

• Pull : Workers can browse, visualize & search data• The workers are active, and

tend to choose tasks in which they have the most expertise, interest & understanding

• Advantage• More effective on platform

with high turn-over

• Disadvantage• Coverage & completion time

• More in “Advertisement”


FINDING THE RIGHT CROWD• Crowd selection by ranking the members of a social group

according to the level of knowledge that they have about a given topic

[Bozzon2013b]


MAIN RESULTS• Profiles are less effective than level-1 resources

• Resources produced by others help in describing each individual’s expertise

• Twitter is the most effective social network for expertise matching – sometimes it outperforms the other social networks

• Twitter most effective in Computer Engineering, Science, Technology & Games, Sport

• Facebook effective in Locations, Sport, Movies & TV, Music

• Linked-in never very helpful in locating expertise

Groundtruth created trough self-assessment. For expertise need, vote on 7 Likert Scale. EXPERTS expertise above average


PICK-A-CROWD

Djellel Eddine Difallah, Gianluca Demartini, and Philippe Cudré-Mauroux. Pick-A-Crowd: Tell Me What You

Like, and I'll Tell You What to Do.

In: 22nd International Conference on World Wide Web (WWW 2013)


LIKE VS ACCURACY


INCENTIVES

“money, love, or glory”

T. W. Malone, R. Laubacher, and C. Dellarocas. Harnessing Crowds: Mapping the Genome of Collective Intelligence. Working paper no. 2009-001, MIT Center for Collective Intelligence, Feb. 2009.

Sourcing


INCENTIVESINTRINSIC VS. EXTRINSIC

People would prefer activities where they can pursue three things.

• Autonomy: People want to have control over their work.• Mastery: People want to get better at what they do.• Purpose: People want to be part of something that is bigger than they are

Intrinsic Motivations

• Enjoyment, desire to help out,

Extrinsic Motivations

• Money, praise, promotion, preferment, the admiration of peers (social rewards), etc.

Intrinsic motivations are typically more powerful than extrinsic ones, but the two classes have a strong interplaySUGGESTED VIEW http://www.ted.com/talks/dan_pink_on_motivation.html


MONETARY INCENTIVE VS. PERFORMANCE• “Rational choice” in economic theory: Rational workers will choose to improve their

performance in response to a scheme that rewards such improvements with financial gain

• Chocking effect

• [Herzberg1987] financial incentives undermine actual performance e.g., hampering innovations

• [Horton2010; Farber2008; Fehr2007] Workers may ignore rational incentives to work longer when they have accomplished pre-set targets

• [Lazear200] Autoglass factory, install windshields• Switched from time-rate wage (pay per hour) to piece-rate wage (pay per unit) brought

a 20% increase in productivity• Performance based pay scheme is a powerful tool for eliciting improved performance

=> but at what risk?

• [Gneezy2000] [Heyman2004] Under certain circumstance the provision of financial incentives can undermine “intrinsic motivation” (e.g., enjoyment, altruism), possibly leading to poorer outcome


MONEY AND TROUBLE• No expectation of financial reward

• effort motivated by other kinds of rewards• e.g.

• social• non-profit SamaSource contracts workers refugee

• Monetary compensation expected

• the anticipated financial value of the effort will be the driving mechanism

• Careful: Paying a little often worse than paying nothing!

• Price commensurate with task effort

• Ex: $0.02 for yes/no answer• Small pay now locks future pay

• $0.02 bonus for optional feedback


MONEY AND TROUBLE• Payment replaces internal motivation (paying kids to collect

donations decreased enthusiasm)• Lesson: Be the Tom Sawyer (“how I like painting the fence”),

not the scrooge-y boss…

• Paying a little:

• No interest or slow response

• Paying a lot:

• People focus on the reward and not on the task• On MTurk spammers routinely attack highly-paying tasks


EXPERIMENT: WORD PUZZLE[MASON2005]

• Want to further investigate payment incentives

* Shown a list of 15 possible words (not all of the words listed are in the puzzle)* Select a word: click the first and last letter (if correct, it will turn red)

* Two wage models: quota vs. piece rate* Quota: every puzzle successfully completed* Piece: every word they found* Pay levels: low, medium, high, (no pay) -- Puzzle: $0.01, $0.05, $0.10 -- Word: $0.01, $0.02, $0.03



• Payment incentives increase speed



Acc

ura

cy

(fra

ctio

n o

f w

orl

ds

fou

nd

per

pu

zzle

)

Cost

per

Word

No Contingent Pay Pay per Puzzle Pay per Word

Accuracy

Cost per word

High accuracy per puzzle means low cost per word

Low accuracy per puzzle, but workers find as many words as they can

Intrinsic motivation (enjoyment)


INCENTIVESSOCIALIZATION AND PRESTIGE

• Public credit contributes to sense of participation

• Credit also a form of reputation

• e.g. Leaderboards (“top participants”) frequent motivator [Farmer 2010]• Newcomers should have hope of reaching top• Should motivate correct behavior, not just measurable behavior• Whatever is measured, workers will optimize for this

• Pro:

• “free” • enjoyable for connecting with one another – can share infrastructure across tasks

• Cons:

• need infrastructure beyond simple micro-task – need critical mass (for uptake and reward

• social engineering more complex than monetary incentive • Anonymity of MTurk-like settings discourage this factor


INCENTIVESALTRUISM

• Contributing back (tit for tat): Early reviewers writing reviews because read other useful review

• Effect amplified in social networks: “If all my friends do it…” or “Since all my friends will see this…”

• Contributing to shared goal

• Help Others Who need knowledge (e.g. Freebase http://www.freebase.com/)

• Help workers (e.g. http://samasource.org/)• Charity (e.g. http://freerice.com/)

• Pro

• “Free”• Can motivate workers for a cause

• Cons

• Small workforce


INCENTIVESPURPOSE OF WORK

• Contrafreeloading: Rats and animals prefer to “earn” their food

• Destroying work after production demotivates workers. [Ariely2008]

• Showing result of “completed task” improves satisfaction

• Workers enjoy learning new skills (often cited reason for Mturk participation)

• Design tasks to be educational

• DuoLingo: Translate while learning new language [vonAhn, duolingo.com]• Galaxy Zoo, Clickworkers: Classify astronomical objects [Raddick2010;

http://en.wikipedia.org/wiki/Clickworkers]

• On MTurk [Chandler2010]

• Americans [older, more leisure-driven] work harder for “meaningful work”

• Indians [more income-driven] were not affected • Quality unchanged for both groups


INCENTIVESFUN

• Gamify the task (design details later)

• Examples

• ESP Game: Given an image, type the same word (generated image descriptions)

• Phylo: aligned color blocks (used for genome alignment)

• FoldIt: fold structures to optimize energy (protein folding)

• Fun factors [Malone, 1982, 1980]:

• timed response, • score keeping, • player skill level, • highscore lists, • and randomness


ADVERTISEMENT• Your task needs to be found!

• Mechanical Turk UI is very primitive

• Users constantly refresh the web page to find most recent HITs

• Quality of description is paramount!

• Clear title, useful keywords

• Tricks needed in order to promote tasks

• Workers pick tasks that have large number of HITs or are recent [Chilton2010]

• VizWiz optimizations [Bingham2011] :• Posts HITs continuously (to be recent) • Makes big HIT groups (to be large)


EFFECT OF #HITS: MONOTONIC, BUT SUBLINEAR

• 10 HITs 2% slower than 1 HIT

• 100 HITs 19% slower than 1 HIT

• 1000 HITs 87% slower than 1 HIT or, 1 group of 1000 7 times faster than 1000 sequential groups of 1


REPUTATION MANAGEMENT• Word of mouth effect

• Forums, alert systems

• Trust

• Pay on time?• Fair rejections?

• Clear explanation if there is a rejection

• Opportunity

• Workers looks for good tasks (time vs. reward)• – Experiments tend to go faster – Announce forthcoming

tasks (e.g. tweet)


OCCUPATIONAL HAZARDS

• Costs of requesters and admin errors are often borne by workers

• Defective HITs, too short time to finish, etc.• Worker’s rating can be affected due to such errors

• Staying safe online: phishing, scamming

• Some reports from Turker Nation: “Do not do any HITs that involve: secret shopping, ….; they are scams”

• How to moderate such instances? (in a scalable way?)

• Employers who don’t pay


HELPING WORKERS?• Augmenting M-Turk from the outside

• Few external Turking tools

• Building alternative human computation platforms?

• Offering workers legal protections (human rights)?

• Humans or machines? • Legal responsibilities?• Intellectual properties?

• Offering fair wage?

• Minimum wage? (or fair wage?)



COMPLEX PROBLEMS


COMPLEX PROBLEMS• Sometimes the problem at hand is too complex to be managed

by a single task

• Examples:

• Text transcription / summarization• Open descriptions• Dynamic planning

• Need for orchestration of several tasks

• Possibly performed with the help of humans


CONTROLS

True False

Sequence Conditional Repetition

Parallel


LOGICAL UNITS

Generate / Create

Find

Improve / Edit / Fix

Vote for accept-reject

Vote up, vote down, to generate rank

Vote for best / select top-k

Split task

Aggregate

Creation

Quality Control

Flow Control


EXAMPLE: FREE-FORM ANSWERS

• Create-Vote pattern. Break task into two HITs

• “Create” HIT• “Vote” HIT

• Vote HIT controls quality of Creation HIT

• Redundancy controls quality of Voting HIT

• Note: If “creation” very good, workers just vote “yes”

• Solution: Add some random noise (e.g. add typos)

Creation HIT

(e.g. find a URL about a

topic)

Voting HIT:

Correct or not?

Tur

kIt

tool

kit

[Litt

le e

t al

., U

IST

201

0]:

http

://g

roup

s.cs

ail.m

it.ed

u/ui

d/tu

rkit/

http://groups.csail.mit.edu/uid/turkit/


EXAMPLE: FREE-FORM ANSWERS

• Create-Improve-Compare pattern. Break task into three HITs

• “Create” HIT• “Improve” HIT• “Compare” HIT

Creation HIT

(e.g. describe the image)

Improve HIT

(e.g. improve description)

Compare HIT (voting)

Which is better?


version 1:

A parial view of a pocket calculator together with some

coins and a pen.

version 2:

A view of personal items a calculator, and some gold and copper coins, and a round tip pen, these are all pocketand wallet sized item used for business, writting, calculating prices or solving math problems and purchasing items.

version 3:

A close-up photograph of the following items: A CASIO multi-function calculator. A ball point pen, uncapped. Various coins, apparently European, both copper and gold. Seems to be a theme illustration for a brochure or document cover treating finance, probably personal finance.

version 4:

…Various British coins; two of £1 value, three of 20p value and one of 1p value. …

version 8:

“A close-up photograph of the following items: A CASIO multi-function, solar powered scientific calculator. A blue ball point pen with a blue rubber grip and the tip extended. Six British coins; two of £1 value, three of 20p value and one of 1p value. Seems to be a theme illustration for a brochure or document cover treating finance - probably personal finance."


EXAMPLE: SOYLENT• Word processor with crowd embedded [Bernstein2010]

• “Proofread paper”: Ask workers to proofread each paragraph

• Lazy Turker: Fixes the minimum possible (e.g., single typo)• Eager Beaver: Fixes way beyond the necessary but adds

extra errors (e.g., inline suggestions on writing style)

• Find-Fix-Verify pattern

• Separate Find and Fix, does not allow Lazy Turker• Separate Fix-Verify ensured quality

http://www.youtube.com/watch?v=n_miZqsPwsc


Find

Fix

Verify

“Identify at least one area that can be shortened without changing the meaning of the paragraph.”

“Edit the highlighted section to shorten its length without changing the meaning of the paragraph.” Soylent, a prototype...

“Choose at least one rewrite that has style errors, and at least one rewrite that changes the meaning of the sentence.”

Independent agreement to identify patches

Randomize order of suggestions



HYBRID PROBLEMS


HYBRID PROBLEMS

David De Rourehttp://www.slideshare.net/davidderoure/social-machinesgss


KEY ISSUES• The role of machine (i.e., algorithm) and humans

• use only humans?• both? • Who’s doing what?

• Quality control

• Optimization: What to crowdsource

• Scalability: How much to crowdsource


EXAMPLEINTEGRATION WITH MACHINE LEARNING

• Crowdsourcing is cheap but not free

• Cannot scale to web without help

• We need to know when and how to use machines along with humans

• Solution

• Build automatic classification models using crowdsourced data• Humans label training data• Use training data to build model


TRADE-OFF FOR MACHINE LEARNING MODELS

• Get more data

• Active Learning, select which unlabeled example to label [Settles, http://active-learning.net/]

• Improve data quality

• Repeated Labeling, label again an already labeled example [Sheng et al. 2008, Ipeirotis et al, 2010]


ITERATIVE TRAINING• Use model when confident, humans otherwise

• Retrain with new human input => improve model => reduce need for humans


HOW OFTEN WE CAN REFER TO THE CROWD?

• Interaction protocol

• Upfront: Ask all the B queries at once • Iterative: Ask K queries to the crowd and use them to

improve the system. Repeat this B/K times

All Human Intelligent Tasks (HIT) are NOT equally difficult for the machine

• Measures used for selection

• Uncertainty: Asking hardest (most ambiguous) questions• Explorer: Ask questions with potential to have largest

impact on the system


EXAMPLEHYBRID IMAGE SEARCH

Yan, Kumar, Ganesan, CrowdSearch: Exploi?ng Crowds for Accurate Real-?me Image Search on Mobile Phones, Mobisys 2010.


EXAMPLEHYBRID DATA INTEGRATION

Generate Plausible Matches

• Paper = title, paper = author, paper = email, paper = venue• Conf = title, conf = author, conf = email, conf = venue

Ask Users to Verify

McC

ann, Shen, D

oan: Matching S

chemas in O

nline

Com

munities. IC

DE

, 2008


EXAMPLECROWDQ: CROWDSOURCED QUERY UNDERSTANDING

• Understand the meaning of a keyword query

• Build a structured (SPARQL) query template

• Answer the query

over Linked Open

Data

Gianluca Demartini, Beth Trushkowsky, Tim Kraska, and Michael Franklin. CrowdQ:

Crowdsourced Query Understanding. In: 6th Biennial Conference on Innovative Data

Systems Research (CIDR 2013)`

124

ICWE 2013 - An Introduction To Human Computation and Games With a Purpose

FRAMEWORKS

PART 1


CROWD-SOURCING DB SYSTEMSHow can crowds help databases?

• Fix broken data• Entity Resolution, inconsistencies • Add missing data• Subjective comparison

How can databases help crowd apps?

• Lazy data acquisition• Game the workers market• Semi-automatically create user interfaces• Manage the data sourced from the crowd

Existing systems mainly academic

• CrowdDB (Berkley, ETH)• Qurk (MIT)• Scoop (Stanford)


GENERIC ARCHITECTURE


CROWDDB

GOAL: crowd-source comparisons, missing data

• SQL with extensions to the DML and the Query Language


CROWDSQL SEMANTICS


UI EXAMPLES


CROWDDBUSER INTERFACE VS. QUALITY


CROWDSEARCHER• Given that crowds spend times on social networks…

• Why don’t use social networks and Q&A websites as additional human computation platforms?

• Example:

search task

[Bozzon2012][Bozzon2013b]

http://crowdsearcher.search-computing.org


SEARCH ON SOCIAL NETWORKS

Embedded application

Social/ Crowd platformNative

behaviours

External application

Standalone application

API

Embedding

Community / Crowd

Generated query template

Native


MULTI-PLATFORM DEPLOYMENT


MODEL• Support for several types of task operations

• Like, Comment, Tag, Classify, Add, Modify, Order, etc.

• Several strategies for

• Task splitting: the input data collection is too complex relative to the cognitive capabilities of users.

• Task structuring: the query is too complex or too critical to be executed in one shot.

• Task routing: a query can be distributed according to the values of some attribute of the collection

• Output aggregation

• Platform/community assignment

• a task can be assigned to different communities or social platforms based on its focus


REACTIVE CONTROL• Controlling crowdsourcing tasks is a fundamental issue

• Cost• Time• Quality

• A conceptual framework for modeling crowdsourcing computations and control requirements

• Reactive Control Design• Active Rule programming framework• Declarative rule language

• A reactive execution environment for requirement enforcement and reactive execution


RULE EXAMPLE


WORKFLOWS WITH MECHANICAL TURK

HIT

HIT

HIT

HIT

HIT

HIT

Data Collected in CSV File

Requester posts HIT Groups to Mechanical Turk

Data Exported for Use


CROWDFORGE

Map-Reduce framework for crowds [Kittur et al, CHI 2011]

My Boss is a Robot (mybossisarobot.com), Nikki Kittur (CMU) + Jim Giles (New Scientist)

• Easy to run simple, parallelized tasks.

• Not so easy to run tasks in which turkers improve on or validate each others’ work.

http://mybossisarobot.com/


CROWDWEAVER

[Kittur et al. 2012]


TURKOMATIC• Crowd creates workflows [Kalkani et al,

CHI 2011]:

• Turkomatic interface accepts task requests written in natural language

• Ask workers to decompose task into steps (Map)

• Can step be completed within 10 minutes?

• Yes: solve it. • No: decompose further (recursion)

• Given all partial solutions, solve big problem (Reduce)


DECOMPOSITION


EVALUATION• Tasks:

• Producing a written essay in response to a prompt: “please write a five-paragraph essay on the topic of your choice”

• Solving an example SAT test “Please solve the 16-question SAT located at http://bit.ly/SATexam”

• Payment: $0.10 to $0.40 per HIT• Each “subdivide” or “merge” HIT received answers within 4

hours; solutions to the initial task were completed within 72 hours

• Essay: the final essay (about “university legacy admissions”) displayed a reasonably good understanding of a topic; yet the writing quality is often mixed

• SAT: the task was divided into 12 subtasks (containing 1-3 questions); the score was 12/17


TURKITHuman Computation Algorithms on

Mechanical Turk [Little2010]

• Arrows indicate the flow of information.

• Programmer writes 2 sets of source code:

• HTML files for web servers• JavaScript executed by TurKit

• Output is retrieved via a JavaScript database.

• TurKit: Java using Rhino to interpret JavaScript code, and E4X2 to handle XML results from MTurk

• IDE: Google App Engine3 (GAE)


CRASH-AND-RERUN PROGRAMMING MODEL• Observation: local computation is cheap, but the external class

cost money

• Managing states over a long running program is challenging

• Examples: Computer restarts? Errors?

• Solution: store states in the database (in case)

• If an error happens, just crash the program and re-run by following the history in DB

• Throw a “crash” exception; the script is automatically re-run.• New keyword “once”:

• Remove non-determinism• Don’t need to re-execute an expensive operation (when re-run)


EXAMPLE: QUICK SORT


CROWD-POWERED SEARCH• Users ask questions on Twitter

• An hybrid system provide answers

• Workers used for

• label tweets as “rhetorical question” or not

• Median 3.02 mins

• produce responses to question

• Median 77.4 mins

• Voting responses• Median 82.1 mis

A Crowd-Powered Socially Embedded Search Engine. Jin-Woo Jeong, Meredith Ringel Morris, Jaime Teevan, Daniel Liebling. ICWSM 2013

Median time =>162.5 minutes

Cost => $0.95 per tweet

156


FUTURE

PART 1


WHAT LIES AHEAD OF US?What would it take for us to be proud of our children growing up to be crowd workers*?

*any work that could be sent down a wire

• Ethics, entangled with methods and tools

• Should workers be treated as undifferentiated and discardable?

• Should requesters be viewed as distant and wielding incredible power to deny payment or harm reputations?

• Work is complex, creative, and interdependent

• Could a crowd compose a symphony?

A.Kittur et al. The future of crowd work. CSCW '13. ACM, New York, NY, USA, 1301-1318.


A FRAMEWORK FOR IMPROVEMENT


IMPROVEWORKER EXPERIENCE• Reputation system for workers

• More than financial incentives

• Education? Recognition? Status?

• Recognize worker potential (badges)

• Paid for their expertise • Steering User Behavior with Badges [WWW2013]

• Train less skilled workers (tutoring system)

• Can we facilitate this process and deliver work suited to the person’s expertise, all the way along that process?


IMPROVE WORK• Promote workers to management roles

• Create gold labels• Manage other workers• Make task design suggestions (first-pass validation)

• Career trajectory (based on reputation):

1. Untrusted worker

2. Trusted worker

3. Hourly contractor

4. Employee


IMPROVE WORKTASK RECOMMENDATION

• Content-based recommendation

• Find similarities between worker profile and task characteristics

• Collaborative Filtering

• Make use of preference information about tasks (e.g. ratings) to infer similarities between workers

• Hybrid

• A mix of both


IMPROVE PLATFORMS

• What is a platform?

• Know your crowd: Model workers

• Work-flows

• Enforce Quality

• Ubiquitous crowdsourcing


HOW TO BUILD SOCIAL SYSTEMS AT SCALE?

Social Machines!!

David Rourehttp://www.slideshare.net/davidderoure/social-machinesgss

164


REFERENCES


PAPERS• [Grier2005] When Computers Were Human”

• [Turing1950] http://www.loebner.net/Prizef/TuringArticle.html

• [VonAhn2005] A.M. Turing. Computing Machinery and Intelligence. http://reports-archive.adm.cs.cmu.edu/anon/2005/abstracts/05-193.html

• [Mason2009] Winter Mason and Duncan J. Watts. 2009. Financial incentives and the "performance of crowds". In Proceedings of the ACM SIGKDD Workshop on Human Computation (HCOMP '09), ACM, New York, NY, USA, 77-85.

• [Lazear200] Lazear, E. P. Performance pay and productivity. American Economic Review, 90, 5 (Dec 2000), 1346-1361.

• [Gneezy2000]Gneezy, U. and Rustichini, A. Pay enough or don't pay at all. Q. J. Econ., 115, 3 2000), 791-810.[Heyman2004] Heyman, J. and Ariely, D. Effort for Payment: A Tale of Two Markets. Psychological Science, 15, 11 2004), 787-793.

• [Herzberg1987] Herzberg, F. One More Time: How do You Motivate Employees? Harvard Business ReviewSeptember-October, 1987), 5-16.

• [Kittur2013] Aniket Kittur, Jeffrey V. Nickerson, Michael Bernstein, Elizabeth Gerber, Aaron Shaw, John Zimmerman, Matt Lease, and John Horton. 2013. The future of crowd work. In Proceedings of the 2013 conference on Computer supported cooperative work (CSCW '13). ACM, New York, NY, USA, 1301-1318.


PAPERS• [Farber2008] Farber. Reference-dependent preferences and labor supply: The case of New

York City taxi drivers. American Economic Review, 2008.

• [Fehr2007] Fehr and Goette. Do workers work more if wages are high?: Evidence from a randomized field experiment. American Economic Review, 2007

• [Chandler2010] Chandler and Kepelner, Breaking Monotony with Meaning: Motivation in Crowdsourcing Markets , 2010

• [Farmer2010] Farmer and Glass, Building Web Reputation Systems, O’Reilly 2010

• [Horton2010] Horton and Chilton: The labor economics of paid crowdsourcing. EC 2010

• [Quinn2011] Alexander J. Quinn and Benjamin B. Bederson. 2011. Human computation: a survey and taxonomy of a growing field. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '11). ACM, New York, NY, USA, 1403-1412.

• [Snow2008] Snow, Rion and O'Connor, Brendan and Jurafsky, Daniel and Ng, Andrew. Cheap and Fast -- But is it Good? Evaluating Non-Expert Annotations for Natural Language Tasks, Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, October 2008, Honolulu, Hawaii.

• [Chi2006] M.Chi. Two approaches to the study of experts’ characteristics. In K.A.Ericsson,N.Charness, P. J. Feltovich, and R. R. Hoffman, editors, The Cambridge handbook of expertise and expert performance, pages 21–30. Cambridge University Press, 2006. Cited on page(s) 35


PAPERS• [Kittur2012] Aniket Kittur, Susheel Khamkar, Paul André, and Robert Kraut. 2012.

CrowdWeaver: visually managing complex crowd work. In Proceedings of the ACM 2012 conference on Computer Supported Cooperative Work (CSCW '12). ACM, New York, NY, USA, 1033-1036.

• [ImageNet] http://www.image-net.org/about-publication

• Mason and Watts, Financial Incentives and the “Performance of Crowds”, HCOMP 2009

• Yan, Kumar, Ganesan, “CrowdSearch: Exploiting Crowds for Accurate Real-time Image Search on Mobile Phones”, MobiSys 2010

• Ipeirotis, Analyzing the Mechanical Turk Marketplace, XRDS 2010

• Wang, Faridani, Ipeirotis, Estimating Completion Time for Crowdsourced Tasks Using Survival Analysis Models. CSDM 2010

• Chilton et al, Task search in a human computation market, HCOMP 2010

• Bingham et al, VizWiz: nearly real-time answers to visual questions, UIST 2011

• Horton and Chilton: The labor economics of paid crowdsourcing. EC 2010


PAPERS• Huang et al., Toward Automatic Task Design: A Progress Report, HCOMP

2010

• Quinn, Bederson, Yeh, Lin.: CrowdFlow: Integrating Machine Learning with Mechanical Turk for Speed-Cost-Quality Flexibility

• Parameswaran et al.: Human-assisted Graph Search: It's Okay to Ask Questions, VLDB 2011

• Mitzenmacher, An introduction to human-guided search, XRDS 2010

• Marcus et al, Crowdsourced Databases: Query Processing with People, CIDR 2011

• Raykar, Yu, Zhao, Valadez, Florin, Bogoni, and Moy. Learning from crowds. JMLR 2010.

• Mason and Watts, Collective problem solving in networks, 2011

• Dellarocas, Dini and Spagnolo. Designing Reputation Mechanisms. Chapter 18 in Handbook of Procurement, Cambridge University Press, 2007


TUTORIALS• Ipeirotis (WWW2011)

• http://www.slideshare.net/ipeirotis/managing-crowdsourced-human-computation

• Omar Alonso, Matthew Lease (SIGIR 2011)• http://www.slideshare.net/mattlease/crowdsourcing-for-information-retrieval-

principles-methods-and-application

• Omar Alonso, Matthew Lease (WSDM 2011)• http://ir.ischool.utexas.edu/wsdm2011_tutorial.pdf

• Gianluca Demartini, Elena Simperl, Maribel Acosta (ESWC2013)

• https://sites.google.com/site/crowdsourcingtutorial/

• Bob Carpenter and Massimo Poesio• http://lingpipe-blog.com/2010/05/17/lrec-2010-tutorial-modeling-data-annotation/


TUTORIALS• Bob Carpenter and Massimo Poesio

• http://lingpipe-blog.com/2010/05/17/lrec-2010-tutorial-modeling-data-annotation/

• Omar Alonso• http://wwwcsif.cs.ucdavis.edu/~alonsoom/crowdsourcing.html

• Alex Sorokin and Fei-Fei Li• http://sites.google.com/site/turkforvision/

• Daniel Rose• http://videolectures.net/cikm08_rose_cfre/

• A. Doan, M. J. Franklin, D. Kossmann, T. Kraska (VLDB 2011)

• Crowdsourcing Applications and Platforms: A Data Management Perspective.

• List by Matt Lease http://ir.ischool.utexas.edu/crowd/


BLOGS AND ONLINE RESOURCES

• [B_Tibbets2011] http://soa.sys-con.com/node/1996041

• [B_Wired2006] http://www.wired.com/wired/archive/14.06/crowds.html

• [B_NextWeb2013] http://thenextweb.com/shareables/2013/01/16/verizon-finds-developer-outsourced-his-work-to-china-so-he-could-surf-reddit-and-watch-cat-videos/

• [B_Chronicle2009] http://chronicle.com/blogs/wiredcampus/duke-professor-uses-crowdsourcing-to-grade/7538

• [B_DemoTurk] http://behind-the-enemy-lines.blogspot.com/2010/03/new-demographics-of-mechanical-turk.html


BOOKS, COURSES, AND SURVEYS

• E. Law and L. von Ahn. Human Computation. Morgan & Claypool Synthesis Lectures on Artificial Intelligence and Machine Learning, 2011

• http://www.morganclaypool.com/toc/aim/1/1

• S.Ceri, A.Bozzon, M.Brambilla, P.Fraternali, S.Quarteroni. Web Information Retrieval. Springer.

• Omar Alonso, Gabriella Kazai, and Stefano Mizzaro. Crowdsourcing for Search Engine Evaluation: Why and How.

• To be published by Springer, 2011.• Deepak Ganesan. CS691CS: Crowdsourcing - Opportunities

& Challenges (Fall 2010). UMass Amherst

• http://www.cs.umass.edu/~dganesan/courses/fall10Credits: Matt Lease http://ir.ischool.utexas.edu/crowd/


BOOKS, COURSES, AND SURVEYS• Matt Lease. CS395T/INF385T: Crowdsourcing: Theory, Methods,

and Applications (Spring 2011). UT Austin.

• http://courses.ischool.utexas.edu/Lease_Matt/2011/Spring/INF385T• Yuen, Chen, King: A Survey of Human Computation Systems, SCA

2009

• Quinn, Bederson: A Taxonomy of Distributed Human Computation, CHI 2011

• Doan, Ramakrishnan, Halevy: Crowdsourcing Systems on the World-Wide Web, CACM 2011

• Uichin Lee

• http://mslab.kaist.ac.kr/twiki/bin/view/CrowdSourcing/• Jerome Waldispuhl, McGill University

• http://www.cs.mcgill.ca/~jeromew/comp766/

an introduction to human computation and games with a purpose - part i

Technology

human computation games

future of human computation

human effort

purpose icwe

hot topics human computation

human microlabor markets

crowd icwe

quinn2009 icwe