anatomy of large-scale human-computation engine
TRANSCRIPT
The Anatomy of a Large-Scale Human-Computation Engine
Shailesh Kochhar, Stefano Mazzocchi, Praveen Paritosh
freebase.com, google.com
HCOMP'10
July 25, 2010 HCOMP'10
1: Freebase & Human Computation
2: Example – Stanford Library
3: RABJ
4: Lessons
Freebase
Structured database
12 MM entites, 300 MM triples/facts
July 25, 2010 HCOMP'10
Where does the data come from?
July 25, 2010 HCOMP'10
Community contributions
Mass Data Loads
July 25, 2010 HCOMP'10
Human Judgments Improve Both
July 25, 2010 HCOMP'10
Community
Simplifying contribution through games
July 25, 2010 HCOMP'10
Mass Data Loads
Precision: QA for >99% accuracy
Recall: increase coverage
July 25, 2010 HCOMP'10
1: Freebase & Human Computation
2: Example – Stanford Library
3: RABJ
4: Lessons
July 25, 2010 HCOMP'10
Reconcile Stanford Library Catalog with freebase.com
July 25, 2010 HCOMP'10
Stanford Library Catalog
4.4MM book editions
1.3MM English book editions
1.2MM English books
600K authors
July 25, 2010 HCOMP'10
For freebase, identity is key
match books, match authors
July 25, 2010 HCOMP'10
Automatic matching insufficient
Trained judges needed to decide hard cases
July 25, 2010 HCOMP'10
How to get this?
July 25, 2010 HCOMP'10
RABJRedundant Array of Brains in a Jar
July 25, 2010 HCOMP'10
Abstraction
Powers human judgment applications
1.8MM judgments in 16 months of
operation
July 25, 2010 HCOMP'10
Provides primitives for more sophisticated
HJ applications
July 25, 2010 HCOMP'10
Questions
Judgments
Queues
Agents
July 25, 2010 HCOMP'10
Design Constraints
July 25, 2010 HCOMP'10
Content-agnostic
Dynamic data
Low latency
July 25, 2010 HCOMP'10
Architecture
July 25, 2010 HCOMP'10
Questions contain pointers to data, pushed
to a store
Questions added to queues
Metadata allows slicing and dicing
July 25, 2010 HCOMP'10
JS applications pull questions from broker
Broker matches judge to work
Apps render question, collect judgment
Broker writes judgments back to store
July 25, 2010 HCOMP'10
Declarative consensus
Yes: 3, No: 3, Skip: 2, Bad: 2, Max: 4
Broker notifies agents of consensus
July 25, 2010 HCOMP'10
Applications
July 25, 2010 HCOMP'10
matchmaker
http://matchmaker2.freebaseapps.com/
July 25, 2010 HCOMP'10
Book Edition QA
July 25, 2010 HCOMP'10
typewriter
http://typewriter.freebaseapps.com/
July 25, 2010 HCOMP'10
Scale
July 25, 2010 HCOMP'10
1.8 MM questions
2.8MM judgments
500 queues
20+ applications
July 25, 2010 HCOMP'10
1: Freebase & Human Computation
2: Example – Stanford Library
3: RABJ
4: Lessons
July 25, 2010 HCOMP'10
Relationships, relationships, relationships
This is not controversial
July 25, 2010 HCOMP'10
Spam, collusion, gaming: $0
Skill development
Communication, documentation
July 25, 2010 HCOMP'10
Don't have to pay per-judgment
Yes, this is controversial
July 25, 2010 HCOMP'10
July 25, 2010 HCOMP'10
There are always leftovers
July 25, 2010 HCOMP'10
Working on formalizing workflows
July 25, 2010 HCOMP'10
More in the paper
RABJ Architecture
Learning through feedback loops
July 25, 2010 HCOMP'10
http://rabj.freebaseapps.com/
http://wiki.freebase.com/wiki/RABJ_API/
Questions?