faitcrowd: fine grained truth discovery for crowdsourced data aggregation fenglong ma 1, yaliang li...
Post on 24-Dec-2015
223 Views
Preview:
TRANSCRIPT
1/61
FaitCrowd: Fine Grained Truth Discovery for Crowdsourced Data Aggregation
Fenglong Ma1, Yaliang Li1, Qi Li1, Minghui Qiu2,
Jing Gao1, Shi Zhi3, Lu Su1, Bo Zhao4, Heng Ji5, Jiawei Han3
Presenter: Jing Gao1SUNY Buffalo; 2Singapore Management University; 3University of Illinois Urbana-Champaign; 4LinkedIn;
5Rensselaer Polytechnic Institute
Which of these square numbers also happens to be the sum of two smaller numbers?
16 25
36 49
https://www.youtube.com/watch?v=BbX44YSsQ2I
A B C D
50%
30%19%
1%
3
A Straightforward Aggregation Method
• Voting/Averaging– Take the value that is claimed by majority of the
sources (users)– Or compute the mean of all the claims
Which of these square numbers also happens to be the sum of two smaller numbers?
16 25
36 49
https://www.youtube.com/watch?v=BbX44YSsQ2I
A B C D
50%
30%19%
1%
5
A Straightforward Aggregation Method
• Voting/Averaging– Take the value that is claimed by majority of the
sources (users)– Or compute the mean of all the claims
• Limitation– Ignore source reliability (user expertise)
• Source reliability– Is crucial for finding the true fact but unknown
6
Source 1 Source 2 Source 3 Source 4 Source 5
Aggregation
Object
7/61
Truth Discovery
• Principle– To learn users’ reliability degree and discover
trustworthy information (i.e., the truths) from conflicting data provided by various users on the same object.
• A user is reliable if it provides many pieces of true information
• A piece of information is likely to be true if it is provided by many reliable users
8/61
Existing Work on Truth Discovery
• Existing methods– Assign single expertise (reliability degree) to each
user (source).E
xper
tise
Barack Obama
Albert Einstein
Michael Jackson
Example--Existing Truth Discovery Methods
• Input– Question Set – User Set – Answer Set
• Output– Users’ Expertise– Truths
User u1 u2 u3
Expertise 5.00E-11 0.961 3.989
Question q1 q2 q3 q4 q5 q6
Truth 1 2 2 2 1 2
QuestionUser
u1 u2 u3q1 1 2 1q2 2 1 2q3 1 2 2q4 1 2 2
q5 2 1
q6 1 2 2
Question q1 q2 q3 q4 q5 q6
Ground Truth 1 2 1 2 1 2
10/61
Overview of Our Work
• Goal– To learn fine-grained (topical-level) user expertise
and the truths from conflicting crowd-contributed answers.
Politics
Physics
Music
Example--Our Model
• Input– Question Set – User Set – Answer Set– Question Content
• Output– Questions’ Topic– Topical-Level
Users’ Expertise– Truths Question q1 q2 q3 q4 q5 q6
Truth 1 2 1 2 1 2
QuestionUser
Wordu1 u2 u3
q1 1 2 1 a b
q2 2 1 2 b c
q3 1 2 2 a c
q4 1 2 2 d e
q5 2 1 e f
q6 1 2 2 d f
Question q1 q2 q3 q4 q5 q6
Ground Truth 1 2 1 2 1 2
User u1 u2 u3
ExpertiseK1 2.34 2.70E-4 1.00K2 1.30E-4 2.34 2.35
Topic Question
K1 q1 q2 q3
K2 q4 q5 q6
FaitCrowd Model
• Overview
qmw qz qua qbqM QqN
K
eUK
u
2
qt q
'2'
'
qmy
Input Output HyperparameterIntermediate
Variable
Modeling Content Modeling Answers
– Jointly modeling question content and users’ answers by introducing latent topics.
– Modeling question content can help estimate reasonable user reliability, and in turn, modeling answers leads to the discovery of meaningful topics.
– Learning topic-level user expertise, truths and topics simultaneously.
Modeling Question Content
• Word Generation– Assume that each question is about
a single topic (the length of each question is short).
• Draw a topic indicator
qmw qz qua qbqM QqN
K
eUK
u
2
qt q
'2'
'
qmy
Modeling Question Content
• Word Generation– Assume that each question is about
a single topic (the length of each question is short).
• Draw a topic indicator
– Assume that a word can be drawn from topical word distribution or background word distribution.
• Draw a word category
qmw qz qua qbqM QqN
K
eUK
u
2
qt q
'2'
'
qmy
Modeling Question Content
• Word Generation– Assume that each question is about
a single topic (the length of each question is short).
• Draw a topic indicator
– Assume that a word can be drawn from topical word distribution or background word distribution.
• Draw a word category
• Draw a word
qmw qz qua qbqM QqN
K
eUK
u
2
qt q
'2'
'
qmy
Modeling Answers
• Answer Generation– The correctness of a user’s answer
may be affected by the question’s topic, user’s expertise on the topic and the question’s bias.
• Draw user’s expertiseqmw qz qua qb
qM QqN
K
eUK
u
2
qt q
'2'
'
qmy
Modeling Answers
• Answer Generation– The correctness of a user’s answer
may be affected by the question’s topic, user’s expertise on the topic and the question’s bias.
• Draw user’s expertise
• Draw the truth
qmw qz qua qbqM QqN
K
eUK
u
2
qt q
'2'
'
qmy
Modeling Answers
• Answer Generation– The correctness of a user’s answer
may be affected by the question’s topic, user’s expertise on the topic and the question’s bias.
• Draw user’s expertise
• Draw the truth
• Draw the bias
qmw qz qua qbqM QqN
K
eUK
u
2
qt q
'2'
'
qmy
Modeling Answers
• Answer Generation– The correctness of a user’s answer
may be affected by the question’s topic, user’s expertise on the topic and the question’s bias.
• Draw user’s expertise
• Draw the truth
• Draw the bias
• Draw a user’s answer
qmw qz qua qbqM QqN
K
eUK
u
2
qt q
'2'
'
qmy
Inference Method
• Gibbs-EM– Gibbs sampling to learn the hidden variables and .– Gradient descent to learn hidden factors and .
qmw qz qua qbqM QqN
K
eUK
u
2
qt q
'2'
'
qmy
Datasets & Measure
• Datasets– The Game Dataset
• Collected from a crowdsourcing platform via an Android App based on a TV game show “Who Wants to Be a Millionaire”.
• 2,103 questions, 37,029 sources, 214,849 answers and 12,995 words
– The SFV Dataset• Extracted from Slot Filling Validation (SFV) task of the NITS Text Analysis
Conference Knowledge Base Population (TAC-KBP) track.• 328 questions, 18 sources, 2,538 answers and 5,587 words
• Measure– Error Rate
• The lower the better
Baseline Methods
• Basic Method– MV
• Truth Discovery– Truth Finder– AccuPr– Investment– 3-Estimates– CRH– CATD
• Crowdsourcing– D&S– ZenCrowd
• Variations of FaitCrowd– FaitCrowd-b– FaitCrowd-b-g
Performance Validation
• Analysis– For easy questions (from Level 1 to Level 7), all
the methods can estimate most answers correctly.
– For difficult questions (from Level 8 to Level 10) , the performance of FaitCrowd is much better than that of the baseline methods.
– FaitCrowd performs well on both Game and SFV datasets.
Table 1: Performance on the Game Dataset.
Table 2: Performance on the SFV Dataset.
Model Validation
• Goal– Illustrate the importance of joint modeling
question content and answers by comparing with the method that conducts topic modeling and true answer inference separately.
• Explanation– Dividing the whole dataset into sub-topical
datasets will reduce the number of responses per topic, which leads to insufficient data for baseline approaches.
Table 3: Results of Model Validation.
Topical Expertise Validation
• Goal– Validate the correctness of topical expertise learned by FaitCrowd.– Ideally, the expertise estimated by the proposed method is
consistent with the ground truth accuracy.
Figure 1: Topic 2 on the Game Dataset. Figure 2: Topic 4 on the SFV Dataset.
Expertise Diversity Analysis
• Goal– Demonstrate that the topical expertise for each source varies on
different topics. – Ideally, the topical expertise should correspond to the ground
truth accuracy, i.e., the higher expertise, the higher the ground truth accuracy.
Figure 3: Source 7 on the Game Dataset. Figure 4: Source 16 on the SFV Dataset.
27/61
Summary
• Problem– Recognize the difference in source reliability among topics
on the truth discovery task and propose to incorporate the estimation of fine grained reliability into truth discovery.
• Solution– Propose a probabilistic model that simultaneously learns
the topic-specific expertise for each source, aggregates true answers, and assigns topic labels to questions.
• Results– Empirically show that the proposed model outperforms
existing methods in multi-source aggregation with two real world datasets.
28/61
Thank you!Questions?
top related