cristian danescu-niculescu-mizil 1, gueorgi kossinets 2, jon kleinberg 1, lillian lee 1 1 dept. of...
TRANSCRIPT
How Opinions are Received by Online Communities- Case study on Amazon helpfulness votes
Cristian Danescu-Niculescu-Mizil1, Gueorgi Kossinets2, Jon Kleinberg1, Lillian Lee1
1Dept. of Computer Science, Cornell University, 2Google Inc.
WWW 2009
Emin Sadiyev
Cmpe 493
Amazon.com layout
2
Average star rating
Helpfulness ratio
OutlineUsers’ evaluation on online reviews:
Helpfulness votesMake hypothesisProving their validityComing up with a mathematical model that
explains these behaviors
Introduction
OpinionWhat did Y think of X?
4
Introduction
Meta-OpinionWhat did Z think of Y’s opinion of X?
5
The Helpfulness of ReviewsWidely-used web sites include not just reviews,
but also evaluations of the helpfulness of the reviewsThe helpfulness vote
“Was this review helpful to you?”Helpfulness ratio:
“a out of b people found the review itself helpful”
b
a
Flow of Presentation
Hypothe-siz-ing
Verify-ing
Model-ing
7
Flow of PresentationHypoth-esizing•Con-formity
•Individ-ual-bias
•Bril-liant-but-cruel
•Quality-only
Verifying Modeling
8
Hypotheses: Social MechanismsWell-studied hypotheses for how social effects
influence group’s reaction to an opinionThe conformity hypothesisThe individual-bias hypothesisThe brilliant-but-cruel hypothesisThe quality-only straw-man hypothesis
9
HypothesesThe conformity hypothesis
Review is evaluated as more helpful when its star rating is closer to the consensus star rating Helpfulness ratio will be the highest of which
reviews have star rating equal to overall average
The individual-bias hypothesisWhen a user considers a review, he or she will
rate it more highly if it expresses an opinion that he or she agrees with
10
Hypotheses (contd.)The brilliant-but-cruel hypothesis
Negative reviewers are perceived as more intelligent, competent, and expert than positive reviewers
The Quality-only straw-man hypothesisHelpfulness is being evaluated purely based on
the textual content of reviewsNon-textual factors are simply correlates of
textual quality
11
Flow of Presentation
Hypothe-siz-ing
Verifying•Absolute deviation of helpful-ness ratio
•Signed de-viation of helpfulness ratio
•Variance of star rating and help-fulness ra-tio
•Making use of plagia-rism
Model-ing
12
HypothesesConformity•A review is evaluated as more helpful when its star rating is closer to the average star rating
Individual-bias•A review is evaluated as more helpful when its star rating is closer to evaluator’s opinion
Brilliant-but-cruel•A review is evaluated as more helpful when its star rating is below to the average star rating
Quality-only•Only textual infor-mation affects help-fulness evaluation
13
Absolute Deviation from Average Consistent with
conformity hypothesisStrong inverse
correlation between the median helpfulness ratio and the absolute deviation
Reviews with star rating close to the average gets higher helpfulness ratio
14
HypothesesConformity•A review is evaluated as more helpful when its star rating is closer to the average star rating
Individual-bias•A review is evaluated as more helpful when its star rating is closer to evaluator’s opinion
Brilliant-but-cruel•A review is evaluated as more helpful when its star rating is below to the average star rating
Quality-only•Only textual infor-mation affects help-fulness evaluation
15
Signed Deviation from AverageNot consistent with
brilliant-but-cruel hypothesisThere is tendency
towards positivityBlack lines should not be
sloped that way if it is valid hypothesis
16
HypothesesConformity•A review is evaluated as more helpful when its star rating is closer to the average star rating
Individual-bias•A review is evaluated as more helpful when its star rating is closer to evaluator’s opinion
Brilliant-but-cruel•A review is evaluated as more helpful when its star rating is below to the average star rating
Quality-only•Only textual infor-mation affects help-fulness evaluation
17
Addressing Individual-bias EffectsIt is hard to distinguish between the
conformity and the individual-bias hypothesisWe need to examine cases in which individual
people’s opinions do not come from exactly the same distributionCases in which there is high variance in star
ratingsOtherwise conformity and individual-bias are
indistinguishable Everyone has same opinion
18
Variance of Star Rating and Helpfulness Ratio
19
Helpfulness ratio is the highest with reviews of which rating is slightly-above the average
Two-humped camel plots: local minimum around average
Helpfulness ratio is the highest when star ratings of reviews have average value
HypothesesConformity•A review is evaluated as more helpful when its star rating is closer to the average star rating
Individual-bias•A review is evaluated as more helpful when its star rating is closer to evaluator’s opinion
Brilliant-but-cruel•A review is evaluated as more helpful when its star rating is below to the average star rating
Quality-only•Only textual infor-mation affects help-fulness evaluation
20
Quality-only hypothesisPossible other methods
Human annotation Could be subjective
Classification using machine learning methods We cannot guarantee the accuracies of algorithms
Plagiarized reviewsAlmost(not exact) same text
same text could be considered as spam reviewsDifferent non-textual information
If the quality-only straw man hypothesis holds, helpfulness ratios of documents in each pair should be the same
21
PlagiarismMaking use of plagiarism is effective way to
control for the effect of review textDefinition of plagiarized pair(s) of reviews
Two or more reviews of different productsWith near-complete textual overlap
Author takes %70 textual overlap as plagiarism
22
An ExampleSkull-splitting headache guaranteed!•If you enjoy thumping, skull splitting migraine headache, then Sing N Learn is for you.As a longtime language instructor, I agree with the attempt and ef-fort that this series makes, but it is the execution that ultimately weakens Sing N Learn Chinese.To be sure, there are much, much better ways to learn Chinese. In fact, I would recommend this title only as a last resort and after you’ve thoroughly exhausted traditional ways to learn Chinese …
Migraine Headache at No Extra Charge•If you enjoy a thumping, skull splitting migraine headache, then the Sing N Learn series is for you.As a longtime language instructor, I agree with the effort that this series makes, but it is the execution that ultimately weakens Sing N Learn series. To be sure, there are much, much better ways to learn a foreign language. In fact, I would recommend this title only as a last resort and after you’ve thoroughly exhausted tradi-tional ways to learn Korean …
23
Experiments with PlagiarismText quality is not the only explanatory factor
Statistically significant difference between the helpfulness ratios of plagiarized pairs
24
The plagiarized reviews with deviation 1 is significantly more helpful than those with deviation 1.5
HypothesesConformity•A review is evaluated as more helpful when its star rating is closer to the average star rating
Individual-bias•A review is evaluated as more helpful when its star rating is closer to evaluator’s opinion
Brilliant-but-cruel•A review is evaluated as more helpful when its star rating is below to the average star rating
Quality-only•Only textual infor-mation affects help-fulness evaluation
25
Flow of Presentation
Hypothe-siz-ing
Verify-ing
Model-ing•Based on in-divid-ual bias andmix-tures of distri-butions
26
Authors’ ModelBased on individual bias and mixtures of distributionsTwo distributions: one for positive, one for negative
evaluatorsBalance between positive and negative evaluators: Controversy level:
Density function of helpfulness ratios of positive evaluators Gaussian distribution of which average is -centered
Density function of helpfulness ratios of negative evaluators Gaussian distribution of which average is -centered
27
)( q
)( p
)()()( xqgxpfxh
)(xg
)(xf
p
Validity of the ModelEmpirical observation and model generated
28
ConclusionA review’s perceived helpfulness depends not just on its
content, but also the relation of its score to other scoresThe dependence of the score is consistent with a simple and
natural model of individual-bias in the presence of a mixture of opinion distributions
Directions for further researchVariations in the effect can be used to form hypotheses about
differences in the collective behaviors of the underlying populations
It would be interesting to consider social feedback mechanisms that might be capable of modifying the effects authors observed here
Considering possible outcomes of design problem for systems enabling the expression and dissemination of opinions
29
DiscussionsSo, how can we use this?
In which cases would this information be helpful?
Available information is very limited Star ratings Helpfulness ratios
Conclusion is rather trivialDoes not present new discoveries
30