self-selection. self-selection and information role of online product reviews xinxin li, lorin hitt...
Post on 26-Dec-2015
216 Views
Preview:
TRANSCRIPT
Self-Selection
Self-Selection and Information Role of Online Product Reviews
Xinxin Li, Lorin HittThe Wharton School, University of PennsylvaniaWorkshop on Information Systems and Economics (WISE 2004)
,
Outline
IntrodutionData CollectionTrend in Consumer reviewsImpact of Consumer Reviews on Book SalesTheory Model and ImplicatonsConclusion
Introduction
Word of mouth has long been recognized as a major drivers of product sales.
eBay-like online reputation systems : a large body of work
product review websites : very little systematic research
Self-Selection ProblemThe efficacy of consumer-generated product reviews may be limited for at least two reasons.
Firms may manipulate online rating services by paying individuals to provide high ratings.
There are possibilities that the reported ratings are inconsistent with the preferences of the general population.
Ratings of products may reflect both consumer taste as well as quality.
Major Research Questions
Early adopters may have significantly different preferences than later adopters which will create trends in ratings as products diffuse.
We consider whether consumers account for these biases in ratings when making product purchase decisions.
Data CollectionA random sample of 2651 hardback books was collected from “Books in Print” covering books published from 2000-2004 that also have reviews on Amazon.
Book characteristic informationtitleauthorpublisherpublication datecategorypublication date for corresponding paperback editionsconsumer reviews
Sales-related data (every Friday from March to July in 2004)sales rankpricethe number of consumers reviewsthe average reviewshipping availability
Trend in Consumer ReviewsThe Box-Cox model :
AvgRatingit : the average review for book i at time t,T : the time difference between the date the average review was posted and the date the book was releasedui : the idiosyncratic characteristics of each individual book that keep constant over time.
Trend in Consumer Reviews (contd.)
Impact of Consumer Reviews on Book Sales
Sales rank is a log-linear function of book sales with a negative slope.
iii
ic
ii
iii
mmiesShippingDummiesCategoryDuT
omotionPLogviewNumofLog
PLogAvgRatingSalesRankLog
765
432
110
Pr][]Re[
][][
Impact of Consumer Reviews on Book Sales (contd.)
All estimates are significant and have the right sign.
With other demand-related factors controlled for, the time variant component RT has a significant impact on book sales when consumers compare different books at the same time period
]746.0exp[45.0
90.3
]746.0exp[45.090.3^^
^^
TR
R
TAvgRating
T
itii
itiit
Theory Model and Implicatons
An individual consumer’s preferences over the product can be characterized by two components (xi, qi).
The element xi is known by each consumer before purchasing and represents the consumer’s preference over product characteristics that can be inspected before purchaseThe element qi measures the quality of the product for consumer Iqe: expected quality
Conclusion
Our findings suggest the significance of product design and early period product promotion
Self-Selection Bias in Reputation Systems
Mark Kramer MITRE CorporationIFIPTM ’07
Outline
IntroductionExpectation and Self-SelectionAvoiding Bias in Reputation Management SystemsConclusion
Motivation
Can a reputation system based on user ratings accurately signal the quality of a resource?
Ratings Bias
Reputation systems appear to be inherently biased towards better-than-average ratings
Amazon: 3.9 out of 5Netflix prize data set: 3.6 out of 5 stars
Ratings Bias (contd.)
87% of ratings are 3 or higher
Possible Reasons for Positive Bias
People don’t like to be criticalPeople don’t understand the rating system or cannot calibrate themselvesLake Wobegon Effect: Most movies are better than average
Number of ratings for quality movies far exceeds number of ratings of poor movies
The SpongeBob Effect
Oscar Winners 2000-2005 : Average Rating 3.7 Stars SpongeBob DVDs : Average Rating 4.1 StarsIf SpongeBob effect is common, then ratings do not accurately signal the quality of the resource
What is Happening Here?
People choose movies they think they will like, and often they are right
Ratings only tell us that “fans of SpongeBob like SpongeBob”Self-selection
Oscar winners draw a wider audienceRating is much more representative of the general population
What is Happening Here? (contd.)
There might be a tendency to downplay the problem of biased ratings
you already "know" whether or not you would like the SpongeBob movieyou could look at written reviewsone could get personalized guidance from a recommendation engine
Importance of Self-Selection BiasBizrate 44% of consumers consult opinion sites before making online purchasesHigh ratings are the norm, contain little informationWritten reviews also can be biasedDiscarding numerical (star) ratings would eliminate an important time-saverConsumers have no idea what “discount” to apply to ratings to get a true idea of qualityNo recommendation engine will ever totally replace browsing as a method of resource selection
Model of Self-Selection BiasTwo groups:
Evaluation group E Feedback group F where F E
Consider binary situation:E = Expect to be satisfied (T/F)S = Are satisfiedR = Resource selected (and reviewed)P(S) = probability of satisfaction with resource in EP(S|R) = probability of satisfaction within F
If P(R|E) > P(R|~E) Self-SelectionAnd P(S|E) > P(S|~E) Realization of expectationsThen P(S|R) > P(S) Biased Rating
Utility and Self-SelectionSome distribution of expected utility in evaluation group E Resource will be selected only if expected utility is positive
Very high reviews can shift the expected utility curve to the right and increase the number of people selecting the resource
“Swing” group has a greater chance of disappointment
Select
# people
Expected Utility
(Evaluation Group)
Effect of Biased Rating: Example10 people see SpongeBob’s 4-star ratings
3 are already SpongeBob fans, rent movie, award 5 stars6 already know they don’t like SpongeBob, do not see movieLast person doesn’t know SpongeBob, impressed by high ratings, rents movie, rates it 1-star
Result:Average rating remains unchanged: (5+5+5+1)/4 = 4 stars9 of 10 consumers did not really need rating systemOnly consumer who actually used the rating system was misled
Paradox of Subjective Reputation“Accurate ratings render ratings inaccurate”
The purpose of reputation systems is to increase consumer satisfaction
Do better than random selectionThe mechanism is self-selection
If self-selection works, ratings will become positively biased
In the limit, all ratings will be 5-star ratingsSelf-Selection bias (SpongeBob Effect) distorts the information needed for accurate self-selectionRating system defeats itself
Dynamics of Ratings Paradox
Inaccurate or biased prior information
Accurate, complete prior information
Happy consumers
Positively biased ratings
Good self-selection
Mix of happy and unhappy consumers
Unbiased ratings
Poor self-selection
Example of Reputation Dynamics
Resource with uniformly distributed satisfaction between 0 – 100Successive groups decide whether to use the resource, based on rating# selecting resource is proportional to average rating
Example of Reputation Dynamics(contd.)
Fans first
Random people first
Ideas for Bias-Resistant Reputation Systems
Use more demographicsKids like SpongeBob, most adults do notSelf-selection is still at work within demographic subgroupDemographics might not create useful groups with different preferences
Make personalized recommendationsYes, but people still like to browseRecommendations based on biased ratings might failNetFlix recommendation engine has large error
Use written reviewsSelf-selection bias is still present
Bias-Resistant Reputation System
Want P(S) but we collect data on P(S|R)S = Are satisfied with resourceR = Resource selected (and reviewed)
However, P(S|E,R) P(S|E)Likelihood of satisfaction depends primarily on expectation of satisfaction, not on the selection decision If we can collect prior expectation, the gap between evaluation group and feedback group disappears
• whether you select the resource or not doesn’t matter
Bias-Resistant Reputation SystemBefore viewing:
I think I will:• Love this movie• Like this movie• It will be just OK• Somewhat dislike this
movie• Hate this movie
After viewing:I liked this movie:
• Much more than expected• More than expected• About the same as I
expected• Less than I expected• Much less than I expected
Big fans
Everyone else
Skeptics
ConclusionsSelf-selection bias exists in most cases of consumer choice
Bias means that user ratings do not reflect the distribution of satisfaction in the evaluation group
Consumers have no idea what “discount” to apply to ratings to get a true idea of quality
Many current rating systems may be self-defeatingAccurate ratings promote self-selection, which leads to inaccurate ratings
Collecting prior expectations may help address this problem
top related