Self-Selection

Self-Selection and Information Role of Online Product Reviews

Xinxin Li, Lorin HittThe Wharton School, University of PennsylvaniaWorkshop on Information Systems and Economics (WISE 2004)

，

Outline

IntrodutionData CollectionTrend in Consumer reviewsImpact of Consumer Reviews on Book SalesTheory Model and ImplicatonsConclusion

Introduction

Word of mouth has long been recognized as a major drivers of product sales.

eBay-like online reputation systems : a large body of work

product review websites : very little systematic research

Self-Selection ProblemThe efficacy of consumer-generated product reviews may be limited for at least two reasons.

Firms may manipulate online rating services by paying individuals to provide high ratings.

There are possibilities that the reported ratings are inconsistent with the preferences of the general population.

Ratings of products may reflect both consumer taste as well as quality.

Major Research Questions

Early adopters may have significantly different preferences than later adopters which will create trends in ratings as products diffuse.

We consider whether consumers account for these biases in ratings when making product purchase decisions.

Data CollectionA random sample of 2651 hardback books was collected from “Books in Print” covering books published from 2000-2004 that also have reviews on Amazon.

Book characteristic informationtitleauthorpublisherpublication datecategorypublication date for corresponding paperback editionsconsumer reviews

Sales-related data (every Friday from March to July in 2004)sales rankpricethe number of consumers reviewsthe average reviewshipping availability

Trend in Consumer ReviewsThe Box-Cox model :

AvgRatingit : the average review for book i at time t,T : the time difference between the date the average review was posted and the date the book was releasedui : the idiosyncratic characteristics of each individual book that keep constant over time.

Trend in Consumer Reviews (contd.)

Impact of Consumer Reviews on Book Sales

Sales rank is a log-linear function of book sales with a negative slope.

iii

ic

ii

iii

mmiesShippingDummiesCategoryDuT

omotionPLogviewNumofLog

PLogAvgRatingSalesRankLog

765

432

110

Pr][]Re[

][][

Impact of Consumer Reviews on Book Sales (contd.)

All estimates are significant and have the right sign.

With other demand-related factors controlled for, the time variant component RT has a significant impact on book sales when consumers compare different books at the same time period

]746.0exp[45.0

90.3

]746.0exp[45.090.3^^

^^

TR

R

TAvgRating

T

itii

itiit

Theory Model and Implicatons

An individual consumer’s preferences over the product can be characterized by two components (xi, qi).

The element xi is known by each consumer before purchasing and represents the consumer’s preference over product characteristics that can be inspected before purchaseThe element qi measures the quality of the product for consumer Iqe: expected quality

Conclusion

Our findings suggest the significance of product design and early period product promotion

Self-Selection Bias in Reputation Systems

Mark Kramer MITRE CorporationIFIPTM ’07

Outline

IntroductionExpectation and Self-SelectionAvoiding Bias in Reputation Management SystemsConclusion

Motivation

Can a reputation system based on user ratings accurately signal the quality of a resource?

Ratings Bias

Reputation systems appear to be inherently biased towards better-than-average ratings

Amazon: 3.9 out of 5Netflix prize data set: 3.6 out of 5 stars

Ratings Bias (contd.)

87% of ratings are 3 or higher

Possible Reasons for Positive Bias

People don’t like to be criticalPeople don’t understand the rating system or cannot calibrate themselvesLake Wobegon Effect: Most movies are better than average

Number of ratings for quality movies far exceeds number of ratings of poor movies

The SpongeBob Effect

Oscar Winners 2000-2005 : Average Rating 3.7 Stars SpongeBob DVDs : Average Rating 4.1 StarsIf SpongeBob effect is common, then ratings do not accurately signal the quality of the resource

What is Happening Here?

People choose movies they think they will like, and often they are right

Ratings only tell us that “fans of SpongeBob like SpongeBob”Self-selection

Oscar winners draw a wider audienceRating is much more representative of the general population

What is Happening Here? (contd.)

There might be a tendency to downplay the problem of biased ratings

you already "know" whether or not you would like the SpongeBob movieyou could look at written reviewsone could get personalized guidance from a recommendation engine

Importance of Self-Selection BiasBizrate 44% of consumers consult opinion sites before making online purchasesHigh ratings are the norm, contain little informationWritten reviews also can be biasedDiscarding numerical (star) ratings would eliminate an important time-saverConsumers have no idea what “discount” to apply to ratings to get a true idea of qualityNo recommendation engine will ever totally replace browsing as a method of resource selection

Model of Self-Selection BiasTwo groups:

Evaluation group E Feedback group F where F E

Consider binary situation:E = Expect to be satisfied (T/F)S = Are satisfiedR = Resource selected (and reviewed)P(S) = probability of satisfaction with resource in EP(S|R) = probability of satisfaction within F

If P(R|E) > P(R|~E) Self-SelectionAnd P(S|E) > P(S|~E) Realization of expectationsThen P(S|R) > P(S) Biased Rating

Utility and Self-SelectionSome distribution of expected utility in evaluation group E Resource will be selected only if expected utility is positive

Very high reviews can shift the expected utility curve to the right and increase the number of people selecting the resource

“Swing” group has a greater chance of disappointment

Select

# people

Expected Utility

(Evaluation Group)

Effect of Biased Rating: Example10 people see SpongeBob’s 4-star ratings

3 are already SpongeBob fans, rent movie, award 5 stars6 already know they don’t like SpongeBob, do not see movieLast person doesn’t know SpongeBob, impressed by high ratings, rents movie, rates it 1-star

Result:Average rating remains unchanged: (5+5+5+1)/4 = 4 stars9 of 10 consumers did not really need rating systemOnly consumer who actually used the rating system was misled

Paradox of Subjective Reputation“Accurate ratings render ratings inaccurate”

The purpose of reputation systems is to increase consumer satisfaction

Do better than random selectionThe mechanism is self-selection

If self-selection works, ratings will become positively biased

In the limit, all ratings will be 5-star ratingsSelf-Selection bias (SpongeBob Effect) distorts the information needed for accurate self-selectionRating system defeats itself

Dynamics of Ratings Paradox

Inaccurate or biased prior information

Accurate, complete prior information

Happy consumers

Positively biased ratings

Good self-selection

Mix of happy and unhappy consumers

Unbiased ratings

Poor self-selection

Example of Reputation Dynamics

Resource with uniformly distributed satisfaction between 0 – 100Successive groups decide whether to use the resource, based on rating# selecting resource is proportional to average rating

Example of Reputation Dynamics(contd.)

Fans first

Random people first

Ideas for Bias-Resistant Reputation Systems

Use more demographicsKids like SpongeBob, most adults do notSelf-selection is still at work within demographic subgroupDemographics might not create useful groups with different preferences

Make personalized recommendationsYes, but people still like to browseRecommendations based on biased ratings might failNetFlix recommendation engine has large error

Use written reviewsSelf-selection bias is still present

Bias-Resistant Reputation System

Want P(S) but we collect data on P(S|R)S = Are satisfied with resourceR = Resource selected (and reviewed)

However, P(S|E,R) P(S|E)Likelihood of satisfaction depends primarily on expectation of satisfaction, not on the selection decision If we can collect prior expectation, the gap between evaluation group and feedback group disappears

• whether you select the resource or not doesn’t matter

Bias-Resistant Reputation SystemBefore viewing:

I think I will:• Love this movie• Like this movie• It will be just OK• Somewhat dislike this

movie• Hate this movie

After viewing:I liked this movie:

• Much more than expected• More than expected• About the same as I

expected• Less than I expected• Much less than I expected

Big fans

Everyone else

Skeptics

ConclusionsSelf-selection bias exists in most cases of consumer choice

Bias means that user ratings do not reflect the distribution of satisfaction in the evaluation group

Consumers have no idea what “discount” to apply to ratings to get a true idea of quality

Many current rating systems may be self-defeatingAccurate ratings promote self-selection, which leads to inaccurate ratings

Collecting prior expectations may help address this problem