the many faces of reputation: towards a discipline of web 2.0 reputation system design

1

Copyright © C. Dellarocas 2008-09

The Many Faces of Reputation:

Towards a discipline of Web 2.0 reputation system design

Prof. Chris DellarocasRobert H. Smith School of BusinessUniversity of [email protected]


Suddenly everything Web 2.0 became so sexy…

2


Except, perhaps, reputation systems…

RepuNomics

How Online Reputation

Enables Markets

Sustains Communities

and Creates

New Business

Opportunities

This book

does not exist


But reputation systems are everywhere

eCommerceeBayAmazonEpinions

Web 2.0YelpSlashdot

CrowdsourcingYahoo! AnswersiStockPhoto

GamingXbox Live

3


Reputation Systems are the Unsung Heroes

of the Web


Outline of this talk

Reputation Systems Serve Business Objectives

Four areas where practice runs ahead of research

Our challenge: Developing a rigorous discipline of real-life reputation system design

4


Some Definitions

Reputation = a summary of one’s past actionsdefined within the context of a specific community

Reputation system = a system that mediates and automates the process of assessing one’s reputation

Keeps track of a user’s actionsAggregates and displays summary statistics


Bird’s eye view of a reputation system

Aggregation

Outputs

Inputs

Users

(Inter)actions

Artifacts

COMMUNITY

5


Reputation Systems serve a variety of

Business Objectives


Why do communities need reputation systems?

TrustEncourage “good” and discourage “bad” behavior

QualityProvide incentives for quality contributions Recognize best contributors

Matching and FilteringAssist users in finding suitable partnersReduce information overload

Participation and loyaltyGive user reasons to join and stay in your community

6



7



8



Four examples

++++++++Yahoo!Answers

+++++++++++Yelp

++++++++Amazon

+++++++eBay

LoyaltyFilteringQualityTrust

9


The design space of Reputation Systems

is very rich


Four dimensions of reputation system designwhere practice runs ahead of research

1. What inputs should be solicited2. What outputs should be presented3. How transparent should the rules be4. How should reputation evolve over time

10


What inputs should be solicited?

What internal actions to keep track ofWhat external feedback to solicitRatings of artifacts vs. ratings of people


11


Determining inputs

Perform an audit of your systemList all actions a user can takeAssess how each action relates to each of the four classes of objectivesTrack/report actions that:

you want to encourage/discourageprovide the most information related to one or more of your key objectives


A story of a site that got it wrong: Consumating.com

12


An interesting question: Should you allow ratings of users….

… separately from the artifacts they create?

Pluses:Makes a person feel like a person… which might increase site loyalty

Minuses:Can distract from quality of one’s workMight encourage personal attacks


13



1. What inputs should be solicited2. How outputs should be presented3. How transparent should the rules be4. How should reputation evolve over time


What outputs should be presented?

Simple statisticsStar ratingsScoreNamed levelsAchievement badgesSocial network

14



Simple Statistics

15


Statistics

Score


SocialNetwork

16



17



18



To what extent should the system allow comparison across users?

Display of relative performance statisticsTop N lists“Leaderboards”

19



20



21



22


To compare or not to compare?

PlusesIncreases incentives to contributeEnhances filtering role of reputation

MinusesInstills a culture of competitionObsession with rankings might lead to manipulative, disruptive behaviorLow ranked users might be discouraged and exit


A cautionary tale

Digg.com

23


How you display reputation information affects the spirit of your community

Reputation system design must be consistent with the community’s overall culture

Statistical evidence

RankingLeaderboards

Point-based scoreBadges

NamedLevels

CORDIAL COMPETITIVE


Some ideas

Implement multiple ways of ranking usersNo single measure that people obsess onAllows users with different qualities to feel goodParticularly relevant in systems where matching and filtering is an important objective

24





25


How transparent should be the aggregation rules?

Transparency breeds trust

… but also facilitates gaming


26



27





28


How should reputation evolve over time?

Accumulate and growDecay and refresh

TradeoffsAccumulation breeds complacencyAccumulation discourages new entrants

BUTAccumulation promotes loyalty and lock-in


Accumulation

Decay and Refresh

29


A case study:Yelp


30



31



The challenge:Mapping Business Objectives

to Design

32


Observation #1: Everything should be driven from a clear understanding of objectives

Trust building/Community PolicingIncentives for Quality Matching and FilteringSite Loyalty

It is important for the designer to be very clear with respect to how these goals are prioritizedIt is important to understand how every aspect of design affects each of these goals


Observation #2: The design of a reputation system can affect a community’s culture

Reputation system design must be consistent with the community’s overall culture

Statistical evidence

RankingLeaderboards

Point-based scoreBadges

NamedLevels

CORDIAL COMPETITIVE

33


Observation #3: Intangible aspects of reputation matter

Users care about their reputation beyond the tangible benefits it confers to themReputation as pure status

Status is zero-sumUsers who are not recognized might get upsetSo, introduction of a reputation system might make some users worse off and cause them to leave


Observation #4: A reputation system can be a site’s competitive weapon

A way to lock-in users to our site

Lock-in properties of reputation system can sometimes be in conflict with its other objectives

Understanding the design of competitors’reputation mechanisms matters

34


In summary

Much of what we know about reputation system design in Web 2.0 communities is still anecdotal

We need research-driven guidelines on how to design such systems

A lot of open areas where theoretical and experimental work needs to be done


In the meantime…

Reputation systems designers will be the poor lonesome cowboys of the WebNot necessarily such a bad place to be (hey, it can get you to places like Gargonza)

the many faces of reputation: towards a discipline of web 2.0 reputation system design

Education

web copyright

chris dellarocas robert

key objectives copyright

ratings of people copyright

summary statistics copyright

personal attacks copyright

ones reputation

reputation information