Download - Reputation Systems
Reputation Systems
Guest LecturePaul Resnick
Associate ProfessorUniv. of Michigan
School of [email protected]
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Learning Objectives Understand
– What a reputation system is– Theory about when and why it should work– Open research questions
Participate in design– Recognize situations when it might be helpful– Raise some of the difficult design challenges
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Outline What is a reputation system? Theory: when/why they should work Empirical results Design space Case study: NPAssist recommender
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Definition A Reputation System…
– Collects– Distributes– Aggregates
…information about behavior
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Examples BBB Bizrate eBay Expertise sites
– Epinions “top reviewers”– Slashdot karma system
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Why Reputation Systems Interacting with strangers Sellers (Exchange Partners) Vary
– Skill– Effort– Ethics
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Other Trust-Inducing Mechanisms in E-commerce Insurance Escrow Fraud Prosecution
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
How Reputation Systems Should Work Information Incentive Self-selection
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Some Issues Anonymity Name changes Name trades Lending reputations Eliciting evaluation Honesty of evaluations
Interaction Type
ID changes
Anonymous every xaction
Pseudonyms
at will
Identified never
Anonymity Analysis
Interaction type
ID changes
Reputation Sharing
Trust/ cooperation
Anonymous every xaction
Pseudonyms
at will
1L Pseudonyms each arena
Identified never
Interaction type
ID changes
Reputation Sharing
Trust/ cooperation
Anonymous every xaction
none none
Pseudonyms
at will + only + only
1L Pseudonyms each arena
Identified never + and – + and 0
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
1L Pseudonyms Third-party issues pseudonyms
– No cost– Not replaceable– Reveal name to third party– Don’t reveal mapping of name to
pseudonym
Interaction type
ID changes
Reputation Sharing
Trust/ cooperation
Anonymous every xaction
none none
Pseudonyms
at will + only + only
1L Pseudonyms
each arena
+ and – within arena
+ and 0 within arena
Identified never + and – + and 0
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Empirical Results: eBay Feedback is provided It’s almost all positive Reputations are informative Reputation benefits
– Effect on probability of sale– Effect on price
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Provision of Feedback
Negatives: paid but did not receive; seller cancelled; not as advertised; communication
Neutrals: slow shipping, not as advertised, communication
Buyer of Seller Seller of Buyer Frequency Percent Frequency Percent
negative 111 0.3 353 1.0 neutral 62 0.2 60 0.2 positive 18,569 51.2 21,560 59.5 none 17,491 48.3 14,260 39.4 Total 36,233 36,233
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Feedback Profiles of Buyers and Sellers
Group N (Sellers)
Percent neutral and negative
(Sellers)
N (Buyers)
Percent neutral and negative
(Buyers) 0-9 positive 4,018 2.83% 13,306 1.99%
10-49 positive 3,932 1.25% 7,366 1.09% 50-199 positive 3,728 0.95% 3,678 0.76%
200-999 positive 1,895 0.79% 738 0.60% 1000+ 122 1.18% 15 0.92%
All 13,695 0.93% 25,103 0.83%
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Predicting Problematic Transactions
Logistic Regression
f(0,0) = 1.91% f(100,0) = .18% f(100,3) = .53%
N = 36233Beginning Block Number 0. Initial Log Likelihood Function-2 Log Likelihood 2194.3468-2 Log Likelihood 2075.420
Dependent Variable.. NEGNEUT---------------------- Variables in the Equation -----------------------
Variable B S.E. Wald df Sig R Exp(B)
LNNPOS .7712 .1179 42.7907 1 .0000 .1363 2.1624LNPOS -.5137 .0475 116.8293 1 .0000 -.2288 .5983Constant -3.9399 .1291 931.3828 1 .0000
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Predictive Value
1-specificity (% of unproblematic transactions rejected)
Sensitivity (% of problematic transactions rejected)
Cutoff predicted probability
% of accepted transactions that are problematic
75% 94.2% .20% .11% 50% 81.5% .31% .18% 25% 57.2% .54% .27% 10% 32.4% 1.09% .36% 0% 0% Accept all .48%
Predicting Problem Transactions
1 - Specificity
1.00.75.50.250.00
Sen
sitiv
ity
1.00
.75
.50
.25
0.00
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Some Recently Completed Work Experiment: does reputation affect
profit?– Many positives: Yes, but only a little (8.1%)– One or two negatives: No
Incentives for quality feedback provision– Can pay based on agreement among
raters
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Studies Currently Underway Feedback provision (empirical)
– Reciprocation, altruism, and free riding Dynamics: learning and selection
(empirical) Geography: trust and trustworthiness by
state
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Design Space Rating scales Aggregation of ratings Who rates? Incentives for raters Identification/Anonymity
– Exchange partners– Evaluation providers
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Case Study Goal: help Michigan non-profits select
consultants and other service providers Is this a good candidate for a reputation
system?
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Case Study Goal: help Michigan non-profits select
consultants and other service providers Is this a good candidate for a reputation
system?Interacting with strangersSellers (Exchange Partners) Vary
SkillEffortEthics
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Case Study Design Choices Rating scales Aggregation of ratings Who rates? Incentives for raters Identification/Anonymity
– Exchange partners– Evaluation providers
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
SCHOOL OF INFORMATION
UNIVERSITY OF MICHIGANsi.umich.edu
Summary RS inform, incent, select Opportunity for RS: interactions with
strangers Design space
– Scales, aggregation, raters, incentives, anonymity