the wisdom of crowds - wordpress.com...wisdom of crowds is not a very well de ned term or topic. it...
TRANSCRIPT
-
The Wisdom of Crowds
Timo Freiesleben
GSN/MCMP
CTPS
Timo Freiesleben The Wisdom of Crowds CTPS 1 / 36
-
Overview
1 Experiments
2 What is the wisdom of crowds?
3 What do we require for a wise crowd?
4 Explanations for wisdom of crowdsThe Diversity Prediction TheoremCondorcet Jury TheoremsInformational Cascades
5 Opinion Leaders, Polarizers, and InfluencersThe Russian-CaseThe US-CaseThe European-Case
6 Other Interesting Networks
Timo Freiesleben The Wisdom of Crowds CTPS 2 / 36
-
Experiments
How many red M&M’s are in the glass?
Timo Freiesleben The Wisdom of Crowds CTPS 3 / 36
-
Experiments
For this experiment, please close your eyes while voting.
Which of the following countries has the highest population?
1 Turkey
2 Pakistan
3 Japan
4 Mexico
Timo Freiesleben The Wisdom of Crowds CTPS 4 / 36
-
Experiments
For this experiment, you can keep your eyes open.
Which of the following Norwegian cities has the most inhabitants?
1 Stavanger
2 Narvik
3 Trondheim
4 Bergen
Timo Freiesleben The Wisdom of Crowds CTPS 5 / 36
-
What is the wisdom of crowds?
Wisdom of crowds is not a very well defined term or topic. Itdescribes a bunch of phenomena for which groups:
perform better than each of the individuals in the group.can do tasks that individuals could not.with mediocre agents outperform groups of experts.
Timo Freiesleben The Wisdom of Crowds CTPS 6 / 36
-
Examples
Francis Galton and the weight of an ox
Timo Freiesleben The Wisdom of Crowds CTPS 7 / 36
-
Examples
Who wants to be a millionaire
Timo Freiesleben The Wisdom of Crowds CTPS 8 / 36
-
Examples
Wikipedia
Timo Freiesleben The Wisdom of Crowds CTPS 9 / 36
-
What do we require for a wise crowd?
Diversity of opinion
Independence
Decentralization/Specialization
Aggregation
Trust
See Surowiecki (2004)
Timo Freiesleben The Wisdom of Crowds CTPS 10 / 36
-
Surowiecki’s ideas summarized?
It is possible to describe how people in a group think as a whole.
In some cases, groups are remarkably intelligent and are often smarterthan the smartest people in them.
The three conditions for a group to be intelligent are diversity,independence, and decentralization.
The best decisions are a product of disagreement and contest.
Too much communication can make the group as a whole lessintelligent.
There is no need to chase the expert
Timo Freiesleben The Wisdom of Crowds CTPS 11 / 36
-
Explanations for wisdom of crowds
The two pictures
Truth is out there and accessible but it is blurred. Everyone draws hisor her guess from this blurred distribution.
People observe different relevant features and build a model based onthese.
Timo Freiesleben The Wisdom of Crowds CTPS 12 / 36
-
The Diversity Prediction Theorem
In one informal sentence the Diversity Prediction Theorem says thefollowing:
For a group to perform well in an estimation task the individuals’performance needs to be inversely proportional to the group’s diversity.
Timo Freiesleben The Wisdom of Crowds CTPS 13 / 36
-
The Diversity Prediction Theorem
Assume we model the agents’ guesses as as random variables Xi with
i = 1, . . . , n distributed around a value T := E(Xi ) and let C :=∑n
i=1 Xin
be the crowds average guess. Then, the following holds:
E[(T − C )2]︸ ︷︷ ︸Group
Accuracy
=1
n
n∑i=1
E[(Xi − T )2]︸ ︷︷ ︸Mean Squared
Error
− 1n
n∑i=1
E[(Xi − C )2]︸ ︷︷ ︸Diversityof Crowd
Timo Freiesleben The Wisdom of Crowds CTPS 14 / 36
-
The Diversity Prediction Theorem
E[(T − C )2]︸ ︷︷ ︸Group
Accuracy
=1
n
n∑i=1
E[(Xi − T )2]︸ ︷︷ ︸Mean Squared
Error
− 1n
n∑i=1
E[(Xi − C )2]︸ ︷︷ ︸Diversityof Crowd
Group Accuracy≥ 0 ⇒ (MSE≥ Diversity of Crowd≥ 0)
Thus, Group Accuracy is good if
MSE and Diversity are both low or
MSE and Diversity are both high.
Timo Freiesleben The Wisdom of Crowds CTPS 15 / 36
-
Example, Remark, and Critique
Remarks and Critiques:Interestingly, the theorem is not restricted to the Brier score as our distancemetric but holds true for any Bregman divergence. (Pettigrew (2007))The Theorem does not make any assumptions about the variables nor any ofthe numbers, thus, one should be very skeptical about the power of thestatement.
Timo Freiesleben The Wisdom of Crowds CTPS 16 / 36
-
Condorcet Jury Theorems
Assume you have a binary decision problem in which one option iscorrect and one option is wrong. Assume moreover you have a group ofagents who vote independently and each of them has a probability p > 12to pick the right option (competence). Also, assume that the groupvotes via majority vote. Then,
group competence is monotonically increasing with group size and
group competence converges to 1 for n→∞.
Timo Freiesleben The Wisdom of Crowds CTPS 17 / 36
-
Formal Version
Let (Ji )i∈N be a sequence of binary independent random variables withJi ∼ Ber(p) for all i ∈ N and p > 12 . Assume Ji : Ω→ {0, 1} wherewithout loss of generality 1 denotes the better/correct option and 0 theworse/wrong option. We define
MV2n+1 :=
1 iff2n+1∑i=1
Ji > n
0 else
Then,
for all odd numbers j holds P(MVj = 1) < P(MVj+2 = 1) and
limn→∞
P(MV2n+1 = 1) = 1.
Timo Freiesleben The Wisdom of Crowds CTPS 18 / 36
-
Critiques
Attacking the premises/setting
Usually, we are not only facing binary decision problems where oneoption is correct/better than the other and also the options are oftennot given.Voters very rarely vote independent from each other.Competence might be violated.Why majority vote?Independence and Competence counteract each other. (See Dietrich(2008))
Attacking the conclusions
Even very large groups fail in picking good options empirically(democracies)Larger groups can perform worse than small groups empirically(responsibility problem)
Timo Freiesleben The Wisdom of Crowds CTPS 19 / 36
-
Reactions and Generalizations
Defending the premises/settingThe Theorem can be generalized to n-ary decision problems. (See Listand Goodin (2001))We can allow for a bunch of dependencies. (See Peleg and Zamir(2011) or Pivato (2016))The theorem can be generalized to agents having differentcompetences as long as the average of competences is greater one half.(See Niztan and Paroush (1982))Other voting methods might be better in accuracy but are perceived asless fair. (See Nurmi (2002))Independence can be made plausible by conditional independence onstates. (See Dietrich & Spieckermann (2013))
Defending the conclusionsDemocracies seem to work better than dictatorships.In many contexts crowds do perform better than individuals. (SeeSurowiecki (2004))
Interestingly, the reasoning of the theorem is used in modern AItechnologies, if you are interested then have a look intoboosting/ensemble learning.Timo Freiesleben The Wisdom of Crowds CTPS 20 / 36
-
Informational Cascades: How groups fail
The following gives a very simple scenario of how groups can fail even in avery simple task. Assume there are two urns in a tent, a black and a white
urn.
Will this group certainly find the correct urn if the queue is infinitely longand all of the participants are Bayesians?
Timo Freiesleben The Wisdom of Crowds CTPS 21 / 36
-
Informational Cascades: How groups fail
Informational cascades are one example of how groups can fail terribly.They are used as an explanation in the social sciences in a variety of
contexts as for instance for analyzing the financial crisis, the spread of fakenews, etc.
Timo Freiesleben The Wisdom of Crowds CTPS 22 / 36
-
Opinion Leaders, Polarizers, and Influencers
In very many groups there is one agent present who is particularlyinfluential. Examples are democratic leaders, managers, group leaders,
class representatives, etc. Even though they usually have only one vote,they influence others due to their power, charisma, media presence, etc.
Does Condorcet’s Theorem still hold? How do groups with such influentialagents perform epistemically?
Timo Freiesleben The Wisdom of Crowds CTPS 23 / 36
-
Opinion Leaders, Polarizers, and Influencers
We can model this scenario by the following Bayesian Network:
IA
MV
J1 J2 · · · J2m
Our assumption here is:
Ji |= J1, . . . , Ji−1, Ji+1, . . . , J2m|IA.
Timo Freiesleben The Wisdom of Crowds CTPS 24 / 36
-
Scenario 1: Opinion Leaders (The Russian Case)
Assume our group is very homogeneous and the network is full specified bythe following assignments:
P(OL=1)=p, P(Ji |OL)=j and P(Ji |¬OL)=l with p,j ,l∈[0,1].
Then,
P(MV2m+1=1)=
(2m∑i=m
(2mi )ji (1−j)2m−i
)p+
(2m∑
i=m+1(2mi )l
i (1−l)2m−i)(1−p).
And for z:= limm→∞
P(MVm=1), we get
z l > 12 l =12 l <
12
j > 12 11+p2 p
j = 12 1−p2
12
p2
j < 12 1− p1−p2 0
Timo Freiesleben The Wisdom of Crowds CTPS 25 / 36
-
Scenario 1: Opinion Leaders (The Russian Case)
In this scenario neither part (i) nor part (ii) of Condorcet’s Theoremhold in general.We can even have very pathological cases:
All agents are competent while the group is incompetent.All agents are incompetent while the group as a whole is competent.
Groups can have an optimal number of group members.
0 5 10 15 200.55
0.6
0.65
0.7
0.75
0.8
m (Number of agents is computed by 2m+1)
P(M
V_{
2m+
1}=
1)
Probability of the Majority Vote being coorect with j=0.45, l=0.9 and p=0.6.
0 20 40 60 80 1000.7
0.72
0.74
0.76
0.78
0.8
0.82
m (Number of agents is computed by 2m+1)
P(M
V_{
2m+
1}=
1)
Probability of the Majority Vote being coorect with j=0.6, l=0.45 and p=0.8.
Timo Freiesleben The Wisdom of Crowds CTPS 26 / 36
-
Scenario 2: Polarizers (The US-Case)
Assume we have one polarizer and two groups influenced in different ways:
PONegatively Correlated Positively Correlated
MVs
D1 · · · Dns Rms· · ·R1
Timo Freiesleben The Wisdom of Crowds CTPS 27 / 36
-
Scenario 2: Polarizers (The US-Case)
Assume our group is split and the network is fully specified by thefollowing assignments:
P(PO=1)=p, P(Ri |PO)=j1, P(Ri |¬PO)=l1, P(Di |PO)=j2 and P(Di |¬PO)=l2
with p,j1,j2,l1,l2∈[0;1].Then,
P(MVm,n=1)=
m+n∑k=m+n2
k∑i=0
( mk−i)(ni)j
k−i1 (1−j1)m−k+i j i2(1−j2)n−i
p+ m+n∑
k=m+n2 +1
k∑i=0
( mk−i)(ni)l
k−i1 (1−l1)m−k+i l i2(1−l2)n−i
(1−p).For z:= lim
s→∞P(MVs=1) we get dependent on m,n∈N and l1,l2,j1,j2∈[0;1] the
following values:z l1
mm+n + l2
nm+n >
12 l1
mm+n + l2
nm+n =
12 l1
mm+n + l2
nm+n <
12
j1m
m+n + j2n
m+n >12 1
1+p2 p
j1m
m+n + j2n
m+n =12 1−
p2
12
p2
j1m
m+n + j2n
m+n <12 1− p
1−p2 0
Timo Freiesleben The Wisdom of Crowds CTPS 28 / 36
-
Scenario 2: Polarizers (The US-Case)
Given that we have similar group sizes and strong negative correlationbetween the groups, part (ii) of Condorcet’s Theorem is retained,however, it is not necessarily very stable.Also, more pathological cases occur:
Most agents in the group are competent while the group is insane.Most agents in the group are incompetent while the group reachesexpert level.
0 5 10 15 200.2
0.3
0.4
0.5
0.6
0.7
0.8
s
P(M
V_{
s}=
1)
Probability of the Majority Vote being coorect with j_1=0.95j_2=0.4, l_1=0.35, l_2=0.8, m=3, n=5 and p=0.3.
0 5 10 15 200.25
0.3
0.35
0.4
0.45
0.5
0.55
s
P(M
V_{
s}=
1)
Probability of the Majority Vote being coorect with j_1=0.95j_2=0.4, l_1=0.3, l_2=0.8, m=3, n=5 and p=0.3.
Timo Freiesleben The Wisdom of Crowds CTPS 29 / 36
-
Scenario 3: Influencers (The European-Case)
Assume we have one influencer and several groups influenced in differentways:
IN
MVn
G1
G2
. ..
Gn
Variety of dependencies
Timo Freiesleben The Wisdom of Crowds CTPS 30 / 36
-
Scenario 3: Influencers (The European-Case)
Assume our group is split into n different subgroups and the network isfully specified by the following assignments:
P(IN=1)=p, P(Jg,i |IN)=jg , P(Jg,i |¬IN)=lg
with p,jg ,lg∈[0;1] for all 1≤g≤n.Then,
P(MVn=1)=
w1+···+wn∑k=
w1+···+wn2
∑A∈Bk
∏i∈A
j∗i∏
u∈AC(1−j∗u )
p+ w1+···+wn∑
k=w1+···+wn
2 +1
∑A∈Bk
∏i∈A
l∗i∏
u∈AC(1−l∗u )
(1−p) (1)For z:= lim
n→∞P(MVn=1) we get dependent on wg∈ and lg ,jg∈[0;1] for all g∈N:
z∞∑i=1
si li >12
∞∑i=1
si li =12
∞∑i=1
si li <12
∞∑i=1
si ji >12 1
1+p2 p
∞∑i=1
si ji =12 1−
p2
12
p2
∞∑i=1
si ji <12 1− p
1−p2 0
Timo Freiesleben The Wisdom of Crowds CTPS 31 / 36
-
Scenario 3: Influencers (The European-Case)
Condorcet Diversity Theorem
Assume p ∈ [0, 1] and every subgroup has the same size wi = wj for alli , j ∈ N. Let α > 12 denote the competence of each individual. If for everysubgroup g ∈ holds that
jg ∼ Unif([max(0,α + p − 1
p),min(1,
α
p)])
with lg :=α−jgp1−p we obtain
1 limn→∞
P(MVn = 1) = 1 and
2 E(MVn) ≤ E(MVn+1)
Hence, infallibility remains valid as long as dependencies vary andindividuals are competent. Even for finite groups we have an epistemicgain in expectation by adding another group with above properties.
Timo Freiesleben The Wisdom of Crowds CTPS 32 / 36
-
A lot of highly un(der)explored networks are awaiting. Ideas?
Cambridge Analytica
Facebook
Election
U1
U2
. ..
Un
Timo Freiesleben The Wisdom of Crowds CTPS 33 / 36
-
Rumor Spread
P1
P5P2
P3
P4
P6
Timo Freiesleben The Wisdom of Crowds CTPS 34 / 36
-
Insider Trading
E1
EZBE2
T3
T4
BlackRock
Timo Freiesleben The Wisdom of Crowds CTPS 35 / 36
-
Thank you for your attention!
Timo Freiesleben The Wisdom of Crowds CTPS 36 / 36
-
References
Timo Freiesleben The Wisdom of Crowds CTPS 1 / 1
-
Berend D. and Paroush J. (1998), When Is Condorcet’s Jury TheoremValid?, Social Choice and Welfare 15: pp 481-488.
Berg, S. (1993), Condorcet’s jury theorem, dependency among jurors,Soc Choice Welf 17: pp 87-96.
Berg, S. (1993), Condorcet’s jury theorem revisited, Eur J PoliticalEcon 9: pp 437-446.
Boland P. J., Proschan F., and Tong Y. L. (1989), ModellingDependence in Simple and Indirect Majority Systems, Journal ofApplied Probability 26: pp 81-88.
Boland P. J. (1989), Majority Systems and the Condorcet JuryTheorem, Statistician 38: pp 181-189.
Jean-Antoine-Nicolas de Caritat Condorcet, marquis de: Essai surl’application de l’analyse à la probabilité des décisions rendues à lapluralité des voix. Imprimerie royale, Paris 1785
Dietrich, F. and List, C. (2004), A model of jury decision where all thejurors have the same evidence, Synthese 142: pp 175-202.
Timo Freiesleben The Wisdom of Crowds CTPS 1 / 1
-
Dietrich, F. (2008), The premises of Condorcet’s theorem are notsimultaneously justified, Episteme Volume 5 Issue 1: pp 56-73.
Dietrich, F. Spiekermann, K. (2013) “ EPISTEMIC DEMOCRACYWITH DEFENSIBLE PREMISES”, Economics and Philosophy,29(1),pp 87-120.
Estlund, D.M. (1994), Opinion Leaders, Independence, andCondorcet’s Jury Theorem, Theory and Decision 36(2): pp 131-162.
Galton, F. (1907). ”Vox populi”. Nature. 75 (1949): 450–451.Bibcode:1907Natur..75..450G. doi:10.1038/075450a0.
Grofman, B., Guillermo O. and Scott L. F. (1983), Thirteen Theoremsin Search of the Truth, Theory and Decision 15: pp 261-278.
May, K. (1952) “A set of independent necessary and sufficientconditions for simple majority decisions”, Econometrica, Vol 20 (4):pp. 680-684.
Niztan, S. and Paroush, J. (1982), Optimal decision rules indichotomous choice situations. Int Econ Rev 23: pp 289-297.
Timo Freiesleben The Wisdom of Crowds CTPS 1 / 1
-
Nurmi, Hannu (2002), Voting Procedures under Uncertainty,Springer-Verlag.
Page, S. E. (2007). Making the difference: Applying a logic ofdiversity, Academy of Management Perspectives, 21(4), 6-20.
Peleg, B and Zamir, S. (2011), Extending the Condorcet JuryTheorem to a general dependent jury, Soc Choice Welf 39: pp 91-125.
Shapley, LS and Grofman, B. (1984), Optimizing group judgementalaccuracy in the presence of interdependencies, Public Choice 43: pp329-343.
Shirari, S.N. (1982), Reliability analysis of majority vote systems,Inform. Sci. 26, pp. 243-256.
Surowiecki, James. The Wisdom of Crowds. Doubleday, 2004. ISBN978-0-385-50386-0.
Timo Freiesleben The Wisdom of Crowds CTPS 1 / 1
ExperimentsWhat is the wisdom of crowds?What do we require for a wise crowd?Explanations for wisdom of crowdsThe Diversity Prediction TheoremCondorcet Jury TheoremsInformational Cascades
Opinion Leaders, Polarizers, and InfluencersThe Russian-CaseThe US-CaseThe European-Case
Other Interesting NetworksAppendix