university of michigan santa fe institute leveraging · pdf filechanging nature of work . pair...
TRANSCRIPT
![Page 1: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/1.jpg)
Leveraging Diversity
Scott E Page University of Michigan Santa Fe Institute
![Page 2: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/2.jpg)
![Page 3: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/3.jpg)
Outline
Background: Two Inescapable Trends Identity and Cognitive Diversity Prediction
Case: Netflix Prize
Problem Solving Takeaways
![Page 4: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/4.jpg)
Trend 1 Increasing Diversity
![Page 5: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/5.jpg)
![Page 6: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/6.jpg)
![Page 7: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/7.jpg)
![Page 8: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/8.jpg)
Chicago
![Page 9: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/9.jpg)
New York
![Page 10: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/10.jpg)
LA
![Page 11: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/11.jpg)
The D
![Page 12: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/12.jpg)
Equalizing Opportunity. Addressing past disadvantages.
Normative Argument
![Page 13: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/13.jpg)
Promoting diversity enlarges the pool.
Pragmatic Advantages
![Page 14: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/14.jpg)
Diversity Ability
![Page 15: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/15.jpg)
Trend 2 Changing Nature of Work
![Page 16: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/16.jpg)
![Page 17: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/17.jpg)
![Page 18: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/18.jpg)
Pair Progamming
![Page 19: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/19.jpg)
![Page 20: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/20.jpg)
![Page 21: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/21.jpg)
First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen 1902 Hendrik A. Lorentz, Pieter Zeeman 1903 Antoine Henri Becquerel, Pierre Curie, Marie Curie 1904 John W. Strutt 1905 Philipp E. A. von Lenard 1906 Sir Joseph J. Thomson 1907 Albert A. Michelson 1908 Gabriel Lippmann 1909 Carl F. Braun, Guglielmo Marconi 1910 Johannes D. van der Waals 1911 Wilhelm Wien 1912 Nils G. Dalen 1913 Heike Kamerlingh Onnes 1914 Max von Laue 1915 William Bragg
![Page 22: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/22.jpg)
42
![Page 23: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/23.jpg)
![Page 24: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/24.jpg)
Diverse +
Group
![Page 25: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/25.jpg)
Rethinking
![Page 26: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/26.jpg)
≠ + +
![Page 27: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/27.jpg)
Want to understand how diversity affects group performance
![Page 28: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/28.jpg)
Diversity Ability
![Page 29: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/29.jpg)
![Page 30: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/30.jpg)
E[SqE(c)] =
€
B2 +1n
Var +n −1
nCov
![Page 31: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/31.jpg)
![Page 32: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/32.jpg)
![Page 33: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/33.jpg)
Identity and Cognitive Diversity
![Page 34: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/34.jpg)
![Page 35: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/35.jpg)
![Page 36: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/36.jpg)
![Page 37: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/37.jpg)
![Page 38: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/38.jpg)
![Page 39: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/39.jpg)
The Ketchup Question
![Page 40: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/40.jpg)
![Page 41: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/41.jpg)
![Page 42: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/42.jpg)
Soda, Pop, or Coke By Region
![Page 43: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/43.jpg)
Cultural Blinders
![Page 44: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/44.jpg)
![Page 45: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/45.jpg)
Gunter Blobel: The exception
![Page 46: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/46.jpg)
Coumadin.
![Page 47: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/47.jpg)
Prediction
![Page 48: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/48.jpg)
X lb?
![Page 49: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/49.jpg)
Lightning
Smoke and Fire
Flight of Birds
Neighing of Horses
Tea Leaves and Coffee Grounds
Passages of Sacred Texts
Numbers
Guessing
Stars and Planets (astrology)
Rolling Dice
Tarot Cards
Palm Reading
Crystal Balls
Head Shape (Phrenology)
Atmospheric Conditions
Dreams
Animal Entrails
Moles on the body
David O
rrell “The Future of Everything. Model
Methods of
Divination
![Page 50: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/50.jpg)
Diverse Categorizations
![Page 51: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/51.jpg)
Linnean Classification Sort
![Page 52: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/52.jpg)
Source: www.promotega.org
![Page 53: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/53.jpg)
Pile Sort
![Page 54: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/54.jpg)
BOBO Sort
![Page 55: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/55.jpg)
Airstream Sort
![Page 56: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/56.jpg)
Crowd Error = Average Error - Diversity
Diversity Prediction Theorem
![Page 57: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/57.jpg)
The Spherical Cow
![Page 58: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/58.jpg)
The Gateway Cow
![Page 59: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/59.jpg)
Crowd Error = Average Error – Diversity
0.6 = 2,956.0 - 2955.4
Galton’s Steer
![Page 60: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/60.jpg)
Case: Netflix Prize
![Page 61: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/61.jpg)
Outline
Netflix Prize: Background Predictive Models
Factor Models
Ensembles of Models Ensembles of Teams The Value of Diversity
![Page 62: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/62.jpg)
Netflix Prize
November 2006, Netflix offers a prize of $1 million to anyone who can defeat their Cinematch recommender system by 10% or more.
![Page 63: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/63.jpg)
Some Details
Netflix users rank movies from 1 to 5 Six years of data Half million users 17,700 movies Data divided into (training, testing) Testing Data dived into (probe, quiz, test)
![Page 64: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/64.jpg)
Interesting Asides
Lost in Translation and The Royal Tenenbaums had the highest variance
Shawshank Redemption had the highest rating Miss Congeniality had the most ratings.
![Page 65: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/65.jpg)
Singular Value Decomposition
Each movie represented by a vector: (p1,p2,p3,p4…pn)
Each person represented by a vector:
(q1,q2,q3,q4…qn) Rating: rij = mi + aj + p�q Training: choose p,q to minimiize
(actualij –rij)2
+ c( ||p||2+ ||q||2)
![Page 66: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/66.jpg)
BellKor’s Initial Models
Approximately 50 dimensions, 107 Models Best Model: 6.8% improvement Combination of Models: 8.4% improvement
![Page 67: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/67.jpg)
Why Do More Work Better?
SqE(c) = SqE(s) - PDiv(s)
![Page 68: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/68.jpg)
BellKor’s Pragmatic Chaos
More is Better: Seven person team Functional Diversity: statisticians, machine learning experts and
computer scientists Identity Diversity: United States, Australia, Canada and Israel. Difficult be build a “grand” model (800 variables) but possible to
build lots of “huge” models
![Page 69: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/69.jpg)
Ensemble Effects
Best Model 8.4% Ensemble: 10.1% Rules: Once someone breaks 10%, then the contest ends in 30
days.
![Page 70: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/70.jpg)
Enter ``The Ensemble’’
23 teams from 30 countries who blended their predictive models who tried in the last moments to defeat BellKor’s Pragamatic Chaos
![Page 71: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/71.jpg)
The Ensemble
“The contest was almost a race to agglomerate as many teams as possible,” said David Weiss, a Ph.D. candidate in computer science at the University of Pennsylvania and a member of the Ensemble.
“The surprise was that the collaborative approach works so well, that trying all the algorithms,
coding them up and putting them together far exceeded our expectations.”
New York Times 6/27/09
![Page 72: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/72.jpg)
And The Winner is…
RMSE for The Ensemble: 0.856714 RMSE for Bellkor's Pragmatic Chaos: 0.856704 By the rules of the competition the scores are rounded to four decimal
places so it was a tie. However, BellKor’s Pragmatic Chaos submitted 20 minutes earlier so they
won. (and they had the lower error)
![Page 73: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/73.jpg)
Oh, by the way..
BellKor’s Pragmatic Chaos 10.06% The Ensemble 10.06% 50/50 Blend 10.19%
![Page 74: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/74.jpg)
Problem Solving
![Page 75: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/75.jpg)
Diverse Perspectives
![Page 76: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/76.jpg)
Page vs de Marchi 3/17/98
Page de Marchi
![Page 77: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/77.jpg)
Page de Marchi
Page vs de Marchi 3/17/98
![Page 78: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/78.jpg)
Page de Marchi
Page vs de Marchi 3/17/98
![Page 79: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/79.jpg)
Page de Marchi
Page vs de Marchi 3/17/98
![Page 80: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/80.jpg)
Page de Marchi
Page vs de Marchi 3/17/98
![Page 81: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/81.jpg)
Page de Marchi
Page vs de Marchi 3/17/98
![Page 82: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/82.jpg)
Page de Marchi
Page vs de Marchi 3/17/98
![Page 83: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/83.jpg)
Page de Marchi
Page vs de Marchi 3/17/98
![Page 84: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/84.jpg)
15
A Magic Square
15
![Page 85: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/85.jpg)
Play it Again
P
D
P
D
P
![Page 86: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/86.jpg)
Diverse Cognitive Models/Heuristics
![Page 87: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/87.jpg)
A Test
• Create a bunch of agents with diverse perspectives and heuristics • Rank them by their performance on a problem. • Note: all of the agents must be “smart”
![Page 88: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/88.jpg)
Experiment
Group 1: Best 20 agents Group 2: Random 20 agents Have each group work collectively - when one agent gets stuck at a point, another agent tries to find a further improvement. Group stops when no one can find a better solution.
![Page 89: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/89.jpg)
The IQ View
75
121 84
135
111
9
Alpha Group
138 137 139
140 136 132
Diverse Group
![Page 90: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/90.jpg)
The diverse group almost always outperforms the group of the best by a substantial margin.
See Lu Hong and Scott Page Proceedings of the National Academy of Sciences (2002)
![Page 91: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/91.jpg)
The Toolbox View
EZ
AHK FD
BCD
AEG
IL ADE BCD BCD
ABC ACD BDE
Alpha Group Diverse Group
![Page 92: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/92.jpg)
![Page 93: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/93.jpg)
Calculus Condition: Problem solvers must all be smart- -we must be able to list their local optima
Diversity Condition: Problem solvers must have
diverse heuristics and perspectives Hard Problem Condition: Problem itself must be difficult
What Must be True?
![Page 94: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/94.jpg)
Realizing Diversity’s Benefits
![Page 95: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/95.jpg)
![Page 96: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/96.jpg)
Kevin Dunbar
Out of the Bathtub and In the Group
![Page 97: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/97.jpg)
GRE:???
![Page 98: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/98.jpg)
![Page 99: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/99.jpg)
Group Accuracy & Perceived Effectiveness Phillips, Liljenquist, & Neale, 2009
5.6
5.7
5.8
5.9
6
6.1
6.2
6.3
6.4
Homogeneous Diverse 30
35
40
45
50
55
60
65
70
75
Homogeneous Diverse
% Groups Accurate Perceived Effectiveness
Controls for individual performance & how well group members know one another
![Page 100: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/100.jpg)
Group Objective Accuracy vs Confidence in Decision
40
45
50
55
60
65
70
75
80
Homogeneous Diverse 60
62
64
66
68
70
72
74
76
78
Homogeneous Diverse
% Groups Accurate Confidence in Decision
![Page 101: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/101.jpg)
The Prepared Mind
![Page 102: University of Michigan Santa Fe Institute Leveraging · PDF fileChanging Nature of Work . Pair Progamming . First 15 Nobels in Physics: 19 Winners 1901 Wilhelm C. Röntgen ... Stars](https://reader034.vdocuments.us/reader034/viewer/2022051721/5a8465007f8b9a001c8b8bd6/html5/thumbnails/102.jpg)
The Prepared Mind Community