it's all a game: the twin fallacies of epistemic purity and the scholarly invisible hand
TRANSCRIPT
![Page 1: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/1.jpg)
���It’s all a game: ���
���The twin fallacies of epistemic purity
and the scholarly invisible hand
ICIS Gaming Metrics Carl T. BergstromUC Davis Department of BiologyFebruary 4th, 2016 University of Washington#GamingMetrics @CT_Bergstrom
![Page 2: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/2.jpg)
Responsible science Gaming the System
![Page 3: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/3.jpg)
Responsible science Gaming the System
![Page 4: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/4.jpg)
Image: ASU Center for Biology and Society
1) The fallacy of epistemic purity���
2) The fallacy of the invisible hand
![Page 5: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/5.jpg)
Whatmotivatesscientists?
Imagina7on
Reality
Epistemically pure
Epistemically sullied
![Page 6: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/6.jpg)
The field of economics in four words:
peoplerespond toincentives
![Page 7: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/7.jpg)
E.g. Dasgupta and David (1994) Research Policy
• Non-rivalrous
Scientific discoveries (made public) are public goods.
![Page 8: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/8.jpg)
E.g. Dasgupta and David (1994) Research Policy
• Non-rivalrous
• Non-excludable (Arrow’s paradox)
Scientific discoveries (made public) are public goods.
![Page 9: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/9.jpg)
Dasgupta and David (1994) Research Policy
Markets typically under-allocate to the production of public goods because producers are not sufficiently rewarded.
![Page 10: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/10.jpg)
Dasgupta and David (1994) Research Policy
Scientists are rewarded for discoveries in recognition and prestige which are rivalrous and excludable.
![Page 11: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/11.jpg)
Dasgupta and David (1994) Research Policy
Scientists may enjoy solving puzzles and learning about the world, but they are highly motivated to seek recognition for their discoveries.
![Page 12: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/12.jpg)
Research strategy
Publi
catio
n str
ateg
y
Epistemically pure
Socially op7mal
Fraud
Plagiarist
Epistemically sullied
A hypothetical space of scholarly ���
practice
![Page 13: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/13.jpg)
Research strategy
Publi
catio
n str
ateg
y
Epistemically pure
Socially op7mal
Fraud
Plagiarist
most of us most of the 7me
A hypothetical space of scholarly ���
practice
![Page 14: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/14.jpg)
Research strategy
Publi
catio
n str
ateg
y
Science as usual
Social op7mality
Norms of behavior���attempt to carve
up the space of practice
![Page 15: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/15.jpg)
We are struggling to define these norms and figure out where various practices fall.
RESEARCH STRATEGIES
Outcome switching P-‐hacking Garden of forking paths
Uncorrected mul7ple comparison
Icons from Jeff Leek, http://simplystatistics.org/2016/02/01/
![Page 16: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/16.jpg)
We are struggling to define these norms and figure out where various practices fall.
PUBLICATION STRATEGIES
Courtesy authorship Salami slicing Medical ghostwri7ng Cita7on rings
CC images: pixabay.comBroader authorship Cita7on infla7on Self-‐cita7on
![Page 17: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/17.jpg)
Careful: some seemingly questionable strategies���may have beneficial consequences.
Broader authorship
Salami slicing
Cita7on infla7on CC images: pixabay.com
More rapid publication
Better, more interdisciplinary work
Denser citation network
![Page 18: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/18.jpg)
“I’m exploring the fit of my mathematical models of scientific activity to historical data because I think an empirical component will make it easier to find funding for my work.”
“I’m exploring the fit of my mathematical models of scientific activity to historical data because I think an empirical component will make it easier to find funding for my work.”
![Page 19: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/19.jpg)
“I’m exploring the fit of my mathematical models of scientific activity to historical data because I think an empirical component will make it easier to find funding for my work.”
Gaming the system?
![Page 20: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/20.jpg)
“
Are scientist, like Adam Smith’s���economic actors,
led by an invisible hand to ��� promote an end which was ��� no part of [their] intention ?
David Hull 1988; 1997,Brad Wray 2000, etc.
”
![Page 21: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/21.jpg)
Research strategy
Publi
catio
n str
ateg
y
Science as usual
Social op7mality
Epistemically sullied
The invisible hand is the hopethat somehow individual
incentives will lead to social optimality ���
![Page 22: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/22.jpg)
E.g. Kitcher (1993) The Advancement of Science
Modeling efforts attempt to support the invisible hand hypothesis, e.g. by showing how competition can cause scientists to disperse across problems instead of clumping on the most promising one.
![Page 23: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/23.jpg)
Song, Bergstrom, and Foster in prep.
But we’ve recently developed an explicit social welfare analysis of problem selection by scientists.
We find that individual incentives do not, in general lead to socially optimal outcomes.
![Page 24: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/24.jpg)
Oren and Kleinberg 2013
Another model, by Oren and Kleinberg, finds that if scientists differ in aptitude for different problems, there is no way society can incentivize socially optimal behavior.
![Page 25: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/25.jpg)
In science, the invisible hand doesn’t do its job.
Even the good guys are trying to work the system.
Let’s design our norms and institutions accordingly.
![Page 26: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/26.jpg)
Example: self-citation ������
Responsible practice?������
Or gaming the system?
![Page 27: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/27.jpg)
Self-citation rate by gender
●
●
●●●
●●
●
●●●
●●
●●
●●●
●
●●●●
●●●
●●●●
●●●●●●●●●
●●●
●●●●●●●●●●●●●●●
●
●●
●●
●●●
●●●
●●●●●●
●
●●●●●
●●●●●
●●●●●●
●●●●●●●
●●●●●●●●●●●●●
●●●●●●
●●●
●
●
●●
Women's and men's rates of self-citation
���� ���� ���� ���� ���� ��������
���
���
���
���
���
���
����-����� / ����������
Based on > 3 million papers from JSTOR King et al. in prep.
![Page 28: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/28.jpg)
Rates of rates of self-citation do make a difference to impact metrics, particularly the h-index.
– Cameron et al 2016 Bioscience
Self-citations per authorship
”“
King et al. in prep.
![Page 29: It's all a game: The twin fallacies of epistemic purity and the scholarly invisible hand](https://reader031.vdocuments.us/reader031/viewer/2022030318/5a6d78d37f8b9a04428b57f3/html5/thumbnails/29.jpg)
If men self-cite more, are they ���gaming the system?���
������
We face a complicated interplay of norms, strategic incentives and gender differences.���
������
Gaming is a spectrum; the ambiguities matter. ������