fallacies of rankings and ratings - european commission · the stiglitz report, on page 65,...
TRANSCRIPT
![Page 1: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/1.jpg)
Fallacies of rankings and ratings
SAMO 2013University Nice Sophia Antipolis, Valrose
Campus, Nice, France
Paolo Paruolo, Michaela Saisana, Andrea Saltelli, European Commission,
Joint Research Centre
![Page 2: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/2.jpg)
About indicators
![Page 3: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/3.jpg)
… and composite indicators
![Page 4: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/4.jpg)
[…] composite indicators as an object populating a multidimensional space whose main axes are advocacy, analysis and quality […]
Saltelli, A., and Saisana, M., Advocacy, analysis and quality. The Bermuda triangle of Statistics, International Statistical Institute Conference, Hong Kong, August 2013, Statistics and Policy
Advocacy, analysis and quality
![Page 5: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/5.jpg)
These three dimensions (advocacy, analysis and quality) are not independent from one another.
[…]most developers adopt for transparency and simplicity linear aggregation procedures to build composite indicators which are fraught with considerable difficulties […]
In this case quality may suffer at the expenses of advocacy.
ibidem
Advocacy, analysis and quality
![Page 6: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/6.jpg)
THE ROLE OF COMPOSITE INDICATORS FOR MEASURING SOCIETAL PROGRESS
Ubiquitous; 5-fold increase in 6 yStatistics' best known face (to general public & media) Open the floor to plurality of norms and views Can provide analytic input to policy
Features of composite indicators
![Page 7: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/7.jpg)
The Stiglitz-Sen-Fitoussi report
![Page 8: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/8.jpg)
“the role of statistical indicators has increased over the last two decades”
(Stiglitz report, 2009)
More Statistical Indicators
![Page 9: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/9.jpg)
Why?
(i) more literacy,
(ii) more complexity,
(iii) more information society(Stiglitz report, 2009)
More Statistical Indicators
![Page 10: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/10.jpg)
Caveats
![Page 11: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/11.jpg)
The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite
indicators, i.e. the arbitrary character of the procedures used to weight their various components.
Adding: […] The problem is not that these weighting procedures are hidden, non-transparent or non-replicable –they are often very explicitly presented by the authors of the indices, and this is one of the strengths of this literature. The
problem is rather that their normative implications are seldom made explicit or justified.
Caveats
![Page 12: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/12.jpg)
Quality
![Page 13: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/13.jpg)
Testing (composite) indicators: two approaches
Michaela Saisana, Andrea. Saltelli, and Stefano Tarantola, 2005, Uncertainty and sensitivity analysis techniques as tools for the quality assessment of composite indicators. J. R. Statist. Soc. A 168(2), 307–323.
Paolo Paruolo, Michaela Saisana, Andrea Saltelli, 2013, Ratings and rankings: Voodoo or Science?, J. R. Statist. Soc. A, 176 (2), 1-26
Quality of composite indicators
![Page 14: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/14.jpg)
First: The invasive approach, University ranking example
Michaela Saisana, Béatrice d’Hombres, Andrea Saltelli, Rickety numbers: Volatility of university rankings and policy implicationsResearch Policy (2011), 40, 165-177
Quality of composite indicators
![Page 15: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/15.jpg)
15
(Invasive) Sensitivity Analysis
Simulation Model
parameters
Resolution levels
data
errors model structures
uncertainty analysis
sensitivity analysismodel output
feedbacks on input data and model factors
![Page 16: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/16.jpg)
Robustness analysis, of ARWU and THES
Assumption Alternatives
Number of indicators all six indicators included or
one-at-time excluded (6 options)
Weighting method original set of weights,
factor analysis,
equal weighting,
data envelopment analysis
Aggregation rule additive,
multiplicative,
Borda multi-criterion
![Page 17: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/17.jpg)
Space of alternatives
Including/excluding variables
Normalisation
Missing dataWeights
Aggregation
Country 1
10
20
30
40
50
60
Country 2 Country 3
Sensitivity analysis
![Page 18: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/18.jpg)
Question:Can we say something about the quality of the university rankings and the reliability of the results?
Relative uncertainty of the two rankings 1
51
101
151
201
251
301
351
401
451
501Med
ian
rank
(and
99%
con
fiden
ce in
terv
al) a
ccou
ntin
g fo
r m
etho
dolo
gica
l unc
erta
intie
s
Seoul National UniversityUniversity of Frankfurt
University of Hamburg
University of California-Davis
University of Alaska-Fairbanks
Hanyang University
54 universities outside the interval (total of 503) [43 universities in the Top 100]
1
51
101
151
201
251
301
351
401
Med
ian
rank
(and
99%
con
fiden
ce in
terv
al) a
ccou
ntin
g fo
r m
etho
dolo
gica
l unc
erta
intie
s
250 universities outside the interval (total of 400) [61 universities in the Top 100]
University of California, Santa Barbara
Stockholm School of Economics
University of st. Gallen
University of Tokyo
University of Leichester
University La Sapienza, Roma
Source: Saisana, D’Hombres, Saltelli, 2011, Research Policy 40, 165–177
ARWU THES
![Page 19: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/19.jpg)
ARWU: simulated ranks – Top20
Harvard, Stanford, Berkley, Cambridge, MIT: top 5 in more than 75% of our simulations.
Univ California SF: original rank 18th but could be ranked anywhere between the 6th and 100th position
Impact of assumptions: much stronger for the middle ranked universities
Legend:Frequency lower 15%Frequency between 15 and 30%Frequency between 30 and 50%Frequency greater than 50%Note: Frequencies lower than 4% are not shown
1-5
6-10
11-1
516
-20
21-2
526
-30
31-3
536
-40
41-4
546
-50
51-5
556
-60
61-6
566
-70
71-7
576
-80
81-8
586
-90
91-9
596
-100 Original
rankHarvard Univ 100 1 USAStanford Univ 89 11 2 USAUniv California - Berkeley 97 3 USAUniv Cambridge 90 10 4 UKMassachusetts Inst Tech (MIT) 74 26 5 USACalifornia Inst Tech 27 53 19 6 USAColumbia Univ 23 77 7 USAPrinceton Univ 71 9 11 7 8 USAUniv Chicago 51 34 13 9 USAUniv Oxford 99 10 UKYale Univ 47 53 11 USACornell Univ 27 73 12 USAUniv California - Los Angeles 9 84 7 13 USAUniv California - San Diego 41 46 9 14 USAUniv Pennsylvania 6 71 23 15 USAUniv Washington - Seattle 7 71 21 16 USAUniv Wisconsin - Madison 27 70 17 USAUniv California - San Francisco 14 9 14 11 7 10 6 6 18 USATokyo Univ 16 16 49 20 19 JapanJohns Hopkins Univ 7 54 21 17 20 USA
Simulated rank range - SJTU 2008
![Page 20: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/20.jpg)
THES: simulated ranks – Top 20
Impact of uncertainties on the university ranks is even more apparent.
M.I.T.: ranked 9th, but confirmed only in 13% of simulations (plausible range [4, 35])
Very high volatility also for universities ranked 10th-20th position, e.g., Duke Univ, John Hopkins Univ, Cornell Univ.
Legend:Frequency lower 15%Frequency between 15 and 30%Frequency between 30 and 50%Frequency greater than 50%Note: Frequencies lower than 4% are not shown
1-5
6-10
11-1
516
-20
21-2
526
-30
31-3
536
-40
41-4
546
-50
51-5
556
-60
61-6
566
-70
71-7
576
-80
81-8
586
-90
91-9
596
-100
HARVARD University 44 56 1 USAYALE University 40 49 11 2 USAUniversity of CAMBRIDGE 99 3 UKUniversity of OXFORD 93 7 4 UKCALIFORNIA Institute of Technology 46 50 5 USAIMPERIAL College London 74 24 6 UKUCL (University College London) 73 23 7 UKUniversity of CHICAGO 80 19 8 USAMASSACHUSETTS Institute of Technology 14 13 17 16 11 11 7 9 USACOLUMBIA University 6 13 17 11 10 7 10 14 10 USAUniversity of PENNSYLVANIA 37 56 6 11 USAPRINCETON University 6 59 27 9 12 USADUKE University 27 11 9 7 10 6 9 6 13 USAJOHNS HOPKINS University 20 10 9 9 7 10 6 6 7 6 13 USACORNELL University 6 24 11 7 6 7 9 9 7 15 USAAUSTRALIAN National University 10 30 29 31 16 AustraliaSTANFORD University 10 14 7 10 9 10 6 6 7 17 USAUniversity of MICHIGAN 6 27 17 9 10 7 14 6 18 USAUniversity of TOKYO 16 7 13 7 6 6 19 JapanMCGILL University 7 19 41 13 9 7 20 Canada
Simulated rank range - THES 2008
![Page 21: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/21.jpg)
Second: The non-invasive approach
Comparing the weights as assigned by developers with ‘effective weights’ derived from sensitivity analysis.
And the linear aggregation paradox (weights are used as if they were importance coefficients while they are trade off coefficients)
Non invasive Sensitivity analysis
![Page 22: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/22.jpg)
The linear aggregation paradox: weights are used as if they were importance coefficients while they are trade off coefficients
![Page 23: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/23.jpg)
An example. A dean wants to rank teachers based on ‘hours of teaching’ and ‘number of publications’, adding these two variables up she sees that teachers are practically ranked by publications.
The linear aggregation paradox
![Page 24: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/24.jpg)
Dean’s example: y=x1+x2.
Estimated R12 = 0.0759, R2
2 = 0.826,
corr(x1, x2) =−0.151, V(x1) = 116, V(x2) = 614, V(y) = 162.
X1: hours of teaching X2: number of publications
![Page 25: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/25.jpg)
To obviate this the dean substitutes the model
y=1/2(x1+x2)
with
y=0.7x1+0.3x2
A professor comes by, looks at the last formula, and complains that publishing is disregarded in the department …
X1: hours of teaching X2: number of publications
The linear aggregation paradox
![Page 26: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/26.jpg)
Ratings and Rankings 26
Using these points we can compute a statistics (Si) that tells us:How much (on average) would the variance of the ARWU scores be reduced if I could fix the variable ‘Papers in Nature & Science’?
Statistical coherence
ARWU score
![Page 27: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/27.jpg)
Ratings and Rankings 27
Si [linear/non linear] is the variance of the [linear/non linear] interpolation curve
index
variable
![Page 28: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/28.jpg)
First order sensitivity index
Pearson’s correlation ratio
Smoothed curve
Unconditional variance
![Page 29: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/29.jpg)
University Rankings
Comparing the internal coherence of ARWU versus THES by testing the weights declared by developers with ‘effective’ importance measures.
![Page 30: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/30.jpg)
THES
X1_Academic opinion: 6354 academics 40%X2_Recruiters’ opinion: 2339 recruiters 10%X3_Full-time equivalent faculty/student ratio 20%X4_Total citation/full time equivalent faculty 20%X5_Percentage of full-time international staff 5%X6_Percentage of full-time international students 5%
Issues with THES:a) ‘Opinion’ variables’ weight overall: >60% instead of 50
b) Faculty/student ratio: 10% instead of 20%
![Page 31: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/31.jpg)
HDI
2009
declared weight importance
Life expectancy, 33%
Adult literacy, 22%
Enrollment education, 11%
GDP per capita, 33%
![Page 32: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/32.jpg)
HDI
2010
declared weight
importance
Life expectancy, 33%
Education, 33%
GNI per capita, 33%
![Page 33: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/33.jpg)
HDI 2010 more coherent than HDI 2009
declared weight importance
![Page 34: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/34.jpg)
The Sustainable Society Index (SSI-2008) van de Kerk, G. and A. R. Manuel (2008). A comprehensive index for a sustainable society: The SSI, sustainable society index. Journal of Ecological Economics 66(2-3), 228-242.
See also
http://www.beyond-gdp.eu
![Page 35: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/35.jpg)
SSI
2008
declared weight
importance
Personal development, 0.13%
Healthy environment, 0.13%
Well-balanced society, 0.13%
Sustainable use of resources, 30%
Sustainable World, 30%
![Page 36: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/36.jpg)
![Page 37: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/37.jpg)
Ratings and Rankings 37
2012 Environmental Performance Index (EPI)
• Developed for 132 countries • Based on 22 indicators grouped in • Ten Policy Categories and Two
Objectives
![Page 38: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/38.jpg)
Ratings and Rankings 38
EPI 2012 Framework
Weights for the two objectives in EPI 2012: 30-70
But in EPI 2010 they were 50-50
![Page 39: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/39.jpg)
Ratings and Rankings 39
The JRC analysis focused on:
1. Conceptual & statistical coherence in the EPI framework
2. Impact on EPI ranks of modeling assumptions (e.g. change of weights, aggregation formula)
3. Most sensitive (…to be read as least reliable) country ranks Next we discuss
the first point
![Page 40: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/40.jpg)
Ratings and Rankings 40
EPI relatively balanced in the two
objectives
ButForestry and Marine
are “silent” indicators
![Page 41: Fallacies of rankings and ratings - European Commission · The Stiglitz report, on page 65, mentions: […] a general criticism that is frequently addressed at composite indicators,](https://reader034.vdocuments.us/reader034/viewer/2022050309/5f71b2630252d12c9f56654f/html5/thumbnails/41.jpg)
END