positional impact of reputation in global university rankings
DESCRIPTION
Positional impact of reputation in global university rankings. Philippe Vidal and Ghislaine Filliatreau Observatoire des Sciences et Techniques, Paris. Reputation is an idle and most false imposition, oft got without merit, and lost without deserving Shakespeare, Othello, II, 3. - PowerPoint PPT PresentationTRANSCRIPT
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Positional impact of reputation
in global university rankings
Philippe Vidal Philippe Vidal andand Ghislaine Filliatreau Ghislaine Filliatreau
Observatoire des Sciences et Techniques, ParisObservatoire des Sciences et Techniques, Paris
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Reputation is an idle and most false imposition, oft got without
merit, and lost without deserving
Shakespeare, Othello, II, 3
IREG Forum University Rankings, Warsaw 16 -17 May 2013
In Global university rankings and their impact. Report II, EUA Report on rankings 2013
• 2.3 Superficial descriptions of methodology and poor indicators
• …. Use of poor indicators also persists. In spite of widespread criticisms, reliance on reputation is becoming more and more widespread…..
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Rankings graphical comparison as an approach
for deconvoluting ranking patterns
Reputation surveys: ARWU: 0%, THE: 34.5%, QS: 40%
Webometrics taken as a proxy for reputation
Facts vs reputation
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Methodology 1Ternary diagrams
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Methodology 2Ranking A vs Ranking B
0
50
100
150
200
250
300
350
400
450
500
0 50 100 150 200 250 300 350 400 450 500
rank
ing
A
ranking B
Ranking A vs Ranking B - Europe
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Methodology 3Absolute and relative differences
Diagrams
(ARWU-Y) vs ARWU
and
(ARWU-Y)/ARWU vs ARWU
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Methodology 4Diagram ARWU/X vs ARWU
A
B
C
0
0,5
1
1,5
2
2,5
0 100 200 300 400 500
Rank
ing
ARW
U/Ra
nkin
g X
Ranking ARWU
forbidden zone
A B
C
0
100
200
300
400
500
0 100 200 300 400 500
Rank
ing
X
Ranking ARWU
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Methodology 5Rankings reproducibilities
- years selected: 2010 - 2011 - 2012
- average deviations to the average from one year to next:
ARWU: 2-3 %THE: 10-12 %QS: 6-7 %Webometrics: 25-30 %
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Methodology 6 Filtering rules
90 % of the universities (259) selected
62 % ranked in 4 rankings6 % in 3 rankings22 % in 2 rankings
First 250 ARWU ranks: 131 universities
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Targets
United States
United Kingdom
Northern Europe (Germany, Scandinavia, Netherlands)
Southern Europe (France, Spain, Portugal, Italy)
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Results 1 Ternary diagrams
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Results 2 Ternary diagramsleft: ARWU 1->500, right:1->250
US
UK
Northern Europe
Southern Europe
ARWU
Webometrics QSbetter ARWU score
better QS score
better Webometrics score
ARWU
Webometrics QS
better ARWU score
better QS score better Webometrics score
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Results 3 ARWU vs THE 1
(ARWU - THE) vs ARWU
-250
-200
-150
-100
-50
0
50
100
150
200
0 50 100 150 200 250
ARWU
ARW
U - T
HE
USA
United Kingdom
Northern Europe
Southern Europe
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Results 4 ARWU vs THE 2
(ARWU/THE) vs ARWU
0
1
2
3
4
5
6
7
0 50 100 150 200 250
ARWU
ARW
U / T
HE
USA
United Kingdom
Northern Europe
Southern Europe
forbidden zone
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Results 5 ARWU vs QS 1
(ARWU - QS) vs ARWU
-350
-300
-250
-200
-150
-100
-50
0
50
100
150
200
0 50 100 150 200 250
ARWU
ARW
U - Q
S
USA
United Kingdom
Northern Europe
Southern Europe
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Results 6 ARWU vs QS 2
(ARWU / QS) vs ARWU
0
0,5
1
1,5
2
2,5
3
3,5
4
4,5
0 50 100 150 200 250
ARWU
ARW
U/Q
S
USA
United Kingdom
Northern Europe
Southern Europe
forbidden zone
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Results 7 ARWU vs Webometrics 1
-500
-400
-300
-200
-100
0
100
200
0 50 100 150 200 250
ARWU
ARW
U - W
ebom
etric
s
US
UK
Northern Europe
Southern Europe
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Results 8 ARWU vs Webometrics 2
(ARWU/Webometrics) vs ARWU
0
0,5
1
1,5
2
2,5
3
3,5
4
4,5
0 50 100 150 200 250
ARWU
ARW
U / W
ebom
etric
s
USA
United Kingdom
Northern Europe
Southern Europe
forbidden zone
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Results 9US: Relative differences
ordinate scale: 1 = 100%
-7
-6
-5
-4
-3
-2
-1
0
1
2
0 50 100 150 200 250
ARWU
US: (ARWU - THE, QS, Webometrics) / ARWU
THE
QS
Webometrics
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Results 10 Europe: Relative differences
THE-QS: Continental Europe Webometrics=whole Europe
ordinate scale: 1 = 100%
-12
-11
-10
-9
-8
-7
-6
-5
-4
-3
-2
-1
0
1
2
0 50 100 150 200 250
ARWU
Europe: (ARWU -THE, QS, Webometrics) / ARWU vs ARWU
THE
QS
Webometrics
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Summary
Underestimatation of universities best ranked for researchOverestimation of universities less well ranked
for researchClear for THE and QS, even clearer for Webometrics
And for the best research universities:
US: underestimation THE/QS, overestimation WebometricsUK: overestimation THE/QS, underestimation WebometricsN + S Europe: underestimation THE/QS/Webometrics
IREG Forum University Rankings, Warsaw 16 -17 May 2013
Conclusions
Reputation concept still on its infancy
Necessity to do research on the concept as well as its modalities of implementation