1
From Individual Ranking to National Benchmarking
Jamil Salmi and Michaela Saisana (JRC)Shanghai, 5 November
2
New developments
More national and international rankings
Ranking audits
Rankings of national higher education systems
Rankings influencing policy-makers
Allocation of scholarships
Allocation of visas
Mergers
Excellence initiatives
3
Outline
Limitations of university rankings
Innovative rankings of national systems
System performance indexes
4
5
6
s
Absolute achievement
vs. value added
A
B
C
D
E
Fach
ieve
me
nt
entry graduation
highest ranked
greatest value added
7
Highly-cited scientists
186
Canada
187
Harvard University
Patents
401
Spain (2010)
811
Professor Robert Langer (MIT)
8
What do rankings tell us about a country’s
performance?
Top 50 universities (2013)
ARWU 2013
THE
2012-13
JAPAN, 2CANADA, 2
UK, 5
WESTERN EUROPE, 6
USA , 35
JAPAN, 1AUSTRALIA, 2
CANADA, 3
OTHER ASIA, 4
WESTERN EUROPE, 4
UK, 7
USA, 29
9
ARWU 2013 ranking related to population
CountryNo, Top
500spopulation (000s)
people required to produce each top 500 U
Ireland 3 4,528 1,509
Denmark 4 5,591 1,398
Norway 4 5,088 1,272
Austria 7 8,484 1,212
Australia 19 22,967 1,209
Switzerland 7 8,050 1,150
Israel 7 7,867 1,124
Finland 5 5,454 1,091
New Zealand 5 4,509 0,902
Sweden 11 9,540 0,867
Number of ranked universities / 1 million people
0.02
0.03
0.16
0.21
0.38
0.39
0.69
0.82
0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90
China
Malaysia
Japan
South Korea
Singapore
Taiwan
Hong Kong
Australia
10
Evolution 2004-2013
Year 2004 2013
S. Korea 8 11
Brazil 4 6
Distribution of scientists
23.1%
59.0%
4.7%
8.7%
72.1%
32.3%
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
80.0%
Brazil S. Korea
Firms
Research Institutes
Universities
11
How about equity?
No higher education system should be an outstanding system if it is not achieving excellence for its
most disadvantaged students
How about institutional differentiation?
12
outline
Limitations of individual university rankings
Innovative rankings of national systems
Systems rankings
Lisbon Council (2008)
Universitas 21 (2012, 2013)
13
Lisbon Council ranking
17 OECD countries
6 main indicators
Proportion of graduates
Pisa math scores of incoming students
Wage premium of graduates
Proportion of foreign students
Lifelong learning
Ability to reform
14
Universitas 21 country ranking
48 countries (50 in 2013)
4 main groups of indicators
resources (25%)
environment (25%)
connectivity (10%)
output (40%)
Universitas 21 country ranking United States
Sweden
Canada
Finland
Denmark
Switzerland
Norway
Australia
Netherlands
United Kingdom
15
Methodological challenges
overall ranking with arbitrary weights
lack of statistical robustness?
mix of results and determinants of these results
selection of indicators partly based on data availability, not meaningfulness (equity)
mix of data from various years
Positive features
system-wide focus
large range of indicators (multi-dimensional)
control for size
16
Outline
Limitations of university rankings
Innovative rankings of national systems
System performance indexes
Performance Index
Professor Aghion et al.
National Performance Index
Number of ARWU-ranked universities weighted by their rank divided by population, relative to USA (= 100)
17
Aghion Performance Index
Country 2004
Switzerland 1
US 2
Sweden 3
UK 4
Denmark 5
Norway 6
Netherlands 7
Canada 8
Finland 9
Australia 10
Salmi/Saisana Research Excellence Index
Building on Aghion’s National Performance Index
Number of ARWU-ranked universities weighted by their actual score divided by population, relative to USA (= 100)
18
Country 2013 2004
Switzerland 1 1
Sweden 2 2
Israel 3 3
Denmark 4 4
Netherlands 5 6
Australia 6 12
New Zealand 7 13
Norway 8 9
Finland 9 7
UK 10 5
Belgium 10 11
Canada 12 10
USA 13 8
Country Salmi/Saisana Universitas 21 Lisbon Council
Switzerland 1 3 14
Sweden 2 2 6
Israel 3 3 n.r.
Denmark 4 5 3
Netherlands 5 7 13
Australia 6 8 1
New Zealand 7 13 n.r.
Norway 8 11 n.r.
Finland 9 6 4
UK 10 10 2
Singapore n.r. 9 n.r.
Canada 12 4 n.r.
USA 13 1 5
19
Contribution of rankings
Put performance on the map
20
Contribution of rankings
Put performance on the map
Role of universities in economic development
Need for reliable data to measure all dimensions
University rankings can be complemented by systems rankings
But still zero-sum game (Red Queen effect)
Move towards benchmarking
21
Benchmarking
Comparing your performance to competitors or good practices
Purpose is to identify weaknesses for improvement
No hierarchy
You choose your own indicators – don’t have to follow the selection of the ranking
Theoretical framework
• distinction between performance and health of system
• how good are the system’s actual outcomes?
• does it operate under conditions known to lead to high performance?
22
political & economic stability, rule of law, basic freedoms
resources & incentives
quality assurance & enhancement
telecommunications & digital
infrastructure
governance & regulatory framework
articulation & integration mechanisms
vision, leadership & reform capacity
location
attainment
learning
equity
research
technology transfer
values
results
44
23
The rankings
businessA ranking of league tables
September 10, 2013