1

From Individual Ranking to National Benchmarking

Jamil Salmi and Michaela Saisana (JRC)Shanghai, 5 November

2

New developments

More national and international rankings

Ranking audits

Rankings of national higher education systems

Rankings influencing policy-makers

Allocation of scholarships

Allocation of visas

Mergers

Excellence initiatives

3

Outline

Limitations of university rankings

Innovative rankings of national systems

System performance indexes

4

5

6

s

Absolute achievement

vs. value added

A

B

C

D

E

Fach

ieve

me

nt

entry graduation

highest ranked

greatest value added

7

Highly-cited scientists

186

Canada

187

Harvard University

Patents

401

Spain (2010)

811

Professor Robert Langer (MIT)

8

What do rankings tell us about a country’s

performance?

Top 50 universities (2013)

ARWU 2013

THE

2012-13

JAPAN, 2CANADA, 2

UK, 5

WESTERN EUROPE, 6

USA , 35

JAPAN, 1AUSTRALIA, 2

CANADA, 3

OTHER ASIA, 4

WESTERN EUROPE, 4

UK, 7

USA, 29

9

ARWU 2013 ranking related to population

CountryNo, Top

500spopulation (000s)

people required to produce each top 500 U

Ireland 3 4,528 1,509

Denmark 4 5,591 1,398

Norway 4 5,088 1,272

Austria 7 8,484 1,212

Australia 19 22,967 1,209

Switzerland 7 8,050 1,150

Israel 7 7,867 1,124

Finland 5 5,454 1,091

New Zealand 5 4,509 0,902

Sweden 11 9,540 0,867

Number of ranked universities / 1 million people

0.02

0.03

0.16

0.21

0.38

0.39

0.69

0.82

0.00 0.10 0.20 0.30 0.40 0.50 0.60 0.70 0.80 0.90

China

Malaysia

Japan

South Korea

Singapore

Taiwan

Hong Kong

Australia

10

Evolution 2004-2013

Year 2004 2013

S. Korea 8 11

Brazil 4 6

Distribution of scientists

23.1%

59.0%

4.7%

8.7%

72.1%

32.3%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

70.0%

80.0%

Brazil S. Korea

Firms

Research Institutes

Universities

11

How about equity?

No higher education system should be an outstanding system if it is not achieving excellence for its

most disadvantaged students

How about institutional differentiation?

12

outline

Limitations of individual university rankings

Innovative rankings of national systems

Systems rankings

Lisbon Council (2008)

Universitas 21 (2012, 2013)

13

Lisbon Council ranking

17 OECD countries

6 main indicators

Proportion of graduates

Pisa math scores of incoming students

Wage premium of graduates

Proportion of foreign students

Lifelong learning

Ability to reform

14

Universitas 21 country ranking

48 countries (50 in 2013)

4 main groups of indicators

resources (25%)

environment (25%)

connectivity (10%)

output (40%)

Universitas 21 country ranking United States

Sweden

Canada

Finland

Denmark

Switzerland

Norway

Australia

Netherlands

United Kingdom

15

Methodological challenges

overall ranking with arbitrary weights

lack of statistical robustness?

mix of results and determinants of these results

selection of indicators partly based on data availability, not meaningfulness (equity)

mix of data from various years

Positive features

system-wide focus

large range of indicators (multi-dimensional)

control for size

16

Outline

Limitations of university rankings

Innovative rankings of national systems

System performance indexes

Performance Index

Professor Aghion et al.

National Performance Index

Number of ARWU-ranked universities weighted by their rank divided by population, relative to USA (= 100)

17

Aghion Performance Index

Country 2004

Switzerland 1

US 2

Sweden 3

UK 4

Denmark 5

Norway 6

Netherlands 7

Canada 8

Finland 9

Australia 10

Salmi/Saisana Research Excellence Index

Building on Aghion’s National Performance Index

Number of ARWU-ranked universities weighted by their actual score divided by population, relative to USA (= 100)

18

Country 2013 2004

Switzerland 1 1

Sweden 2 2

Israel 3 3

Denmark 4 4

Netherlands 5 6

Australia 6 12

New Zealand 7 13

Norway 8 9

Finland 9 7

UK 10 5

Belgium 10 11

Canada 12 10

USA 13 8

Country Salmi/Saisana Universitas 21 Lisbon Council

Switzerland 1 3 14

Sweden 2 2 6

Israel 3 3 n.r.

Denmark 4 5 3

Netherlands 5 7 13

Australia 6 8 1

New Zealand 7 13 n.r.

Norway 8 11 n.r.

Finland 9 6 4

UK 10 10 2

Singapore n.r. 9 n.r.

Canada 12 4 n.r.

USA 13 1 5

19

Contribution of rankings

Put performance on the map

20

Contribution of rankings

Put performance on the map

Role of universities in economic development

Need for reliable data to measure all dimensions

University rankings can be complemented by systems rankings

But still zero-sum game (Red Queen effect)

Move towards benchmarking

21

Benchmarking

Comparing your performance to competitors or good practices

Purpose is to identify weaknesses for improvement

No hierarchy

You choose your own indicators – don’t have to follow the selection of the ranking

Theoretical framework

• distinction between performance and health of system

• how good are the system’s actual outcomes?

• does it operate under conditions known to lead to high performance?

22

political & economic stability, rule of law, basic freedoms

resources & incentives

quality assurance & enhancement

telecommunications & digital

infrastructure

governance & regulatory framework

articulation & integration mechanisms

vision, leadership & reform capacity

location

attainment

learning

equity

research

technology transfer

values

results

44

23

The rankings

businessA ranking of league tables

September 10, 2013