Ranking ≠ Comparability

Barcelona – September 21st 2011

Professor Tony StevensonPro Vice Chancellor – Planning and Resources

University of Newcastle

Introduction

• Most ranking systems aim to rank top 50 or top 250 worldwide......

• But there are at least 15,000 Higher Education Institutions worldwide

• National Ranking systems – some 50 : e.g. CHE; Times• International rankings – some 10: e.g. Jiao-Tong; QS;

THES etc; Webometrics etc

Thought experiment!

• After extensive consultation we invented a rating system for restaurants:

• 4* = World-class • 3* = Of European renown • 2* = Of national renown • 1* = Acceptable. Popular in the city • 0* = Poor

Restaurant qualitypercentage of restaurants submitted in each category

0%

10%

20%

30%

40%

50%

4* 3* 2* 1* 0*

Someburgh

0%

10%

20%

30%

40%

50%

4* 3* 2* 1* 0*

Otherville

0%

10%

20%

30%

40%

50%

4* 3* 2* 1* 0*

Thistown

0%

10%

20%

30%

40%

50%

4* 3* 2* 1* 0*

Elseport

Which City is best?• Weighted Grade Point Average (GPA)

– 4:3:2:1 as advocated by Gastronomy Times or 16:9:4:1 as advocated by Restaurant Fortnight

• Medals tables - only 4*; 4*+3*; all medals as advocated by Association for Haute Cuisine

Rank orderGastronomy Times: ESTORestaurant Fortnight: TEOSAHC Gold medal table: TOSEAHC Gold and Silver medal table: T(EO)SAHC all medal table: ET (OS)

Which City is best?Power – Tourist Board decided that its the number of restaurants submitted that is important

– Someburgh submitted 40– Otherville submitted 30– Thistown submitted 20– Elseport submitted 20

– Impact of Power on the rankings – Multiply the score by the number of restaurants submitted for each city

Power Rank orderGastronomy Times: SOETRestaurant Fortnight: SOTEAHC Gold medal table: (OT)SEAHC Gold and Silver medal table: (OS)TEAHC all medal table: SOET

Which City is best?BUT: the Chamber of Commerce point out that cities may have played a game

and only submitted what they thought were good restaurants They discover that their database reveals that:• Someburgh submitted 40 out of 64 restaurants in total• Otherville submitted 30 out of 33• Thistown submitted 20 out of 40• Elseport submitted 20 out of 30

So you can adjust game playing by altering (downweighting) the power scores by the proportion submitted

Ratio Rank orderGastronomy Times: OESTRestaurant Fortnight: OETSAHC Gold medal table: OTSEAHC Gold and Silver medal table: O(ET)SAHC all medal table: OEST

Which is right?

Rank order Power rank Ratio RankGastronomy Times: ESTO SOET OESTRestaurant Fortnight: TEOS SOTE OETSAHC Gold medal table: TOSE (OT)SE OTSEAHC Gold and Silver medal table: T(EO)S (OS)TE O(ET)SAHC all medal table: ET(OS) SOET OEST

Comparability

0

5

10

15

20

25

4* 3* 2* 1* 0* NS

Someburgh

0

5

10

15

20

25

4* 3* 2* 1* 0* NS

Otherville

0

5

10

15

20

25

4* 3* 2* 1* 0* NS

Thistown

0

5

10

15

20

25

4* 3* 2* 1* 0* NS

Elseport

• Any resemblance of this to the UK’s Research assessment exercise is entirely coincidental

Conclusions

• Difference does not necessarily reflect ordering• The interests of the ranker affect the results as much as

any intrinsic qualities of those being ranked• Ranking systems create reality – constrain University

Missions• Is a University Excellent – how measured; universities

are differentiated in their missions

Conclusions ctd

• Diversity replaced by Hierarchy – networks of power

• Is a University Good or Good at what it does? - How Measured

• Simple graphical representations can be richer than a linear scale to compare quality

• Rankings are here to stay – but lets have lots of them