mapping diversity – the u-multirank approach to rankings gero federkeil workshop universidade nova...

Mapping Diversity –

The U-Multirank Approach to Rankings

Gero FederkeilWorkshop Universidade Nova de Lisboa,

29th June 2012

Presentation

Universidade Nova de Lisboa, 2012/06/29 2

Portuguese Universities in Global RankingsPortuguese Universities in Global Rankings

A (Short) Critique of Existing Global RankingsA (Short) Critique of Existing Global Rankings

An Alternative Approach: U-MultirankAn Alternative Approach: U-Multirank

Portuguese Universities in Global

Rankings


Shanghai Ranking QS Ranking THE Ranking

401- 500 U Lisboa 394 U Coimbra 301-350 U Aveiro

301-400 U Porto 401-450 U Nova de Lisboa 351-400 U Porto

401-450 U Porto

501-550 U Catolica Portuguesa

Portuguese universities are not well represent edin international rankings

Hence those rankings are not a good instrument to look at the Portuguese HE system

Presentation



A (Short) Critique of Existing Global RankingsA (Short) Critique of Existing Global Rankings


The coverage of Global Rankings


Existing global rankings cover only a small minority of all universities

EUA: Global University Rankings and their Impact. Bruxelles, 2011.

The Indicators


Existing global rankings are rankings of internationally oriented research universities only

Shanghai Jiaotong Ranking QS Indicator Weight Indicator Weight

SCI publications 20 % Reputation among scholars 40 %

Publications Science & Nature 20 % Reputation among employers 10 %

Highly cited authors 20 % Citations 20 %

Nobel Prizes & Field Medals 20 % Student-staff-ratio 20 %

Alumni with NobelPrizes 10 % International students 10 %

Size 10 % International staff 10 %

http://www.arwu.org/index.jsp

The methodology: Ranking orthodoxy


Ranking of whole institutions

Ranking of whole institutions

Composite overall score

Composite overall score

League table approach

League table approach

Most users are interested in information about “their” field”

Institutional rankings give misleading averages across fields/units

Most users are interested in information about “their” field”

Institutional rankings give misleading averages across fields/units

• Composite indicators blur profiles and strengths & weaknesses

There are neither theoretical nor empirical arguments for assigning specific pre-defined weights to single indicators

• Composite indicators blur profiles and strengths & weaknesses

There are neither theoretical nor empirical arguments for assigning specific pre-defined weights to single indicators

• Small differences in the scores of indicators lead to big differences in league table positions

Give false impression of exactness (“Number 123 is better than number 127”)

• Small differences in the scores of indicators lead to big differences in league table positions

Give false impression of exactness (“Number 123 is better than number 127”)

Conclusions


1. Existing global rankings cover only a small minority of universities

2. Due to their indicators they only measure research and cover internationally oriented research universities only

3. They devaluate other profiles and missions (teaching, LLL, regional engagement)

4. They are a threat to diversity in higher education

Presentation



A Critique of Existing Global RankingsA Critique of Existing Global Rankings


The Project


• Commissioned by the European Commission• 2-year feasibility project, 2009 – June 2011• Report now available:

http://ec.europa.eu/education/higher-education/doc/multirank_en.pdf

• Ján Figel, the former European Commissioner for Education, Training, Culture and Youth:“- to allow stakeholders to make informed choices; - to help institutions to position themselves and improve their performance”

• Two phases:o Design of new instrumento Testing the feasibility of new instrument



Specification of the Project


• Five dimensions:o Teaching & learningo Researcho Knowledge transfero International orientationo Regional engagement

• Development of a list of indicators to be tested in pilot project

• Development of data collection tools and processes (question-naires, definitions, FAQs, communication + feedback processes)

• Done by CHERPA consortium: CHE, CHEPS, CWTS, INCENTIM; OST

Specification of the project


• Two levels:• Institution (FIR)• Fields (FBR)

• Global sample of higher education and research institutions: • 159 (target: 150), 2/3 Europe, • 109 completed institutional questionnaires

• Two pilot fields: • Business studies• Engineering (electrical and mechanical)

The Basic Methodology


Multi-dimensional ranking

Multi-dimensional ranking

Multi-level ranking

Multi-level ranking

Grouping approachGrouping approach

• Different levels of analysis are relevant for different users

Field-based and institutional rankings

• Different levels of analysis are relevant for different users

Field-based and institutional rankings

• Rank groups instead of league tables provide more meaningful and valid information

• Rank groups instead of league tables provide more meaningful and valid information

There is no single objective ranking Each ranking reflects the ideas and preferences

of those doing them The decision about the relevance of indicators

should be left to the user

There is no single objective ranking Each ranking reflects the ideas and preferences

of those doing them The decision about the relevance of indicators

should be left to the user

++

++

++

Mapping Diversity


• A basic aim of U-Multirank is to make visible the diversity of Higher Education Institutions

• and to show excellence beyond research excellence

HOW ?

Universidade Nova de Lisboa, 2012/06/29

15

Starting point: Diversity of higher education institutions in Europe & the world

Starting point: Diversity of higher education institutions in Europe & the world

Identifying comparable institutions that can be

compared in one ranking

Identifying comparable institutions that can be

compared in one ranking

Description of horizontal diversity

Types/profiles

Description of horizontal diversity

Types/profiles

Assessment of vertical diversity

Performance

Assessment of vertical diversity

Performance

Complementary instruments of transparencyComplementary instruments of transparency

+

Mapping and Ranking

Universidade Nova de Lisboa, 2012/06/2916

1. Step: Mapping: Selection of a comparable set of universities based on institutional profiles

1. Step: Mapping: Selection of a comparable set of universities based on institutional profiles

Teaching and learning

Teaching and learning

Research involvementResearch

involvement

Knowledge exchange

Knowledge exchange

Regional engagement

Regional engagement

International orientation

International orientation

Student profileStudent profile

Example:

• Comprehensive, teaching oriented institution• Mainly undergraduate education• Low research orientation• Low international orientation• Regionally embedded (e.g. recruiting)

2. Step: Rankingof subset of comparable institutions with similar profiles

2. Step: Rankingof subset of comparable institutions with similar profiles

Multi-Dimensional Ranking for Subset of

Comparable Institutions


No composite

indicator!No composite

indicator!

No number 1 !

No number 1 !

User-driven, Personalised Ranking


A Personalised Ranking …


… Helping to make an informed choice

based on invidual preferences

… Helping to make an informed choice

based on invidual preferences

Results of the feasibility study


• Generally the concept, indicators and instruments of U-Multirank are feasible both on the institutional and the field level

• There are some problems concerning indicators, mainly on• Issues of employability• Knowledge transfer, and,• Regional engagment

• There has to be some refinement of concepts and indicators

Outlook : U-Multirank II


• New call for tender launched by the European Commission• CHE applied with partners (CHEPS, CWTS, …)• Decision at the end of July (?)• Start of the project on 1 October (?)

• 2 * 2 years• Implementation of the concept: First publication of ranking at

the end of 2013• Min. 500 institutions, institutional and 4 fields• Annual extension of number of institutions and fields• Development of a business model for a sustainable system

But after all, there still might be some

limits to ranking…

22„You‘re kidding! You count publications?“

Muito obrigado!

For more information:

www.u-multirank.eu

[email protected]

http://www.u-multirank.eu/

mailto:[email protected]

Mapping Diversity –

The U-Multirank Approach to Rankings

Gero Federkeil, CHE Centre for Higher Education

Workshop Universidade Nova de Lisboa,

29th June 2012

mapping diversity – the u-multirank approach to rankings gero federkeil workshop universidade nova...

Documents

u nova

presentationuniversidade

projectuniversidade

umultirankuniversidade

conclusionsuniversidade

universitiesuniversidade

indicatorsuniversidade

incentim ostuniversidade