quality assurance and university rankings

www.aqu.cat

La qualitat, garantia de millora.

www.aqu.cat

La qualitat, garantia de millora

Quality Assuranceand

RankingsJosep A. Ferré

DirectorSeptember 2011

www.aqu.cat


Quality assurance and rankings?

www.aqu.cat


Some HEI in Spain have read this, but not many seem to have understood it !

The QA problemOUR MESSAGE:1. Our objective is that you (the HEI) have such a good Internal Quality Assurance

System (IQAS) that we (the QA agency) may develop external quality assurance processes that are of very low intensity.

2. Meanwhile … if you do not have an IQAS we’ll help you to conceive, design and implement one, the one that you choose for you (AUDIT program).

3. In addition, we’ll help you to apply the IQAS to your programs through the life cycle defined in the Framework for Accreditation of programmes of AQU.

4. After, we’ll come back to evaluate how this IQAS is operating. However we’ll not analyze directly the IQAS, rather the focus will be on the outputs of your programmes. If they are “within control limits” we’ll conclude that …

• your choice of this IQAS was correct and it is working• your processes are under control

5. If you have a good IQAS and good results then you can become more transparent and step into higher accountability, and then ….

• your stakeholders will grant more autonomy to your institution• more resources will be committed to you (… perhaps!)• you will strengthen your social leadership and recognition

www.aqu.cat


Framework for the Accreditation of programmes

Accreditation

Plan

Develop

Monitor

Improve

Plan

Develop

Monitor

ImprovePlan

Develop

Monitor

Improve

Plan

Develop

Monitor

Improve

MonitoringMonitoring report

Monitoring report

External evaluation

Monitoring report

Ex-ante accreditation

www.aqu.cat


Monitoring report

Annual plan

Quantitative indicators

Assessment of the indicators

Improvement actions

Modification

Plan

Develop

Monitor

ImproveHow would you improve it?

What do you want to do? Why?

How will you do it?

Why that way?

How do you know it’s OK?

How will you measure it?

How have you done it?

Would you repeat it?

www.aqu.cat


OUTPUT


MissionVisionValues

OUTPUTPROCESSINPUTH

R

MR

OR

G

GOVERNANCE

€GOVUNIVEST

SOC

Specifications

Quantitative indicators: • Characteristics of the input • Contributions of the university to the teaching-learning process• Characteristics of the output

Which quantitative indicators should we monitor?

www.aqu.cat



1. Previous qualifications and degrees of the incoming students2. Personal, family and social characteristics of the students

3. Qualifications and competence of the teaching staff4. Hours of lectures, seminars, labs, etc.. and average group size5. Classrooms, labs, libraries and other material resources used6. Use of the virtual campus (volume of text rich documents, downloads, etc.)7. Availability, updatability and usability of the web information8. Evidence of the assessment methods used (exams, reports, etc.)9. Student internship, mobility and career advice10. Student satisfaction, teachers satisfaction

11. Academic results (performance, residence time, abandon rate, etc)12. Personal results (personal and inter-personal skills)13. Employability (occupation and profession)

www.aqu.cat


Quality assurance and rankings?

Rankings are not an ingredient of the QA recipe … but ……what would happen if you add a ranking to your QA recipe?To add a ranking to your QA recipe, first you should check that both are “compatible”. A very nice approach to this compatibility test are the …

You will find them and many more in this recent document of the EUA …..

I recommend reading it with great care without skipping over a single word !!

www.aqu.cat


An example of a bibliometric ranking

www.aqu.cat


If I was to make a ranking (1) …

2 – How HEI’s stakeholders (society, public authorities, students, HEI’s employees) understand these three missions are best achieved? How they evaluate the quality of any/all the three missions?

Fitness for purpose? Value for money?Excellence? Perfection? Transformation?

1 - Which are the relevant missions of the institutions to be ranked?• Teaching or “only teaching”?• Research or “also research”?• 3rd mission? (… what’s that?)

4 - How homogeneous/comparable is the data? to which domain can it be extended without lacking context?(regional, national, international) × (field)

3 – How the success in each of the assigned missions is measured? • which data we already have? (do we use it for decision making?)• which data we do not have?• do we have plans to collect the missing data? (do we plan to use it?)

www.aqu.cat


3rd missionresearch

teaching

soci

ety

publ

ic a

utho

ritie

s

stud

ents

HE

I em

ploy

eesfitness for purpose

value for money

excellence

perfection

transformation 5 × 4 × 3 = 60 !Which data we have on this?Which data we do not have?

Multidimensional reality perception / conceptual domain

www.aqu.cat


Who is interested in having rakings that span this range of domains?….Our stakeholders are interested in all the physical domains?… Or, are we thinking about future customers rather than current stakeholders?

Which data we have on this?Which data we do not have yet?Which data we’ll never have?

3rd missionresearch

teaching

soci

ety

publ

ic a

utho

ritie

s

stud

ents

HE

I em

ploy

eesall the world

international

national

regional

field

Multidimensional physical domain

www.aqu.cat


If I was to make a ranking (2) …

5 – Raw data of pre-processed data?Pre-processing options:• how do we make it “size invariant”? … OUTPUT/SIZE ?• how do we align the zero level? …. (OUTPUT - INPUT) ?• how do we relate to cost? …. (OUTPUT-INPUT)/€ ?• only € matters? … (OUT-IN) / (€+LEGISLATION+GOVERNANCE) ?

7 – Moving average, current value or rate of change data?

6 – Keep it multi-dimensional or cook into a single number?• will users be allowed to select the weights?• poor teaching can be replaced by good research?• no-research can be replaced by good teaching?• can we skip the 3rd mission from the list? …

www.aqu.cat


And now we have it !

8 – What’s for? … How we’ll use it?- it’s a drive for continuous improving?- connects with the strategic management of the institution?- is related to some data driven decision making process?

… If quality in higher education is a core success factor for institutional success, the quality assurance processes should not be considered as something “extra” in addition to the other core processes in higher education, but should be integral. Guaranteeing a certain quality or enhancing the quality of a programme becomes an integral part of the regular management of a HEI. There is a trend of moving from quality assurance towards quality oriented higher education management, which means that former quality assurance processes get directly linked to core management processes.

www.aqu.cat


- or is just part of the marketing (reputation management) of the HEI?

And now we have it !

8 – What’s for? … How we’ll use it?- it’s a drive for continuous improving?- connects with the strategic management of the institution?- is related to some data driven decision making process?

- It’s a part of the public policy of the government?- It’s the “placebo” of the public policy of the government?- It’s the substitute of the policy that public authorities …

~ do not have? (…not yet developed)~ do not want to have? (…neo-liberal approach)~ do not have time to elaborate? (…too fast!)~ are happy to have externalized (…no comment!)

Are there some connections here?

Do HEI prefer rankings to an IQAS or to transparency & accountability?Have QA agencies failed not putting transparency & accountability on top of the list?

www.aqu.cat


Surveys

MEDIA

… quality assurance… transparency

… enhancement of quality… accountability

HE

IDAT

A

QA(ESG 1.6 and 1.7)

DATA

Data driven decision making and strategic management

Transparency … “to show list of all the things HEI do”

Accountability … “to check the list of all the things HEI do .vs. stakeholder’s expectations”

Accountability+ … “to check with the stakeholders the list of (output-input)/resources”

Communication … “to publish the list of all the things HEI do”

QA, rankings and transparency tools

RA

NK

ING

GO

VE

RM

EN

T

SO

CIE

TY

ES

TUD

.

Surveys

Should QA have pushed up ESG 1.6 and 1.7 to the top of

the list?

www.aqu.cat


Why transparency and accountability?

Intended mission(s)

Resources and

autonomy

Transparency and

accountability

What activities stakeholders expect from

universities to develop?

How universities demonstrate that they develop their mission(s) with effectiveness and efficiency?

What resources (€ and rules) are

allocated to universities?

The resources allocated and the transparency and accountability methods are always an evidenceof the true intended missions (even if these have not been formulated explicitly).Therefore it is easy to detect inconsistency when niceintended missions are associated to insufficient resources or poor transparency and accountability tools, for example, rankings!

What universities

do?

What society expects from universities?

How universities serve the

social needs?

www.aqu.cat


QA and rankings …. revisited

When you read carefully the ranking recipe is difficult to find there any reference to …• “fitness for purpose”… the mission of the HEI ... the effectiveness • “value for money” … the resources allocated to the HEI … efficiency

???The QA recipe and the rankings recipe share some common ingredients…

• “a continuous improving strategy” … the internal QA effort • “a self-evaluation report” … transparency & accountability

• but are cooked in a very different way• are served with different dressings• point to opposite goals

• “experts committee” … the external QA review

www.aqu.cat


Rankings are like …

When we say “food” we do not figure up the same things!

www.aqu.cat


Thank you very much!

20

quality assurance and university rankings

Education