rankings and the emergence of new forms of ......rankings and the emergence of new forms of...

www.dit.ie/researchandenterprise

RankingsandtheEmergenceofNewFormsofAccountability

ProfessorEllenHazelkornPolicyAdvisortotheHigherEducationAuthority(Ireland)Director,HigherEducationPolicyResearchUnit(HEPRU)

130th NEASCAnnualConferenceandMeetingDecember2015

Boston

www.dit.ie/researchandenterprise

WhicharetheBestUniversities&Colleges?

Are the best universities those which best match the criteria established bythe different rankings OR those that help the majority of students earn thecredentials for sustainability living and employment?Are the best universities those that contribute to new scientific discoveriesand highly trained PhDs OR those that “emphasize the obligations studentshave to serve their communities and the nation at large”?Are the best universities those that reinforce an elite knowledge society(where progress depends on the cutting-edge knowledge of the few) ORthose that help build-up a mass knowledge society (where progress dependson the “wisdom of the many”)?Should we define quality according to reputation/prestige, studentsatisfaction, salary level of graduates OR how students have beentransformed by their educational experience, e.g. learning gain?

www.dit.ie/researchandenterprise

Themes

• RankingsandtheirLegacy

• NewFormsofAccountability

• ImplicationsforHigherEducation

www.dit.ie/researchandenterprise

1.RANKINGSANDTHEIRLEGACY

www.dit.ie/researchandenterprise

Rankings’Legacy

• Global rankings have become a game-changer;• Immediately – and subsequently – attracted attention of policymakers and

the academy:– Choice of indicators has set parameters for what constitutes quality;– Visible measure of global competitiveness and multi-polar character;– “Top 100” has transformed “world-class” into a strategy, a language, a

topic of study;– Driven profound transformation of our HEIs and HE systems;– Placed HE/R&D investment high on political and policy agenda.

• Today, less about student choice and more about geopolitical positioning.And in the process, a whole industry has been created.

www.dit.ie/researchandenterprise

WorldOrderAccordingtoRankingsTop100,2004-2015

www.dit.ie/researchandenterprise

MeasuringanElite

• Over18,000HEIsworld-wide,butglobalrankingsfocusoncharacteristicsandperformanceofthetop100universities;- lessthan0.5percentoftotal;- ~0.4percentoftotalworldpopulationof196mtertiarystudents(UIS,

2012).

• AsdemandforHEgrowsworldwide,selectivityisaccelerating.

www.dit.ie/researchandenterprise

InfluenceofRankings

• Restructuringofnationalsystems;• Reshapingofnationalpriorities;• Refocusingofinstitutionalpriorities;• ReorganisingtheHEI,institutionaldepartmentsandhierarchyof

disciplines;• Emphasisonresearchvs.teaching;postgraduatevs.undergraduate– with

implicationsfortheacademicprofession;• Changesinresearchpractice:language,publication,orientation,

basic/applied,etc.• Influenceonstakeholders– students,governments,business/employers,

investors,public,etc.

www.dit.ie/researchandenterprise

MeasuringQuality

• Despite all the criticism, rankings have succeeded in placing HE within awider comparative and international framework;

• Quality and excellence are key differentiators in national/global market;– National geo-political positioning and pride;– Beacon to attract/retain investment, business and talent;– Institutional reputation and status;– Performance assessment of scientific-scholarly research;– Link between qualification and career opportunities and life-style;– Value-for-money and return-on-(public) investment;

• “All the things wrong with the rankings matter considerably less than theplain fact that the rankings matter” (Locke, 2011, 226)

www.dit.ie/researchandenterprise

GlobalSignificance

Global economy critical to understanding significance and impact:• Early years associated with tail-end of long period of economic growth

driven by unregulated finance capital;• Latter years marked by lingering effects of global economic crisis – and

search for sustained recovery and economic growth:– Intensification of competition for a greater share of mobile capital and

talent;– Noticeable shifts in “world order”;– Significant demographic changes;– R&D investment patterns and geographic imbalances;– HE is big business for nations and institutions;

• Internationalisation• Deregulation of HE market: for-private/cross-border providers.

www.dit.ie/researchandenterprise

QualitynowaGeopoliticalIssue

• As globalisation accelerates and market principles intrude further, growingneed, arguably out of necessity, to regulate the marketplace;– Supra-national governments, national governments/US states, HE

agencies, commercial media, HE organisations;– Global intelligence business leading to common international data set;– In global labour market, quality credentials are essential for

international recognition.• Greater government involvement either directly in the process and/or use

of outcomes in policy and decision making;- Signals extent to which HE is losing role as primary guardian of quality

(Harman, 2011, 51;Dill and Beerkens, 2010, 313-315).

– Explains why global rankings have assumed such significance at ageopolitical level.

www.dit.ie/researchandenterprise

2.NEWFORMSOFACCOUNTABILITY

www.dit.ie/researchandenterprise

AlternativeRankings/AlternativestoRankings

• Rankings have been the dominant instrument but there are a range ofother tools being developed:– Alternative rankings by competitors, as new products and services;– Alternatives to rankings by governments, agencies, HE, and others.

• Traditionally,qualitywasmeasuredininput(e.g. studententry,academicqualifications,budget/income,libraryresources) andreputation;

• Asthe“private”and“public”goodrolesofhighereducationcomeunderincreasingscrutiny,thefocushasshiftedtooutputs, outcomes,impact,benefitandrelevance;

• Shows different ways to measure quality.

www.dit.ie/researchandenterprise

Transparency&AccountabilityInstruments

Rankings

Classification&Profiling

QualificationsFrameworks

QA&LearningOutcomes

Scorecards&Websites

Globalintelligence

SocialNetworking

www.dit.ie/researchandenterprise

Rankings

Rankings

Classification&Profiling

QualificationsFrameworks

QA&LearningOutcomes

Scorecards&Websites

Globalintelligence

SocialNetworking

www.dit.ie/researchandenterprise

GlobalRankings

• AcademicRankingofWorldUniversities (ARWU)(ShanghaiJiaoTongUniversity,China),2003

• Webometrics(SpanishNationalResearchCouncil,Spain),2004• NationalTaiwanUniversityRankings(formerlyPerformanceRankingof

ScientificPapersforResearchUniversities, HEEACT),2007• LeidenRanking(CentreforScience&TechnologyStudies,Universityof

Leiden),2008• SCImagoJournalandCountryRank(SJR)(Spain),2009• UniversityRankingbyAcademicPerformance(URAP) (InformaticsInstitute

ofMiddleEastTechnicalUniversity,Turkey),2009• QSWorldUniversityRankings(QuacquarelliSymonds,UK),2010• THE WorldUniversityRanking(TimesHigherEducation,UK),2010• U-Multirank (EuropeanCommission, Brussels),2014• BestGlobalUniversitiesrankings (USNWR,US),2014

www.dit.ie/researchandenterprise

SelectNationalRankings(red=governmentsponsored)

INSTITUTIONALDISCIPLINE/

SUB-CATEGORIESSPECIALIST

• UniversityRankingSystem(Bulgaria)• CHE-HochschulRanking (Germany)• ExpertUniversityRanking(Russia)• GoodUniversityGuide(Australia)• GuardianUniversityGuide(UK)• UniversityRankingsofIslamicCountries(Iran)• HigherEducationCommissionRankings(Pakistan)• LaRepubblicaGrandeGuidaUniversità (Italy)• Maclean’s OnCampus(Canada)• NationalRankingsof BestUniversities(Kazakhstan)• Netbig ChineseUniversityRanking(China)• NigeriaUniversities CommissionRanking• OHEC(Thailand)• PerspektywyUniversityRanking(Poland)• RankingU-Sapiens(Colombia)• Sunday TimesGoodUniversityGuide(Ireland)• TimesHigherEducationUniversityGuide(UK)• Top200UniversityRankings(Ukraine)• URANK-rank(Sweden)• USNewsandWorldReport(USNWR)CollegeRankings(US)

• Dataquest(India)• India Today(India)• Outlook(India)• LeNouvelObservateur (France)• SherifMagazine(Iran)• NationalResearchCouncilRankingof

Doctoral Programmes(US)• Toplawschools.com(US)• AmericanUniversities Admission

Programme:UndergraduateAmericanUniversities Rankingsfor InternationalStudents (US)

• USNewsandWorldReport(USNWR)TopMed Schools(US)

• WPROSTMBA (Poland)

• CollegeNET SocialMobilityIndexRanking(US)

• GeorgetownPublicPolicy ReviewPlacementEfficiencyRanking(US)

• Metroversities (US)• NewYorkTimesMost

Economically DiverseTopColleges(US)

• OnlineStudyAustraliaOnlineUniversityRankingList(Australia)

• PrincetonReview(US)• SavioursofOurCities (US)• SocialMobilityIndex(CollegeNet

andPayscale,US)• WashingtonMonthlyCollege

Guide(US)• WashingtonMonthlyRankingof

CommunityColleges(US)

www.dit.ie/researchandenterprise

U-Multirank

www.dit.ie/researchandenterprise

www.dit.ie/researchandenterprise

Classification&Profiling

Rankings

Classification&Profiling

QualificationsFrameworks

QA&LearningOutcomes

Scorecards&Websites

Globalintelligence

SocialNetworking

www.dit.ie/researchandenterprise

U-Map

• Profilingtooltofacilitateeasycomparisonbetweendifferent institutionsandinformstudentchoiceorstrategicdecisionmakingbyinstitutionsorgovernments.• SimilarinitiativeshavebeendevelopedbyNorway,Ireland,Australia

www.dit.ie/researchandenterprise

www.dit.ie/researchandenterprise

QualificationsFrameworks

Rankings

Classification&Profiling

QualificationsFrameworks

QA&LearningOutcomes

Scorecards&Websites

Globalintelligence

SocialNetworking

www.dit.ie/researchandenterprise

www.dit.ie/researchandenterprise

QA&LearningOutcomes

Rankings

Classification&Profiling

QualificationsFrameworks

QA&LearningOutcomes

Scorecards&Websites

Globalintelligence

SocialNetworking

www.dit.ie/researchandenterprise

www.dit.ie/researchandenterprise

LessonsfromBologna

• Bologna places quality within broader educational framework;• Big idea is the “accountability loop”:

– Qualification framework facilitates flexible learning paths and sets out clearstatement of what students must demonstrate at each qualification level

– “Tuning” aids (re-)design, development, implementation, evaluation andenhancementof quality degree programmes within context

– Formalized concept of learning outcomes rather than measuring inputs (e.g.credit hours, classroom teaching, entry grades)

– QA systems sets evaluation standards and guidelines for institutional self-assessment and quality processes, and externalmonitoring

– Diploma Supplement provides description of academic career and thecompetencies acquired

www.dit.ie/researchandenterprise

Measuring&ComparingAchievementsofLearningOutcomes

• CALOHEEisaEUfundedprojectdrawingontheTuningexperiencebeginning2017;

• SeekstostudywhetherstudentsenrolledinHEaroundEuropedeveloptheappropriatecompetencesbylookingatstudyprogrammes,andwhethertheydeliver theirpromises;– UnlikeAHELOwhichlooksatinstitutionallevel,thisprojectlooksat

theprogrammelevel• Willfocuson5domain:Engineering (CivilEngineering),HealthCare

(Nursing),Humanities(History),NaturalSciences(Physics),andSocialSciences(Education).

www.dit.ie/researchandenterprise

AHELO

• AssessmentofLearningOutcomesdevelopedbyOECDinresponse tocriticismofrankings;

• Attentiononoutputsoflearning,– Genericskills commontoallstudents

(suchascriticalthinking,analyticalreasoning,problem-solving,andwrittencommunication).

– Discipline-specificskills ineconomicsandengineering

– Contextualquestionnaires arealsobeinggivenatthestudent,facultyandinstitution leveltolinkthedatatostudentbackgroundsandlearningenvironments.

www.dit.ie/researchandenterprise

TeachingExcellenceFramework(UK)

• TEFwillreplaceQAA;• NewOfficeofStudent(OfS);• Spotlightonteaching,andencourage

excellentteaching;• Provideextensive informationto

studentsandemployers;• Metrics:

– Teachingquality– Learningenvironment– Studentoutcomesandlearninggain

• InstitutionswhichperformagainstTEFcanraisetuitionfee;– Institutionswhichdonot- willfail.

www.dit.ie/researchandenterprise

Scorecards&Websites

Rankings

Classification&Profiling

QualificationsFrameworks

QA&LearningOutcomes

Scorecards&Websites

Globalintelligence

SocialNetworking

www.dit.ie/researchandenterprise

www.dit.ie/researchandenterprise

www.dit.ie/researchandenterprise

www.dit.ie/researchandenterprise

QualityIndicatorsforLearningandTeaching(Australia)

• PurposebuiltwebsiteknownasQILT(www.qilt.edu.au).– ReplacesMy-Uni

• Helpsstudentsandfamiliesmakeinformedchoices;

• Dataon:– Overallsatisfactionofcurrentstudents– Overallsatisfactionofrecentgraduates– Outcomesofstudentsmovingintofull-

timeemploymentaftergraduation– Mediansalaryreceivedbyrecent

graduates.

www.dit.ie/researchandenterprise

GlobalIntelligence

Rankings

Classification&Profiling

QualificationsFrameworks

QA&LearningOutcomes

Scorecards&Websites

Globalintelligence

SocialNetworking

www.dit.ie/researchandenterprise

GlobalInitiatives

• GlobalInstitutionalProfilesProject– ThomsonReuters- Dataprofilesof700+globallysignificant institutions, combiningreputational

assessment, scholarlyoutputs,fundinglevels,facultycharacteristicsandmuchmoreinonecomprehensivedatabase.

• GlobalResearchUniversityProfiles(GRUP)– ARWU- Comprehensivedatabaseandbenchmarkingtoolcovering1200research

universitiesacross40indicators,allowinguserstolearnaboutperformanceoflocalandinternationalpeersforvariouspurposes.

• TimesHigherEducationGlobalProfilesProject– THE- PreviouslyassociationwithTR,nowwithScopus– planstodevelopitsown

globaldatabase.

• EuropeanTertiaryEducationRegister(ETER)– EuropeanUnion- DatabaseofEuropeanHEIs ,36countriesand2,673HEIs;underheadingse.g.

staffandstudents,subjectdomains,researchactivityandexpenditures.

www.dit.ie/researchandenterprise

SocialNetworking

Rankings

Classification&Profiling

QualificationsFrameworks

QA&LearningOutcomes

Scorecards&Websites

Globalintelligence

SocialNetworking

www.dit.ie/researchandenterprise

Student“TripAdvisor”

www.dit.ie/researchandenterprise

3.IMPLICATIONSFORHIGHEREDUCATION

www.dit.ie/researchandenterprise

ShapeofThingstoCome

Brief survey of recent developments shows what is coming down the track –what is happening internationally, and puts US experience in broader context.

• HE has always been competitive, but it must now learn to survive in thismore complex, and globally competitive environment in which:

- Quality and Excellence are key mantra;- Greater public scrutiny and transparency are normalised;- National prominence is no longer sufficient;- Survival no longer guaranteed.

• There is likely to be a “savage sorting of winners and losers” (Sassen, 2011)

www.dit.ie/researchandenterprise

WouldtheDebateHappenedOtherwise?

• Academy has been slow to engage meaningfully in discussion aboutquality;- Efforts at obfuscation, “gaming” and boycott have not helped;- Toomuchemphasisonprocess;- Complaintsaboutexcessivebureaucracy,form-filling,andcostof

compliance;- Pointedtothel’Oreal factor:becausewe’reworthit;

• Information deficit has created opportunities for the public andgovernments – and commercial interests – to define quality for their ownpurposes.

www.dit.ie/researchandenterprise

BeCarefulWhatYouAskFor

• Publicdiscourseinrecentyearshasfocusedattentionon:– Meetingneedsofnationaleconomies,– Concernsaboutgraduateemployabilityandaffordability– Response toconcernsaboutvalueandimpact;

• Noticeableshifttomeasuringteachingandlearningoutcomesthatallowpublictojudgewhethergraduateshavethethresholdqualitiesexpected;

• Rankingsinitiallyfilledthisgapbuttheyfocustoonarrowlyoneliteuniversitiesandresearch;

• Arguably– manyofthenewdevelopmentsarearesponse totheover-emphasisonresearch ratherthanT&L.

www.dit.ie/researchandenterprise

StartDrivingtheBus

• Institutional and country-based evaluations are rapidly being overtaken byinternational efforts.

• Public/political support for universities and colleges only maintained byassurances of quality because society, students and others have a right toknow whether the institutions are capable of meeting expectations.

• Little doubt that determining and measuring quality is complex;- Lots of “good practice” but no agreed definition;- Difficult to compare diverse institutions across jurisdictions;

• BUT – however worthy, criticisms of government/international efforts are– they miss the ‘politics’ of the situation.

• Challenge is how does HE respond? Can we afford to wait?

Ifdon’t cookowndinner– don’tcomplainwhat’sservedtoyou.

www.dit.ie/researchandenterprise

HigherEducationPolicyResearchUnit(HEPRU)

Ellen.hazelkorn@dit.iehttp://www.dit.ie/hepruhttps://www.researchgate.net/profile/Ellen_Hazelkorn