___ I%g 83

23 LCSHD Paper Series

Department of Human Development

I

Quality Assurance in Higher Education:Recent Progress; Challenges Ahead

Elaine El-KhawasRobin DePietro-Jurand

*Lauritz Holm-Nielsen

September 1998

The World Bank

Latin America and the Caribbean Regional Office

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

Pub

lic D

iscl

osur

e A

utho

rized

LCSIID Paper Series

No. i Van der Gaag and Winkler, Children of the Poor in Latin America and the Caribbean

No. 2 Schneidman, Targeting At-Risk Youth: Rationales, Approaches to Service Delivery andMonitoring and Evaluation Issues

No. 3 Harrell, Evaluacion de los Programaspara NiAosy Jovenes Vulnerables

No. 4 Potashnik, Computers in the Schools: Chile's Learning Network

No. 5 Barker and Fontes, Review and Analysis of International Experience with Programs Targetedon At-Risk Youth

No. 6 Lewis, Measuring Public Hospital Costs: Empirical Evidence from the Dominican Republic

No. 7 Edwards, Bruce and Parandekar, Primary Education Efficiency in Honduras: What Remains tobe Done?

No. 8 Winkler, Descentralizacidn de la Educacion: Participacidn en el Manejo de las Escuelasal Nivel Local

No. 9 Meza, Descentralizacidn Educativa, Organizacidn y Manejo de las Escuelas al Nivel Local:EDUCO, el Caso de El Salvador

No. 10 Espinola, Descentralizacidn Educativa, Organizacion y Manejo de las Escuelas al Nivel Local:El Caso de Chile

No. 11 Guedes, Lobo, Walker and Amaral, Gesti6n Descentralizada de la Educaci6n enel Estado de Minas Gerais, Brasil

No. 12 Cominetti and Ruiz, Evolucion del Gasto Puiblico Social en America Latina: 1980 - 1995

No. 13 Bedi and. Edwards, The Impact of School Quality on the Level and Distribution of Earnings:Evidence from Honduras

No. 14 Duthilleul, Do Parents Matter? The Role of Parental Practices on Fourth Graders' ReadingComprehension Achievement in Montevideo Public Schools

No. 15 Villegas-Reimers, The Preparation of Teachers in Latin America: Challenges and Trends

No. 16 Edwards and Liang, Mexico's Preschools: Coverage, Equity and Impact

No. 17 Soares, The Financing of Education in Brazil: With Special Reference to the North, Northeastand Center-West Regions

No. 18 Salmi, Equity and Quality in Private Education: The Haitian Paradox

No. 19 Waiser, Early Childhood Care and Development Programs in Latin America: How muchdo they cost?

No. 20 Tulic, Algunos Factores del Rendimiento: las Expectativasy el GCnero

No. 21 Delannoy, Reformas en Gestidn Educacional en los 90s

No. 22 Barro, The Prospects for Developing Internationally Comparable Education Finance Statisticsfor Latin American Countries: A Preliminary Assessment

No. 23 El-Khawas, DePietro-Jurand, Holm-Nielsen, Quality Assurance in Higher Education: RecentProgress; Challenges Ahead

Human Development DepartmentLCSHD Paper Series No. 23

Quality Assurance in Higher Education:Recent Progress; Challenges Ahead

Elaine El-KhawasRobin DePietro-JurandLauritz Holm-Nielsen

September 1998

pre.en pr:i .. a. , noi.e..ulso . ........ r .s::.. ......... i.circulated to~~~~~~.. e ..ourag di:sceussion arandt~~ ormment; any ciati onad s f s ae

sbudtk ---eco fitspoisoa -hrctr Th nd--ngs- intrrtL s -ait~~~~~~~~~~~~~~~~~~~~.....j. ...... ............. . ........ .j ... ...... . .. ............ ....... .. ........ ..

conclusions expressed in this paper areetiely those oftheauthorsd.should...t.be-i - .d'' " ' '' ''' ' '' ' 'i ' '' ''''' '' -' ''' ''''' ' '''' -" ' ::'' :'::........... ......-:-'.. ........ ... .':.'.: '....... .......................

P ji i;,, ,,Si''''' '''' iS 1ii i' 'i,,f; -j -, j;,, iii ,,,, , .'. ...... ..... ............... 'it,h, ji£i,h, -j ......ii,,,,jjs.,i... .i ii j

attrbutdin miiany imanner toi t iheii Word Bank a iilts affilate jiorganzto mmeso 1;er of; ExAecuiv Direr ori th ei countriesth -r;e resenti;

The World BankLatin America and the Caribbean Regional Office

TABLE OF CONTENTS

I. INTRODUCTION ............................................... 1

. RECENT TRENDS ............................................... 4

III. EMERGING AREAS OF CONSENSUS ............................................... 6

IV. PUBLIC REPORTING ............................................... 6

V. AREAS OF CONTINUING DEBATE ................................................ 8

VI. CHALLENGES AHEAD: NEW FORMS OF EDUCATION DELIVERY ....10

VII. THE KEY CHALLENGES FOR QUALITY ASSURANCE .......................... 11

VIII. TOWARD QUALITY ASSURANCE FOR THETWENTY-FIRST CENTURY ............................................... 13

SELECTED BIBLIOGRAPHY ............................................... 16

L INTRODUCTION

Economic development is correlated with the development of higher education:enrollment ratios in higher education average over fifty percent for countries belonging tothe Organization for Economic Cooperation and Development (OECD), compared totwenty-one percent in middle income countries, and six percent in low income countries(World Bank, 1998a). The World Bank realizes the importance of investment in highereducation for economic growth and social development and its publication HigherEdwcafion: Yhe Lessons of Experience, focused its attention on the challenges and theconstraints facing higher education institutions around the world and recognized these as asymptom of a crisis. The report found that developing countries were particularly hard hitby the crisis in higher education. In fact, the fiscal constraints faced by many countries,coupled with increasing demand, has led to overcrowding, deteriorating infrastructure,lack of resources for non-salary expenditures, such as textbooks and laboratoryequipment, and a decline in the quality of teaching and research activities.

Based on a review of country experience, Lessons of Experience suggested fourkey directions for reform:

* Encouraging greater differentiation of institutions* Providing incentives for public institutions to diversify sources of funding* Redefining the role of government in higher education, and* Introducing policies explicitly designed to give priority to quality and equity

objectives

Since the release of Lessons of Experience, some exciting changes have takenplace in higher education. The use of technology has expanded access and spawned newways of teaching and learning that were only imagined just four years ago. But it has alsobrought with it new challenges and concerns for quality control. This paper addressessome of the challenges to quality assurance faced by higher education as we enter the 21stcentury.

ifU~~~~~~~~~~~~~~~~~~.......

0~~~~~~i~

:pp~~~~

~~~~~~~~~~r

~~~~~~~~~IAV ~ ~

rA~0 ~~~~~~~~

c~~~~~~~~>1

. .. . . ....[itlith g [9~~~~~~~~~~~.......

. .... .

3

Box 2. Romanian Higher Education: Institutional Management as a Means toQuality Assurance

A 1994 education sector efom tatey adopted by the Romanian Coveemment led to the 1995 EducationLaw. The- new Eduation -w :ieplaced centralized Ministry of Education E o systemovesgtI thoug iermedayconis, or buffr organ2ations'.i Thiiswaacivdbdeoin-itS = -r o rmfo~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~. ... ...professil and licy funcions firomn l:the MOE to four, semi-autonomous iNatkrnol Cowwi( on AccriedfaUo d'Acadm Eval ation, the National ouncil "on Ac e,.d De~reeg, the H Zher Edzrcn ,,finanOng.Counl, and, the 7iVerSityResearc Cou'c*

.~~~~~~~~~~~~. ........ .... .... .... .................................................. ...

The higher education system is now mate responsive to the needs of the emerging MArket eronom. Thishas been accomplirshed by hanging'the content of programs, readjusting .te sizeo.f: pmgorams and

i . m. .. e .. ibiit. N fielwds, swh as business and moder macroAnd nucro-economnes weIntr d w h fields su as central planing, were eliminated, .Overealtn d oer-.enrollmen- in certain technical and ing fi ave bn a iad neSaplinarY Programs ham bee nroue. Feiblt has: been increse truhthe. introdtinfshrprra,retni.an ..... co-tinuing. eco.: - -- - : - .::.~~~~~~~~~~~~~~~~~~~~~~~~... ...... ,; ..... .................... .. .

Inadiio,reiaitionof.academic programs, through the Accreditation Cowidil, s ensun higheruy standrds, especy in the newly developin gpublic and private univsities Quality of fculty, in

fast growing fields, bengurde throuh the dev I i elopment tgraduate -proms o t ain the nextgeneaion if a'caemicsta, 'whfl,eh:fie Niioual UniversitResearchC.Council isifunding the delopme-of newpostgate-trograMsh.

:A set(oTO atns rns to be taken tob srengtben the quality e mechanims alrad irouced.Accountability fur maintainng -stan will be- achieved throuh peri iewasprovided for ithe' Accreditation iLa. Standards d D re for piodicl quality e o i be.developed and applied. Incenties ;for quality improvements at both the undergraduate p dualevels ill be lntroduced throuh competjiive grnts for program innovatio and esach. Public

ed t ill b allocated to ike p for past neglect by increasilg resources to, devlopment,-nnations ?and capital investment to Cat least 20 percent of total public spending on higher education.

Post uate studies will be concentrated in selected institutions so -that resources can be focused ondeveop- n hg ? ? iW . -ty p--g:-.:

The-E;ducation Lw provides for the establishment of internal governance structures and efforts began into li aminisave and academic functions. Eventualy,

prae eonal ndelopm plns as p of thei part on the compeitie proThe manages of ittutions Wi ermine institutio 'priorities through internal vie gran' proposals snbniiue by the institutions.F measures'to be itrodudin te ne : eare thebuilding of professional management through separate- sastreams for admistras adtheeinof management inlbrmation systems to sot in titiio and sytem d recison making. -.

Soue'Y World Bank, S96a.,

There are wide differences among countries in their approaches to quality. Insome countries, governments have taken steps to strengthen quality by introducing newreporting requirements or other mechanisms of management control. Argentina, forexample, has introduced quality assurance mechanisms that depend on an enhancedinformation and evaluation system and new rules for funding the universities (Box 1).Many countries have developed accreditation systems, while others have establishedevaluation committees or centers that carry out cycles of external review. In manycountries, independent bodies have been established, often a single national agency butsometimes, as in the Netherlands, Mexico, or Romania (Box 2), separate agencies are

4

responsible for different types of institutions, regions, or purposes. Such variation inapproach reflects political and cultural preferences within each country, differences ingovernmental leadership, as well as varying stages of development for the highereducation sector.

The scope of responsibility given to quality assurance systems also has rangedwidely. Scotland and England, for example, have procedures to monitor teachingeffectiveness, while Hong Kong is focusing on high-quality management processes. Othersystems have been established to license new institutions or to certify educationcredentials. Still other efforts have been directed toward rewarding research productivity,either of individual scholars (as in Mexico) or of entire acadernic departments (as in theUnited Kingdom). There is wide variation too in the extent to which quality assuranceagencies have addressed issues related to student transfer and to study in other countries,as well as the issues related to the expansion of new modes of educational delivery,including video-based education, interactive transmission to remote sites or, most recently,Internet-based learning.

This paper reviews the current status of national policies for quality assurance. Itoffers a brief assessment of recent trends and identifies some areas of emerging consensusas well as issues still being debated. The paper also considers some of the unresolvedissues that are likely to shape policy debate over the next decade. It concludes with somesuggestions for ways that governments and international organizations might hasten thepace of change.

IIL RECENT TRENDS

Although circumstances vary among countries, several broad trends havecontributed to growing governmental interest in establishing policy mechanisms to ensurequality and accountability in higher education. Particularly significant has been the trendtoward mass higher education. Many countries have seen a doubling or tripling of post-secondary enrollments in the last few decades, along with increased participation rates foryoung people. In most middle income countries today, between 25 and 45 percent ofyoung people enroll for higher education. Greater diversity of educational offerings hasemerged in response to mass higher education; most countries have introduced differenttypes of institutions, different lengths of study programs, and varying modes ofinstructional delivery as ways to accommnodate rapid enrollment expansion. Otherresponses are seen in the growth of a private, entrepreneurial sector of higher educationand, in many countries, an expansion of educational offerings by distance learning.Further, as universities have become larger and more specialized, they have adopted morediverse roles, often seeking to serve needs of business and industry or to strengthen theircontributions to local, regional or national needs.

5

Box 3. Two Cases of Diversification in Higher Education: Brazil and Chile

Bra-il.bas ong traition of university education and a lage segwented systemu of lgher educaon.Pubhc hier -daccounts for- 28% of higher education enrollment, Is promnipl stae and fedeca gover whih operate, parallel, essentially autonomous systemsT

ng? e.Si? :, ?on hint ae emo it e ~ Scto.: lllt.11whh he bome he govenment's instrument for :expandig accs ar g:se s' nanc d gene-lly

. ,: ; :,: : ::: : :? ??: ,',1' :52 ,?>:'t ' .''.' ....................... ?. :: .. ; ..... '~~~~~~~~~~~~~~~~~~~~. . .. ...

The prncipalpolicyechaismfor toi the highr education systemi F Council aEducation ( whs chief fuinction Sis t approve the charters of newinti io c touis6s.ofS9t?i and set ition, ees,' ad enollmentlevels in both. public and p institutons. How ,:EC doe: -n ooss v sufficienttaffresorces to exercise theirp owers nordoe it have teaut fordire~ -rourcs to thstituionsthatareplacedunder i ei hat woud g r lait toits ao powers........... vate ;unuversige ivdirect supo fro th* government in the-orm ori: to studentsattending he inlStitution butonl ffiTh sI

loans. ?e.loan..chemeis adminiserd bytMinistryfEducai no by t wrdiig fucionsand thus could use this poolicy devit prove the low staodardsp f ny lival.. ',-?'' .'.. -. ..'...'- . -.- ........... :- .....- ........-

institutions'~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~.....

Chiles'hiher ducaion ystem Js. at a,crossrods between an elite! hihreducation: system, :adacI?. e d m h e ecation sysem. the govenmen lac a.cm}p.rehensive seriesof strwucht :and financial reforms. Tertiaryeducation was diversifiedandstratifed

intothre ties: Uiveritie, Prfessonal Instituties (P)and Techncal Trann eters iTC Fej: .- ti ? W;tIs, W em. ?0 I t .0 d T (.ech .tt.we.e n ed pu st . A loan sheie for stdets in public institutions Therocedures for S.anc publi universities were cnged t encourage income genero a

... :S.. ...... . : .savin and wardresearc fundin oacopttive basiis. Public university personneFl6lsaterreua

civil ser~~~~~~~~~~~ce status.c z,;orat

T fr ed at.......... deregulaing the sub-secor, have -d a profound impac on fi.its.size. o Zaiafui;anciX and management They hav?e led to.more than.doubling th-;e ermt increasng goss

enr:JlmenLfm lQ8% on? 198 9-to 28.2% in. 1995. TheyS alsoresulted iahs. e prolf at o i ns .ituio.rom eight universities to a 0total o 27t) institution,J including 200 7C's and Pr a f hih i

eios :y all f the :expansion bee Inanced by private sources. Fundi bi andprivte ititions,omes from student fes (33%),4 income from services (26%), and0 b bsidies

-(41N). - he Sme time. the Chilean, higher. eucation s y3-stemn: has becoe ..f...r......... .. Tre isinc-mplet. diversification4 ihWow enrohlent at the technical leel andiin sience andtc l-.dergraduate and graduate Pro transfer between the :diffieret

ilstitutona ll leves is lhiited, haperg tghe absprtiAoS of learners and adversely affcting equit as s ligwit the public and priva and egional developmen

The: Governent stratey to address ithese issues consists of developing a vi?sion for the highr eduticaio,syste ith jpuali e i iency as piority objectives. To this end the currentaccetation and

evaluation willbe e? nhanced d ded to over all higher education institutions... .. .. ... . . .... .. . .. .....................

-- i e Eis'=semon and H!m4fielser, 1995; World Bank, 1998b.

Governmental initiatives on issues of quality assurance represent another responseto a larger and more diverse higher education sector. Traditional, often informalprocedures for quality assurance, deemed suitable with only a few institutions andrelatively small enrollments, are now seen as insufficient for the more diversecircumstances of a larger system of higher education.

6

The shift to formal systems of quality assurance, evident across many countries andregions, is among the most significant trends affecting higher education over the last twodecades. Today, many countries have organizations or agencies responsible forconducting quality assurance reviews of academic institutions. Indicative of this shift isthe emergence of international networks, journals, conferences and newsletters thataddress issues related to quality assurance.

In international settings and within individual countries, debate continues over theproper role and form of quality assurance for higher education. The issues have evolvedover the years, however, from an initial questioning of whether new forms of qualityassurance were needed to current debates on what are the more effective approaches toquality assurance. Many academic leaders criticized early approaches and defendedacademe's traditional methods for quality assurance even though they were largely internaland not transparent to external audiences. More recently, academics seem to haveconceded that the pressures of mass higher education and financial constraints havechanged the conditions of higher education sufficiently that formal, externally validatedmethods of quality control must be a central component of higher education systems.Recent debate in England reflects this shift. While approaches to quality assurance are stillwidely discussed and analyzed -- by special commissions and in published articles,conference papers, evaluation studies and the media -- much of the debate focuses onwhether one approach is better than another, whether a certain procedure should besimplified, or whether a procedure unfairly advantages some institutions over others.

IIL EMERGING AREAS OF CONSENSUS

Although the policy debate continues, some elements of consensus have emerged,especially around the core elements of an approach to quality assurance that is appropriatefor higher education. Box 4 lists some of these consensus elements of quality assurance.In part, this convergence reflects wide-scale cultural "borrowing" among countries. Inpart too, quality assurance structures share common features because they representmodifications of traditional academic review processes.

Box 4. Core Elements of Quality Assurance

*+ :E,q,iit xstandardsan expectationlsit:2402tiSfEtt40t:2t:::rS ::-t: :* Self-study theAcade micintiuton orunt

* Attention .o both procs (.e., cap:aiyand :

IV. PUBLIC REPORTING

Key elements of today's models, having emerged from processes of negotiationbetween policy leaders and academic leaders, have political support from both sides. The

7

academic side considers them to be appropriate, not overly intrusive, while policy officialsconsider them to be satisfactory and useful. As can be noted, the typical approach is toallow the initiative for review to rest first with the academic institution or unit beingreviewed, through the self-study.

Box 5. Indonesia Case Study

la 1937 th, World Bank supported the introduction of accreditation mechani,s, in a ptojec 1tolimprovetaher-a tandards ipic institi. A pilot program was established to delo a c e to

~er4itteche-trinng:programs In seeced institutions aftlr theovrendcidtop-ae lpiF-seMce teacher-training nsStitutions to university status.' Tei djectiv was :to agreeuon a setof:sanda :iw llteachetraning institutions could be evaluated as well' as establish a baseline forinstitutioiial 4lopent s i were, identified rough a competitive mech'nim. Splanning grants were made available:to each institution to enable tbem to do, a sel,-s'udy',- which'w' asexteay aluaeda validiatedby,professionals:and education praioners. -The, pilot study was? mse?ul in genering acceptan_e Sor a tion as a mechanism to hnprove ticl a r' ? .

ine Ae, origal c, tree higher education rjects, inmvol ng on eeie funng aareit;¢ati have: been suessfuly, implemented in. Indonesia: University Resear. h for G'raduate'Eduaion `(URGE);, Development f ?Undergraduate Education (DUE) and: Qualty of 'UndegraduateEduca,ion .QE.All three roMects focus o mp ing ityand i tr co ti e eelopment grants. Instititions write development proposals based oth. results fa self-vaution hch is prepared according to explic:standards: nd .ex tions. e plans are:

'" n? -,w,??d'-r o:'erm -'bir:tion to: :ad''acmr~vie~d b PaW pae fdnsan: and'international etxpet o teto opoesa otoeSnccesstl institutions- rceive: deelopinent grants to implement their Vpropsals. Progress is monitored

on a regularbasis and continued tbnding of«the? pla isbased on Su lie ent t

.J Eisemo 1992. World Bank, 1997, 4996b and 1994b.

Academic perspectives also predorminate in the external review, particularly if thevisiting team is mainly comprised of academnics. Tilting toward governmental interests,however, is the development of explicit standards, the requirement for public reportingand the attention that is paid to actual results, or accomplishments. Overall agreement onthese core elements still leaves much room for variation, certainly, and importantdifferences can be seen when specific settings are compared

Some agreement is emerging about the general effects of quality assurance, at leastin systems that have had extensive experience and that have undergone systematicevaluations. Several studies have documented, for example, that quality assurancesystems have caused academic institutions to give greater attention to issues of effectiveteaching and learning. Degree completion rates have improved in some systems, asstudent services and advising have received more attention. Quality assurance systemsthat focus on institutions, as in France, have reported that institutional management hasimproved, that strategic planning has been strengthened and that programs have becomemore responsive to changing needs. In Indonesia (Box 5), an emerging process intendedto encourage internal program review and revitalization has had considerable success.Some analysts see these changes as evidence that, among university leaders, there isgreater acceptance today of the need for having open and accountable managementprocedures. Drawbacks have also been identified, including concern that quality assurancesystems have led to "compliance" behavior and inordinate paperwork burdens. Quality

8

assurance systems also impose considerable administrative and financial burdens ongovernments that, in many countries, can be difficult to sustain over an extended periodand as political leadership changes.

V. AREAS OF CONTINUING DEBATE

Significant differences still exist on many policy and implementation issuesregarding quality assurance. The same issues tend to appear in the debates occurringacross many countries, while other issues, equally as important, receive much lessattention. Generally speaking, issues of policy and practice have received more attentionthan some of the educational or learning issues that underline approaches to qualityassurance.

Box 6. The British Open University

The IXlOpen University (0 ,createdin 1 , iS a separate, autonomous', instiutionmoffeng u tyleve. eucao topr-tim Sadultg learners. Itis the laresgtunversity in the nied'igdom,

n fo appro ly seven perent o higherieducation enrollmen and:20 pret of par-timeS eroThuents.- -Thej Open..UniversityEis sthe ysame, iim nany ilmporJtant ways,. aEany tfradi.lionaf

.0,, ... ̂ t tS, '. -. 000 .-0D,, :: Wj ,g, g ', .. .E't' -0'S - -,:-.' tl00!00t'E'# ! ... . . . .' .' .......44000-

|g...g.-....>i j....0.A S -j z.4;j | t le W :9::iX-o dgt4td.- t.. .........* . It iaperats withn Te sWame. relator environmentr and wlha fthe -same satu fas otherulniversitis

Master and Doctoral levelaYSchemesii ii E< to &ifltatetasfer to iother institutions and gstudentsi may. obtan ceadit--for:prior

ii Tu-i.i iti on i 'fiees are chargedin taccordance 4to lcoure ofsuyad t limitdfinciaglasistanc isf'

Wh:ere the CU difrs drinaticty; from other higher education institutions in the UK is i its stuentprofile. andin ts thod of courdelivery. The tpical OU:studeut is much more likely t be enrolledpan-time and tote older then student at otieruuniversities.C Consient wi it policy oop lerinthemx tre no aadeic ntracerequiremients for study at the OW AU (0-series ofmdules i0s offered t& assis.potentia students in ass,es,singthir capactyforhier level,inepndentstdy. -

iE)i iji-... . ij....iLE.--i.E -.-i, -ELE-t .H ... i iiS.iEE . ERji.i ij::::.i...EE g;:tE... .. .......... i- ti t E

The hasioneered the use.offdifirent technologiesito meetThe needs and schedules 6o;f ts. suetCourses use a combination of materials for independent stu4d-p radio andlisiobradcsts,.ct .bnased learmninand shod residental courses heldaover thesunimer. .

Subject to wide debate, for example, is whether the entire academic institutionshould be reviewed or whether, instead, individual academic programs or disciplinesshould be the focus of quality review. Countries in Western Europe, despite a generalconsensus among them on other elements of quality assurance, vary widely on whetherprograms or institutions are the focus of review. Denmark, the Netherlands, and Portugalhave focused on reviews of academic programs. France, in contrast, began withinstitutional evaluations but now conducts both types of review. Some universities inGermany have adopted institutional audits or other institution-wide reviews. In the United

9

Kingdom, both institutional and program reviews have been conducted, although underthe sponsorship of different bodies while, in Ireland, one agency conducts both types ofreview. Outside of Europe, many countries have begun with an approach to qualityassurance issues based on an institutional review but, as their systems experienced growthin professional fields of study, the pressures have been strong to conduct quality assurancereviews on a program-by-program basis.

Much debate also continues on the appropriate use of quantitative information inmonitoring the operations and accomplishments of higher education institutions. Publicofficials in most countries have advocated the development of a few relevant performanceindicators, with comparisons among institutions and over time. England took a furtherstep by linking the amount of research funding to performance scores of academicdepartments. Whatever the specific approach, academics have resisted the move towardperformance indicators, arguing that such indicators are reductionist, offer inaccuratecomparisons, and are unduly burdensome. In the face of such opposition, somegovernments have adopted an incremental stance, requiring that universities provide anincreasing amount of quantitative information to quality assurance bodies. Others haveintroduced perfornance indicators, not in quality assurance systems, but as part of theirinstitutional contracts or in other forms of conditional funding. Thus far, only a fewcountries link performance informnation to governmental decisions on levels of funding oreligibility for funds. In Romania, a set of coordinated improvement efforts includes bothaccreditation reviews and the establishment of a competitive fund to support innovation(Box 2).

These issues, currently subject to discussion in both policy and academic circles,are only a few of the many complex issues that could be considered in developing a qualityassurance system in higher education. A comprehensive approach needs to reflectthoughtful decisions on a range of questions, including: what to inspect and how; whodoes the inspecting and who administers the process to ensure it is conducted fairly; whatis reported following the inspection and what are the consequences of the report.Consensus seems to have been achieved mainly on the division of labor in qualityassurance, especially the respective roles of government and of academic voices. Thedebate over whether to emphasize programs or institutions involves some decisions onwhat should be inspected, but many design questions are still unresolved. Even lessagreement appears with respect to what actions should follow from inspection results and,especially whether bad results should have financial consequences. Some want to rewardgood results, possibly through supplemental funding or incentive systems. Others want topunish bad results, for example, by withholding funds or not allowing a program to enrollnew students. Still others want to shape results so that they lead to voluntaryimprovements.

Despite the many unsettled issues, most countries can be expected to continue tosponsor quality assurance systems for years to come. Differences across countries are alsolikely to remain in specific approaches to quality assurance, which are responsive todifferent conditions and felt needs within each country. Nevertheless, the next decade islikely to see some movement toward consolidation and greater coherence of procedures in

10

quality assurance systems. These systems will be under continuing scrutiny in mostcountries and, therefore, are likely to modify their procedures as various issues gainattention. Pressures for greater efficiency can be expected, in light of the administrativeburden created by many current systems. Further, as their results have greater publicimpact, quality assurance systems will face greater demands for consistent treatment of alltypes of institutions and for greater transparency in their procedures and decisions. Infight of these pressures, the current period of experimentation - with ideas and approachesbeing tried and debated - may be superseded by a trend toward stable structures andsettled routines for many quality assurance systems.

VL CHALLENGES AHEAD: NEW FORMS OF EDUCATION DELIVERY

Quality assurance systems will find continuing challenges in the decade ahead, notonly on their procedural decisions but also on the fundamental educational issues theymust address. Complex questions about how to measure educational quality are gainingnew urgency because of two recent developments: the widening use of educationaltechnology and the burgeoning interest in global delivery of educational services. Therehas been some effort to address each of these emerging trends, especially in some majorsystems of quality assurance, but much remains to be addressed.

Some quality assurance agencies have begun to adapt their review procedures toaddress a growing variety of distance education and other electronic learningopportunities, including television-based courses, and instruction over the Internet. In theUnited States, for example, accrediting agencies and other organizations have developedstatements on ways to evaluate distance learning. However, most statements offer alimited response, mostly following procedures that were developed for conventionallearning settings, rather than developing new assessment processes to fit these new modesof education. Such adaptations may be appropriate when electronic learning is used by afew institutions, involves small numbers of students or, as in Australia, where electronicdelivery is undertaken primarily by traditional institutions with long experience in distanceleamning. Limited oversight mechanisms also can be suitable when distance learning is theresponsibility of one public organization, as in India or the United Kingdom. However,there is growing awareness that the review procedures developed for conventionallearning settings are not sufficient for electronic delivery methods, especially as these newdelivery vehicles grow in importance and reach. Yet, little clarity has emerged about howelectronic delivery can be better evaluated.

Because of the rapid development of electronic media and delivery infrastructureover the last decade, distance learning has now reached a stage where substantial growthcan be expected. Prospects for expansion are especially great in view of the burgeoningenrollments forecast for many countries and the limited prospects they have to invest inphysical facilities to match rising demand. Over the next decade, an increased number ofinstitutions can be expected to offer distance learning, and with much higher numbers ofstudents enrolled. This growth poses serious implications for quality assurance agencies: agreater number of providers of distance learning must be monitored, in a greater variety of

11

learning sites and modes; and an increasing number of students will be seeking recognitionfor distance-based courses to fit a greater diversity of study objectives.

Similar issues arise in considering the implications for quality assurance of thegrowing trend of international student mobility. Whenever students enroll in othercountries as part of a degree program, their study plan must be evaluated for how it can becompared to the academic work they might have completed in their home country orinstitution. In considering how to monitor such learning, most quality assurance systemsfollow a decision strategy broadly similar to their stance on distance learning, i.e.,assuming that small adaptations of conventional procedures can accommodate thisvariation in study pattern. In Europe, several universities cooperated on a multi-yearEuropean Commission project to develop guidelines for student transfer and creditrecognition, but the project resulted in the type of inter-institutional agreements that haslong been used with cross-national study. In the United States, accrediting agenciesgenerally rely on each university to extend its own internal procedures to ensure that studyabroad is compatible with its regular program. These approaches may be more suitable toshort-term study than to an extensive period of study abroad, however.

Efforts to evaluate and recognize extended periods of academic study acrosscountry borders have been problematic. Many countries refuse to recognize academicstudy outside their borders. Most of the bi-lateral agreements that do exist are foundamong the few countries with large enrollments of foreign students and are not easilyextended to other countries. U.S. procedures, for example, are based primarily ondetailed analyses that compare U.S. programs and degrees and those of another country,looking to details of course and curricular structure, examination systems, etc. Heateddisputes have arisen between countries with respect to whether degrees in one country canbe recognized as employment credentials in another country. Similar disagreements havearisen regarding what level of prior work will be recognized when students wish to take afurther degree in another country.

Recognizing the flaws in today's arrangements, one can expect increased pressuresfor better solutions in the future as more students seek to extend their studies in othercountries. Important forces of globalization - symbolized by multi-country compacts inEurope, in North America, and in South Asia - already portend a much greater mobility ofstudents and scholars in the next century. Thus, difficult as the issues are, qualityassurance agencies must address the assessment and monitoring issues that will arise asincreasing numbers of students seek to complete an entire degree program, or a specificcomponent of a program, in another country.

VIIL THE KEY CHALLENGES FOR QUALITY ASSURANCE

The challenges to quality assurance raised by electronic learning and byinternational student mobility are closely related. Both force attention to questions of howlearning can be assessed and monitored when it takes place apart from the instructionalprocess or away from the sponsoring institution. With electronic delivery methods,learning need not occur at the same time or place as instruction. How is this learning

12

monitored when the instructor does not have face-to-face contact with a student, and howwell can learning be assessed by remote methods that are not buttressed by direct contact?With international study, learning occurs at a different time and place and the sponsoringinstitution has limited control over the circumstances in which it takes place. How is thislearning monitored and how can the coherence of a program be assessed across differentinstructional settings, including cultural and linguistic differences or differences in howeducational programs are structured?

In both circumstances, the critical evaluative task for quality assurance agencies isto focus on what remains central, i.e., what the student learns. Educators, as well asquality assurance agencies, must look to the actual results, or outcomes of an instructionalprocess. Furthermore, as many quality assurance agencies have recognized, these sarnechallenges also apply to study in traditional settings, especially when students enroll part-time, change field of study, interrupt their studies or transfer to a new institution. Whethera unit of learning takes place in a workplace, in another country or university, or over theInternet, the instructor or assessor may not have the full range of knowledge about thestudent that is possible in a traditional classroom situation.

Yet, it is also true that giving attention to learning and outcomes raises new andcomplex issues likely to concern quality assurance agencies for the next decade or more.They will need to work with higher education institutions to develop effiective ways toassess learning accomplishment, not time spent in a certain place or with a certainprogram. They also will need to develop inspection methods appropriate to a new focuson learning, regardless of its setting or provider.

This sets out an ambitious agenda, but it is also an area in which educationscholars have already conducted much sophisticated research. In a number of countries,potentially useful approaches and models exist, including examinations and other methodsthat test students when they complete a degree program as well as modularized methodsthat assess specific units of learning. Some developmental work also has been conductedto develop assessment and inspection methods for different learning circumstances.Procedures to allow credit transfer or to give recognition to degrees earned in othercountries also offer useful precedents for further work to equate electronic learning orstudy abroad with traditional units of acadernic learning.

With new attention to learning, it is also necessary for quality assurance systems tore-examine their conventional expectations for monitoring characteristics of highereducation institutions. In traditional terms, many attributes of a university wereconsidered when its education was assessed: its library and classroom quality, the strengthof its staff credentials, or the reputation it had for providing good education. Electronicmethods of instructional delivery call into question whether and how such institutionalcharacteristics matter. So too, international study challenges the relevance of physicalsetting and raises questions about the purpose of inspection visits to an institution.

Thus, a second key challenge stemming from electronic learning and studentmobility is for quality assurance agencies to clarify their assumptions and to have

13

appropriate reasons for looking to an institution's capacity to offer a good educationalprogram. In the past, quality assurance agencies may have confused some traditionalthinldng about institutional capacity and reputation with a more defensible focus oneducational capacity and effectiveness, i.e., whether an institution or other provider candemonstrate the resources and the ability to offer an entire program and its related servicesat an expected level of quality.

Such clarification is also called for by the growing variety of "providers" of highereducation. In some countries, private higher education institutions can be organized on aprofit-making basis, may offer primarily short-cycle courses, and may offer instruction inmultiple sites and without permanent staff. Such organizations may offer electronicdelivery of instruction and, too, could sponsor study abroad programs using distanceeducation modes of delivery. For such new types of higher education, quality assuranceagencies face the same necessity to clarify what issues relate to the capacity to offerprograms of sufficient educational quality.

VIIL TOWARD QUALITY ASSURANCE FOR THE TWENTY-FIRSTCENTURY

Differences by country and region can be expected on the pace of transitiontoward a quality assurance paradigm based on learning outcomes, regardless of setting.Some countries can rely on a substantial base of scholarly research on learning as well ason extensive project experiences of individual universities or national organizations.Others have little direct experience with outcomes assessment. Some governments havealready taken steps to shift their systems of quality assurance toward learning, most oftenthrough the use of performance measures but in several recent instances through calls forattention to standards of learning (e.g., in England) or for the development of overarchingqualifications frameworks for higher education (e.g., in New Zealand and South Africa).

Developing a system of quality assurance based on learning is a major task forevery country. Decisions are needed on the learning objectives to be assessed, as well asthe evidence that would demonstrate its accomplishment; methods must be developed forapplying evaluation techniques to actual learning situations. With a focus on learning,rather than on the setting in which learning takes place, all of these new decisions andmethods must be made applicable to a wide variety of learning circumstances, includingdifferent fields of study, different degree levels and stages of learning accomplishment.

To be successful, such an effort requires collective action by universities and bygovernmental agencies, along with scholars in educational research. Cooperative action isneeded both within and across many countries. Fortunately, several precedents exist forsuch cooperative approaches to quality assurance and might offer a basis for building stillfurther cooperation. In Europe, for example, a pilot project was supported by theEuropean Commission between 1991 and 1995 to explore aspects of joint cooperation onquality assurance; a Europe-wide quality assurance network has been formed to follow upon the pilot project's work and to take further steps. In the United States, the separateregional commissions that accredit higher education have recently given their joint

14

endorsement to a set of guidelines for the evaluation of distance learning, guidelines thatearlier had been developed by a consortium of states in the western part of the UnitedStates.

Such national and multinational steps are promising and, if sustained and extended,could offer a framework for developmental action in other countries as well. Individualgovernments and agencies could contribute to this larger effort by sponsoring conferencesor other efforts to learn from the developmental work underway in several countries. Amechanism to encourage this developmental process also exists in a recently formedinternational network of quality assurance agencies, which sponsors an annual conferenceto discuss mutual issues and to share progress on developmental efforts such as those justmentioned. This network, or a similar structure, could facilitate discussion of howprogress in one country might translate across national borders.

Concerted international action is needed, however, to hasten what will otherwisebe a slow, long-term shift by quality assurance agencies from a focus on settings to a focuson learning. Because the problems posed by electronic delivery and by the increasing paceof international student mobility will affect all countries, an uneven pattern of change --with some countries adopting new practices while others do not have access to these newpractices - will undercut the overall objective on ensuring that learning, in any place, canbe assessed appropriately.

Wide-scale international cooperation is needed to spur the process of developingnew alternatives and building inter-country agreements for using new ways to assess andvalidate learning across settings. Strong multinational organizations should assumeleadership roles to further the development of truly international standards for assessingquality in higher education. The OECD might be asked to take on such a role, based onits respect and legitimacy and on its expertise in conducting country reviews of nationaleducation policies and, recently, in completing a twelve-nation review of higher education.A project sponsored by the institutional management program of the OECD offers a goodprecedent: this IMBE program helped to develop and test a quality review processappropriate for reviewing the quality of internationalization initiatives within universities.Also underway is another relevant IMHE project which makes cross-national comparisonsof the impact of quality assurance on institutions of higher education.

Other international and non-governmental organizations also could be called on tohelp develop new models of quality assurance. A network of quality assurance agenciesthat has recently formed in Europe could offer assistance, possibly by identifying certaintasks where it could undertake developmental projects. The international network ofquality assurance agencies could also be a resource for identifying special expertise forcertain tasks.

Before any substantial amount of developmental work can move forward, aseparate effort is needed to develop greater clarity and consensus on what types of newstructures might be appropriate for assessing learning, regardless of setting. Aninternational approach is needed for this task, too, and needs to be organized by a

15

multinational body. Various options need to be debated for what the broad developmentalobjective should be; areas of agreement and cooperative research need to be identified.The options to be considered might range widely, including:

* Harmonization of degree levels across countries, for example, to organizelevels of learning to more closely conform with the ISCED levels developed byUNESCO;

* Identification of general competencies or skills that should characterize holdersof a degree, regardless of where or how they earned the degree;

* International standards for student achievement, specified for each professionand discipline;

* International leaving examinations, comparable to the internationalbaccalaureate that has been established at the high-school-leaving stage andthat becomes more feasible worldwide as computer-assisted examinationtechniques are developed;

* Assessments of institutional capacity based on international standards andreview teams, organized separately for research universities, technicalinstitutes, etc.; or

* Internationally recognized methods for assessing educational capacity.

Many other options are possible, particularly oh the assumption that widening useof Internet resources will facilitate communication among universities and agenciesthroughout the world. Consistent performance indicators might be developed acrossseveral countries, for example, and their development, experimentation and eventual usecould all be carried out and made available through the Internet. The Internet also lendsitself to the development of a common approach to student transcripts and records, or toan internationally available resource that offers descriptive detail on study programs andcourses offered at universities anywhere in the world.

International agencies should consider ways in which they could contribute to thedevelopment of these or other new models for quality assurance. The issues are global,and will be increasingly shared across countries. Solutions are needed that are consistentthroughout the world of higher education.

16

SELECTED BIBLIOGRAPHY

Brennan, J. (1997). Quality Assurance in Higher Education: An Overview. Paperpresented at the OECD/UNAM International Dissemination Conference on QualityAssessment in Higher Education, Mexico City, 1-2 December.

Curran, C. (1997). ODL and Traditional Universities: Dichotomy or Convergence?European Journal of Education, 32, no 4 (December), pp. 335-344.

Deupree, J. and Lenn, M. P. (eds.) (1997). Ambassadors of U.S. Higher Education:Quality Credit-BearingProgramsAbroad. New York: College Board.

Ehrmann, S. C.(1996). Adult Learning in aNew Technological Era. Paris: OECD.

Eisemon, T. (1992). "Language Issues in Scientific Training and Research in DevelopingCountries." World Bank, Population and Human resources Department, Educationand Employment Division Background Paper Series PHREE/92/47. Washington,D.C.

Eisemon, T. and Holm-Nielsen, L. (1995). "Reforming Higher Education Systems: SomeLessons to Guide Policy Implementation." Education and Social PolicyDepartment, ESP Discussion Paper Series 60. Washington, D.C.

El-Khawas, E. and Shah, T. (1998). Internal Review to Assure Quality: ComparativePerspectives on Evolving Practice, Tertiary Education and Management, 4, no. 3.

El-Khawas, E (1998). Accreditation's Role in Quality Assurance in the United States,Higher Education Management, 10, no. 2.

European Commission (1995). The European Report: European Pilot Project forEvaluating Quality in Higher Education.

Fames, N. (1997). New Structures to Reform Higher Education in Central and EasternEurope: The Role of Distance Education. European Journal of Education, 32,no 4 (December), pp. 379-396.

Gibbons, M. (1998). Higher Education Relevance in the 21st Century. Paper presented atThe World Bank Human Development Week '98, Alexandria, Virginia, 4 - 6March.

Hoffert, M. (1997). Quality Management, Quality Assessment and the Decision-MakingProcess. Paper presented at the OECD/UNAM International DisseminationConference on Quality Assessment in Higher Education, Mexico City, 1-2December.

17

IMAE (1997). Internationalisation: The Quality Issue. In: lMHE Info, Summer 1997, p.1. Paris: Program on Institutional Management in Higher Education, OECD.

Massaro, V. (1997). Institutional Responses to Quality Assurance: A Separation ofPowers? Paper presented at the OECD/UNAM Intemational DisseminationConference on Quality Assessment in Higher Education, Mexico City, 1-2December.

Massaro, V. (1996). Quality Measurement in Australia: An Assessment of the HolisticApproach. Higher Education Management, vol. 7, no. 1, pp.

Schofield, A. (ed.) (1998). Benchmarking in Higher Education: an international review,Commonwealth Higher Education Management Service.

Shah, T. (1997). Quality Management, Quality Assessment and the Decision-MakingProcess. In: Brennan, J., deVries, P. and Williams, R (eds.) Standards andQuality in Higher Education. London: Jessica Kingsley.

Staropoli, A. (1997). A French Approach to Evaluation. Paper presented at theOECD/UNAM International Dissemination Conference on Quality Assessment inHigher Education, Mexico City, 1-2 December..

Thune, C. (1997). The European Systems of Quality Assurance. Paper presented at theOECD/UNAM International Dissemination Conference on Quality Assessment inHigher Education, Mexico City, 1-2 December.

Thune, C. (1996). The Alliance of Accountability and Improvement: the DanishExperience. Quality in Higher Education, vol. 2, no. 1, 1996, pp.

van Vught, F. and Westerheijden, D. F. (1996). Institutional Evaluation and Managementfor Quality: The CRE Program: background, goals, and procedures. CRE-action,no. 107, pp. 9-40.

Wahlen, S. (1997). Is There a Scandinavian Model of Evaluation? Paper presented at theOECDJUNAM International Dissemination Conference on Quality Assessment inHigher Education, Mexico City, 1-2 December.

Westerheijden, D. F., Brennan, J., and Maassen, P. A. M. (eds.) (1994). ChangingContexts of Quality Assessment: Recent Trends in West European HigherEducation. Utrecht: Lemma.

World Bank. (1998a). EdStats Data.

World Bank. (1998b). Chile: Higher Education Project. Project Pre-Appraisal Document.

18

World Bank. (1997). Indonesia - Quality of Undergraduate Education Project. ProjectAppraisal Document, report no. PIC1883.

World Bank (1996a). Romania - Reform of Higher Education and Research Project. StaffAppraisal Report, report no. 15525.

World Bank. (1996b) Indonesia Higher Education Support: Development ofUndergraduate Education Project. Project Appraisal Document, report no. P6866.

World Bank (1995). Argentina Higher Education Reform Project. Staff Appraisal Report,report no. 13935-AR.

World Bank. (1994a). Higher Education: 7he Lessons of Experience. Development inPractice Series. World Bank: Washington, D.C.

World Bank. (1994b). University Research for Graduate Education. Sector AppraisalReport, report no. P6298.

Latin America and the Caribbean RegionDepartment of Human Development (LCSHD)The World Bank1818 H Street, N.W.Washington, D.C. 20433

Fax: 202-522-0050E-mail: laced(worldbank.org