researchdegree examining–common principlesand...

Research degreeexamining – commonprinciples anddivergent practices

Stuart Powell andClaire McCauley

Overview

Over the past 18 months the authors have

co-ordinated a number of national seminars

and workshops on research degree examining

involving active researchers and research

supervisors/examiners from across the UK

and from a range of institutions and academic

disciplines. Focussed discussions at these

national seminars have indicated that

institutional practices concerning research

degree examination vary significantly across

the sector. It was also noted that some issues

relating to this variation may be largely

outside the remit of the current QAA (1999)

Code of Practice for the Assurance of Academic

Quality and Standards in Higher Education:

Postgraduate Research Programmes. It also

became clear that the issue of thresholds may

be contentious when applied to research

degree examining, and indeed, that current

notions of assessment may be inadequate

when applied to practice with regard to the

examination of theses. The notion of the vivawas particularly problematic in discussions. It

was noted that in many European countries

the viva was a matter of public defence and

part of a ritual ’’welcoming into the Guild’’ –

in the UK it tended to have a range of

functions, some overt but others remaining

hidden. One area of consensus was that there

is a need to focus on the function of research

degree examination as a form of assessment;

for example, it was suggested that other

aspects of the process such as ’’ rite of

passage’’ sometimes obscure this function.For the purposes of this paper, issues

discussed are grouped under twoheadings:(1) research degree examination as

assessment; and(2) examiners and examining panels.

In each case the original issues as set out in

the briefing paper that preceded the seminars

and workshops are noted – these issues are

followed by the key questions that were

presented to participants, and finally the main

points of the subsequent discussion are

summarised.

The authors

Stuart Powell is Director of Research Degrees, University

of Hertfordshire, Hatfield, UK.

Claire McCauley is Administrator, UK Council for

Graduate Education, Dudley, UK

Keywords

Qualifications, Research, Assessment

Abstract

Reports on discussions that took place at a series of

specialist seminars and workshops on research degree

examining organised by the UK Council for Graduate

Education during 2000/2001 at various venues in the UK.

Debates the processes and procedures of research degree

examination in the UK in terms of variations in practice

that exist along with principles that signal a common

identity. Takes account of the effects of developments in,

for example, professional doctorates and the PhD by

published work and on perceptions of the `̀ traditional’’

examination. Issues addressed include: the composition of

PhD examining panels and the roles of individual

examiners; the training and qualification of examiners;

the purpose and nature of the oral examination; the

tension between examination of the process of training

and that of the `̀ finished product’’ (which the thesis may

be seen to represent) . Argues for the need for more

transparency about examination processes, for challenge

to common assumptions and for a refocusing on research

degree examination as a process of assessment.

Electronic access

The research register for this journal is available at

http://www.emeraldinsight.com/researchregisters

The current issue and full text archive of this journal is

available at

http://www.emeraldinsight.com/0968-4883.htm

The authors wish to acknowledge the invaluablecontributions made to this paper by Howard Greenand Malcolm Shaw.

104

Quality Assurance in Education

Volume 12 . Number 2 . 2002 . pp. 104±115

# MCB UP Limited . ISSN 0968-4883

DOI 10.1108/09684880210423591

http://www.emeraldinsight.com/researchregisters

http://www.emeraldinsight.com/0968-4883.htm

Research degree examination asassessment

Setting out the issuesThere are many commonly expressed notionsabout research degree examination and it isfrequently seen in a different light to otheraspects of examining in higher education. Ithas developed its own custom and practiceand mystique with the viva (or oralexamination) taking on a significance,particularly for the candidate, which hasmoved it some way from being merely anassessment of the achievement of researchoutcomes, if it ever was!

We might usefully reflect on what we meanby assessment and ask whether the PhDexamination stands scrutiny as a form ofassessment. The QAA’s (2000a, p. 3)definition of assessment will suffice tochallenge us in our approach to PhDexamination:

Assessment is a generic term for a set ofprocesses that measure the outcomes ofstudents’ learning, in terms of knowledgeacquired, understanding developed, and skillsgained. It serves many purposes. Assessmentprovides the means by which students aregraded, passed or fail. It provides the basis fordecisions on whether a student is ready toproceed, to qualify for an award or todemonstrate competence to practise. It enablesstudents to obtain feedback on their learning andhelps them improve their performance. Itenables staff to evaluate the effectiveness of theirteaching.

Additionally, we may wish to question theelements of the assessment, as most, if not all,documents and artefacts submitted forassessment will contain the words ’’ submittedin partial fulfilment of the requirements . . . ’’ Ifthe document or artefact is part of thesubmission for examination, what is the otherpart? What is the relationship between the twoor possibly more parts, in terms of weighting?Does the candidate have to pass both parts inorder to be awarded the degree? Mostimportantly, where are any of these questionsanswered in institutional regulations?

The role of oral examination/vivaAn inspection of their regulations shows thatinstitutions vary in their stance with regard tothe obligatory nature of the viva, with someinsisting on it being held regardless ofperceived qualities in the thesis and othersoffering various kinds of dispensation, e.g.

’’where no useful purpose would be served’’ .

In one sense at least, attitude to the necessityfor an oral defence relates to the underlyingperception of whether the examination is ofthe thesis or the process of research training.A product might be examined in the absenceof its author, whereas to judge the outcomesof a process of training requires somequestioning of the individual who has beentrained.

There is wide variation in practice overwhether or not a supervisor should be presentat the oral examination and their role if theyare allowed/encouraged to be present. Insome institutions where supervisors may bepresent, they are only invited at the discretionof the examiners, in others only at thediscretion of the candidate, and yet in othersthey may be required to be available onrequest but normally not be expected toattend. In some institutions attendancerequirements or restrictions relate only to theprincipal, or first, supervisor. Contribution ofsupervisors during the examination variesbetween being expected to speak and beingallowed to speak, but only at the request/discretion of the examiners. Commonly, thesupervisor’ s unspoken role is that of acting as’’candidate’ s friend’’ .

Few institutions make the viva a publicevent, though some departments claimopenness within certain categories of staff.The merits and demerits of public defence arerarely discussed within institutions. Practicein other countries differs greatly from that inthe UK. It may be argued that practice in theUK needs to be debated in the context ofpotentially increasing European convergence.Clearly, differences in current UK practicerelate largely to differences in the kind andquantity of research training and researchactivity seen as necessary for the award ofPhD, and subsequently to differences in thebases for assessment. This area is explored inmore detail in Powell (1999).

There are now possibilities for making theviva a more flexible event by means of videoconferencing. There may, however, be issuesarising from the use of technology in this waywithin the process of assessment, for examplewhere confidentiality may be threatened.

Generally there is lack of clarity about thepurpose of the viva. For example, whether theviva is intended to be an examination in thebroadest sense of the term or merely a

105

Research degree examining ± common principles and divergent practices

Stuart Powell and Claire McCauley


Volume 12 . Number 2 . 2002 . 104±115

verification of authenticity. Indeed, in reality,many vivas seem to become effectivelyopportunities to fine-tune the written work ofthe candidate in order that it reaches anotional standard acceptable for scrutiny bythe wider intellectual community.

Assessment criteriaThere also appears to be little consistency interms of the level or kind of guidance that isgiven to examiners on criteria for assessingsubmissions. Many institutions cite the needfor criteria such as originality andpublishability, but detailed checklists ofcriteria for assessing theses are not inevidence. This area is explored in some detailin Shaw and Green (1996). This kind ofvariation reflects the underlying lack of clarityabout threshold performances that mightdefine differences between degrees at Masters(e.g. MRes), MPhil and PhD levels. Also,there is lack of clarity concerning anyweighting that might be given to differentcomponents within a thesis and to differentaspects of the examination. For example,should critical self-appraisal be allowed tocompensate for poor research design?

Assessment of process or product?Differing examiners’ views indicate a tensionin the PhD examination over the extent towhich it is the process of research training orthe finished product that should be examined.Different views in this respect will lead todifferences in notions of the way in which theviva ought to proceed, and indeed, thebreadth of the remit given to examiners. Forexample, should examiners be:

able to ’’ explore records of researchsupervision’’ (CVCP, 1993) or otherdocuments such as progression ortransfer reports;asked to comment on quality ofsupervision, training opportunities andfacilities provided by departments – seenin the BPS document, UCoSDA/BPS(1995), as ’’ an important subsidiary roleof the examination board’’ ;given access to a candidate’ s criticalself-appraisal of their own learning as wellas the research findings andinterpretations?

In order to explore these sorts of issues morecarefully with practitioners, the key questions

indicated below were compiled and used tofocus discussion and group activity duringseminars/workshops:

What is being assessed? When assessing atresearch degree level – is it the candidate(i.e. the sum total of the learningoutcomes for the individual (includingperhaps any such outcomes from generictraining)) or the substantive nature of anycontribution to knowledge (i.e. asexpressed in the thesis) that is beingassessed?Who is being assessed? Should a judgementbe made as to the quality of supervisionthat has led to the submission and if socould ’’negligent supervision’’ (CVCP,1993) influence decisions when awardingthe degree? Should judgements be madeby the examiners about overall provisionwithin the institution (e.g. of resourcesand training opportunities for students)?Are there identifiable (and usable)assessment criteria for PhD? What mightsuch criteria be? Should criteria for theaward of PhD be made explicit? Shoulddifferent criteria for different parts of thethesis and/or the examination beweighted? Should such criteria be madeavailable to all students? What mightcount as ’’publishability’’ (but see below)?How should the assessment beconducted? Is a viva necessary foreffective assessment of a PhD? Is morethan one examiner necessary? What is therole of supervisors in the examinationprocess?

Summary of discussion on researchdegree examination as assessment

The discussions were allowed to befree-ranging and the foci and responses werenot always neatly confined to the keyquestions posed. The summary is presentedhere under theme headings that include thekey questions, but that range beyond them.

What or who is being assessed?It was generally agreed that it is more than thewritten word that is being examined. It shouldbe the case that the key skills, reflected by thework, are being judged and therefore thestudent himself/herself is the real point ofscrutiny. Examiners should be asking

106




Volume 12 . Number 2 . 2002 . 104±115

themselves – has this student integrated keyskills and understanding? The viva issupposed to be the most satisfactory place toinvestigate this, yet this point of focus is oftenlost as the marking of the thesis and the viva,as already noted, become contexts forchecking text and improving literary qualities.

Special mention was made that thesesshould in one sense be valued ’’ in their ownpart of the academy’’ , but this is problematicwhere the work is genuinely interdisciplinary– an area which is growing. Indeed, one keyarea of potential difficulty was identified asthe nomination of an examination team thatcould properly and fairly examine a thesis thatcrosses disciplinary boundaries and that thusemploys constructs and methodologies thatmay not be universally recognised.

Assessment criteriaIt was generally agreed that criteria might bedifficult to standardise across HEIs and thatin any case such standardisation would notnecessarily be desirable. But it was alsoargued strongly that there was a need formore transparency. At present the situationacross the sector is not only varied but it isalso opaque. This makes quality andstandards auditing a difficult exercise anddisadvantages the student who may enter anassessment scenario unaware of the hiddenagenda in operation.

The notions of ’’ publishability’’ , ’’ originaland significant contribution to knowledge’’ ,that are commonplace criteria used byinstitutions, often depend on a number offactors and are linked to discipline. It isquite common for there to be no agreementmade between examiners with regard tocriteria – by default, decisions are thereforebased on implicit criteria, which mayremain idiosyncratic to individualexaminers.

Some seminar attendees made theargument for ’’ threshold’’ levels for criteriarather than standardisation and it wasgenerally agreed that there should not besubclasses for PhDs. It was generally agreedthat the PhD is a ’’ threshold’’ qualification.

Status and nature of the viva and thethesisThe thesis was seen to be prioritised as themedium for assessment. It was felt that theviva, as currently operated in the UK, could

not function adequately as a means of

reflecting the student’ s expertise accuratelyenough for judgement to be made andtherefore the student’ s individual learning wasseen to be secondary. It is then the thesis thatis assessed, though discussants did claim thatexaminers were making judgements about thecandidate on the basis of what was presentedin the thesis.

There was general agreement that the vivashould be an area for learning for allconcerned. But there was recognition that itgenerally functioned as a means of’’ polishing’’ the thesis and tuning the clarityof the contribution to knowledge and overallliterary quality of the written work.Throughout the various discussion groupsthere was an emerging consensus that thelearning opportunities offered for allparticipants by research degree examinationwere not regularly pursued to fulladvantage. Certainly there was littleevidence of the process of examinationbeing reflected on in any formal sense, norwas evaluative feedback on the processregularly obtained.

It was felt by many that the nature of theviva depends on the nature of the thesis.Some colleagues felt that if the document wasstrong then the viva functioned as asummative process whereby the examinerwould sum up the strengths of the thesis anddiscuss post-doctoral progression. If, on theother hand, the document was felt to befailing, then the viva functioned as aformative process whereby the examinerwould inform the students of the weaknessesthat would lead to resubmission and possiblere-examination. Time would then be spent ongiving formative guidance on what is requiredto amend current failings. We should notehere that the converse view was alsoexpressed, namely that the function issummative when the thesis fails (the examineris making a judgement) and formative when itis successful – the examiner is consideringprogression beyond the PhD and into theacademic community.

The view was expressed that the vivashould function also as a way of assessingwhether students have integrated key researchskills and understanding of methodology as aresult of research training. This was seen as anew aspect of the modern PhD along withother value-added aspects such as the

107




Volume 12 . Number 2 . 2002 . 104±115

importance of a PhD resulting in a newartefact, product or innovation.

Monitoring and evaluation of processThere was a suggestion from some discussiongroups that the final examination of a PhDcould be informed by various earlier stages inthe process, such as a first year or transfer/upgrading report. Some colleagues felt thatexaminers should have access to these earlierreports. Again, opinions were divided alongsimilar fault lines to those that existed in the’’ teacherly-confrontational’’ debate (see ’’Roleof the examiner’’ section later in this paper).There was general agreement that asatisfactory assessment process was likelywhen each assessment point during astudent’ s progress acted clearly as a device tokeep the student on track. In particular, theupgrading process should not been seen as anautomatic progression to PhD status. Inshort, it was generally agreed that thereshould be delineated points in the process of aresearch degree programme which indicate ifthe student is likely to complete or fail, andthat a more rigorous evaluation of the processwould affect the type of viva and ratio ofpass/fails.

As already noted, there was some strongsupport for the notion that the sameexamining panel should examine throughoutthe process of a programme of study and atthe final thesis stage. In this sense theexamination of the final thesis was seen as asynoptic assessment of a complete process.This was seen as analogous with assessmentsof taught postgraduate courses whereexaminers might have access to recordsindicating the whole of the progress of anindividual student through a programme ofstudy.

There was some agreement, though notunanimous, for the notion that a reportshould be included at the end of the process(i.e. at the final examination stage) whichdescribes ’’ special measures’’ encounteredand perhaps overcome successfully during theresearch process.

The main conclusions and areas ofconsensus emerging from the discussiongroups are presented concisely in Table I.The rows indicate function and role whilethe columns show aspects of the process.What is clear from this table is that functionand role shift as one considers the different

aspects of the whole process. In particular,there are tensions between what is valuedwithin the written (or alternativelypresented) ’’ product’’ itself and the processof the student’ s learning.

Examiners and examining panels

Setting out the issuesDifferent notions of what is being examinedand how examinations should be carried outlead to different notions of the composition ofpanels. Some institutions have anindependent chair of PhD examination panels– many do not. Sometimes a senior internalmember of staff, not necessarily possessingsubject expertise, acts with a brief to ensurethe good conduct of the examination processand the dissemination of good practice.Institutions differ in the extent to which theyallow or encourage assessors at anyprogression stage to become examiners at thefinal assessment stage. Some institutionsroutinely appoint two examiners – oneinternal and one external – other institutions,and many outside the UK, routinely appointthree examiners. Institutions vary in whetheror not they require that an examiner shouldhave a PhD himself/herself.

The question has been raised as to whetheror not the external examiner should’’comment on the remainder of theexamination board’’ (UCoSDA/BPS, 1995).Additionally, the extent and nature of anystudent input into the appointment of theexamining team differs.

The differing status of examiners within thepanel is also far from well defined. In theCVCP (1993) document Handbook forExternal Examiners in Higher Education, it isstated that:

. . . an external examiner of a PhD is theexaminer. He or she decides whether the thesispasses or fails (VCP, 1993, p. 73, theiremphasis).

This is a view held in some universities butnot in others. In some the external examinerovertly holds the key vote, whereas in othersall examiners are equal – though again somemay be more equal than others! Clearly, therole of the internal examiner is defined inrelation to that given to the external.

The procedures for nominating examiners,and the mechanisms that enable institutions

108




Volume 12 . Number 2 . 2002 . 104±115

to monitor them, are generally seen to beimportant if independent and fair assessmentis to be achieved, see, for example, QAA(1999) precept 10a which stresses – ’’ themechanisms used for communicatingprocedures relating to the nomination ofexaminers’’ . Yet there seems to be littleconsistency in practice. Certainly mostinstitutions take particular care over thisphase of the administration of research degreeexamination.

Many institutions do not train examiners,yet most workshop attendees thought of theprocess as complex and important enough forsome minimal training to be desirable. Withregard to qualifications for examinership,UCoSDA/BPS (1995, p. 8) suggests that ’’anexternal examiner should possess a PhD orother evidence of a similar level of scholarshipand should normally have been principalsupervisor of at least one successful PhDstudent’’ . The wording of thisrecommendation lacks precision and allowssome considerable latitude and, whilst thismay not appear to be serious, it does leave theissue of just what is expected of an examiner

largely unresolved. The relationship betweenexamining, scholarship and supervising mightusefully be explored further. Manyinstitutions require an examining team tohave experience of examining but are lessconcerned to consider supervisory experience,presumably on the grounds that these twoactivities are discrete.

The key questions set out below weresuggested by the issues identified above andwere used to focus the discussions in thenational seminars and workshops:

What are the qualifications andexperience required of an individualexaminer?Should examiners be trained?How should external scrutiny functionwithin the assessment?How should an examining panel benominated?Should the process of examination bemonitored/evaluated?Should any one examiner have the castingvote?Should the student have any role in theprocess of determining panels?

Table I Key factors in research degree assessment in the UK

The finished product The research process The viva

Function Verification of learned research skills

Indicate contribution to research

community

Indicate contribution to knowledge

Facilitate research

apprenticeship

Polishing/fine tuning the literary work

Examination of the candidate’s

knowledge and skills

Verification of authenticity

Rite of passage

Status Prioritised as medium for

assessment

Rarely prioritised during

assessment

Varies from absolute to probable to

possible

Mode Written thesis

Alternative forms of doctoral

submission

Culmination of discrete parts of a

professional doctoral programme

Collection of published work with

exegisis

Supervision

Taught elements (including

training in methodology and in

generic skills)

Retrospectively acknowledged

(PhD by published work)

Real time

Video conferencing

Appraisal of work in situ (e.g. art

submission)

Assessmentcriteria

Publishability (does not always apply,

e.g. in practice-based doctorates)

Originality

Innovation

Significance of contribution to

knowledge

Level of scholarship

May form part of assessment in

professional and practice-based

doctorates

Records of progress may be

referred to

Determined by evaluation of finished

product

Ability to communicate ideas and

defend positions

Evidence of authorship

Progression andimprovement

Increased transparency/clarity (from

earlier submissions where made)

Threshold markers for different awards

Weighting for different components

Evaluation of final thesis may

be interpreted as a synoptic

assessment of the complete

process

May lead to amendments and further

supervisor input

109




Volume 12 . Number 2 . 2002 . 104±115

Summary of discussion on examiners andexamining panelsOnce again the themes emerging in discussiondid not match exactly the set of questions in aneat one-to-one correspondence, but theheadings below do give a clear indication ofthe preoccupations of the participants.

Experience/qualifications of the examiner. Itwas generally agreed during the variousworkshops that examiners only gainexperience of examining by examining, butthat training would help to bridge the gapbetween novice and expert. It was also felt bysome participants that examiners should havea record of successful supervision. Thegrounds cited were that in order to passjudgement on the outcomes of researchdegree study it is necessary to understand theprocess of that study, and in turn it istherefore necessary to understand therelationship between supervisor, student andproject. Examiners should therefore betrained to understand the formative nature ofsupervision and the relationship between thatsupervision, individual student achievementand tangible research outcomes.

But it was also noted that there is nonationally recognised forum for exploring theissues of examiner training across the UK –although several universities now offertraining to their own staff on their role in theexamination of the PhD. It was also notedthat there are severe limits to what can beexpected of examiners given the level ofremuneration and the amount of workinvolved.

While it was agreed that the examinationshould act as a learning process for examiners,as noted elsewhere in this paper, it was alsorecognised that a context needed to bedeveloped in which examiners could reflectand develop such learning. Suchopportunities did not exist at present formany colleagues

Role of the examiner. There was a generalconsensus in the workshops that whileexamining should be seen as a distinct ’’ role’’with associated specific skills, all too oftenexaminers are chosen for their individualsubject specialism and their expertise in thisfield. If perceived as a ’’ role’’ , the notion ofexaminer training could be moremeaningfully explored and hence betterunderstood. The issue of ’’ super examiners’’who can step outside their particular ideology/specialism to judge scholarly work more

widely was raised. There was some debateabout whether or not it was practicable tomake use of an independent chairperson inthis respect and it was suggested by some thatthe burden on particular individuals mightbecome onerous if this latter scenario were todevelop.

It was generally agreed that examiningshould not be driven by an examiner’ s ownpersonal research agenda and that examinersmust be aware that other ideologies andmethodologies exist outside their ownpreferences. They should not be influenced tofail a thesis purely on the basis of their ownpersonal convictions, feelings or passion.

Some colleagues conceptualised the role ofexaminer as either ’’ teacherly’’ or’’confrontational’’ . In the first instance thenotion of ’’ role’’ is highlighted and examinersof this type act as positive gatekeepersallowing access into the research community.In the second instance the notion of theindividual is highlighted and the role assumesmore of a function of negative gatekeeping:keeping people out of the researchcommunity; disallowing those students whodo not share the same ideology/methodologyand passions. Along with this negativegatekeeping is suggested the notion that someexaminers may even use the examinationprocess, consciously or otherwise, as a chancefor ’’ settling old scores’’ .

Notions of using the individual who judgedthe progress of the work at a transfer (orprogression or upgrading) stage as anexaminer at the final examination stage variedalong with the view taken of the gatekeepingrole. Those who saw the role as a teacherlyone tended to see the role of formativeassessor as positive and as compatible withthat of final assessor. Conversely, those with amore confrontational view tended to see aneed for the examination at the final stage tobe ’’blind’’ to processes that had gone before.Judgements here were to be made on the finalproduct only. Roles tended to correspondwith ’’assessment of process’’ (teacherly) or’’examination on the day as single event’’(confrontational).

Most participants agreed that examinersneed to be informed about the work they areexamining and need to understand theapproach used, even if they do not practise itthemselves. It was clear that problems werelikely to arise where the examiner had notbeen made fully aware of the nature of the

110




Volume 12 . Number 2 . 2002 . 104±115

substantive content of the thesis or theprocedures and methods employed in thestudy at the time of appointment. Theusefulness of briefing the potential examinersby giving them access to an abstract (orsimilar) of the thesis was suggested.

Views on the remit that should be given toexaminers tended to be constrained byrecognition that remuneration levels arealready either poor or derisory. Therefore toexpect the examiner to comment on morethan the work in front of him/her (e.g. tocomment on supervisory arrangements atthe university or on departmentalresourcing) was seen as unrealistic by manyparticipants. This whole issue of the statusof examining and its relationship to theworkings of the academic community as aself-perpetuating guild was noted, but littleconsensus was achieved.

Examination panels. The role of anindependent chair was felt to be useful insome situations but not others. Somecolleagues felt that in 95 per cent of casesthere was no need for a chair, as the result ofthe examination process was clear-cut andtherefore any panel members in addition tothe actual examiners would only add a burdenof unnecessary bureaucracy.

However, others felt that an independentchair could enable the sound running of theexamination process. He/she could act as athird party or witness in any appeal process –particularly if they have been involved in theexamination process since its inception. Thosein favour of an independent chairperson tendedto see the role as that of a ’’ super’’ examinerwho would judge the work on its scholarly valuerather than be biased towards any one kind ofideology or methodology.

A majority of discussants felt that theexternal examiner should not be all-powerfulnor have the casting vote because this raisedquestions regarding the purpose and role ofthe internal examiner.

Student role in the nomination of theexamination panel. There were differingopinions concerning the strength of the studentvoice: the student should have a veto; thestudent should have one veto; the studentshould ’’have a say’’ in the examination panel.But there was a fairly consistent view that thestudent should have a voice of some kind, i.e.not be totally shut out of the process. Onereason given was that the student might need atthe very least to declare that he/she has had

contact with an individual examiner in terms ofcomments on work. There was some agreementthat while the student should not be in aposition to choose their own examiner theirviews on the kind of expertise that would beneeded to ensure fair and rigorous examinationof their work should be taken into account.

A summary of some of the main issuesraised in relation to examiners and examiningpanels is set out in Table II. In general, whatis clear from the table is that there isconsiderable variation in terms of function,status, level of experience and qualificationthat permeates the constitution of and theroles of the various participants within theexamination scenario.

Recent significant developments

The QAA (2001) has published its finalversion of the framework for higher educationqualifications. Awards are contained withinthree undergraduate levels (levels C, I, H)and two postgraduate levels – Masters (levelM) and Doctorate (level D). A set ofqualification descriptors which indicate thegeneral learning outcomes that should beachieved within the range of qualification ateach of the five levels are also indicated withinan Annex to their framework paper.

By carefully cutting and pasting thedescriptors, it is possible to construct a matrixsuch as is indicated, for levels M and D, inFigure 1 (reproduced here with the kindpermission of Professor Malcolm Shaw whohas been responsible for its development anduse at Leeds Metropolitan University). Thisprovides a more convenient means to scan thedescriptors both within and across levels. Thematrix also has allowed some grouping of thedescriptors into categories (knowledge andunderstanding, problem solving,communication, etc.) which helps withattempts to trace notions of progressionthrough the levels. It should be noted that thismatrix is not in any way a part of QAApromotion of the framework but it does use,for obvious reasons, the precise wording oftheir descriptors.

These descriptors at level M and level Dclearly impinge considerably on our view ofwhat should be achieved within awards such asMRes (level M), MPhil (also level M), PhD(level D) and other doctorates (also level D). As

111




Volume 12 . Number 2 . 2002 . 104±115

such, they impact on the function of ourassessment of students and the criteria that weuse to assess student achievement in theseawards. They suggest a more systematic andconsistent approach to assessment andstandards across all examiners and across allawards in the HE sector. The descriptors doprovide an explicit device with the potential tobegin to identify and explore answers to some ofthe key questions already posed. Inevitablyhowever, these descriptors raise a number ofrelated fundamental questions of their own,such as:

Do the descriptors differentiateadequately between Masters andDoctorate levels?Do the outcomes present new factors thatare not currently examined?What implications do the outcomespresent for the role of the thesis and theviva?Do the outcomes accommodate thedifferent types of masters and doctoralawards?

These are questions that require a more

extensive and rigorous examination than has

been possible in the seminars to date. Certainly

it becomes clear, from even a cursory glance at

the matrix, that QAA’s descriptors provide

more scope for a systematic approach to the

assessment of the PhD whilst differentiating it

from other awards than has typically existed to

date. At the same time, most of these

descriptors at doctoral level might well be seen

to be implicit in much PhD assessment that is

currently carried out, but whether they are

valid, adequate and sufficient is another matter.So we are not suggesting here that the

descriptors should be directly translated intoassessment criteria, but that they might forma basis for discussion leading to a morestructured approach to assessment and hencethe possibility of more transparency andequity across the sector.

Conclusions

Codes of practiceThere was some discussion during seminars

of the need for a ’’Code of practice for

assessing research awards’’ . It is worth noting

here that the QAA (2000b) in its code of

practice on external examining gives little

specific attention to the examination of the

highest award that most universities

commonly make. Similar lack of clarity and of

detail is found within precepts 10a and 10b of

the QAA (1999) code of practice for

postgraduate research awards.

Table II Constitution of and roles within examination panels

External examiner Internal examiner Chair Supervisor Student

Composition Invariably necessary Often but not always

necessary

External examiner

External to panel

Chosen on the day

Not necessary in examination

process

May attend viva at discretion of

examiners or student

Presence necessary if

viva required

Status ofindividuals

Expert in field of

research?

Experience of

examining?

Has casting vote?

Aware of university

regulations?

Candidate’s friend?

Chair independent of

panel ± aware of

university

regulations?

External?

Candidate’s friend

Advisor on process

Voice in composition

of panels?

No voice ± merely

subject of enquiry

Function ofindividuals

To gatekeep?

To be teacherly?

To endow credibility

To ensure

comparability across

institutions

To represent

institution?

To ensure fairness

for candidate?

To balance against

external?

To mediate?

To control process of

oral?

To enable fair

process?

To act as note-taker for candidate

in respect of any amendments

To interpret for the candidate for

the examiners?

To authenticate

research?

To prove research

skill as an individual?

To give evidence of

learning

Experience andqualifications

Successful

supervision?

Experience of

examining?

Holder of a PhD?

Domain expert

Awareness of

regulations

Awareness of

regulations

Research skills?

First/pg degree?

112




Volume 12 . Number 2 . 2002 . 104±115

Figure 1 Matrix of QAA descriptors for qualifications at levels M and D

113




Volume 12 . Number 2 . 2002 . 104±115

Figure 1

114




Volume 12 . Number 2 . 2002 . 104±115

PrinciplesThe following principles for assessment andexamination emerged strongly from thediscussions:

Institutions need to ensure balancebetween research degree examination as anassessment of the process of training andthe final product that may well be thethesis. Examiners need to recognise that itis the candidate who is being examined,albeit on the evidence of learning andachievement presented in the thesis and theviva.The composition of the examinationpanel, its remit and roles of individualsshould be clear to all participants.The relationship between the variouselements of the examination must beclearly specified along with a statement ofthe role of each element in the overallassessment.In any oral examination situation thegeneral procedure and the criteria formaking judgements should be laid downin advance by the institution.There should be some way of monitoringthe process of an oral examination toenable fairness to be judged and goodpractice to be identified anddisseminated.There are skills in examining which can betaught – assumptions to the contraryshould not be made. Training in researchdegree examining should become ascommonplace as training in the processand methods for assessment itself.There is need for transparency inconceptions of threshold performanceand in the weighting of assessmentcriteria.New forms of research degree require arethink of assessment processes.There should be comparability in thelevel, if not the kind, of assessmentapplied to traditional and to newer formsof research degree.

Finally, we must bite the bullet, accept that weare assessing a total programme of research anddevelop appropriate criteria. Why should therigour of the assessment of doctoral work beregarded any differently to that pursued in other

aspects and levels of student achievementwithin higher education?

References

CVCP (1993), Handbook for External Examiners in Higher

Education, UK Universities Staff Development Unit,

Sheffield.

Powell, S.D. (1999), `̀ A Swedish disputation: reflectionson PhD examination’’, Newsletter, UK Council for

Graduate Education.

QAA (1999), Code of Practice for the Assurance ofAcademic Quality and Standards in Higher

Education: Postgraduate Research Programmes,

QAA, Gloucester.

QAA (2000a), Code of Practice for the Assurance ofAcademic Quality and Standards in Higher

Education: Assessment of Students, QAA,

Gloucester.

QAA (2000b), Code of Practice for the Assurance ofAcademic Quality and Standards in Higher

Education: External Examining, QAA,

Gloucester.

QAA (2001), The Framework for Higher EducationQualifications in England, Wales and Northern

Ireland, QAA, Gloucester.

Shaw, M. and Green, H. (1996), `̀ Standards in researchawards: length, weight or quality? Developing an

approach for resolving the dilemma’’, Innovation &

Learning in Education: The International Journal for

the Reflective Practitioner, Vol. 2 No. 3, pp. 4-10.

UCoSDA/BPS (1995), Guidelines for the Assessment of thePhD in Psychology and related Disciplines,

Universities’ and Colleges’ Staff Development

Agency, Sheffield.

Further reading

Hartley, J. and Jory, S. (in press), `̀ Lifting the veil on the

viva: the experiences of psychology PhD candidates

in the UK’’, paper submitted for publication (copies

from the authors ± Keele University).

Jackson, C. and Tinkler, P. (2000), `̀ Examining thedoctorate: institutional policy and the PhD

examination process in Britain’’, Studies in Higher

Education, June (note: available from CarolynJackson at Department of Education, University of

York).

Johnston, S. (1997), `̀ Examining the examiners: an

analysis of examiners’ reports on doctoral theses’’,Studies in Higher Education, Vol. 22 No. 3,

pp. 333-48.

Underwood, S. (1999), What is a PhD?, Higher Education

Development Centre, Lancaster University,Lancaster.

115




Volume 12 . Number 2 . 2002 . 104±115

researchdegree examining–common principlesand...

Documents