Conclusions

Therapeutic and toxic plasma concentrations ofcommonly measured drugs

Concentration below Concentrationabovewhich a therapeutic which a toxic effect

Drug effect is unlikely is more likelyAminoglycosides:Amikacin 34 ,umol/l (20 ,ug/ml) 55 ,umol/I (32 ,ug/ml)

(at peak) (at peak)17 ,imol/I (10,g/ml)(at trough)

Gentamicin 5 ;Lg/ml (at peak) 12 ,ug/ml (at peak)2,ug/ml (at trough)

Kanamycin 50 ,umol/l (25 ,ug/ml) 80 ,umol/I (40 ,ug/ml)(at peak) (at peak)20 jimol/I (10 ,ug/mI)(at trough)

Cardiac glycosides:Digitoxin 20 nmol/I (15 ng/ml) 39 nmol/l (30 ng/ml)Digoxin 1.0 nmol/l (0-8 ng/ml) 3.8 nmol/l (3 ng/ml)

Cyclosporin* 80-200 nmol/I 170-330 nmol/I(100-250 ng/mI) (200-400 ng/ml)

Lithium 0.4 mmol/l 1-0 mmol/lPhenytoin 40 jimol/l (10 ,ug/ml) 80 ,umol/l (20 ,ug/ml)Theophylline 55 ,umol/I (10 jig/ml) 110 ,umol/l (20 ,ug/ml)

*Measured in whole blood by specific radioimmunoassay or highperformance liquid chromatography. The actual results depend on thelaboratory in which the measurement is made.

In this series we have outlined the uses ofmeasuring the plasma concentrations ofsomedrugs and given guidelines on how suchmeasurements should be made and interpreted.The box summarises the target plasma

concentrations for each of the drugs. In each casethere is a concentration below which atherapeutic effect is unlikely and a concentrationabove which the risk of toxicity is high. Thesetwo concentrations imply a therapeutic range foreach drug, but remember that there arecircumstances in which strict adherence to arange of this kind is inappropriate. The plasmaconcentration should always be interpreted in thelight of factors which may alter the effectivetherapeutic range.Nor is it always necessary to measure plasma

concentrations to achieve satisfactory drugtherapy. Routine measurement without a clearpurpose is as bad as no measurement at all. Theapplication of the principles we have outlinedshould allow the rational use ofplasmaconcentration measurement in optimising drugtherapy.

1 Vozeh S. Cost-effectiveness of therapeutic drug monitoring. Clin Pharmacokin 1987;13:131-40.2 Spector R, Park GD, Johnson GF, Vesell ES. Therapeutic drug monitoring. Clin Pharmacol Ther

1 988;43:345-53.3 McInnes GT. The value of therapeutic drug monitoring to the practising physician-an hypothesis in

need of testing. BrJ Clin Pharmacol 1989;27:281-4.4 Sjoqvist F. Interindividual differences in drug responses: an overview. In: Rowland M, Sheiner LB,

Steimer JL, eds. Vanrability in drug therapy. New York: Raven Press, 1985:1-10.5 Larkin JG, Herrick AL, McGuire GM, Percy-Robb IW, Brodie MJ. Antiepileptic drug monitoring at the

epilepsy clinic: a prospective evaluation. Epilepsia 1991;32:89-95.

Dr JK Aronson is clinical reader in clinical pharmacology, and Dr D JM Reynolds is clinicallecturer in clinical pharmacology, Radcliffe Infirmary, Oxford.

Medical Education

Assessment ofstudents

Stella Lowry

This is thefifth in a series ofarticles examining theproblems in medical educationand their possible solutions

British Medical journal,LondonWCIH 9JRStella Lowry, assistant editor

BMJ 1993;306:51-4

Student assessment is often described as "the tail thatwags the dog" of medical education. It is seen as thesingle strongest determinant of what students actuallyleam (as opposed to what they are taught) and isconsidered to be uniquely powerful as a tool formanipulating the whole education process. Sir WilliamOsler summed up the power of examinations in 1913:"At the best means to an end, at the worst the enditself, they may be the best part of an education or theworst-they may be its very essence or its ruin."' But isassessment as powerful as we think, and, if it is, aremost medical educators using it effectively?

Why assess?Few people formally question why we assess medical

students, and many who do think no further than usingassessment as a means of checking that requiredinformation has been leamt. Certainly in an overloadedcurriculum students will pay attention to topics thatthey know will feature in examinations.2 A recent study

of surgical students at the Flinders University of SouthAustralia found that when no clear guidelines andcourse objectives were given in a self directed learningprogramme the students-far from exploring thetopic widely and pursuing personal interests-tried to"guess" what would feature in the final examinationand concentrated on that (D J Prideaux, paperpresented at fifth Ottawa international conferenceon assessment of clinical competence, Dundee, Sep-tember 1992). This tendency allows staff to directstudents' attention to important topics but alsoincreases the risk that unexamined areas will beignored.

Unfortunately, the fact that students can success-fully answer examination questions on a topic is noguarantee that they will retain their knowledge of thesubject. Assessments that are based on a one off factualrecall are notoriously unreliable as indicators of reallearning,3 and if assessment is to be used to ensurelearning more complex approaches are needed. Onemethod is to retest the same information at regular

BMJ VOLUME 306 2 JANUARY1993 51

on 20 March 2020 by guest. P

rotected by copyright.http://w

ww

.bmj.com

/B

MJ: first published as 10.1136/bm

j.306.6869.51 on 2 January 1993. Dow

nloaded from

intervals. At the University of Maastricht, for example,the entire medical school sits an identical multiplechoice "progress test" at various points during thecourse.4 This allows students to document theirincreasing knowledge as they progress through thecourse and ensures that essential information is notconveniently forgotten once the exam is out of the way.McMaster University in Canada, which has never

had formal examinations during the medical course, isnow planning to introduce a Maastricht-style progress

test. This is largely in response to frequent complaintsfrom students that the complete lack of formal assess-

ment actually adds to their stresses rather than reducesthem. Students never quite know whether they havelearnt enough about a subject, and it is hoped that a

progress test will give them some idea of when it is safeto stop.

Assessment as an aid to learningAn important function of assessment is to aid

learning by providing students with a check on theirprogress and an opportunity to improve. If studentsare given an opportunity to give feedback to theirtutors assessment can also be useful in refining the waya curriculum is taught and ensuring that course

objectives are met. Too often assessment is used purelyas a tool for staff to regulate progression of studentsthrough the system and rank them in order of achieve-ment.We tend to rely heavily on summative assessment-

testing acquired knowledge at the end of a course

(when it is too late to correct deficiencies). Studentsneed formative assessment-regular checks on howthey are doing with detailed feedback on the resultsand an opportunity to try again after remedial learning.This is a powerful tool in focusing students' learning5and can be tailored to individual needs and wants. TheCentre for Medical Education, University of Dundee,for example, has devised computer banks of multiplechoice questions that students can use as and whenthey wish to monitor their own progress. Simplefeedback can be incorporated into such programmes.More complex feedback may have implications forstaff time, but if our aim is to teach rather than to testthat must be accepted.Some medical schools still refuse to provide any

feedback on their examinations on the grounds thatthey need to use the test again the next year andmustn't allow the answers to leak out. A bank of threepapers used in rotation should get round this problem,and if that is impossible then by all means hold a

summative test without feedback but ensure that

separate formative tests are also available. It isinexcusable that many students who have failed anexamination are still expected to retake the whole topicwith no idea ofwhere they went wrong last time and noguidance on where to focus their attention. That is awaste of their time and is not educational.

Ensuring minimum standardsBecause undergraduate medical education is a pro-

fessional training the minimum standards of profes-sional-practice must be acquired during the course.The protection of the public demands this &atekeepingfunction of assessment, even if it places constraints onthe educational experience.This requirement is universal. McMaster in Canada

has no summative assessment during the course,6 butany graduate who wishes to practise medicine muststill pass the national licensing exams. Indeed, onecriticism often made of the McMaster course is that itsstudents do not do as well as those from other schoolsin the licensing exams. Dr Barbara Ferrier, a foundermember of the medical faculty at McMaster, thinksthat this is an oversimplification. For one thing therelative ranking of McMaster students varies from yearto year, and over several years no consistent patternemerges. McMaster students tend to do a bit worse

-Studentsmling in norm referenced testC Students-Sling in criterion referenced teit

With norm referencedmarking studens,mayfail exam despit

achieving requiredminimum standard

With norm referencedmarking students may passexam despite falling far shortof required minimumstandard

* . - . _ . v az

Test xcbr*Minimum d;sired standard

Norm referenced marking cannot ensure minimum standards are

attained; criterion referenced marking can. Proportions of studentsfailing in the two types of test

BMJ voLuME 306 2 JANUARY 1993

Tail wagging the dog ofmedicaleducation. Medical studentssittingfinals at end oftheir course

52

on 20 March 2020 by guest. P

rotected by copyright.http://w

ww

.bmj.com

/B

MJ: first published as 10.1136/bm

j.306.6869.51 on 2 January 1993. Dow

nloaded from

than average on the multiple choice question paper andrather better than average in the patient managementsection. Secondly, McMaster allows all of its studentsto enter the licensing exams as soon as they completethe course whereas many other schools screen out thosethey think might fail and allow them to enter only afterfurther tuition.The only sensible way to ensure that minimum

standards are met is to define them and fail anystudent who does not reach them. Why, then, do so

many medical schools persist in marking studentsagainst their peers rather than against an externally setstandard?7 Most British medical schools use so callednorm referenced assessment, in which the students'scores in an exam are ranked on a normal distributionand the pass mark is adjusted to provide a predeter-mined percentage failure rate. This means that theabsolute standard achieved may vary widely from yearto year (figure). Because most medical studentsmanage to learn the absolute essentials of a course thesebasic points have little discriminatory power in a norm

referenced examination and are often omitted. Theexams thus concentrate on the rarities that will dis-criminate between the best and worst students, and thebasic principles may never be tested.Norm referenced assessment also raises the possi-

bility that a particularly poor group of students will"pass" the exam despite falling far short of the desiredminimum achievement. Conversely, an excellentadaption in the way a course is taught may havedramatic effects on the amount that students learnabout a subject, but this will not be reflected in anyimprovement in the "pass rate."

In criterion referenced assessment, however, therequired minimum attainment is determined inadvance and students who score less than this standardfail. Norm referenced marking cannot ensure thatminimum standards are attained; criterion referencedassessment can. Norm referenced assessment is usefulwhen scores are being ranked before allocation of a

limited commodity, such as house jobs, but should notbe the main system for assessing medical students'achievements.

What should we assess?If assessment is so important what should be

assessed? Traditionally the emphasis in medical educa-tion has been on acquiring a body of essential facts oneach topic, but now more attention is given to the skillsand attitudes thought to be important in a "gooddoctor."89 No single examination can be expected toassess such a wide range of features. Medical examinersshould identify those aspects that they wish to test andthen provide a range of appropriate formats.There is often a high correlation between students'

performance in different types of examination, whichhas led some people to conclude that they can rely on

whatever system is cheap and administratively simple.Professor David Newble, of the University of Adelaide,however, emphasises that this correlation may simplybe reflecting the relative ability of students to study forexams rather than saying much about their actualknowledge. Most experts now agree that a range ofmethods is needed to assess a range of clinical skills.'0The multiple choice question paper, for example, is anexcellent, cheap, reliable, and reproducible way oftesting factual knowledge but tells us almost nothingabout clinical skills. The objective structured clinicalexamination (box 1) is expensive, administrativelycumbersome, and lacks high reliability but is an

excellent way of assessing practical skills."-'4 Evencomplex processes like skill at self directed learningcan be tested; McMaster has devised the "triple jumptest" to do just that" (box 2).

Features ofa good test

Given the plethora of techniques for assessingstudents, there is a real danger of becoming obsessedwith the methods and neglecting the content. ProfessorCees van der Vleuten, from Maastricht, remindeddelegates at a recent conference on the assessment ofclinical competence that ultimately the care with whichany examination is devised is more important than theform that is chosen (paper presented at fifth Ottawainternational conference on assessment of clinicalcompetence, Dundee, September 1992). What, then,are the cardinal features of a good test?A good test must be acceptable to those using it,

feasible, valid, and reliable. A test may be acceptable to

BMJ VOLUME 306 2 JANUARY 1993

Box 2McMaster "triple jump" testAn important aspect of education is learning how tolearn. Many medical courses now use large elements of"self directed learning" so that students becomefamiliar with the skills and resources needed to keepup to date with the rapid changes in medical know-ledge. The "triple jump" test was devised as a meansof assessing students' competence at self directedlearning.* Step 1: Student reads written "problem" anddiscusses first impressions with tutor. Student thenselects some tasks for further learning and decideswhat additional information is needed* Step 2: During period of private study the studentuses any relevant sources of information to tackle theseself selected tasks* Step 3: Student reports back to tutor and presentsrevised summary of problem based on his or her recentresearches. Tutor provides feedback on way in whichstudent has tackled problem.

Box 1Objective structured clinical examinationWritten papers may be useful tests of factual know-ledge but are poor at assessing clinical skills. Tradi-tional clinical examinations are difficult to standardise,and how well students perform may depend largely onthe "luck of the draw" in determining who examinesthem, on which patients, and using what questions.The objective structured clinical examination wasdevised to provide a more standardised way of assess-ing clinical competence. 15"6* Objectives of the test are identified and recorded.Objective structured clinical examination is thendesigned to cover all of required aspects* Candidates rotate around series of "stations," ateach of which they are asked to perform clinical task oranswer questions on material provided* Assessors are present at relevant stations to assesscandidate's performance using standardised checklist* Clinical models and simulated patients can be usedduring objective structured clinical examinations toallow large numbers of students to be tested on sameclinical problem without causing fatigue or distress toreal patients* Marking can be completed as the objective struc-tured clinical examination proceeds and prompt feed-back is possible (some centres provide immediatefeedback between each station)* Objective structured clinical examinations can beexpensive and administratively cumbersome to set upbut are easy to mark and allow testing of skills thatmore traditional methods ignore.

53

on 20 March 2020 by guest. P

rotected by copyright.http://w

ww

.bmj.com

/B

MJ: first published as 10.1136/bm

j.306.6869.51 on 2 January 1993. Dow

nloaded from

7W77...... .......

"Excellent way of assessingpractical skills. "Objective ...........stt-uctured clinical examinationin progress at St Bartholomew's........Hospital, London

some of those dealing with it and not to others.Multiple choice tests, for example, may be acceptableto those administering them because they are simple touse, cheap to run, and quick and easy to mark. Theymay be wholly unacceptable to those sitting them if thequestions do not seem to be a fair test of the importantaspects of a course. A carefully thought out objectivestructured clinical examination may be a good way toassess essential clinical skills, but the logistic problemsof running it may make it unfeasible for the simul-taneous assessment of an entire year of students.A valid test measures what we want it to measure and

nothing else. The validity of a test can be assessedunder various headings (see glossary)."

Reliability is a measure of the consistency andaccuracy with which a test measures what it is sup-posed to. In a good assessment system not only shouldthe test be reliable but so also should the system formarking it. A multiple choice question paper is usuallyvery reliable and feasible, but because it essentiallytests factual recall it may not be very acceptable tocandidates and would be unlikely to have high validityas a measure of clinical skills.'9 Essay questions tend tohave low reliability (particularly because of difficultiesin standardising the marking) and surprisingly lowvalidity. Structured short answer papers have highervalidity because more precise instructions can be givento the candidate.20No single examination format will guarantee accept-

ability, feasibility, validity, and reliability, but care inidentifying the strengths and weaknesses of eachapproach and clear objectives for the assessmentshould help staff select a useful range of tests.

ConclusionsAssessing medical students should help them focus

their learning during the course, identify individualstrengths and weaknesses, provide an opportunity forimprovement, highlight deficiencies in the content ordelivery of the medical course, and, ultimately, protectthe public against incompetent graduates. To do allthis the assessment system must contain a large andproperly managed formative element. The summativeassessment must be criterion referenced. There mustbe adequate feedback between staff and students toensure that all potential gains are obtained from theassessment system.

GlossarySummative assessment: Testing acquired knowledge at the end

of a course.

Formative assessment: Regular testing of progress throughouta course.

Norm referenced tests: Scores are ranked on a normal distribu-tion and the pass mark adjusted to achieve a predeterminedpercentage failure rate.

Criterion referenced tests: Pass mark is predetermined, and thepass rate is allowed to vary with absolute achievement.

Validity: Extent to which a test measures what we want it tomeasure and nothing else.

Content validity: Measure of whether the test contains arepresentative sample of the items that we wish to assess.

Construct validity: How well a test measures the feature ofinterest. (If students do better on a test after studying acourse aimed at teaching a particular skill we have someevidence of the construct validity of that test for assessingthat skill.)

Criterion referenced validity of a test is judged by assessing howwell the test results correlate with another accepted assess-ment method.

Reliability: Measure of the consistency and accuracy withwhich a test measures what it is supposed to.

1 Osler W. Examinations, examiners, and examinees. Lancer 1913;ii: 1047-59.2 Newble DI, Jaeger K. The effect of assessments and examinations on the

learning of medical students. Med Educ 1983;17:165-71.3 Tooth D, Tonge K, McManus IC. Anxiety and study methods in preclinical

students: causal relation to examination performance. Med Educ 1989;23:416-21.

4 Verwijnen GM, Imbos T, Snellen H, Stalenhoef B, Pollenmans M, Luyk S,et al. The evaluation system of the medical school of Maastricht. Assessmentand Evaluation in Higher Education 1982;3:225-44.

5 Ende J. Feedback in clinical medical education.JAMA 1983;250:777-81.6 Pallie W, Carr DH. The McMaster medical education philosophy in theory,

practice and historical perspective. Med Teach 1987;9:59-71.7 Turnbull JM. What is ... Normative versus criterion-referenced assessment?

Med Teach 1989;1:145-50.8 General Medical Council Education Committee. Recommendations on basic

medical education. London: GMC, 1980.9 Towle A. Critical thinking: the future of undergraduate medical education.

London: King's Fund Centre, 1991.10 Jolly B, Wakeford R, Newble D. Implications for action and research. In:

Newble D, Jolly B, Wakeford R, eds. Certification and recertification inmedicine. Cambridge: Cambridge University Press (in press).

11 Petrusa ER, Blackwell TA, Rogers LP, Saydjari C, Parcel S, Guckian JC. Anobjective measure of clinical performance. AmYMed 1987;83:34-42.

12 Newble DI. Eight years' experience with a structured clinical examination.Med Educ 1988;22:200-4.

13 Malik SL, Manchanda SK, Deepak KK, Sunderam KR. The attitudesof medical students to the objective structured practical examination.Med Educ 1988;22:40-6.

14 Roberts J, Norman G. Reliability and learning from the objective structuredclinical examination. Med Educ 1990;24:219-23.

15 Harden RMcG, Stevenson M, Downie WW, Wilson GM. Assessment ofclinical competence using objective structured examination. BMJ 1975;i:447-51.

16 Harden RM, Gleeson FA. Assessment of clinical competence using anobjective structured clinical examination. Med Educ 1979;13:41-54.

17 Painvin C, Neufeld V, Norman G, Walker I, Whelan G. The triple jumpexercise-a structured measure of problem solving and self-directedlearning. Annual Conference on Research in Medical Education 1979;18:73-83.

18 Newble DI, Hoare J, Elmslie RG. The validity and reliability of a newexamination of the clinical competence of medical students. Med Educ1981;15:46-52.

19 Levine HG, McGuire CH, Nattress LW. The validity of multiple choiceachievement tests as measures of competence in medicine. AmericanEducational Research oumrnal 1970;7:69-82.

20 Feletti GI, Engel CE. The modified essay question for testing problem-solvingskills. MedjAust 1980;i:79-80.

54 BMJ VOLUME 306 2 JANUARY 1993

on 20 March 2020 by guest. P

rotected by copyright.http://w

ww

.bmj.com

/B

MJ: first published as 10.1136/bm

j.306.6869.51 on 2 January 1993. Dow

nloaded from