education&debate - bmj

EDUCATION & DEBATE

MRCGP: examining the exam

Fiona Godlee

Medical colleges have a reputation for being elitist,exclusive, and closed to scrutiny. Not so the RoyalCollege ofGeneral Practitioners. Recently I went at thecollege's invitation to observe the proceedings of itsmembership oral exam. Having taken the membershipexam of the Royal College of Physicians and watchedwhile friends were made miserable by the cramming,grilling, and failing, I was intrigued to see how generalpractitioners-examiners and examinees-would be-have under exam conditions.

I was due to sit in on some vivas but started the dayby joining the examiners at their briefing session. Avideo of a previous oral exam was being shown tocalibrate the examiners with each other. By a show ofhands they registered their opinions of the candidate'sperformance. Most judged it "satisfactory" but therewas a broad range from "very good" to unsatisfactory."The convener of the oral exam, Peter Tate, a GP inOxfordshire, greeted me at the end of the meeting."We want to be perfectly open," he said. "We don'twant to hide anything from you." And no, he wasn'tworried by the range of opinion we had just witnessed.He saw it as a potent reminder of the potential forsubjective judgments.

Exam content and reliabilityPeter Tate explained that it was to reduce the

potential for subjectivity that the college first set up theexam. In its early days the college selected its membersby recommendation and interview. Now GPs have topass a multiple choice paper testing factual knowledge,a modified essay paper testing ability to define andsolve problems, and a critical reading paper beforebeing allowed to sit the vivas.

First set up in 1965, when all five candidates passed,the exam has changed remarkably little despiteconstant scrutiny from college examiners and outsideadvisers. Its -design was the result of advice soughtmainly from the Royal College ofPhysicians with inputfrom American psychometric experts. The examinersseem pleased with the result. "It has the advantage ofhaving been thought out from scratch," said MikeRuscoe, an examiner I met later in the day, implyingthat other colleges' exams, lumbered with years oftraditionalism, do not.There have, however, been changes. Two thousand

candidates now take the exam each year, and the passrate has fallen from that first all time high to an almostconstant 75%. The traditional essay paper has beenscrapped-"It was poor at discriminating and difficultto mark," said Mike Ruscoe-and replaced by thecritical reading paper. In this the candidates must showtheir grasp of basic statistics and study design. Itincludes a scientific article for criticism, a questionabout recent coverage of a clinical problem in theacademic press, and a circular, such as a letter from thechief medical officer, which candidates must evaluatein terms of its impact on their practice."The new paper is easier to mark and has large

standard deviations, which means that it's good at

Not elitist, exclusive, or closea to scrutiny: the Royal (Gollege ofGeneral Practitioners, London

separating the sheep from the goats," said MikeRuscoe. "We found that candidates were not well read,but already the critical reading paper seems to bechanging behaviour. The standard of reading is goingup."

Standard deviation and statistical reliability are buzzwords among RCGP examiners, and they make nosecret of their own ratings. The reliability of an examreflects the statistical likelihood of the same candidatesachieving the same results if they were to sit the examagain. Calculated from the standard error, the (tcoefficient of reliability should be around 0-8 for a testto be acceptable. The RCGP's (x coefficient has been0-81 for the past two years. The RCGP is the onlymedical college in Britain to make such figures publiclyavailable.

"Reliability is terribly important," said PhilipTombleson, the examination convener and a GPin Sussex, over coffee. But he sees problems withbecoming too rigidly tied to it. "In America reliabilitybecame so paramount that candidates started takingexaminers to court if the figures weren't good enough.The colleges reacted by dropping all forms of clinicalassessment, relying entirely on MCQs. MCQs are veryreliable, good at testing factual knowledge in a repeat-able way. Testing clinical ability is more of a problem."

Testing clinical skillsIt is a problem with which the RCGP is still

grappling, so far with only limited success. Unlike

BMJ VOLUME 303 27 JULY 1991

British Medical Journal,London WC1H 9JEFiona Godlee, MRCP,assistant editor

BMJ 1991;303:235-8

235

on 6 Decem

ber 2021 by guest. Protected by copyright.

http://ww

w.bm

j.com/

BM

J: first published as 10.1136/bmj.303.6796.235 on 27 July 1991. D

ownloaded from

http://www.bmj.com/

other medical colleges, in its exam there is no directclinical assessment. The organisers plead mitigatingcircumstances. "The cornerstone of general practice isthe consultation. More than anything else GPs must begood at communicating with their patients," said oneexaminer. "This is not easily tested in an exam. It'scertainly not tested in the membership." I thoughtback to my membership exam. She was right. We wererepeatedly reminded not to hurt the patient, especiallyif the examiner was looking. But I could remembernothing that might have tested my skills at breakingbad news or counselling difficult patients.

Philip Tombleson believes that the RCGP's oralexam-two half hour vivas with two pairs ofexaminers-does test clinical skills indirectly. But he agreed thatsome better surrogate for bedside manner was needed.One possibility is the OSCE (objective structuredclinical examination), already used by the Irish Collegeof General Practitioners. In this section of theexam, worth 25% of the total marks, candidatesare observed interacting with volunteer role playerstrained to act out standardised scenarios.The Irish exam now includes 13 OSCE "stations."

Brian Coffey, former chairman of the Irish college'sexamination subcommittee and an RCGP examiner, isenthusiastic about the results. "A typical roleplaymight be a mother whose child has scabies. It's anembarrassing problem. She's angry. The candidate

must deal with her anger and give her advice." OSCEsare good at showing up bad candidates, but they aredifficult to judge objectively. Data from the ICGP'slast three years' experience are due to be publishedsoon."The Royal College has a number of objections to

OSCEs," said Brian Coffey, a large, gently spoken manwho, according to his college colleagues, none the lessknows how to put a candidate on the spot. "But themain one is the logistical problem. The Irish college ismuch smaller. Lining up enough patients or volunteersfor 2000 candidates would be quite some under-taking." The OSCE is also thought by some in thecollege to fragment the consultation, making it un-realistic for an established GP who is used to dealingwith the whole patient.A development of the OSCE is the idea of simulated

surgeries, currently being researched by anotherRCGP examiner, Elizabeth Bingham, who practises inBerkshire. Her enthusiasm for the idea was evident.Instead of following the candidate around, sheexplained, the examiner would enter and leave theroom with the patient or volunteer. The candidatewould remain in one place, seeing a number of patient/examiner pairs and being interrupted by simulatedphone calls from health visitors and hospital labora-tories. It all sounded great fun, but wouldn't it besimpler just to go and inspect GPs in their own

Examining the examiners

It may be a comfort to GPs planning to sit this examthat the scrutiny they will be under is nothingcompared to the critical analysis endured by theirexaminers. After lunch I sat in on an examiners'debriefing session. Seven of us-four examiners,Peter Tate, Richard Wakeford (the Cambridgepsychologist who advises the examination board onpsychometric and statistical matters), and me-squeezed into a small attic room already crowdedwith television screensand video recorders. (Earlier,as part of his warm up act to put the first batchof candidates at their ease, Peter Tate had jokedthat without video cameras it wouldn't be theMRCGP.)

Peter and Richard led the discussion. In the videoof the first viva one of the examiners had been, theythought, irritated with the candidate. Her bodylanguage had given her away. She agreed. She hadfelt bad at the time about not being able to hide herannoyance. But the candidate had kept skirting theissue and waffling. The discussion turned fromovercoming a personal reaction to a candidate to theneed to pin candidates down and come back tothings they have managed to avoid answering.These sessions-a combination of peer assess-

ment, psychotherapy, and group bonding-are aregular feature oflife for the college's 125 examiners.But to have reached this stage they must alreadyhave leapt through several hoops. Potentialexaminers are put forward by their local collegefaculty. They must have been in general practice forat least five years and must resit the written exam,achieving not less than one standard deviationbelow the mean. My obvious surprise at this lowlevel of requirement seemed to amuse RichardWakeford. "Examiners don't need to be brilliant,"he said. "But they must be able to make criticaljudgments about other people." To test this theyare asked to mark and rank order sample writtenpapers, with the assessors on the lookout foraberrant judgments.

Would be examiners then undergo mock vivasand further direct assessment by senior examiners.They are then approved or not (about one in eightare not) and embark on an 18 month probationaryperiod. During this time they are always pairedwith an experienced examiner and receive regulardebriefing sessions. GPs are not ideal examinermaterial, according to Richard Wakeford. "Some-times they are just too nice," he said. "They aretrained to be reactive and facilitative. They have tocoax information out of patients. To be an examineryou have to learn to be more adversarial."The college's wholehearted commitment to

psychometric method reflects the influence over theyears of the Oxford Group, Michael Balint, and thepsychologists David Pendleton and David Swanson.Richard Wakeford is the current interpreter of thesacred texts, lacing his pithy contributions to thevarious meetings throughout the day with statisticsfrom validated research. The need for such re-assurance is obvious. The examiners are, afterall, attempting the impossible-the creation ofa standardised, entirely subjective assessment ofhuman beings who are intrinsically unstandardisedby human beings who are intrinsically not objective.How, for example, should one arrive at a final markfor a candidate who has performed inconsistently invarious parts ofthe exam? Here, after much debate,the only answer seemed to be to resort to that oldfashioned, all human criterion, gut feeling. Or, asthe psychometrician would have it, "a gut mediated,weighted summary."

In an individual such relentless self analysismight be thought of as verging on the obsessive.Indeed the RCGP has often been criticised for toomuch contemplating of its navel. I doubt, though,that such comments greatly disturb Richard Wake-ford or the college's examiners. They can at leastclaim, with good justification, to be trying to findthe best way forward. And they remain above alltheir own most radical critics.


on 6 Decem


http://ww

w.bm

j.com/

BM


ownloaded from

http://www.bmj.com/

surgeries? Philip Tombleson thinks not. "A realsurgery may be just a lot of colds and sore throats," hesaid. "Simulation allows you to explore differentaspects of competence."

But some examiners would still like to see the realthing. Peter Tate is looking into the possibility ofcandidates submitting videos showing actual consulta-tions. Getting consent from the patients and thepracticalities of making the video are not, he thinks,insurmountable obstacles.

The vivaThe format of the vivas is another area of contro-

versy. At present they contain factual questions aswell as questions that test more general competence.Examiners have in front of them a grid on which thesubject discussed is set against various areas of compe-tence. These include such things as management,prevention, practice organisation, and communication.The examiner might say, for example, "Tell me abouthow you deal with diabetes, and I'm interested particu-larly in the area of practice organisation." Anothersubject, say hypertension, might be discussed as itrelated to methods of communication. In this way theexaminer aims to obtain an overview of the candidate.A recent report, however, commissioned from the

Centre for Medical Education in Dundee, concludedthat the viva should contain no factual questions.Factual knowledge should be entirely the preserve ofthe written papers, leaving the oral exam to testpersonal interactional skills. The Dundee report wasjust being absorbed when I was at the college, and therewere some dissenting voices. "The danger of taking thefacts out of the viva," said one examiner, as we sat inthe bar at the end of the first session of vivas, "is that asocial worker could get a distinction in our oral exam."The examiners are asked to spend not longer than

five minutes on each subject, allowing coverage of atleast 12 subjects in total. The aim is to sample widelyacross a range of subjects, looking for gaps in thecandidate's knowledge rather than attempting togeneralise from more in depth testing on one or twoareas. The college is well aware that the exam mayachieve breadth at the expense of depth. But, as PeterTate says, "skilled examiners can go very deep veryquickly."

Provided the candidate is equally skilled, orschooled, all goes smoothly. I witnessed a particularlycompetent performance from an army GP practisingnear Salisbury Plain. Phrases like psychotherapeuticdialogue (rarely if ever heard in the colleges ofphysicians and surgeons) were rewarded by a nod ofrecognition from the examiners. Consultation modelsrolled off the well coached examinee's tongue-RogerNeighbour, Stott and Davies, Bendick's therapeuticdialogue, transactional analysis. "They are all the thingat the moment," said the candidate when we talked onthe terrace afterwards. "You have to know thembackwards. "

Learning to passSo after all the MRCGP exam is not so different

from the MRCP or FRCS exams. In place of thestylised examination and presentation of findings,rehearsed on longsuffering friends, partners, pillows,and repeated even in sleep, here are equally styliseddemonstrations of problem solving behaviour andemotional interaction. The examiner picks an anno-tated card from a card index and conjures the scene: aman with a knife in his bag demanding pethidine.What are your options? What are the implications ofeach option? Which would you choose? What are theadvantages and disadvantages of that choice? The

mnemonic OICAD, meaningless but memorable,reminds successive candidates: options, implications,choice, advantages, disadvantages.As with any exam some candidates will pass because

they have learnt how to pass the exam rather thanbecause they are good doctors. The concern of theexaminers must be to distinguish those who will dowhat they say they would do from those who are merelygood at saying they would do it. At the midday meetingthe examiners discussed this difference between"stated and actual behaviour." They agreed that theimportant thing was not simply to accept a correctanswer but to make candidates justify the actions theysay they would take. There was much talk of "nailing"the candidate, not letting him or her off the hook whenthings became uncomfortable.Watching the vivas it was clear to me that these

examiners knew how to nail a candidate if the needarose. And I felt again the sickening discomfort of myown membership viva. I remember resorting to "I'mafraid I don't know, sir," more than a few times as theexaminers' questions breached the limits ofmy know-ledge. Such candid admission of ignorance left themlittle choice but to change the subject.And yet I enjoyed every bit of my membership

exam-yes, even the viva. Whether due to nostalgiaand the thrill of passing, or the enduring confidenceboost of having those letters after your name, or thefeeling-far less enduring-that you were, in some

ways, as learned in medicine as you would ever be, thewhole exercise seemed, and still seems, worth while. Iwas certainly a better doctor after the exam than beforeit. Friends and colleagues have less rosy memories.

Some remain cynical about the whole process ofprofessional exams, especially exams which come afterlong years of training, when failing would require a

complete change of course.But since the days of the barber surgeons profes-


Only half of Britain's 31 000 GPs currentlybelong to the college. The proportion is, how-ever, likely to increase as about 80% ofvocationaltrainees sit the exam each year. Because it relatesexclusively and without apology to Britishgeneral practice it attracts few overseas candi-dates, unlike the more exportable MRCP andFRCS exams.

I wondered what the long term aim of thecollege was; whether it planned to push for allGPs to take the exam or to leave it as it stands-avoluntary form of self assessment with no limiton the number of times a candidate can sit it.Philip Tombleson thinks it could easily beadapted to become an exit exam for vocationaltraining, especially since the introduction of theGMC's indicative register for all specialties. Atpresent, to have the initials GP after yourmedical degree requires no specialist exam.There could, however, be resistance to the

idea of compulsory membership from GPs whoare not members ofthe college and who see it as asource of unwelcome controversy and "aggro."A less politically controversial role for the exam,ofwhich Philip Tombleson strongly approves, isas a stimulus for continued academic progress.The college even offers members the chance totake the exam again free of charge. Apparentlysome have done so. He also welcomes the formalassessment, recently introduced, for fellowshipof the college. In the past fellows were proposedand voted in after a number of years as amember.

237

on 6 Decem


http://ww

w.bm

j.com/

BM


ownloaded from

http://www.bmj.com/

sionalisation has been all about certification and exclu-sion. An exam should exist to improve standards, butbecause it must also acknowledge achievement of arequired standard some people must always fail. Ifeveryone passed where would be the point? As theregistrar of the Royal College of Physicians says in hisspeech to new members, "The MRCP is not an easyexam. Nor, now that you have passed it, would youwish it to be."The question, then, is not whether to examine

doctors' competence, but how to. And it is a questionthat should be openly debated. I have been assured byphysicians and surgeons who examine for the member-ship and fellowship exams that selection and training ofexaminers is rigorous and that both exams undergointense internal audit. But secrecy fuels speculationand might even justify cynicism. The confidence withwhich the RCGP opens its doors to outside scrutinysends an unmistakable challenge to other colleges to dothe same.

The Royal College of Psychiatrists, 150 years on

Henry Rollin

On 27 July the Royal College of Psychiatrists willcelebrate its 150th birthday. With a century and a halfunder its belt the college can legitimately claim to bethe oldest association of psychiatrists in the world.The original association was the brainchild of

Dr Samuel Hitch, resident superintendent of theGloucestershire General Lunatic Asylum. In a circularletter dated 19 June 1841 and addressed to 83 visitingphysicians and resident superintendents of 26 asylumsand hospitals in England, seven in Scotland, and 11 inIreland he suggested that such an association beformed. His opening paragraph is well worth quotingfrom because in essence the aims and objects ofwhat heproposed are, with some additions and modifications,the same as those of today's college. It reads: "Ithaving been long felt desirable that medical gentlemenconnected with lunatic Asylums should be betterknown to each other. .. shall communicate more freelythe results of their individual experience... should co-operate in collecting statistical information relating toinsanity-and, above all, should assist each other inimproving the treatment of the insane...."Ofthe gentlemen approached only 47 replied but, of

these, 44 were in favour and three against. It was areasonably encouraging response, sufficient for Hitchto press ahead and to organise a preliminary meeting athis hospital on 27 July 1841. There it was decided thatan association on the lines suggested in Hitch's letter beestablished and be known as the Association ofMedicalOfficers of Asylums and Hospitals for the Insane.The fledgling association got off to a pretty poor

start. It was described by one of its founder members,Mr John Thurnam, in 1845 as "this little society"-asad description, but apt. The attendance at irregularlyheld meetings was meagre; rarely did it reach doublefigures. But included among those who did attend, orperhaps more exactly, were able to overcome thedifficulties so that they could attend, were stalwartswho have become known as heroes of British psy-chiatry: John Conolly (Hanwell), Samuel Hitch(Gloucester), Samuel Gaskell (Lancaster), JohnThurnam (The Retreat, York) and John Bucknill(Exeter). In 1851, however, under the great JohnConolly a renaissance took place. A meeting at theFreemasons' Tavern in London drew an attendance of26 and was followed by an even more successfulmeeting a year later in Oxford. Since then the sequenceof meetings has been unbroken to this day.Three important and far reaching events occurred in

mid-century. Firstly, at the York meeting in 1844 itwas decided to publish a journal, although the AsylumJ7ournal-the forerunner of today's British Journal ofPsychiatry-did not appear until November 1853,under the spirited editorship of Dr (later Sir) JohnBucknill. The journal, once established, became for so

*A~

0

John Connolly, supernenwdent, ofHantwell asylum-one of the heroesofBritishpsychtiay

many members marooned in their institutions the solemeans of communication; indeed, the "glue" thatbound the association together. Secondly, at theLondon meeting in ,1854 a permanent parliamentarycommittee was formed, the first positive step designedto influience legislation affecting the control of asylumsand the welfare'of the patients commnitted to them. Thethird even't was the institution of the presidency of thecollege. In 1854. Dr A J Sutherland of St Luke'sHospital, London, was the first to be elected., to befollowed by a continuous line of presidents all, withoutexception, acknowledged leaders of British psychiatry.With the building of new county asylums under -the

provisions of the two acts ofparliament of 1845 (8 and 9Victoria, CIOO and, C126) the membership of theassociation rose'to 250 in 1864 and to 523 in 1894. Itwas decided in 1883-as an index of an increasedinterest in the -affairs of the association and possibly asan attempt at decentralisation-that, in additionto annual meetings, quarterly meetings be held inScotland a'nd Ireland. These were the forerunners ofdivisional meetings (the college now has 10 divisions),which have continued to be an important feature of theorganisation of the college.

In 1865, allegedly at the suggestion ~of Henry

Royal College ofPsychiatrists, 17 BelgraveSquare, London SWIX8PGHenry R Rollin, FRCPSYCH,'Quondam honorary librarian

BMJ 1991;303:238-9

238 BMJ VOLUME 303 27 JULY 1991

on 6 Decem


http://ww

w.bm

j.com/

BM


ownloaded from

http://www.bmj.com/

education&debate - bmj

Documents