mcgill workshop

The Development of Multiple-Choice Questions using the

Key-features Approach

Claire Touchie, MD, FRCPCUniversity of Ottawa

Medical Council of Canada

Conflict of Interest

Medical Education: none

2

Workshop Agenda

1. Introductions (10 min)

2. What is the key-features approach (20 min)

3. Exercise 1: Defining key-features (15 min)

4. How to write multiple choice items (25 min)

5. Exercise 2: Writing multiple-choice questions (30 min)

6. Leg Stretch (10 min)

7. Large group discussion: Review of MCQs (30 min)

8. Exercise 3: Technical flaws (15 min)

9. What are technical flaws (15 min)

10. Wrap up and evaluations

3

Workshop ObjectivesDescribe what can be tested with multiple-choice

items

Define the anatomy of a multiple-choice item

Define and identify technical flaws

Create multiple-choice items for own stated purpose

Define and criticize poor performing items

4

Why are we doing this?

Which one of the following is true about pseudogout?

1. It occurs frequently in women.2. Seldom associated with acute pain in a joint3. May be associated with a finding of

chondrocalcinosis.4. It is hereditary in all cases5. It responds well to treatment with allopurinol

Why are we doing this? A 62 year-old woman with a history of confusion and constipation comes to

the office for a follow-up visit. Laboratory investigations reveal a serum

calcium of 2.9mmol/L, a creatinine of 146 µmol/L, and a hemoglobin of 108

g/L.

Which one of the following is the most likely diagnosis?

1. Hyperparathyroidism

2. Chronic renal failure

3. Multiple myeloma*

4. Vitamin D intoxication

5. Renal cell carcinoma

Key Feature Problems•Based on the concept of “Case Specificity”

– The clinical performance on one problem is NOT a good predictor for performance on other problems

•Assessment is best served when focusing exclusively on the unique challenges (key features) in the resolution of each problem

– Essential issues or specific difficulty

Key Feature Problems•First discussed at Cambridge Conference – 1984•Developed by Georges Bordage and Gordon Page for the MCC•First incorporated into the MCCQE Part I – 1992•Known under different names

– Q4, Clinical Reasoning Skills (CRS), Clinical Decision Making (CDM)

So what? Who cares?•Studies show validity evidence and reliability in testing settings•Wenghofer et al. (Med Educ 2009)

– Candidates in the bottom quartile had a 3-fold increase in the risk of an unacceptable quality-of-care assessment outcome (OR 3.41)

Key Feature Problems

Assesses decision-making skills NOT recall of factual information

Knowledge

Application of knowledge

Clinical Decision

Application of Knowledge•To elicit clinical clues•To formulate diagnostic impressions•To order investigative or f/u procedures•To acquire data to monitor a course of action OR evaluate the severity/probability of an outcome•To select a management course

Example of Knowledge question•Which of the following are characteristic of delirium?

Alternate type of question

Should assess the ability to:– Recognize delirium tremens in a specific patient

• An example of a “clinical reasoning” issue

– Prescribe appropriate therapeutic measures• An example of a “clinical decision” issue

Key Feature – Definition

•A critical or essential step in the resolution of a problem•A step in which examinees are most likely to make errors in the resolution of a problem•A difficult or challenging aspect in the identification and management of the problem in practice

Advantages of the Key Feature Approach

•More discriminating•Shifts the emphasis from

– The method of assessment to the object of assessment

– Assessing all aspects of solving a problem to assessing only the essential element

Key Question of Key Features

“What are the critical, essential elements in the resolution of the problem?”

Process in Key Feature Development•Problem definition•Selecting a key feature•Developing a clinical scenario•Identifying the correct answer (in the case of single correct MCQ)•Identifying plausible distractors

How can this be applied?

Let us try…

Problem DefinitionSelect an objective or a clinical problem…delirium/confusion

Select a clinical situation– Undifferentiated complaint– A single typical problem*– A multi-system problem– A life-threatening event*

– Preventive care and health promotion

Selecting a Key Feature•Ask the question

– What are the critical essential elements in the resolution of the problem?

•Key feature 1– Given a patient with post-operative delirium, ask about

EtOH consumption

•Key feature 2– Given a patient with post-operative delirium, recognize

delirium tremens and manage with benzodiazepines

Exercise #1

Writing key features

21

How to write multiple-choice items

1. The What

2. The How

22

The What• What can I test with MCQs?

• Knowledge• Clinical-decision making

• Clearly define the purpose of your exam• Define what it is you want to test

• For the overall test• For your specific question

23

The What

Prior to writing your question, ask the following questions• What concept do I want to test?• Where does the learner go wrong?

• Focus on areas of “challenge” for the learner

24

Example of the What

Purpose: To assess the clinical clerk’s knowledge and decision making capability at the end of an Internal Medicine six week rotation• Concept/Objective: Management of

CHF• Challenge to the learner: ???

25

The How – Anatomy of a MC item• Stem

• Clinical vignette which describes the setting, the patient’s age and complaint along with pertinent historical facts, physical exam details, and/or laboratory findings

• Lead-in question• The task

• Answer and alternatives• The most correct answer and the plausible distractors

26

The How – Anatomy of a MC item

Stem• A 58-year old man presents to the ED with

sudden onset of left-sided chest pain associated with shortness of breath, palpitations and dizziness. His past history is relevant for a recent diagnosis of lung carcinoma. His examination is only remarkable for a heart rate of 112/minute.

27

The How-Anatomy of a MC item

Lead-in question• Which one of the following diagnostic test

would be most useful to confirm the diagnosis?

28

The How – Anatomy of a MC item

Correct/Best answer and distractors

1. Chest radiograph

2. CT of the chest *

3. Sputum culture

4. Electrocardiogram

5. Echocardiogram

29

What did you notice about this question?

Could you answer it without seeing the alternatives?

Could you answer without being given the diagnosis?

30

The Stem• Short description of a clinical scenario

• Common or clinically important• Clear and contains relevant information to the

clinical problem – avoid window-dressing

• Word the stem positively• Avoid EXCEPT questions• Use negative words with caution

• Eg: contraindication, what to avoid

31

The Stem• Provide sufficient information to answer

the item• DO NOT create tricky items by omitting

essential information• DO NOT add extraneous information• Stem should be a clinical vignette

32

Lead-in Question• Ensure the directions are very clear with

a clear task• Can the stem be administered in a short answer

(constructed-response) format? • “Cover answer test”

Lead-in Question

• Different clinical tasks can be tested • Can be done with the same stem (cloning of

question)• History• Diagnosis • Investigations• Management/Treatment/Drug therapy• Counseling

Lead-in QuestionTry asking questions that lead to clinical decision-making?

Which one of the following– … is the most likely diagnosis?– … investigations would you now order?– …is the next step in the work-up of this patient? – … is the most important step in the initial management of

this patient?

Distractors• Only one right choice with distractors• Number of distractors is a policy decision (3? vs. 4?)• Use plausible distractor choices• Keep distractors independent, they should not be overlapping

• E.g.: 1. 11-20; 2. 15-30

• Keep distractors homogeneous in content and grammatical structure

• Keep the length about equal• Avoid specific determiners such as All, Never, Always,

Completely and Absolutely• Do not use All of the Above or None of the Above

Item testing clinical decision-makingA 62 year-old woman with a history of confusion and constipation comes to

the office for a follow-up visit. Laboratory investigations reveal a serum

calcium of 2.9mmol/L, a creatinine of 146 µmol/L, and a hemoglobin of 108

g/L.

Which one of the following is the most likely diagnosis?

1. Hyperparathyroidism

2. Chronic renal failure

3. Multiple myeloma*

4. Vitamin D intoxication

5. Renal cell carcinoma

Item testing clinical decision-makingA 62 year-old woman with a history of confusion and constipation

comes to the office for a follow-up visit. Laboratory investigations

reveal a serum calcium of 2.9mmol/L, a creatinine of 146 µmol/L, and

a hemoglobin of 108 g/L.

Which one of the following would help confirm the diagnosis?

1. Parathyroid hormone

2. Serum protein electrophoresis*

3. 25-OH vitamin D

4. Serum creatinine

5. Abdominal ultrasound

What is wrong with this item?A previously healthy person suddenly presents withpleuritic pain in the left chest and shortness of breath.Which one of the following is the most likelydiagnosis?

1. Mycoplasma pneumonia2. Spontaneous pneumothorax3. Pulmonary embolism4. Acute pericarditis5. Epidemic pleurodynia

What is wrong with this item?Which one of the following is true about pseudogout?

1. It occurs frequently in women.

2. Seldom associated with acute pain in a joint

3. May be associated with a finding of chondrocalcinosis.

4. It is hereditary in all cases

5. It responds well to treatment with allopurinol

40

What is wrong with this item?Aortic insufficiency may be caused by all of the

following, EXCEPT:

1. syphilis

2. Marfan’s syndrome

3. aortic dissection

4. bacterial endocarditis

5. myocardial infarction*

Exercise #2 – 30 minutes

Writing MCQ items• Pair up with a partner• Using the MCQ item development worksheet,

develop MCQ items that will be useful to you

42

Leg Stretch

43

Exercise #2 – Pre-test

A Test of General Rock and Roll Knowledge

44

Technical Flaws

Violations of test item writing principles• Flawed items are usually more difficult• Fail more students

45

Downing, 2002

Technical Flaws• Unfocused items• Negative stem or lead-in question• Heterogeneous options• Logical or grammatical cues• Long correct answer• Word repeats• Convergence strategy

46

Unfocused itemWhich one of the following is true about pseudogout?

1. It occurs frequently in women.

2. Seldom associated with acute pain in a joint

3. May be associated with a finding of chondrocalcinosis.

4. It is hereditary in all cases

5. It responds well to treatment with allopurinol

47

Negative stem or lead-in question

48

Which of the following does not cause aorticinsufficiency? 1. syphilis2. Marfan’s syndrome3. aortic dissection4. bacterial endocarditis5. myocardial infarction*

Heterogeneous option

49

A 24-year-old female presents to a walk-in clinic with fever, flank pain, frequency and dysuria. The urinalysis (urine microscopy) shows 1+proteinuria, 25 white blood cells per high power field and a few granular casts. Which one of the following investigations is the next best step?

1. Intravenous pyelography.2. Intravenous antibiotics.3. Creatinine clearance.4. Midstream urine culture.*5. Oral analgesia.

Logical cuesA 47-year old man present with an acute episode of psychosis. Which one of the following treatment would you consider prescribing?

1. Alprazolam

2. Lorazepam

3. Haloperidol*

4. Diazepam

5. Quetiapine

50

Grammatical cuesA 78-year old man undergoes a thoracentesis for a large pleural effusion. Three hours later, he develops sudden onset of shortness of breath. What is your most likely diagnosis?1. Reaccumulation of fluid

2. Pneumothorax*

3. Lung infection

4. Bleeding

5. Blood clot

51

Word repeatsAlso known as “clang association”:

A 45-year-old woman presents with sudden loss of consciousness. On

exam, her vitals are normal, she is not pale and she is not diaphoretic.

Which one of the following is more typical of “fainting” as a conversion

symptom than of a syncopal attack due to orthostatic hypotension?

1. Bradycardia.

2. Muscle twitching.

3. Absence of pallor and sweating.*

4. Urinary incontinence.

5. Rapid recovery.

52

Convergence strategy

53

An 86-year-old woman fell at the local nursing home and sustained an intertrochanteric fracture of her left hip. On clinical examination, you would expect to find her left leg:

1. Shortened, abducted and internally rotated.2. Lengthened, abducted and internally rotated.3. Shortened, adducted and externally rotated.4. Shortened, abducted and externally rotated.*5. Lengthened, abducted and externally rotated.

Post-test

A Test of General Rock and Roll Knowledge

54

Now review the items you have written using the checklist!

55

In Summary

Prior to writing MCQ items:• Determine the purpose of the test• Determine WHAT you want to test• Use key-features to help you develop your

questions

Write questions avoiding technical flaws

56

References• Guidelines for the Development of Multiple-Choice

Questions at http://www.mcc.ca/pdf/MCQ_Guidelines_e.pdf• Haladyna TM, Downing SM, Rodriguez MC. A Review of

Multiple-Choice Item-Writing Guidelines for Classroom Assessment. Applied Measurement in Education 2002;15:309-334

57

Thank you!

Questions? Comments?

ctouchie@mcc.ca

58