bloom’s levels · mcqs with vignettes can test higher levels of knowledge than simple mcqs. 2....

Scenario-based multiple-choice

that assess knowledge at higher

Bloom’s levels

Janet Lindsley, PhD Associate Professor of Biochemistry,

Assistant Dean for Curriculum

Member of the National Board of Medical Examiners USLME Step 1 Integrated Review Committee

Past Step 1 Item Writing Committee

Workshop objectives

By the end of this workshop, participants will be able to:

1. identify the Bloom's taxonomy levels at which well written multiple choice questions can assess knowledge,

2. distinguish between well written and poorly written multiple choice questions,

3. create well written multiple choice questions which assess knowledge higher Bloom's levels.

Bloom’s taxonomy for the

cognitive domain

Handbook I: Cognitive (Bloom et al. 1956)

Image accessed on 9/21/15 http://www.maggiehosmcgrane.com/2014/09/flipping-grade-4-and-flipping-blooms.html

https://en.wikipedia.org/wiki/Bloom's_taxonomyCITEREFBloom_et_al.1956

Analyze the following NBME

questions to identify key

elements/similarities

A 64-year-old man comes to the physician because of swelling in his feet for the past 2 years. He says that his skin is dry and itchy and his feet "feel heavy." One of his legs is shown. Which of the following is the most likely cause of his condition?

(A) Arteriolar constriction and arteriolar hypertension (B) Arteriolar dilation and venous hypertension (C) Venous constriction and arteriolar constriction (D) Venous hypertension and incompetent valves (E) Venous hypertension and venous constriction

2015samples_step1.pdf

A 12-year-old girl with a 1-year history of systemic lupus erythematosus is brought to the physician for a routine follow-up examination. During the past year she has done well with the exception of occasional mild frontal headaches, fatigue, and arthralgias; the results of regular laboratory evaluations have been stable. She has missed 20 days of school in the past 4 months. Her parents express concern that sending her to school might cause her to be exposed to children with contagious diseases that might exacerbate their daughter's condition. The girl tells the physician that she misses seeing her many friends at school. Passive motion of the elbows, wrists, and knees produces mild discomfort. Physical examination shows no other abnormalities. Which of the following is the most likely cause of this patient's excessive school absences?

(A) Lupus arthritis

(B) Lupus cerebritis

(C) Malingering

(D) School phobia

(E) Vulnerable child syndrome


A placebo-controlled clinical trial is conducted to assess whether a new antihypertensive drug is more effective than standard therapy. A total of 5000 patients with essential hypertension are enrolled and randomly assigned to one of two groups: 2500 patients receive the new drug and 2500 patients receive placebo. If the alpha is set at 0.01 instead of 0.05, which of the following is the most likely result?

(A) Significant findings can be reported with greater confidence (B) The study will have more power (C) There is a decreased likelihood of a Type II error (D) There is an increased likelihood of statistically significant findings (E) There is an increased likelihood of a Type I error


A 32-year-old woman with type 1 diabetes mellitus has had progressive

renal failure over the past 2 years. She has not yet started dialysis.

Examination shows no abnormalities. Her hemoglobin concentration is

9 g/dL, hematocrit is 28%, and mean corpuscular volume is 94 μm3. A

blood smear shows normochromic, normocytic cells. Which of the

following is the most likely cause of her anemia?

(A) Acute blood loss

(B) Chronic lymphocytic leukemia

(C) Erythrocyte enzyme deficiency

(D) Erythropoietin deficiency

(E) Immunohemolysis

(F) Microangiopathic hemolysis

(G) Polycythemia vera

(H) Sickle cell disease

(I) Sideroblastic anemia

(J) β-Thalassemia trait


Key elements/similarities?

Appropriately constructed MCQs can

be effective at assessing higher level

cognitive skills

• MCQs with clinical or research vignettes can assess ability to:

– Analyze information (data interpretation)

– Synthesize information

– Apply knowledge to solving novel problems

Of course, the MCQs should be aligned with curricular learning objectives.

UUSOM current practice of tagging

questions by “Bloom’s level”

• Recall of factual knowledge

• Applied knowledge

• Data Interpretation

– Including interpretation of images

• Scientific (Research) scenario

– Complex interpretation of research data

• Clinical Scenario

– Complex interpretation of clinical data

A 7-month-old infant is brought to the physician's office because of poor

weight gain despite large food intake. He has had two episodes of

pneumonia and has frequent bulky stools. He coughs frequently. X-rays

of the lungs show increased markings and hyperinflation. Trypsin is

absent in a fresh stool sample, and the fat content is increased. Which of

the following is the most likely cause of this infant's disorder?

(A) Autoimmune disorder

(B) Defective ion transport at epithelial surfaces

(C) Disaccharidase deficiency

(D) Inability to synthesize apolipoprotein B

(E) Villous atrophy of the jejunum


The stem poses a clear question; it is usually possible to arrive at the answer with the options covered.

Answer choices are homogeneous.

Anatomy of a USMLE-style MCQ

A 4-year-old boy is brought to the physician because of slow growth during the

past year. He has had recurrent urinary tract infections since the age of 1 year.

He is at the 10th percentile for height and 25th percentile for weight. Physical

examination shows pallor. Laboratory studies show a normochromic, normocytic

anemia and increased serum concentrations of urea nitrogen and creatinine.

Urinalysis shows a low specific gravity. Which of the following sets of additional

serum findings is most likely in this patient?


Calcium Inorganic

Phosphorous

1,25-Dihydroxycholecalciferol Erythropoietin

A

B

C

D

E

F

Tables of answer choices can be

helpful

A 4-year-old boy is brought to the physician because of slow growth during the

past year. He has had recurrent urinary tract infections since the age of 1 year.

He is at the 10th percentile for height and 25th percentile for weight. Physical

examination shows pallor. Laboratory studies show a normochromic, normocytic

anemia and increased serum concentrations of urea nitrogen and creatinine.

Urinalysis shows a low specific gravity. Which of the following sets of additional

serum findings is most likely in this patient?


Calcium Inorganic

Phosphorous

1,25-Dihydroxycholecalciferol Erythropoietin

A

B

C

D

E

F

Balancing answer choices prevents

cuing to the correct answer

Avoid giving away the answer by

cuing

Good questions are not

purposefully tricky or deceptive

Summary: A baker’s dozen of “rules” for writing

effective, NBME-style MCQs 1. MCQs with vignettes can test higher levels of knowledge than simple MCQs.

2. Avoid pseudo-vignettes, in which the vignette is not needed.

3. Question stems should be long and answer choices short and homogeneous (e.g. all mechanisms described in similar depth).

4. Don’t be negative. (No “which of the following is NOT...”)

5. Tables of answers can be useful.

6. Use as many distracters as necessary to avoid giving away the answer.

7. Questions should never be purposefully tricky or deceptive.

8. Order answers alphabetically or numerically.

9. Avoid giving away the answer. (No cuing by having the right answer longest or more detailed, spelling or grammatical errors in distractors, having 2 choices similar and one being correct.)

10.Students should be able to answer the question before reading the options (No “which of the following is true?”)

11.The lead-in question should include “most likely” (or similar).

12.No questions that would result in malpractice.

13. Avoid “buzz” words.

Gather ideas throughout the year

Application of these concepts

How could this question be improved?

A 68-year-old man with no known cardiovascular disease has normal adult hemoglobin that displays cooperative interactions between its polypeptide subunits. Which of the following functional characteristics of hemoglobin does this cooperativity most strongly affect? a) Fraction of hemoglobin present as methemoglobin in the peripheral circulation b) Fraction of hemoglobin present in tetrameric form in erythrocytes c) Fraction of hemoglobin subunits bound to a heme group d) Fraction of hemoglobin's bound oxygen transferred from erythrocytes to tissues

Better? A female newborn was referred for hemoglobin analysis after a newborn screen detected a HbF or HbA variant. DNA sequence analysis revealed a heterozygous alpha-globin Asp126Tyr mutation. Functional analysis of this mutant protein gave the following data. What is the most likely consequence of this mutation?

A. Decreased fraction of globin chains with heme attached

B. Decreased methemoglobin fraction

C. Decreased oxygen transfer from erythrocytes to tissues

D. Increased fraction of globin chains with heme attached

E. Increased methemoglobin fraction

F. Increased oxygen transfer from erythrocytes to tissues

Patient

Reference

Better? A 68-year-old man with chronic obstructive pulmonary disease (COPD) is brought to the ED because of confusion and lethargy. His pulse is 96/min, respirations 22/min, blood pressure 148/82 mm Hg. Physical exam shows erythema of the skin and mucosal membranes, as well as cyanosis of the nail beds. There is no splenomegaly. Lab studies show hemoglobin of 17 g/dL and hematocrit of 49%. Arterial blood gas on room air reveals the following:

pH = 7.33

pCO2 = 58 mm Hg

pO2 = 61 mm Hg

HCO3- = 36 mEq/L

This patient most likely has a decrease in which of the following relative to a healthy patient?

A. affinity of hemoglobin for oxygen

B. ferritin concentration

C. heme synthesis

D. percent HbA1c

E. red blood cell volume

A 26-year-old woman presents with pustular skin lesion on the right arm for 2 days and is found to have acute arthritis of the left knee and right wrist. Gram stain of the pustule is shown in the attached image. Which of the following is the most likely initial portal of entry for this pathogen in this patient? a) Cutaneous b) Gastrointestinal c) Genital d) Pulmonary e) Urinary

Good question?

A 25-year-old woman with cystic fibrosis is receiving care for her condition in a patient-centered medical home. Which of the following practices best exemplifies the patient-centered aspect of care in a medical home model? a) Delivering outpatient and emergency room care b) Facilitating access to the emergency room when symptoms are exacerbated c) Following national guidelines for care d) Reducing cost of prescription medications *e) Teaching self-management of symptoms based on condition severity

How could this question be

improved?

A 25-year-old woman with cystic fibrosis is receiving care for her condition in a patient-centered medical home. Her care is most likely to differ from traditional services by including: a) Coverage for both outpatient and emergency room care. *b) Instruction on how to manage her GI and respiratory symptoms. c) Only protocols that follow US Preventative Services guidelines for care of patients with cystic fibrosis. d) Rapid access to emergency room care when she experiences exacerbated shortness of breath. e) Reduced cost of Pancreatic enzymes, Fluticasone, Albuterol, nebulized DNase and Montelukast.

Better?

A 5-year-old boy was brought to the emergency room, unconscious, and in shock after he accidentally overdosed on his mother's iron pills, mistaking them for sweet-sour candy. He is hyperglycemic with leukocytosis and the empty container recovered from the scene suggests that he had ingested about 33 ferrous sulfate tablets. How would you treat this patient? a) Chelation therapy with dimercaprol b) Detoxification with activated charcoal *c) Gastric lavage with deferoxamine d) IV infusion with British Anti-Lewisite e) Oral succimer, then pennicillamine

Good question?

How could this question be improved?

A 20-year-old man presents to the ER after an automobile accident. Radiography shows extensive fractures of the skull, including a fracture and partial closure of the foramen ovale. This fracture would damage the nerve that supplies all of the following muscles, EXCEPT: *a) Buccinator b) Lateral pterygoid c) Masseter d) Medial pterygoid e) Temporalis

Better? A 20-year-old man presents to the ER after an automobile accident. Vital signs are T 97.3 F, RR 22/min, HR 105/min, BP 110/75 mm Hg. Radiography shows extensive damage to his skull, including a fracture and partial closure of the foramen ovale. This fracture most likely results in nerve damage to which of the following muscles?

Buccinator Lateral pterygoid

Masseter Medial pterygoid

Temporalis

A Yes Yes Yes Yes No

B Yes Yes Yes No Yes

C Yes Yes No Yes Yes

D Yes No Yes Yes Yes

E No Yes Yes Yes Yes

What is one thing that you are

going to start, stop or continue

doing as a result of this workshop?

References

• NBME item writing guide (see: http://www.nbme.org/publications/item-writing-manual-download.html)

Questions?

bloom’s levels · mcqs with vignettes can test higher levels of knowledge than simple mcqs. 2....

Documents