Discipline of General Practice

Item Writing and Quality Assurance in MCQ’s

Andrew W Murphy

Discipline of General Practice

February 25th 2013

Objectives

• By 1030 you will– Know

• Case and Swanson Golden Rules• Angoff Methodology• QuestionMark Metrics

– Be competent• Developing clinical SBA’s

– Be in love with• Virtuous cycle of SBA development, testing, evaluation,

standard setting, SBA deverlopment ….

Acknowledgements

• All at Discipline of General Practice

• St Georges Advanced Assessment Course 2010, 2012

– especially John Patterson (Barts)

– John Swanson (NBME)

– and Kathy Holtzman (NBME)

Format

• 0900 Introductions

• 0910 ‘The Bible’: Case and Swanson

• 0920 Activity #1: Write MCQ’s

• 0940 Discussion

• 0950 Standard setting

• 0955 Activity #2: Angoff Method

• 1010 Exam metrics: QuestionMark

• 1020 Discussion and Conclusions

• 1030 Finish

Case and Swanson:Golden Rules Single Best Answer (Pgs 33; 51-66)

1. Items should test application of knowledge rather than recall of isolated facts

2. Stems should be written in the form of a vignette

3. Items should have the right shape

1. long stems and short options

4. Lead-ins must be focused, clearly defining the task

5. Item stems should satisfy the ‘Cover-the-Options’ Rule:

1. You should be able to pose an answer without looking at the options

6. Options must be homogeneous so that they can be rank ordered along a single dimension

What rules are violated?

• Which of the following is true about pseudogout?

a. It occurs more frequently in women

b. It is seldom associated with acute pain in a joint

c. It may be associated with a finding of chondrocalcinosis

d. It is clearly hereditary in most cases

e. It responds well to treatment with allopurinol

Activity #1:

• Develop a SBA on the Oxford Handbook of General Practice (3rd Edition, 2010) material for

• Management

– Menopausal symptoms (Pg 708)

– Chronic childhood asthma (Pg 882-3)

• Aetiology

– Sore throat (Pg 932)

– Gastroenteritis (Pg 26)

Activity #1Discussion of suggested questions

• Menopausal symptoms

• Chronic childhood asthma

• Sore throat

• Gastroenteritis

Standard setting: Modified Angoff Methodhttp://www.cwea.org/cert_howcert_testresults_howpassdeter.shtml

• Uses expert judgements to determine the difficulty level of the test. The easier the test, the higher the pass point. Likewise, the more difficult the test, the lower the pass point.

• The following is a basic outline of the modified Angoff Method:

– A group of Subject Matter Experts (SMEs) independently rate each test question within a given test. The ratings are defined as the probability that a minimally competent person will answer the question correctly. A minimally competent person is defined as someone who adequately performs all job functions safely and requires no further training.

– The SMEs review each test question as group. A consensus is reached for the rating of each test question.

– After the data are refined, the final step is to calculate the average of all the test question ratings. This becomes the overall pass point estimation.

Activity # 2: Practice a modified Angoff Method

Q. 43 Menopausal symptoms

Elizabeth Gallagher is a 50 year old woman who is attending you for a routine cervical screening smear test. During the procedure she mentions to you that her periods are now only coming every second month. They are not very heavy. She also describes mild hot flushes. She is otherwise very well apart from paroxysmal atrial fibrillation for which she is on warfarin.

Which of the following is the most appropriate next step in her care ?

a. Prescribe clonidine

b. Recommend increased regular exercise

c. Recommend red clover

d. Prescribe HRT

e. Recommend black cohosh

Question 52: Chronic childhood asthma

• Imelda is a 3 year old who has asthma. She presents to you today with her Dad requesting a repeat prescription for her blue salbutamol inhaler. Overall her Dad says that her asthma is not well controlled. He has difficulty getting her to take the inhaler. He has not had any better success using the spacer device, as it frightens her. She is on no other medications. She coughs most days. She is otherwise well.

• Examination of her chest is normal. Her inhaler technique is poor, and she cries hysterically when she sees the spacer.

Which of the following is the most appropriate next step in her care?

a. Add in sustained release theophylline

b. Change her current inhaler to a long acting beta agonist (eg salmeterol)

c. Add in daily oral leukotriene receptor antagonist (eg montelukast)

d. Add in a regular inhaled steroid (eg beclametasone dipropionate)

e. Add in regular oral steroid (eg prednisolone)

Question 26: Aetiology of gastroenteritis

• You are working as a GP on call with ‘Ever-Ready Doc’ and at 0200 you meet Janice, a twenty four year old software developer, who has developed vomiting and diarrhoea acutely over the last 6 hours.

• She has vomited 5 times and has had two spells of watery diarrhoea. She is mildly dehydrated but still drinking and passing urine. Examination is unremarkable and urinalysis is normal. She had dinner at 7 PM in a local restaurant but did notice that her rice and vegetable was mixed warm and cold and tasted poorly. Her husband is well and ate only chicken and sweet corn soup. Her diet was unremarkable up to this meal and she has not been abroad recently.

Which of the following is the most likely pathogen:

a. Campylobacter Jejuni

b. Bacillus Cereus

c. Staphylococous Aureus

d. Salmonella Typhi

e. Escherichia Coli

Question 46: Sore throat

• Siobhan is a 47 year old teacher who has been complaining of a sore throat for two days. Her temperature is 38.5 0C and she has enlarged cervical lymph nodes and no cough. Her throat appears as shown.

Which of the following, according to the Centor criteria, is the most likely pathogen ?

a. Diphtheria

b. Group A β-haemolytic streptococci

c. Group B β-haemolytic streptococci

d. Infectious mononucleosis

e. Viral

Evaluation of examinationhttps://www.questionmark.com/us/perception/Pages/default.aspx

• The Questionmark® Perception® assessment management system enables trainers, educators and testing professionals to author, schedule, deliver, and report on surveys, quizzes, tests and exams.

• Author questions and organize them into exams, quizzes, tests or surveys

• Deliver via browser, secure browser, mobile devices or paper

• Analyze and distribute meaningful reports on results to stakeholders 

2 key metricshttp://www.utexas.edu/academic/ctl/assessment/iar/students/report/itemanalysis.php

• Item difficulty (p-value)– the percentage of students that correctly answered the item. The higher the

value, the easier the item. – The range is from 0% to 100%, or more typically written as 0.0 to 1.00. – Ideal value: Slightly higher than midway between chance (1.00 divided by

the number of choices) and a perfect score (1.00) for the item. • Item discrimination

– the relationship between how well students did on the item and their total exam score. The range is from –1.00 to 1.00.

– The higher the value, the more discriminating the item. A highly discriminating item indicates that the students who had high exams scores got the item correct whereas students who had low exam scores got the item incorrect.

– Acceptable range: 0.2 or higher

And another one!

• Reliability coefficient– a measure of the amount of measurement error associated with a exam

score. – The range is from 0.0 to 1.0. The higher the value, the more reliable

the overall exam score. – High reliability indicates that the items are all measuring the same

thing– Three ways to improve the reliability of the exam are to 1) increase

the number of items in the exam, 2) use items that have high discrimination values in the exam, 3) or perform an item-total statistic analysis

– Acceptable range: 0.60 or higher

QuestionMark Test Analysis Report

Item analysis report summaryAssessment name: gp60questionsDate report produced: 21 January 2012

Item analysis report summaryAssessment name: gp60questions

Discussion

• Developing, supporting and maintaining a bank of clinical questions

• Robust methodology e.g. Student Appeals

• Metric software: QPercom!

• Educational and technical expertise availability

• Student feedback

• Interdisciplinary developments e.g. GP/ Psych/ ENT

Conclusions

• By 1030 you now– Know

• Case and Swanson Golden Rules• QuestionMark Metrics• Modified Angoff Methodology

– Are competent• Developing clinical SBA’s

– Are in love with• Virtuous cycle of SBA development, testing, evaluating and

standard setting, SBA development….