assessment of preclinical medical students’ ‘core’ knowledge: use of it, mcq, emq john morris...
TRANSCRIPT
Assessment of preclinical medical students’ ‘core’
knowledge: use of IT, MCQ, EMQ
John MorrisIT learning group
E-Assessment workshop 29.3.07
Who am I?
• Trained as medic in Bristol 1961-1968, intercalated BSc
• Research & teaching anatomy: Bristol, Oxford since 1977
• NOT a ‘medical educationalist’; NOT an IT expert
• With increase in students to 150 class divided into 2 – half with e-learning programme; half discussing in DR
• Design of a ‘Principles of Clinical Anatomy’ course to move much of the clinically-relevant material as close as possible to the clinical course; 1 week after 3rd year FHS
• Heavily involved in the use of IT in medical education; databanks of questions, anatomical images
• DPS; responsible for moving Core examinations (Part A) questions in 1st BM from “short notes” to computer-markable in all subjects in preclinical medicine
Why do we have to do that?
• An external examiner noted a discrepancy in the marking of short notes answers
Why might we want to do it anyway?
• More reliable marking
• Less time in tedious marking
• Examining “smarter”
• Has a very beneficial effect on question setting; makes the examiner think very clearly what answer is required; ‘core’
• This implies that the course has a very clear definition of its ‘core syllabus
What do we want to assess in medical students?
All thinking processes must be based on underlying information
Do the students have the underlying information?
“Core” information; “Additional information”
Can they use that information in logical, deductive etc ways to arrive a diagnoses, treatments etc.
Can they show that they know the evidence on which their thinking is based?
Why use IT, MCQ, EMQ etc for assessment?
What it is NOT useful for?
Anything that requires students
• to produce examples of evidence
• to give examples of a basic principle from their own head
• to show evidence of wider reading beyond core information
• to make suggestions for ways forward
Why use IT, MCQ, EMQ etc for assessment?
What is it useful for? • Rapid, reliable assessment of ‘core’ factual knowledge
Very flexible in terms of different question types
Can design questions that require calculation, derivative thinking – we are moving much more toward that
Could be very flexible in terms of marking protocols; positive or negative marking for difficult parts, bad mistakes etc.
Formative questions and allowing student self-testing
What does Oxford use e-assessment for?
The Oxford medical school has made extensive use of on-line questions for three different purposes
Preclinical: in the e-learning programmes that are part of each
practical class and occupy students for half of any split class. Material is available for preparation before class (present experiment), and remains available for revision.
in the start of term formative assessments in summative assessments: experience of 3 years in
one course, and one cohort in full preclinical.
Clinical: in summative assessments
In the e-assisted learning packages:
as formative questions, with model answers available
as revision questions on the previous week’s work
For the formative questions, we often want students to produce answers with words they have formulated themselves, not just selected from a bank of options
This raises the problem of text recognition, which appears (to me) to be still in its infancy/puberty
In the start of term formative and summative assessments
Advantages:
colour illustrations
marking and feedback is ‘immediate’
can have statistical analysis of the ‘value’ of questions
staff marking time is minimal (essential for weekly PCA assessments); marking errors are avoided
Disadvantages:
only allows selection until text recognition is better
need in current formats to generate many false answers; can use “Which is not ..” but double-negative problem
takes a great deal of time to create (but future bank)
Problems
One problem with the use of true/false and extended match is the need to create ‘false’/incorrect distractor/options.
In formative assessments this makes it imperative to have immediate feedback of the correct answers so that we do not allow incorrect answers to be remembered and ‘learned’
Can partly overcome this with “Which is not ..” type question, but double negatives.
Problem of uncertainty in the material: those who know most might do less well unless question is phrased appropriately – use of the “single most appropriate answer”
Evidence from two years Clinical Anatomy results
Principles of Clinical Anatomy Perception Exam Marks - June 2004
129 - 133 Students
40
50
60
70
80
90
100
Mar
k (%
)
Week 1 Week 2 Week 3
65%
• Intended pass based on 10-11 of 15 5-part “short notes” marked correct – 65%
• Week 1 seemed fine
• Weeks 2 and 3 had much lower marks
• ? Students just slacking off/tired after finals ?
Study the questions very carefully in relation to the marks achieved
Principles of Clinical Anatomy Perception Exam MarksJune 2005
144 Students
40
50
60
70
80
90
100
Mar
k (%
)
Week 1 Week 2 Week 3 Resit
An unexpected advantage was the information for honing questions
Some illustrations from last year’s Principal of Clinical Anatomy course and assessment
The assessment at the end of each week consisted of 20 questions (2 per session; 2 sessions/day; 5 days)
Each questions whether true/false or extended-match had 5 options.
So, standard 100 questions per week’s work
CORE only for this qualifying exam
Marking was based on % correct answers of 100.
Simple true-false
True-false with illustration; derivative
This slide contained a MRI (abdominal cross section – sagittal plane), which requires Patient Consent and copyright permission in order to reproduce it.
Contacts for original image reference:Vivien Sieber ([email protected])
John Morris ([email protected])
Image reference:MRI abdominal cross section – sagittal plane
Text-based extended match
Illustration-based EMQ (a)
This slide contained a CT Scan (through upper abdomen – transverse plane), which requires
Patient Consent and copyright permission in order to reproduce it.
Contacts for original image reference:Vivien Sieber ([email protected])
John Morris ([email protected])
Image reference:CT Scan upper abdomen – transverse plane
Illustration-based EMQ (b)
This slide contained a medical photograph (Laproscopic view of a female pelvis), which
requires Patient Consent and copyright permission in order to reproduce it.
Contacts for original image reference:Vivien Sieber ([email protected])
John Morris ([email protected])
Image reference:Laproscopic image of female pelvis
Other types of EMQ - matching / single best answer
See “Pastest: EMQs for Medical Students (Feather et al)”A Basal skull fractureB Depressed skull fractureC Diffuse axonal injuryD Extradural haematomaE Glasgow Coma Scale 4 ……F Subarachnoid haemorrhageG Subdural haematoma
The patients below all have sustained head injuries. Please select the most appropriate clinical description from the above list.
1. A 36-year old man was the unrestrained driver of a car. On impact he was thrown against the windscreen. On examination he has bruising to the face and chest and has several obvious limb deformities. He opens his eyes to, and withdraws from pain. He is groaning and no words are discernable……….5, A 28-year-old professional boxer is brought into hospital immediately after a heavyweight world title fight. The paramedics in attendance …
Sophistication in marking
Can be as elaborate as you like – computer quick
negative marking of wrong answers
positive marking of difficult correct answers
variable marking throughout
need to gain e.g. 4 of the 5 parts f a question correct
Important to know; test out the procedures to be used. Can get unexpected results.
Demonstration of Nervous System Summative Assessment
Ranks of percentage scores
30
40
50
60
70
80
90
100
110
0 20 40 60 80 100 120 140 160
Rank
Sco
re (
%)
Histogram of percentage scores
05
10152025303540
0 10 20 30 40 50 60 70 80 90 100
Perentage scores
Fre
qu
ency
Histogram
0
5
10
15
20
25
30
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
Bin
Fre
qu
ency
Number questions passed 4/5
NOT FINAL DATAPercent pass of 16x5 (80)
Difference between overall % and number of questions passed.
Need to define very clearly precisely what is being tested