reflections on the migration of medical simulators from training tools to assessment tools rich...
TRANSCRIPT
Reflections on the Migration of Medical Simulators from Training
Tools to Assessment Tools
Rich ShavelsonStanford Education Assessment Lab
Invited TalkSimulation in Medical Education Seminar
Stanford Hospital
March 14, 2007
Overview
• Motivation for Talk
• Analogy and Implications: Medical Simulators and Job Performance Measurement
• Evaluation of Technical Quality of Simulator Assessments
• Open Discussion
Motivation For Talk• Requests for advice on how to score
performance collected with medical training simulators– SUMMIT—Surgical laparoscopic
simulators• Creating scores from data reported by
simulator• Defining reference groups• Defining performance benchmarks• Technical quality
– Pediatric airway management simulator• Separating training and assessment• Developing a profile of performance• Task sampling• Technical quality
ê ê ê ê êê ê ê ê ê ê ê ê ê ê ê ê ê ê ê ê ê ê ê
2 4 6 8 10
Attempt
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
25.00
50.00
75.00
100.00
Proficiency
W
W
W
W
W
W
WW W W
W 11W 12W 13W 14W 15W 16
W 18W 19W 20W 21W 22W 23
W 24W 26W 27W 28W 30
Subject
ê ê ê ê ê ê êê ê ê ê ê ê
1 2 3 4 5 6 7
Attempt
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
0.00
0.10
0.20
0.30
0.40
Conceptually-developed proficiency score
W
WW
W
W
W
W
W
WW
W
W
W
WW
WW
W
W
W W
W
W
W
W W
W
W
W
W
W
W
W
W
W
W
W
W
W
W
W
W
W
W
W
W W W
W
W
W
W
W
W
WW
W WW
W
W WW
W
W
W W
W
W
W
W
W WW
W W
W
W
W W
W
W
W
W
W WW W
W 11W 12W 13W 14W 15W 16
W 18W 19W 20W 21W 22W 23
W 24W 26W 27W 28W 30
Subject
ê ê ê ê ê ê êê ê ê ê ê ê
1 2 3 4 5 6 7
Attempt
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
ê
-40.00
0.00
40.00
80.00
Proficiency
W
W W W
WW
W
W
W
W
W
W
W
W WW W
W
W
WW
W W
W
WW
WW
W
WW
W
W
W
W
W
W
W
WW
W
W
W
W
WW W
W
W
W
W
WW
W
WW
W
W
W
W
W W
W
W
W
W W
W
W
W
W
W W
W
W W
WW
W W
WW W
W
WW
WW
Before
Ideal & Possible Solution
After Imple-menting Solution
Analogy and Implications: Medical Simulators and Job Performance Measurement
• A great deal of work has been done on job & education performance measurement that applies to medical simulators as assessment tools.
• My enduring interest
Analogy Continued
• Approaches to performance measurement– Construct definition– Task sampling
definition– Rapprochement
• Cold reality
Universe of Performance of Interest
Universe of Possible Tasks for Assessment
Universe of “Do-Able” Tasks
Universe of “Do-Able” Task Formats
Tasks & Scoring on Test
Evaluation Of Technical Quality
• Reliability– Classical test theory (“Cronbach’s Alpha”)—
simple task sampling– Generalizability theory—task sampling– Item response theory—construct driven
• Validity– Construct– Content– Predictive
Thanks…Your Turn!