Medical Decision Making

Jeffrey Young, MD, FACSProfessor of Surgery

Director, UVA Trauma CenterChief Quality Officer

University of Virginia Health System

Problems

• How do we train medical students and residents to make effective medical decisions?

Problems

• Education in Medicine is poorly structured with respect to what is known about medical expertise

• Reliance on standardized tests as a measure of operational competence is, at the least, not supported by any data

• How do we know when trainees are able to accomplish more complicated (and risky) tasks? (other than that the page of the calendar flipped?)

Learning vs. Teaching

• We need to learn from other high-risk processes– Military– Nuclear power– Maritime

• Many people have died over the years to benefit training in all these industries

Military Domain

• Simulation essential to preparation• Recreating mindset that soldier will

face in battle is vital• Use entire range of simulation

– Tabletop to highly complex high-fidelity simulation

• Also after-action examination of decisions is common with aggressive search for errors

Learning vs. Teaching: Medicine

• No other high risk domain would have people right out of training show up for work, hand them a list of responsibilities, and say “good luck!”

• All other high risk domains to try assess some level of competence in the domain before turning people loose

Learning vs. Teaching

• Creating sets of simulations, with minimum performance standards, for the medical domain would not be difficult

• Close supervision where the teacher intimately watches the performance of the trainee, and can correct small errors– Happens in OR, not outside

Expertise

What is expertise?

• Nate Fick on Marine training– Hardness = “The ability to face an overwhelming situation

with aplomb, smile calmly at it, and them triumph through sheer professional pride”

• Ability to face new situations and bring experience to bear to make effective decisions

• Ability to manipulate environment to carry out tasks– Just knowing what to do without the ability to implement

action is not effective

Deliberate Practice

• Key concept in gaining expertise• Practicing situations where the same

mental and physical processes will be used

• Ameliorates anxiety, insecurity• Allows clearer thinking during live

events

Lessons from Other Disciplines

• Baseball• Music

– Outliers – The Beatles

• Military• Fire operations• Dispatch

9-11

• FDNY fire dispatch– Early response to event was executed according to

pre-set plan– First Chief Officer on scene – First day as Battalion

Chief• Assessed situation rapidly and effectively• Organized command post• Dispatched first units

– Initial response was well-organized due to repeated practice, lessons learned from earlier attack, and simulations

Analysis

• FDNY had simulated major incidents at the WTC for many years

• The commander, though junior, was stationed near the WTC, and had participated in many of the drills

• Dispatch had rigid procedures for major incidents and automated actions that went into effect for different alarms

Errors

• Though mistakes were made, they mostly occurred when circumstances went far outside practiced parameters– These situations had not been

simulated – Time pressure turned out to be

extreme, where simulations did not routinely include that variable

Lessons

• “Tactical catastrophes are rarely the outcome of a single poor decision. Small compromises incrementally close off options until a commander is forced into actions he would never choose freely”– Rounds vernacular: “good, OK, whatever”

• Rarely will people be able to improvise effectively, under pressure, with unique unpracticed circumstances

• Mistakes most commonly occur in those settings• Simulation must be robust and complex to be

effective

Our Research Purpose

• To simulate common clinical situations encountered by physicians and nurses to improve medical decision making under pressure

• To avoid epiphenomenon associated with high fidelity simulators

• To provide deliberate practice – “Luck is not method, and neither is hope. Hard work is.”

• To improve decision making and confidence, and decrease cognitive errors

Designing medical simulation to enhance decision making

• Focus on variables in, and orders out– Not a technical simulation

• Wide variety of cases• Pressure and stress should be part of

simulation• Should get subjects outside of comfort zone

so they can begin to practice for those situations

• Provide feedback on performance and teach subjects how to improve performance through heuristics and other techniques

Proactive vs. Reactive: The Effect of Experience on Performance in a

Critical Care SimulatorJeffrey S Young, MD, Robert L Smith, MD,

Jayme B Stokes, MD and Chadrick E Denlinger, MD

Presented, Association for Surgical Education, 2005

Published in the American Journal of Surgery, 193(1): 100-104, 2007.

Hypothesis

• Increased experience, as measured by months of clinical rotations, will decrease errors in critical care simulations

Methods

• Subjects grouped by ICU experience:– Novice < 9 wks ICU– Intermediate (IM) ≥ 9 < 20 wks ICU– Expert > 20 wks ICU

• All scenarios were digitally recorded for later data analysis.

• Recall, cognitive strategy and errors in management were compared.

Demographics

*p<0.05

1Y4.3±3.913.9±2.95*24±5.73 GS, 1 EM32±4.84Expert(>10

weeks ICU

experience)

2Y5.1±2.05.4±3.6*18.2±8.84 GS, 1 EM28±1.65N-I(0-10 weeks ICU

experience)

AOASleep in past 24 hours

ICU WeeksMonths of Postgraduate

Education

ResidencyAgeNGroup

* p<0.05 #p =0.07 21.8±9.9*3±2.73.2±1.30.75±0.96*19.8±2.2#Expert (>10)

4±2.6*4±23.2±1.65.6±3.5*15.2±3.96#N-I (0-10)

Proactive Steps

Backward Reasoning

Forward Reasoning

Cognitive Errors

Key Clinical Tasks

ICU Weeks

* p<0.05 #p =0.07

Weeks of ICU Experience

Months of Residency

9.25±6.44±2.93.3±0.951.75±2.217.25±3.9>24

14±14.53.2±1.83.2±1.84.8±517.2±4.40-24

Proactive StepsBackward Reasoning

Forward Reasoning

Cognitive Errors

Key Clinical Tasks

Months Residency

All comparisons not significant.

Conclusions

• We uncovered a unique finding that residents with >10 weeks of ICU experience initiated a large number of proactive actions immediately after being presented with the patient information, while those with less than 10 weeks experience in the ICU rarely performed these actions.

• In addition, residents with this degree of experience committed significantly fewer cognitive errors.

• These differences may play a role in efficiency, cost, and overall outcome in the care of ICU patients

War Games

Evolution

• These were instituted as educational sessions for residents on the Trauma and SICU services in 2002

• They began as an informal session where I would present cases off the top of my head and take the subject through the case

Level 1 Data PointsProblem Respiratory arrest on floorPresentation 58 year-old female 3 days s\p admission for partial SBO. Called by nurse for difficulty breathingInitial actions Go to room 1

Arrange information and help 1Initial assessment Airway Vomit in mouth, labored respirations, unarousable 1Intubation required Yes 1Intubation actions Etomidate 0.5

Succinylcholine 0.5Suction 0.57.0 ET tube 0.5Check tube position and breath sounds 1Anesthesia backup 0.5

Additional intial interventions IV access 1Vital signs 120/90, P95, T37, Sats 100 1

Secondary Actions ICU bed 1Labs required CBC HCT 42, WBC 18 1

Lytes Bicarb 19 1Troponin 0.5 1

Tests required EKG ST changes laterally 1CXR RLL infiltrate 1ABG 7.34/40/200 1

Primary Diagnosis Aspiration 1Secondary Diagnosis MI 1Follow-up Indicates where patient will go 1

Notifies senior resident 1MaxTotal 20.5Total

PERFORMANCE OF STUDENTS AND RESIDENTS IN BASIC CLINICAL SCENARIOS: A DISTURBING ANALYSIS

Journal of Trauma, 63(3): 556-564, September, 2007

Hypothesis

• In simulations of straightforward urgent clinical situations, subjects with increased critical care experience will perform a higher percentage of recommended actions that more inexperienced subjects

Methods• Medical students and residents rotating

on the trauma and surgical ICU services participated.

• Sessions were conducted by an examiner and included the initial nursing report of an unstable patient. Twelve scenarios were created, focusing on basic floor emergencies. Scores were assigned for clinical actions ordered. These were divided into the initial evaluation (ABC’s), secondary evaluation (labs, studies), diagnosis, follow-up, and total score.

Methods

• The scenarios were validated by two critical care attendings, and EM attending, and two critical care fellows and these scores were used as the expert group.

• Scores were assigned by two examiners, and the average of the grades in each area was used.

• The scores are a ratio of actual to possible correct responses in each section, and in the entire exercise.

Results• Eighty subjects participated in the

simulations. • The group was divided into third-year

medical students (MS3), fourth-year students (MS4), first-year residents (PGY-1), and residents beyond their first year (PGY2+). There were 20 subjects in each group.

• Five experts were used as validation– One Emergency Medicine attending, two critical care

attendings and two critical care fellows

Total Score

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

MS3 MS4 PGY1 PGY2 Expert

Clinical Level

To

tal P

erce

nt

Act

ion

s C

om

ple

ted

*

*

**

Conclusions

• All groups had significant deficits in cognitive performance compared to experts in the areas of secondary evaluation, follow-up of the presenting problem and total performance in simple clinical scenarios.

• We must design educational systems that rapidly enhance the cognitive performance of students and residents before they are left to independently diagnose and intervene in life-threatening clinical situations.

USE OF “WAR GAMES” TO ENHANCE HIGH-RISK CLINICAL DECISION MAKING IN RESIDENTS AND STUDENTS

American Journal of Surgery, June, 2008

Hypothesis

• Subjects who have participated in multiple war game sessions will outperform naïve subjects

• Subjects with greater clinical experience will have augmented gains in performance with multiple sessions

Methods

• Similar to previous study except that we examined all subgroups, including subjects performed after undergoing more than one session

• Each subject always receives a different case at each session

Results

• There were 20 subjects in each group and five experts (n=85).

• In total, 227 simulations were performed and graded.

• In naïve subjects, MS3, MS4, PGY-1, and PGY-2+ performed significantly worse than experts in the simulations.

• However, in those subjects who completed 3 “war games” session, all in different scenarios, their performance in every category did not differ significantly from the experts.

Performance by Sessions

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Initial Evaluation Secondary Evaluation Diagnosis Follow-up Total

Per

cen

t C

om

ple

ted

No Sessions

1-2 Sessions

>2 Sessions

Expert

Discussion• All groups initially had significant deficits in

cognitive performance compared to experts in these relatively simple high-risk clinical scenarios.

• We have designed an educational system that rapidly enhances the cognitive performance of students and residents.

• The clinical benefit of this laboratory finding is unknown but it may represent an important tool in assessing and enhancing the competencies of trainees in a safe environment

Critical Incident Interviews

Interviews

• Based on studies performed by Gary Klein in the late 80’s on NICU nurses, fireground commanders, and marines

• Formed basis of “Naturalistic Decision Making” and “Recognition Primed Decision Making”

RPD

• RPD reveals a critical difference between experts and novices when presented with recurring situations.

• Experienced people will generally be able to come up with quicker decision because the situation may match a prototypical situation they have encountered before.

• Novices, lacking this experience, must cycle through different possibilities, and tend to use the first course of action that they believe will work.

• The inexperienced also have the tendencies of using trial and error through their imagination.

Recognition Primed DM

• Experts rarely (if ever) weigh options against each other (differential diagnosis)

• They see a situation, compare it to situations they have experienced previously, bring up how they acted in those situations and war game it in their head

Recognition Primed DM

• If the mental simulation works out well, they proceed

• If they see something wrong in the mental simulation, they alter the plan and simulate it again (may only take 4-5 seconds)

• If they still are not satisfied, they will go back to traditional comparative analysis of options

Critical Incident Interview

• Ask novice (year 1-2), “expert” residents (year 4-5) and attendings to describe an incident where they feel they made an exceptional good decision, and then an exceptional bad decision

• Interviewer takes the subject through a detailed process eliciting how they made the decision, and what factors were involved

Critical Incident Interview

• Hope this will help describe the “naturalistic” decision making processes in medicine, which up to know are poorly understood.

Lessons learned

What we have seen from novices

• Has learning changed?– One (or several) students finish case– Next person begins simulation

• As if nothing had previously occurred• Same exact mistakes are made

– Requires multiple iterations for changes to take

Final Thoughts

• Should residents be allowed to start on floor or in unit without some basic competency training and evaluation

• Beginning mandatory set of floor and ICU cases for trauma service, hopefully expand to entire residency

• We must find a way to bring this to the subject since getting people together in a room is becoming impossible

Final Thoughts

• In new world of duty hours, closer examination of outcomes, and decreasing clinical experience, diffuse learning experience may no longer be optimal

• Focused teaching through deliberate practice in isolated high risk clinical situations should be considered

Thank you