assessing operative autonomy combining theory and software to make evaluation easy
DESCRIPTION
Assessing operative autonomy Combining theory and software to make evaluation easy. Jonathan Fryer MD, Professor of Surgery, Feinberg School of Medicine, Northwestern University. Disclosures. I have made no financial gains from this project I may in the future - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/1.jpg)
Assessing operative autonomy Combining theory and software to make evaluation easy
Jonathan Fryer MD,Professor of Surgery, Feinberg School of Medicine, Northwestern University
![Page 2: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/2.jpg)
Disclosures
• I have made no financial gains from this project• I may in the future• I intend to continue work on this project regardless
2
![Page 3: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/3.jpg)
What is the most essential goal of surgical training?
3
![Page 4: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/4.jpg)
Operative Autonomy
• The ability to independently perform operations safely and effectively.
4
![Page 5: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/5.jpg)
The Problem
There is growing concern that graduating surgical residents are not achieving operative autonomy with essential procedures.
5
1. Bell RH. Why Johnny cannot operate. Surgery 146, 533–542 (2009).2. Mattar SG et al. General Surgery Residency Inadequately Prepares Trainees for
Fellowship: Results of a Survey of Fellowship Program Directors. Annals of Surgery September 2013 258, 440–449 (2013).
3. Coleman JJ et al. Early Subspecialization and Perceived Competence in Surgical Training: Are Residents Ready? Journal of the American College of Surgeons 216, 764–771 (2013).
4. Chen P. Are Today’s New Surgeons Unprepared? Well (2013). at http://well.blogs.nytimes.com/2013/12/12/are-todays-new-surgeons-unprepared
![Page 6: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/6.jpg)
The Problem
• To be able to fix it…… You have to be able to measure it.
6
![Page 7: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/7.jpg)
The Problem
• We don’t do a very good job of assessing residents in the OR.
7
![Page 8: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/8.jpg)
The Problem
– Currently, summative assessment of OR performance is based on:• # of cases logged by resident
– Role of resident in each case?
• Semi-annual global evaluations – Memory decay?
8
![Page 9: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/9.jpg)
The Problem
9
…asking busy surgical faculty to fill out complex assessment forms in a timely manner, doesn’t work.
![Page 10: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/10.jpg)
The Solution
A simple assessment tool that:• Assesses operative autonomy• Doesn’t impede surgical workflow• Facilitates high compliance and
prompt completion
![Page 11: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/11.jpg)
Theoretical Framework
• Inter-related constructs: – Supervision, Guidance, Autonomy, Performance
• Faculty Supervision (oversight) ≠ • Faculty guidance (physical or verbal help)
• 1 • Faculty Guidance = Resident Autonomy
• Resident Autonomy = ƒ (Resident performance)•
11
![Page 12: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/12.jpg)
The Solution
– With every case faculty:• Provide resident supervision.• Assess and document the
level of operative autonomy achieved by the resident.
• Progressively reduce the level of operative guidance they provide to resident.
![Page 13: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/13.jpg)
The “Zwisch” Scale
• 4 levels of operative guidance– Show & Tell– Active Help– Passive Help– Supervision Only
DaRosa, D. A. et al. A Theory-Based Model for Teaching and Assessing Residents in the Operating Room. Journal of Surgical Education 70, 24–30 (2013).
![Page 14: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/14.jpg)
Our method: PASS (Procedural Autonomy and Supervision System)
![Page 15: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/15.jpg)
Today
![Page 16: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/16.jpg)
Coming soon…
![Page 17: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/17.jpg)
Study Design: Participants and Setting
• Department of general surgery at a large academic hospital
• All teaching faculty underwent formal frame-of-reference training per published protocol1
• All general surgery residents and trained faculty raters eligible for inclusion
• IRB-approved
17
1George et al, J. Surg. Educ. 2013; 70
![Page 18: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/18.jpg)
Results: Feasibility
• A 1 hour rater training session is sufficient to achieve reliable and accurate ratings1
• 92% response rate using PASS
1George, B. C. et al. Duration of Faculty Training Needed to Ensure Reliable OR Performance Ratings. J. Surg. Educ. 70, 703–708 (2013).
![Page 19: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/19.jpg)
Results: PASS Sample (7 mos)
Number of Residents 31 By Year of Residency Year 1 Year 2 Year 3 Year 4 Year 5 9 6 5 5 6Number of Attendings 27Number of Procedures 1490 Number of Types of Procedures
127
19
![Page 20: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/20.jpg)
Results: PASS Sample
Relative Case ComplexityEasiest 1/3 Middle 1/3 Hardest 1/3193 (13.0%) 895 (60.1%) 402 (27.0%)
20
![Page 21: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/21.jpg)
Results: Validity: Zwisch Levels by PGY
21
p-values for sequential pair-wise
distributions
p=<.001
p=<.001
p=<.001
p=0.21
23.2%
![Page 22: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/22.jpg)
Results: Validity: Zwisch Levels by Complexity
22
p-values for sequential pair-wise
distributionsp=<.001 p=<.001
![Page 23: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/23.jpg)
Results: Validity: Zwisch Level by Prior Experience
23
p-values for sequential pair-wise
distributionsp=<.001
![Page 24: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/24.jpg)
Study Design: Data Collection
• Sample 2: Video Sample– 8 procedures video recorded for additional review
(subset of PASS sample)– Rated by operating faculty, in-person OR observer, and
video reviewer using Zwisch scale (blinded to other scores)
– Rated by 2 additional video reviewers using other OR assessment instruments (modified OPRS and O-SCORE)
24
![Page 25: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/25.jpg)
Results: Video Sample
25
Number of Residents 4 (PGY 2 to 5)
Number of Attendings 2Number of Procedures 8Number of Types of Procedures 5
2 Laparoscopic cholecystectomy2 Open inguinal hernia repair2 Parathyroidectomy1 Total thyroidectomy1 Laparoscopic ventral hernia repair
![Page 26: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/26.jpg)
Results: Reliability
• Inter-rater reliability– Zwisch ratings – Operating attending, OR observer, and video rater – ICC = .90, 95% CI = .72 - .98, p < .001.
26
![Page 27: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/27.jpg)
Item ρ p-value
Operative Performance Rating System (OPRS) Degree of prompting or direction -.92 .001 Instrument handling .94 .005 Respect for tissue .94 .005 Time and motion .94 <.001 Operation flow .95 <.001 Overall performance .95 <.001Ottawa Surgical Competency OR Eval. (O-SCORE) Knowledge of procedural steps .94 <.001 Technical performance .93 .001 Visuospatial skills .92 .001 Efficiency and flow .86 .007 Communication .92 .001
Results: Validity: Zwisch Level correlation with other OR assessment tools
27
![Page 28: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/28.jpg)
Benefits
• Faculty and residents constantly reminded of ultimate goal …. i.e. operative autonomy.
• Establishes a conceptual framework for teaching and learning in the OR.
• Data can be used to: – Help faculty and residents to set learning goals.– Help programs monitor operative progress and identify those
who may need additional attention.– Address regulatory requirements for OR supervision and
operative performance assessment.– Establish national norms
![Page 29: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/29.jpg)
Limitations
• So far, studied only at a single institution• Validity analysis based on small convenience sample• Raters not blinded to resident PGY level • Comparison with only selected items of OPRS and O-
SCORE• Unmeasured confounders (time of day, supervising surgeon
experience, etc)
29
![Page 30: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/30.jpg)
Conclusion
• The Zwisch rating scale is a reliable and valid measure of faculty guidance and resident autonomy
• Deployed on PASS the Zwisch scale can be used to feasibly record evaluations for the vast majority of operations performed by residents
30
![Page 31: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/31.jpg)
Vision
• All surgical subspecialties.• Other procedural specialties.• Other medical professionals who need to learn to perform
complex clinical tasks.• Other trades or professions where trainees need to learn
to independently perform complex tasks safely and effectively.
31
![Page 32: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/32.jpg)
AcknowledgementsSurgical Education Research & Development Team
Jay Zwischenberg
er
Eric HungnessShari Meyerso
nDebra DaRosaJonatha
n FryerEzra
Teitelbaum
Brian George
Mary Schuller
Research supported by:
Excellence in Academic Medicine Program from the State of Illinois
Augusta Webster Educational Innovation Grant from the Northwestern University
Center for Education in Medicine
![Page 33: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/33.jpg)
Theoretical basis
• Global assessment of performance is simpler, more accurate, and more reliable than checklists1
• Faculty guidance is related to resident performance2
• Faculty can accurately and reliably rate the amount of guidance provided to residents3
1. Regehr, G., MacRae, H., Reznick, R. K. & Szalay, D. Comparing the psychometric properties of checklists and global rating scales for assessing performance on an OSCE-format examination. Acad Med 73, 993–997 (1998).
2. Chen, X. (Phoenix), Williams, R. G., Sanfey, H. A. & Dunnington, G. L. How do supervising surgeons evaluate guidance provided in the operating room? The American Journal of Surgery 203, 44–48 (2012).
3. George, B., Teitelbaum, E., DaRosa, D., Hungness, E., Meyerson, S., Fryer, J., Schuller, M., Zwischenberger, J. Duration of Faculty Training Needed to Ensure Reliable O.R. Performance Ratings. Journal of Surgical Education 70(6), 703-708 (2013).
![Page 34: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/34.jpg)
Study
• Over 7 months• 1490 evaluations • 27 faculty • 31 residents
![Page 35: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/35.jpg)
Study Design: Rating Scales
• Zwisch• Procedural Complexity• Operative Performance Rating System (OPRS)1
– 6 general items only--excludes items that pertain only to specific procedures
• Ottawa Surgical Competency Operating Room Evaluation (O-SCORE)2
– 5 intra-operative items only--excludes items that did not pertain to intra-operative performance.
35
1Chen et al, The American Journal of Surgery 2012; 2032Gofton et al, Acad. Med. 2012; 87
![Page 36: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/36.jpg)
Results: Validity
• Convergent Validity for Guidance/Autonomy and Resident Performance– Zwisch level vs. PGY– Zwisch level vs. Complexity– Zwisch level vs. Resident Experience
• Construct Validity for Guidance/Autonomy– Zwisch level vs. OPRS guidance item
• Construct Validity for Resident Performance– Zwisch level vs. OPRS performance items– Zwisch level vs. O-SCORE performance items
36
![Page 37: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/37.jpg)
The Team
• Dr. Debra DaRosa• Dr. Brian George• Dr. Shari Meyerson• Dr. Ezra Teitelbaum• Mary Schuller• Dr. Nathaniel Soper• Dr. Joseph Zwischenberger
38
![Page 38: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/38.jpg)
Impact so far
• Over 1000 evaluations collected in 6 months• Response rate > 90%• Changes in teaching• They love to use it!
![Page 39: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/39.jpg)
Next steps
• Dictation of feedback• Reports
![Page 40: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/40.jpg)
Results: Validity
• Convergent Validity for Guidance/Autonomy and Resident Performance– Zwisch level vs. PGY– Zwisch level vs. Complexity– Zwisch level vs. Resident Experience
• Construct Validity for Guidance/Autonomy– Zwisch level vs. OPRS guidance item
• Construct Validity for Resident Performance– Zwisch level vs. OPRS performance items– Zwisch level vs. O-SCORE performance items
41
![Page 41: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/41.jpg)
Theoretical Framework
42
Helping Watching
![Page 42: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/42.jpg)
Next Steps
• I am actively trying to bring this to MGH• It needs additional development before it can be launched
here• Multiple other departments have already committed to
supporting this project
43
![Page 43: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/43.jpg)
Questions?
44
![Page 44: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/44.jpg)
1 2 3 4 5
Supervision Levels for PGY5 Residents0
4080
Num
ber o
f pro
cedu
res
Residents
Results
![Page 45: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/45.jpg)
Results
50% = 60 procedures
![Page 46: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/46.jpg)
Current Status
Milestone Achieved Cost / timeDevelopment of v1.0 mobile app
$200,000 / 8 months
Development of v0.9 administrative interface (beta)
$75,000 / 3 months
Integration with Northwestern EMR
$45,000 / 2 months
Development of v2.0 iOS app
$160,000 / 7 months (ongoing)
Total $480,000 + operational expenses
![Page 47: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/47.jpg)
Road Map
Planned Technical Milestones
Target launch date
v2.0 for iOS at Northwestern February 2014v1.0 Administrative interface at Northwestern
April 2014
v2.0 for Android at Northwestern
June 2014
System integration at MGH June 2014v2.0 iOS at MGH July 2014V2.0 Android at MGH October 2014
![Page 48: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/48.jpg)
12 month budget
Expense Item CostDesign and specification $30,000Software Development $225,000-$300,000QA testing $30,000Server hosting and maintenance
$25,000
User training $5,000Administrative $30,000Total $345,000 -
$420,000
![Page 49: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/49.jpg)
The “Zwisch” Scale
• 4 levels of guidance– Show & Tell– Active Help– Passive Help– Supervision Only
DaRosa, D. A. et al. A Theory-Based Model for Teaching and Assessing Residents in the Operating Room. Journal of Surgical Education 70, 24–30 (2013).
![Page 50: Assessing operative autonomy Combining theory and software to make evaluation easy](https://reader036.vdocuments.us/reader036/viewer/2022070421/56816273550346895dd2e364/html5/thumbnails/50.jpg)
Faculty Guidance ∝ 1Resident AutonomyResident Autonomy ∝ 1Faculty Guidance
Theoretical Framework
52
• Stritter FT et al., Handbook for the academic physician. 1986.
• Chen et al., The American Journal of Surgery 2012; 203• Gofton et al., Acad. Med. 2012; 87