neal e. seymour, md baystate medical center tufts university school of medicine baystate simulation...
DESCRIPTION
Evaluation of Surgical Skill Skills that are uniquely surgical Evidenced in OR Surgical patient management Observed during training events Clinical (e.g. OR) Laboratory Occurs throughout residency Progression of skillsTRANSCRIPT
![Page 1: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/1.jpg)
EVALUATING SURGICAL SKILL AND OR PERFORMANCE
Neal E. Seymour, MDBaystate Medical Center
Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab
2010 APDS Annual Meeting, San Antonio, Texas
![Page 2: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/2.jpg)
DisclosuresNo disclosures
![Page 3: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/3.jpg)
Evaluation of Surgical Skill Skills that are uniquely surgical
Evidenced in ORSurgical patient management
Observed during training eventsClinical (e.g. OR)Laboratory
Occurs throughout residencyProgression of skills
![Page 4: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/4.jpg)
Improved Task Performance
0
50
100
150
200
1 3 5 7 9 11 13 15 17
Tota
l Sco
re
Trial Number
VR Laparoscopic Suturing and Knot-Tying
Residents
Experts
Effect of VR Training on Laparoscopic Suturing Performance: Video Analysis
0
10
20
30
40
50
60
70
Pre-VR Post-VR
Video Score
Res.1Res.2Res.3Res.4Res.5Res.6Res.7Res.8Res.9
Effect of VR Training on Laparoscopic Suturing Performance: Time
0
50
100
150
200
250
300
350
400
Pre-VR Post-VR
Time (sec)
Res.1Res.2Res.3Res.4Res.5Res.6Res.7Res.8Res.9
![Page 5: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/5.jpg)
Proficiency-Based Lab Training
![Page 6: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/6.jpg)
p > 0.006*
Skills Transfer: “VR to OR” Study
0
1
2
3
4
5
6
7
8
9
10M
ean
num
ber o
f erro
rs
VR Trained Standard Training
*
![Page 7: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/7.jpg)
Assessment of Performance in OR
Procedure specific instruments GOALS OSATS Analysis of video recorded tasks Differences between investigative
instruments and assessment in frequent, routine use.Limitations on obtrusive methodsResource intensive
![Page 8: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/8.jpg)
Design of New Assessment Tool
User-friendly in routine use, valid, reliable.Opportunity for delivery of feedbackLeverage technology to manage large
volumes of data generated.Generate useful reports on resident
performance.
![Page 9: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/9.jpg)
Resident OR Performance Assessment at Baystate Medical Center
Objective: Develop OR performance assessment tool for routine use
OpRate: Dept. of Surgery and IS project:Multi-item global assessment of skillAttending surgeon raterEase of use = compliance with use.Interfaced to other databases
![Page 10: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/10.jpg)
Evaluated Items
3 preop (preparedness) questionsPatient, technical and disease-specific knowledge
6 open technical skills questionsTissue, instrument handling, knot tying, efficient
planning and ability to function independently 6 laparoscopic technical skills questions 4 point frequency-based responses for
specific technical performance behaviors
![Page 11: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/11.jpg)
![Page 12: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/12.jpg)
![Page 13: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/13.jpg)
Assessment of Residents with OpRateJuly 2004 - June 2006
15 29 21 36 3863
146101
237
113
050
100150200250
Appen
decto
my
Ventra
l Herni
a
Colon R
esec
tion
Cholecy
stecto
my
Inguin
al Hern
ia
Case
Num
ber
2004-5 2005-6
799 OpRate evaluations logged
579 cases reviewed 29 residents
33 attending surgeon evaluators◦ 6 contributed 75% of
assessments
![Page 14: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/14.jpg)
Preop Preparedness
0
20
40
60
80
100
120
140
160
PGY1 PGY2 PGY3 PGY4 PGY5
Training Year
Num
ber o
f Cas
es
3.5
3.6
3.7
3.8
3.9
4
Perfo
rman
ce S
core
Cases
Performance
Cronbach’s α (3 test items) = 0.80
![Page 15: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/15.jpg)
Laparoscopic Appendectomy
Cronbach’s α (4 test items) = 0.80
0
5
10
15
20
25
30
35
PGY1 PGY2 PGY3 PGY4 PGY5Training Year
Num
ber o
f Cas
es
2
2.2
2.4
2.6
2.8
3
3.2
3.4
3.6
3.8
4
Perfo
rman
ce S
core
Cases
Performance
![Page 16: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/16.jpg)
Laparoscopic Cholecystectomy
Cronbach’s α (4 test items) = 0.80
05
101520253035404550
PGY1 PGY2 PGY3 PGY4 PGY5Training Year
Num
ber o
f Cas
es
22.22.42.62.833.23.43.63.84
Perfo
rman
ce S
core
Cases
Performance
![Page 17: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/17.jpg)
Colon Resection
Laparoscopic (n = 37): Cronbach’s α (4 test items) = 0.82 Open (n = 36): Cronbach’s α (6 test items) = 0.84
0
5
10
15
20
25
30
35
40
PGY1 PGY2 PGY3 PGY4 PGY5Training Year
Num
ber o
f cas
es
2.5
2.7
2.9
3.1
3.3
3.5
3.7
3.9
Perfo
rman
ce S
core
CasesPerformance
![Page 18: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/18.jpg)
VR laparoscopic gall bladder excision (time) vs laparoscopic OR performancep = 0.0066, r2 = 0.4450 (Spearman correlation)
J Surg Res 2009;154:177-178
![Page 19: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/19.jpg)
Conclusions Experience with OpRate confirms predictions of
experience-dependent resident performance increases, but does not validate attending rating behaviors.
Still Required:Systematic rater trainingInter-observer agreementConcurrent validity studies (lab and OR)Mandatory compliance
![Page 20: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/20.jpg)
OpRate Build 2 CQL Server database Edit redundant items Add important items
Bimanual skillsAbility to assistDepth perception in laparoscopic cases
![Page 21: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/21.jpg)
![Page 22: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/22.jpg)
![Page 23: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/23.jpg)
Status of Use Remediation decisions 2 SD below mean for peer group
High likelihood of performance problems Compliance to be augmented
Variable compensation linked to useCompliance monitored and posted monthlyExpectation that residents will advocate for
use
![Page 24: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/24.jpg)
Patient Management and Teamwork Skills
![Page 25: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/25.jpg)
Electrical Burns VTE Septic Shock Hemorrhagic Shock Spinal shock Post-Op MI Anaphylaxis
FPS Curriculum Thoracic trauma Hyperkalemia Intraabdominal
catastrophe Pediatric trauma Pheochromocytoma
![Page 26: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/26.jpg)
Recognition of importance of teamwork orientation to effective clinical careStrong foundation principlesAlignment with growing body of work on “non-
technical skills” Effort to take advantage of simulated patient
care, simulated hospital environments (e.g. OR)Limited precedent in surgery
ACS-APDS Phase III Curriculum
![Page 27: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/27.jpg)
Unique communications rating instrument
Checklisted trauma assessment SPIKES protocol (delivering bad news) NOTECHS Ottawa Global Scale Need to strengthen and consolidate
assessment
Assessment
![Page 28: Neal E. Seymour, MD Baystate Medical Center Tufts University School of Medicine Baystate Simulation Center—Goldberg Surgical Skills Lab 2010 APDS Annual](https://reader035.vdocuments.us/reader035/viewer/2022062504/5a4d1b417f8b9ab0599a0f97/html5/thumbnails/28.jpg)
Conclusions Assessment of surgical skills and OR
performance:Opportunities
Expanded uses:Identification of performance outliers.Definition of areas where focused training is required
for specific residents.Measurement of effects of educational interventions.
Measurement of curriculum effectiveness