sso stat pbr.4 ppt
DESCRIPTION
Given at SSO annual meeting, 2008.TRANSCRIPT
Use of a Novel, Web-based Educational Platform Facilitates
Intra-operative Training in a Surgical Oncology Fellowship
Program
Paul B. Roach, Kevin K. Roggin, Eugene Selkov Jr., James Dignam,
Mitchell C. Posner, Jonathan C. Silverstein
The University of ChicagoDepartment of Surgery
Disclosure Statement
• The principal and contributing authors have no conflicts of interest to report.
• The software program presented has an “open source” copyright, and will be distributed freely to any interested surgical program directors.
Background
• No widely available, comprehensive, easy-to-use platform for evaluating surgical resident/fellow operative performance.
• Current educational reforms depend on one being developed and integrated into training programs.
• Examples exist from within Naval Aviation, and Business Management.
Hypothesis
• Can a system be created for achieving continuous operative performance assessments?
• Can it be integrated into our daily routine / educational curriculum?
“The Surgical Training and Assessment Tool” or “STAT”
1. A web-based software system to de- construct the complex training events of an operative case into distinct elements;
2. Enable methodical, thoughtful, subjective assessments of these distinct elements;
3. Re-construct data into usable formats to impact / improve the surgical training process.
“STAT” Login
4 Sections:
Two structured: General Capabilities Specific Capabilities
Two unstructured: Comments Box Overall Assessment
Results
Results
Training Objectives
Distribution Times
0 20 40 60 80 100
Nine-36
64-92
119-146
174-202
229-256
284-312
339-366
394-422
449-476
504-532
Sec
on
ds
Incidence
Seconds
Median = 39 seconds
Mean = 60 seconds
Time per case entry
Data: SYSTEM USAGESystem Usage: Submissions / Cases
0
50
100
150
200
1 3 5 7 9 11 13 15 17
Month
To
tals Submissions
Cases
18 Month Totals: 1318 submissions832 cases
Submissions: ATTENDING/TRAINEE
Submissions: Attending/Trainee
0
20
40
60
80
100
120
140
160
Month
To
tals Attendings
Trainees
18 Month Totals: 607 Attending 710 Trainee
Overall Assessment Distribution
0
20
40
60
80
100
120
140
160
180
F D- D D+ C- C C+ B- B B+ A- A A+C
ou
nt
Attending
AttendingsTraineesOverall Assessment Distribution
0
50
100
150
200
250
300
F D- D D+ C- C C+ B- B B+ A- A A+
Cou
nt
Resident
7.6 = B- 8.71 = B
Discriminatory Capacity
Do the “General Capabilities” correlate with the “Overall Score”?
• Each of the general capabilities is significantly and strongly correlated with the overall score.
Knowledge 0.6036
Skill 0.7638
Independence 0.6935
Linear correlation with overall score:
Do the general capabilities predict the overall score?
• In a multiple regression model, the general capabilities significantly predict the overall score.
Predictor Coef. (slope) P
Knowledge 0.312 <0.001
Skill 1.255 <0.001
Independence 0.546 <0.001
Model R2 = .62
Multiple regression model for overall score:
How well do attending ratings of a given trainee correlate with one another?
0 1 2 3 4
r1
r2
r3
r4
Average Rank
1.4
2.6
2.8
3.2
How well do the self-ratings of the trainees compare with the ratings their attendings
give them?
Trainee “A”Attending
Trainee
Trainee- “B”
• Abundant trainee performance data• Real-time, transparent, longitudinal
record of trainee performance• Inexpensive & time-efficient• Influences the educational process:
– Reflection & self-awareness– Feedback – Formative & summative
Summary
Conclusion1. STAT is PRACTICAL:
FeasibleComprehensiveRelevant
2. STAT appears to be RELIABLE:Internal consistencyInter-rater reliability
3. STAT is EASILY INTEGRATED into the daily work- flow and educational curriculum.
Voluntary, self-sustaining use
• Surgical Oncology Faculty
• Pritzker School of Medicine
• General Surgery Faculty & Residents
• Bob McBride – Education
• CAPT JJ Lee – Naval Aviation
• Steve Roach – Managerial Science
Acknowledgements
Future Directions1. STAT Validation Study:
4 military training programs + Hadassah University
2. STAT Expansion into other specialtiesOrthopedic, Thoracic, Plastic, Obstetric-
Gynecologic Surgery
3. HIPEC study.Training ComponentReporting Component
3. Society of Surgical Oncology:Possible tool for fellowship training
“STAT” General Capabilities
Criteria Checklist- General
Criteria Checklist
Criteria Checklist- Case Specific
Unstructured Elements