matthew roberts, vanessa bacal, mohammed mahdi, ethan d. grober mount sinai & women’s college...

Matthew Roberts, Vanessa Bacal, Mohammed Mahdi, Ethan D. Grober

Mount Sinai & Women’s College Hospital, Division of Urology, Department of Surgery, University of Toronto

Intra-Operative Assessment of Technical Skill on Live Intra-Operative Assessment of Technical Skill on Live Patients Using Economy of Hand Motion: Patients Using Economy of Hand Motion:

Establishing Learning Curves of Surgical CompetenceEstablishing Learning Curves of Surgical Competence

Technical SkillTechnical SkillAcquired

Objectively, reliably assessed

OSATS: Objective Structured OSATS: Objective Structured Assessment of Technical SkillAssessment of Technical Skill

Discriminate novice & experienced surgeonsDiscriminate novice & experienced surgeons

Improvement with training over timeImprovement with training over time

Skills sets transfer to higher fidelity simulationsSkills sets transfer to higher fidelity simulations

Intra-Operative Assessment of Technical Skill Intra-Operative Assessment of Technical Skill - - Real Patients, Real Operating RoomsReal Patients, Real Operating Rooms - -

Subjective, Unvalidated Subjective, Unvalidated

Intra-Operative Assessment of Technical Skill Intra-Operative Assessment of Technical Skill - - Real Patients, Real Operating RoomsReal Patients, Real Operating Rooms - -

Subjective, Unvalidated Subjective, Unvalidated

??Variable patientsVariable patients

Variable operationsVariable operations

Variable instruments & technologyVariable instruments & technology

OR team changesOR team changes

Resource & time pressuresResource & time pressures

Measurement instrumentsMeasurement instruments

Hand-Motion AnalysisHand-Motion Analysis

Electro-magnetic Electro-magnetic measurement of hand position measurement of hand position & orientation& orientation # of hand movements# of hand movements hand travel distancehand travel distance hand speedhand speed

Hand-Motion AnalysisHand-Motion Analysis

In the In the surgical skills labsurgical skills lab:: Objective, reliable and valid Objective, reliable and valid measure of surgical competence on open, laparoscopic, measure of surgical competence on open, laparoscopic, endoscopic and microsurgical procedures endoscopic and microsurgical procedures

Correlates well with traditional lab-based measures of Correlates well with traditional lab-based measures of surgical competence – global ratings and checklists scores surgical competence – global ratings and checklists scores by trained expertsby trained experts

Pilot work and feasibility studies have been performed in Pilot work and feasibility studies have been performed in real operating rooms real operating rooms

Hand-Motion AnalysisHand-Motion Analysis

Hand-Motion AnalysisHand-Motion Analysis

Study ObjectivesStudy Objectives

1.1. Establish the feasibility of performing live Establish the feasibility of performing live intra-operative intra-operative hand-motion analysis while hand-motion analysis while operating onoperating on real patients real patients

2.2. Validate live, intra-operative assessments of Validate live, intra-operative assessments of economy of hand motion as an economy of hand motion as an objective objective measure of technical skill measure of technical skill

3.3. Use hand-motion analysis to Use hand-motion analysis to establish establish competency-based surgical learning curvescompetency-based surgical learning curves based on standards established by based on standards established by performance of experienced surgeonsperformance of experienced surgeons

MethodsMethods

NoviceNovice

ExperiencedExperienced

2 Surgeons2 Surgeons 2 Standardized Surgical 2 Standardized Surgical TasksTasks

Video recorded Video recorded

Hand motion analysisHand motion analysis

VasectomyVasectomy

Microsurgery-Vasectomy reversalMicrosurgery-Vasectomy reversal

2 blinded, experienced surgeons2 blinded, experienced surgeons Global rating scoresGlobal rating scores Checklists scoresChecklists scores Final product scoresFinal product scores

Methods – Video AnalysisMethods – Video Analysis

Methods – Video AnalysisMethods – Video Analysis

Methods – Video AnalysisMethods – Video Analysis

Hand motion data and blinded expert ratings of video-based surgical performance were:

Graphically compared over time Correlated using Pearson calculations

Methods – Data AnalysisMethods – Data Analysis

NoviceNoviceExperiencedExperienced

Total Hand MovementsTotal Hand Movements

0

50

100

150

200

250

300

350

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Case #

Experienced SurgeonExperienced Surgeon

Novice SurgeonNovice Surgeon

0

100

200

300

400

500

600

700

1 2 3 4 5 6

Experienced SurgeonExperienced Surgeon

Novice SurgeonNovice Surgeon

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Hand Travel DistancesHand Travel Distances

Experienced SurgeonExperienced Surgeon

Novice SurgeonNovice Surgeon

Case #

Han

d T

rave

l Dis

tanc

e (m

)

0

5

10

15

20

25

30

35

40

45

50

1 2 3 4 5 6

Experienced SurgeonExperienced Surgeon

Novice SurgeonNovice Surgeon

Global Rating ScoresGlobal Rating Scores

5

7

9

11

13

15

17

19

21

23

25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Experienced SurgeonExperienced Surgeon

Novice SurgeonNovice Surgeon

Glo

bal R

atin

g S

core

Case #0

5

10

15

20

25

30

35

1 2 3 4 5 6

Experienced SurgeonExperienced Surgeon

Novice SurgeonNovice Surgeon

10

12

14

16

18

20

22

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Checklist ScoresChecklist Scores

Experienced SurgeonExperienced Surgeon

Novice SurgeonNovice Surgeon

Che

ckli

st S

core

Case #

0

2

4

6

8

10

12

1 2 3 4 5 6

Experienced SurgeonExperienced Surgeon

Novice SurgeonNovice Surgeon

0

1

2

3

4

5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Final Product ScoresFinal Product Scores

Case #

Fin

al P

rodu

ct S

core

Experienced SurgeonExperienced Surgeon

Novice SurgeonNovice Surgeon

0

1

2

3

4

5

1 2 3 4 5 6

Experienced SurgeonExperienced Surgeon

Novice SurgeonNovice Surgeon

Correlating Hand Motion & Video PerformanceCorrelating Hand Motion & Video Performance

# Hand # Hand movementsmovements

Hand Travel Hand Travel DistanceDistance

TimeTime

Global Rating Global Rating ScoreScore

r = -0.83 -0.85 -0.86

Checklist ScoreChecklist Score -0.78 -0.81 -0.86

Final Product Final Product ScoreScore

-0.82 -0.87 -0.83

Pearson correlation significant < 0.05 level (2-tailed)Pearson correlation significant < 0.05 level (2-tailed)

# Hand # Hand movementsmovements

Hand Travel Hand Travel DistanceDistance

TimeTime

Global Rating Global Rating ScoreScore

-0.94 -0.95 -0.92

Checklist ScoreChecklist Score -0.88 -0.83 -0.78

Final Product Final Product ScoreScore

-0.90 -0.85 -0.70

BADBAD BETTERBETTER

• Set up < 3minsSet up < 3mins

• Data obtained in 23/25 casesData obtained in 23/25 cases

• Surgical team easily trainedSurgical team easily trained

• HMA software and hardware $3500HMA software and hardware $3500

Hand-Motion Analysis: Hand-Motion Analysis: FeasibilityFeasibility

Conclusions Conclusions

1.1. Establish theEstablish the feasibilityfeasibility of performing live of performing live intra-operative intra-operative hand-motion analysis hand-motion analysis while while operating on real patients operating on real patients

2.2. ValidateValidate live, intra-operative assessments of live, intra-operative assessments of economy of hand motion as aneconomy of hand motion as an objective objective measure of technical skill measure of technical skill

3.3. Use hand-motion analysis to Use hand-motion analysis to establish establish competency-based surgical learning curvescompetency-based surgical learning curves based on standards established by based on standards established by performance of experienced surgeonsperformance of experienced surgeons

Utility of Hand Motion AnalysisUtility of Hand Motion Analysis

0

50

100

150

200

250

300

350

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

Case #

Experienced SurgeonExperienced Surgeon

Novice SurgeonNovice Surgeon

1) Measure & track intra-operative surgical performance over time

2) Tool for immediate feedback to surgical trainees and training programs

3) To establish that different procedures competency targets. HMA can be used to identify the ideal or minimum number of exposures required for technical competence

15

Economy of Hand Motion Economy of Hand Motion - Future Directions - - Future Directions -

Further validate this assessment technology: More surgeons - with various levels of skill &

experience Different types surgeries - basic/complex,

open/laparoscopic/endoscopic

Intra-Operative Assessment of Technical Skill Intra-Operative Assessment of Technical Skill - - Real Patients, Real Operating RoomsReal Patients, Real Operating Rooms - -

Matthew Roberts, Vanessa Bacal, Mohammen Mahdi, Ethan D. Grober

Mount Sinai & Women’s College Hospital, Division of Urology, Department of Surgery, University of Toronto

Intra-Operative Assessment of Technical Skill on Live Intra-Operative Assessment of Technical Skill on Live Patients Using Economy of Hand Motion: Patients Using Economy of Hand Motion:

Establishing Learning Curves of Surgical CompetenceEstablishing Learning Curves of Surgical Competence