Stanford Vascular Surgery

Simulation-based Endovascular Skills Assessment: The Future of

Credentialing?

Maureen M. Tedesco, Jimmy J. Pak, E. John Harris, Jr, Thomas M. Krummel,

Ronald L. Dalman, Jason T. Lee

22nd Annual MeetingWestern Vascular Society

September 10, 2007

Vascular Surgery

Disclosures

Jason T. Lee- educational grant from Cordis Endovascular to study simulation technology.

Drs. Dalman, Krummel and Lee: technical grant from Cordis Endovascular in the form of 2 simulators.

Vascular Surgery

Background

• High-fidelity simulation has become important in surgical education.– Laparoscopy– Endoscopy– Cystoscopy

• Training on simulation improves operating room performance of surgical residents.*

* Seymour et al. Annals of Surgery, 2002

Vascular Surgery

Background

• Simulation required during physician training for carotid angioplasty and stenting.

• Recent applications:– Skills assessment– Technical competency – Board certification

• American Board of Vascular Medicine• American Board of Surgery

Vascular Surgery

Purpose

Does global performance

assessment during endovascular simulation correlate well with self-reported procedural skill and prior

experience level?

Vascular Surgery

Vascular Surgery

Methods• 17 general surgery residents interviewing for

vascular fellowship training • Pre-test questionnaire:

– # of major index vascular procedures– # of specific endovascular procedures

• Diagnostic arteriograms• Aortic stent-grafts• Peripheral angioplasty/stenting• Renal stenting• Carotid stenting

Vascular Surgery

MethodsProcedicus Vascular Intervention System Trainer

(VIST®) simulator: Right Renal angioplasty and stenting (RAS) module

Vascular Surgery

MethodsSubjects were evaluated by an experiencedinterventionalist using a global rating scale.

Vascular Surgery

Angiogram

advance wire into suprarenal aorta without forming a J or pushing against obstruction

place pigtail catheter into renal angiogram position/wire manipulation

knowledge of renal anatomy/perform angiogram

Wire Access select proper catheter/wire for renal canalization safely traverse lesion

Intervention

select guiding catheter select appropriate renal stent deploy renal stent select proper balloon for renal angioplasty post-

stent perform completion angiogram

Global Rating Scale (1-5)

Vascular Surgery

Methods

• VIST® provided objective measurements:– total procedure time– fluoroscopy time– volume of contrast used (mL)– % of lesion covered– placement accuracy– presence of residual stenosis– # of cine loops used

Vascular Surgery

Methods

• Post test questionnaire:– Grade his/her own performance – Opinion about optimal number of cases

Vascular Surgery

ResultsLow

Experience (LE,<20)

Moderate Experience

(ME, 20-100)TOTAL

Subjects 8 9 17

Endovascular Cases

(range)

11.1 ± 6.8

(4-20)

46.6 ± 22.6

(25-89)29.9 ± 24.6

(4-89)

Open Cases

(range)

78.8 ± 38.0

(40-150)

75.0 ± 41.1

(40-150)76.9 ± 38.2

(40-150)

Vascular Surgery

Low Experience

(n = 8)

ModerateExperience

(n = 9)

p value

Globalassessment

2.69 3.55 0.04

Total procedure time (sec)

895.6 947 NS

Fluoroscopy Time 459.6 412 NS

Contrast used (mL)

15.6 19.2 NS

% lesion covered 96.8 94.9 NS

*Placement accuracy (mm)

4.85 6.64 NS

No residualstenosis

(% of group)

75% 89% NS

Number of Cine loops

5.5 4.7 NS

Vascular Surgery

Low Experience

(n = 8)

ModerateExperience

(n = 9)

p value

Globalassessment

2.69 3.55 0.04

Total procedure time (sec)

895.6 947 NS

Fluoroscopy Time 459.6 412 NS

Contrast used (mL) 15.6 19.2 NS

% lesion covered 96.8 94.9 NS

Placement accuracy (mm) 4.85 6.64 NS

No residualstenosis

(% of group)

75% 89% NS

Number of Cine loops

5.5 4.7 NS

Vascular Surgery

R 2 = 0.1645

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

0 1 2 3 4 5 6

self assessment score

average global assessment

score

Post-test questionnaire: poor correlation between the global assessment score and

subjects’ self assessment score.

Results

Vascular Surgery

Results

• Post-test questionnaire:– vascular surgeons = 19.2 ± 14.4 cases– interventional cardiologists = 14.7 ± 14.8

cases– interventional radiologists = 12.3 ± 12.0 cases

• p = NS

Vascular Surgery

Summary

• Significant difference in a global assessment score between two groups of surgical residents with varying levels of self-reported endovascular experience.

• Global rating scale was able to discern even minimal differences in experience.

• No difference between the two study groups with respect to the VIST objective measurements.

Vascular Surgery

Limitations

• Only one “expert” observer, no inter-observer variability.

• Each subject underwent only one session, without the opportunity to practice or learn the equipment.

• Stress may have played a role in this testing situation.

Vascular Surgery

Conclusion

• Correlation between self-reported case completion and global rating score by an observer.

• Objective measures provided by the simulator may not be valid to determine endovascular skills.

• More meaningful criteria to determine how to integrate simulation into skill assessment.

• Future research is required to determine if simulator-based testing should be incorporated into the credentialing of vascular specialists.

Vascular Surgery

Thank you!