© 2001 by the American Society for Dermatologic Surgery, Inc. • Published by Blackwell Science, Inc.ISSN: 1076-0512/01/$15.00/0 • Dermatol Surg 2001;27:370–374

Virtual Reality Simulators for Dermatologic Surgery: Measuring Their Validity as a Teaching Tool

Daniel Berg, MD, FRCPC,* Greg Raugi, MD, PhD,* Hayes Gladstone, MD,

§

Jeff Berkley,

†

Suzanne Weghorst,

†

Mark Ganter,

‡

and George Turkiyyah

‡

*

Division of Dermatology,

†

Human Interface Technology Laboratory, and

‡

Department of Mechanical Engineering,

University of Washington, Seattle, Washington, and

§

UCLA, Los Angeles, California

Surgical simulation is increasingly being considered for training,testing, and possibly credentialing in medicine and surgery. At theUniversity of Washington we have been developing a virtual real-ity (VR) suturing simulator. In the course of development it mustbe realized that expensive new technologies should bear the bur-

den of proof of their effectiveness and reliability before they areput into training programs. The purpose of this article is to definethe concept of surgical skill and to discuss how it can be measuredin the context of validating VR surgical simulators. Specific mea-sures of validity and reliability are reviewed and discussed.

CURRENT METHODS OF surgical training rely pri-marily on the apprenticeship model, which suffersfrom problems including variability of experience in aresidency, difficulty in standardization of training, theneed to “practice on real patients,” and financial con-straints on teaching and operating room time.

1–6

Train-ing on animals is costly, often impractical, and in theUnited Kingdom, for example, forbidden by antivivi-section laws.

7

Despite the problems, there is increasing recogni-tion of the need to train medical students, surgeons,primary care physicians, and dermatologists in basicsurgical techniques prior to their developing advancedskills. These basic techniques include excisional sur-gery of skin and suturing. Approximately 40% of newsurgical interns never performed a closure of an inci-sion in medical school.

8

Existing models for suturinginclude pig’s feet, tie and suture boards,

9

cadavers,and synthetic skin models.

10

Tie and suture boardslack realism; cadavers and pig’s feet are limited by riskof exposure to blood-borne pathogens, postmortemchanges in mechanical properties, and perishability.

11

Despite the limitations a survey taken in 1989 showedthat most U.S. dermatology residency training pro-grams used pig’s feet for training practice.

12

One der-matologic surgical training program used live pigs inits teaching of basic surgical procedures.

12

Surgical simulation is increasingly being consideredfor training, testing, and possibly credentialing in med-

icine and surgery. At the University of Washington wehave been developing a virtual reality (VR) suturingsimulator with real-time haptic feedback (see Figure1).

11,13

In the course of development it must be real-ized that expensive new technologies should bear theburden of proof of their effectiveness and reliabilitybefore they are put into training programs. The pur-pose of this article is to define the concept of surgicalskill and to discuss how it can be measured in the con-text of validating VR surgical simulators.

What Makes a Good Surgeon?

I shall not today attempt further to define (it) . . . but Iknow it when I see it.

—Potter Stewart (1915–1985), U.S. Supreme Court Justice,on pornography

If VR simulation aspires to teach, test, and credential itmust be developed with an idea of what comprisescompetence. Despite the obvious importance of this is-sue, objectively defining competence is difficult. Thegeneral educational literature refers to three domainsof competence: knowledge, attitudes, and skills.

14

Clin-ical competence has been said to include, in addition,decision making, clinical judgment, professional hab-its, and interpersonal skills.

15

Neuropsychological at-tributes of surgical residents correlating with high rat-ings by attending surgeons include complex visuospatialorganization, stress tolerance, and psychomotor abili-ties.

16,17

Education of trainees in the realms of knowl-edge and attitudes are likely beyond the scope of first-generation surgical simulators. Developing dexteritymight be a more suitable goal for a VR surgical trainer,but simple dexterity does not seem to be enough tomake a good surgeon. One study that formally tested

D. Berg, MD, FRCPC, G. Raugi, MD, PhD, H. Gladstone, MD, J.Berkley, S. Weghorst, M. Ganter, and G. Turkiyyah have indicated nosignificant interest with commercial supporters.Address correspondence and reprint requests to: Daniel Berg, MD, Uni-versity of Washington Medical Center, Box 356166, 1959 NE PacificSt., Seattle, WA 98195-6166, or e-mail: danberg@u.washington.edu.

Dermatol Surg 27:4:April 2001

berg et al.: virtual reality simulators

371

dexterity showed no better performance of surgeonscompared with residents or nonsurgeons.

18

Neverthe-less, it has been shown that repeated practice of a man-ual skill does correlate with performance.

19

Theories of motor-skill learning have proven in-structive in trying to understand the important factorsin acquiring surgical skill

20

and point VR trainer de-velopers in the right direction. Three stages have beendescribed in the acquisition of a motor skill:

1

cogni-tion or understanding of the task, integration in whichmotor skills unique to the task are applied to avoid in-efficient movements, and automation in which theskill becomes automatic without requiring consciousthought before each step.

Testing for Competence in Surgical Skills: What Is Done Now?

Clearly technical skills are a critical component of com-petence, yet subjective evaluation and “gut feeling” re-main the predominant method of in-training evalua-tion. Formal testing of surgical residents’ technical skillsis uncommon.

21

Relatively little research has been de-voted to objective measures of technical competence.

5,22

Watts and Feldman

5

have listed the methods of techni-cal skills evaluation as summarized in Table 1.

VR Simulators for Teaching and Testing Surgical Skills: Validation of Their Role

Steps Needed for Validation

Simulation has been most useful to date in the fields oflaparoscopic surgery, cardiology patient simulation,and anesthesia.

23

Prior to widespread acceptance ofVR surgical simulators, their value as either educa-tional tools or tests of competence must be examined.Validation studies will help answer the key question:“Does proficiency in the simulator correlate with pro-ficiency in reality?”

In the development of a VR surgical simulator, thefollowing goals must be met:

1. Task analysis: Identify the skills to be trained ortested. This may require the identification of sub-tasks and delineation of complex tasks into sim-pler, more easily measurable steps.

2. Development of measurement tools: Identify sub-tasks that can be measured (metrics) and then in-vestigate their reliability and validity (see below).

VR simulators may ultimately be used for both eval-uation and education. The usefulness of an evaluationor educational tool depends on the extent to whichthey possess a number of features. These are summa-rized by Neufeld

24

to include credibility, comprehen-siveness, reliability, validity, and feasibility. These defi-

Figure 1. Current appearance of the University of Washingtonsuturing simulator.

Table 1.

Current Methods for Testing Technical Skills

Method of evaluation Description Comment

“Gut feeling” Still a very common component of surgical training evaluation

Subjective and open to multiple biases. What does “she has good hands” really mean?

Procedure lists Tracking numbers of procedures performed (eg, logbook)

Assumes that if trainee has performed the procedure enough, he/she will be competent. Subject to reporting bias.

Direct observation without criteria (on humans, animals, or mechanical models)

Observer(s) gives an evaluation based on undefined criteria

Low inter- and intraobserver reliability

Direct observation with criteria (on humans, animals, or mechanical models)

Observer(s) evaluates based on performance of specific predefined steps making up a procedure

More reliable because of criteria. Validity demonstrated in some studies

2,22,44,45

Observation with criteria on VR models As above, using simulation Category of VR surgical simulation

Modified from Watts and Feldman.

5

372

berg et al.: virtual reality simulators

Dermatol Surg 27:4:April 2001

nitions are reviewed in Table 2. It should be noted thata valid test must first be reliable. A reliable test is notalways valid. Reliability can be influenced by examinervariability, variability in candidates’ performance, orerrors in measurement of the test instrument. For thepurposes of our discussion, an instrument or test refersto any tool being examined (such as a surgical simula-tor). Other considerations exist which are more relatedto a simulator’s value as a teaching rather than a test-ing tool. For example, instructional effectiveness of asurgical simulator implies that repeated use of themodel will improve the user’s performance.

19

Finally, itshould be noted that since clinical competence is multi-factorial, no one simulator should be expected to mea-sure all aspects of competence.

Experience with Validation in VR

Simulators have been shown to improve performance ina variety of medical and surgical simulation settings.

25–27

Some forms of validity have been looked at in a fewsimulators.

28–31

Nevertheless, not all simulators showvalidity or lead to better performance,

32–34

and these ex-amples demonstrate the critical need for simulator vali-dation during development. For example, early experi-ence reported with a VR arthroscopy trainer showed a

lack of construct validity when comparing experiencedorthopedic surgeons with medical students.

35

A recentstudy examined the internal consistency (a measure ofreliability) of test items for an anesthesia simulator. Theauthors found that some items showed very poor inter-nal consistency and recommended further reliabilityand validity testing before the patient simulator is usedas an evaluation tool.

36

In a study of medical studentstraining on a catheterization simulation, improvementon the simulator did not lead to improvement in realcatheterizations.

32

The reasons why a VR surgical simulator may failon tests of validity include

The simulator actually tests one or more of the con-founding factors below rather than surgical skill:eye-hand coordination or manual dexterity, eye-sight, gamesmanship, a subtask of the simulationwhich is a rate-limiting step (eg, inexperience hold-ing a needle driver will negatively impact perfor-mance on all aspects of a suture simulator and notreflect performance on other specific subtasks).

Manual dexterity does not coordinate necessarilywith experience or surgical skill. In one studylooking at standardized tests of hand function,staff surgeons performed worse than staff inter-nists, and medical and surgical residents.

18

Table 2.

Features of an Evaluation or Educational Tool

Property (synonyms in brackets) Definition

Credibility (face validity) A subjective judgment of the degree to which the test appears to measure the attribute of interest (eg, does the simulator look and feel real enough to model the skills being tested?).

Comprehensiveness (content validity) The extent to which an instrument samples all possible aspects of the area of competence being tested (eg, a test of technical competence alone has low content validity for global surgical competence because it does not test clinical judgment).

Reliability (precision, consistency) Reliability is a measure of the ability of the test to return the same result under different conditions (eg, different times, different scorers, etc). Estimation of reliability can include internal consistency,

a

test-retest, and interrater reliability methods. Reliability is measured as an index from 0 to 1.

44

Tests with scores

,

0.5 are unreliable; tests with indices from 0.5 to 0.8 are moderately reliable and indices

.

0.80 reflect strong reliability.

Validity An assessment of whether the instrument actually measures what it claims to measure. Criterion-related validity compares performance on the instrument with a “gold standard” test, and includes concurrent and predictive validity.

• Concurrent validity Performance on the instrument is compared (concurrently) with performance on an external measure. Concurrent validity implies that regardless of other variables, those who demonstrate more proficiency in the gold standard (eg, performance of the skill in physical reality) also demonstrate more proficiency on the test (eg, the VR model).

•

Predictive validity Performance on the instrument is compared with performance on an external measure in the future (eg, seeing whether good dexterity on the simulator at the beginning of residency or fellowship correlates with good evaluations of dexterity from residency supervisors).

•

Construct validity Performance on the instrument by one group is compared with performance by a group different (in ways that should make them perform better or worse at the task that is being measured). There is low construct validity if trained surgeons perform more poorly than medical students on a surgical simulator. Multiple factors can bias construct validity (discussed in text).

Feasibility (practicality, acceptability) How feasible is widespread use or acceptance of the instrument? Costs, politics, logistics play a role here.

a

Internal consistency is a gauge of the extent to which all items on a multi-item evaluation system measure the same attribute. It is a measure of reliability. To be internally con-sistent, a subject must score each item in a similar rank order, compared with other subjects, on the same test items.

36

Dermatol Surg 27:4:April 2001

berg et al.: virtual reality simulators

373

Test subject selection bias: older surgeons do lesssurgery than when younger and may be a poor“gold standard.”

Simulation may be inadequate because the task isnot complex enough (ie, novice can do well, mak-ing it difficult to separate scores from those of ex-pert); the experienced surgeon knows what thereal thing is like and doesn’t suspend disbelief, fo-cusing on the difference between the simulationand reality rather than on completing the task;the simulator may be giving the wrong clues; orthe simulator may be scoring the wrong metrics.

The evaluation process simulator may be inadequatebecause subjects may underperform because of“exam stress.” If involved in the performance aswell as evaluation of the test, evaluators may in-fluence or show bias in rating the subjects’ per-formance.

37

Including end users (surgeons) in designing VR simu-lators is critical to avoid some of these errors.

35

Ulti-mately, true validity will only be determined if perfor-mance or training on a simulator can be shown topositively influence patient outcomes.

Specific Issues for Validation of a Suturing Simulator

We are currently developing at the University of Wash-ington a surgical simulator that we hope to use in the fu-ture for training and testing. Initially the simulator will belimited to simple tasks such as suturing. When testingor validating a suture simulator, being able to measure“good suture technique” in some objective way is critical.

Seki

38–40

has studied the factors involved in obtaininggood sutures. This area of surgery has been poorlystudied and he showed that suture placement is less pre-cise than many surgeons have predicted. He pointed outthat there are no objective good data to allow gaugingof the forces on the tissue that are appropriate for anyorgan such as skin.

41

In one study, Seki

41

measured thedistance deviated and force (expressed as torque on theneedle driver—not tissue) used when placing suturesinto predetermined entry and exit points on a foammodel. He found that a short grip on the needle and in-clination of the tissue model generally enhanced accu-racy and reduced the force needed. There was consider-able variation in results between the six surgeons tested.In this study the ideal result was assumed to be pre-cisely placed sutures producing minimal distortions ofthe tissue. He found also that precision of suturing asmeasured by the distance away from a predeterminedexit point showed that a grip in which all fingers areplaced around the holder is more accurate than if thefirst and fourth fingers are placed in the holder.

39

Given the lack of quantitative data, it is not obviouswhich metrics to employ when evaluating any VR su-ture simulator. To date, several measures have beenexamined in validation studies of VR and mechanicalsuturing models. One study of medical students ona non-VR appendectomy simulator model measuredwhether the subject followed the curve of the needlethrough the tissue, appropriately spaced staples, andproperly everted the skin while applying staples.

31

Inone study of laparoscopic bowel suturing in a pig,

42

subjects were asked to suture as quickly as possible3–5 mm from the enterotomy edge and at 3–5 mm in-tervals. Vertical/horizontal deviations from the desiredentry/exit point of the suture, execution time, and fail-ure to complete sutures were recorded. In a laparo-scopic simulator,

19,43

the time to complete a task wasmeasured. In addition, for each task, a penalty was cal-culated for a predetermined measure of inaccuracy. Forexample, in a suturing task (simple sutures through aPenrose drain), penalty points were derived from mea-suring the number of millimeters that the suture devi-ated from premarked points. The total score for a taskwas the timing score minus the penalty score. Constructvalidity was examined by determining the correlationof the score on the simulator with level of training.

43

Of interest, the correlation was significant only for fourof seven tasks. One study comparing knot tying by med-ical students following either a lecture and seminar or acomputer demonstration videotaped the students doingtwo knots.

34

Blinded surgeons reviewed the tapes andrated whether the knot was square. They also measuredtime and used a rating scale identifying actions neces-sary for optimal performance. Of interest, there was nodifference in performance between the two groups.

A more recent study looking at knot tying in a vas-cular VR model measured the skills of 8 experiencedvascular surgeons and 12 medical students performinga VR suturing task. Eight parameters of the suturingtask were measured: total tissue damage, accuracy ofneedle puncture, peak tissue tearing force, time tocomplete the task, damage to the surface of the tissue,angular error in needle technique, total distance trav-eled by the tool tip, and a measure of overall error.

31

Construct validity was shown in this model. In a re-cent test of the simulator by eight general surgeonswho were surveyed after its use, generally positivefeedback was obtained.

30

Suggested improvements tothe simulation included improvements in three-dimen-sional vision, increasing realism of tissue properties,and placing the tasks in a surgical context.

Conclusion

Virtual reality simulation for teaching and testing sur-gical techniques is in its infancy, but can be expected

374

berg et al.: virtual reality simulators

Dermatol Surg 27:4:April 2001

to grow. Previous hurdles such as being able to pro-vide the feeling of real-time force feedback (“feeling oftouch”) with a model are increasingly being surmountedby improved technology.

11

In addition to technical issues, there are importantissues of validation that will need to be addressedprior to VR simulators being widely accepted. The keyquestion that needs to be answered is, “Does profi-ciency in the simulator correlate with proficiency inreality?” Ultimately it will need to be shown that per-formance or training on a simulator can be shown topositively influence patient outcomes.

References

1. Hamdorf JM, Hall JC. Acquiring surgical skills. Br J Surg 2000;87:28–37.

2. Kopta JA. An approach to the evaluation of operative skills. Sur-gery 1971;70:297–303.

3. McMahon DJS, Chen S, MacLellan DG. Formal teaching of basicsurgical skills. Aust N Z J Surg 1995;65:607–9.

4. Sackier JM. Evaluation of technical surgical skills. Lessons fromminimal access surgery [editorial]. Surg Endosc 1998;12:1109–10.

5. Watts J, Feldman W. Assessment of technical skills. In: Neufeld V,Norman G, eds. Assessing Clinical Competence. New York:Springer, 1985:259–74.

6. Bridges M, Diamond DL. The financial impact of teaching surgicalresidents in the operating room. Am J Surg 1999;177:28–32.

7. Ellis H. Teaching surgeons to stitch [editorial]. Aust N Z J Surg1993;63:919–20.

8. Nakayama DK, Steiber A. Surgery interns’ experience with surgicalprocedures as medical students [see comments]. Am J Surg 1990;159:341–3; discussion 344.

9. Boyle DE, Gius JA. Tie and suture training board. Surgery 1968;63:434–6.

10. Munro A, Park KG, Atkinson D, Day RP, Capperauld I. Skin simu-lation for minor surgical procedures. J R Coll Surg Edinb 1994;39:174–6.

11. Gladstone HB, Raugi GJ, Berg D, Berkley J, Weghorst S, Ganter M.Virtual reality for dermatologic surgery: virtually a reality in the21st century. J Am Acad Dermatol 2000;42(1 pt 1):106–12.

12. Anders KH, Goldstein BG, Lesher JL, Shimp RG, Chalk DK. Theuse of live pigs in the surgical training of dermatology residents [seecomments]. J Dermatol Surg Oncol 1989;15:734–6.

13. Berkley J, Weghorst S, Gladstone H, Raugi G, Berg D, Ganter M.Fast finite element modeling for surgical simulation. Stud HealthTechnol Inform 1999;62:55–61.

14. Neufeld V. Implications for education. In: Neufeld V, Norman G,eds. Assessing Clinical Competence. New York: Springer, 1985:297–310.

15. Miller MD. Office procedures. Education, training, and proficiencyof procedural skills. Prim Care 1997;24:231–40.

16. Schueneman AL, Pickleman J, Hesslein R, Freeark RJ. Neuropsy-chologic predictors of operative skill among general surgery resi-dents. Surgery 1984;96:288–95.

17. Schueneman AL, Pickleman J, Freeark RJ. Age, gender, lateraldominance, and prediction of operative skill among general surgeryresidents. Surgery 1985;98:506–15.

18. Squire D, Giarchino AA, Profitt AW, Heaney C. Objective compar-ison of manual dexterity in physicians and surgeons. Can J Surg1989;32:467–70.

19. Derossis AM, Bothwell J, Sigman HH, Fried GM. The effect ofpractice on performance in a laparoscopic simulator. Surg Endosc1998;12:1117–20.

20. DesCoteaux JG, Leclere H. Learning surgical technical skills [seecomments]. Can J Surg 1995;38:33–8.

21. Martin JA, Regehr G, Resnick R, et al. Objective structured assess-

ment of technical skill (OSATS) for surgical residents. Br J Surg1997;84:273–8.

22. Winckel CP, Reznik RK, Cohen R, Taylor B. Reliability and con-struct validity of a structured technical skills assessment form. Am JSurg 1994;167:423–7.

23. Issenberg SB, McGaghie WC, Hart IR, et al. Simulation technologyfor health care professional skills training and assessment. JAMA1999;282:861–6.

24. Neufeld V. An introduction to measurement properties. In: NeufeldV, Norman G, eds. Assessing Clinical Competence. New York:Springer, 1985:39–50.

25. Tuggy ML. Virtual reality flexible sigmoidoscopy simulator train-ing: impact on resident performance [see comments]. J Am BoardFam Pract 1998;11:426–33.

26. Chopra V, Gesink BJ, de Jong J, Bovill JG, Spierdijk J, Brand R.Does training on an anaesthesia simulator lead to improvement inperformance? Br J Anaesth 1994;73:293–7.

27. Taffinder N, Sutton C, Fishwick RJ, McManus IC; Darzi A. Vali-dation of virtual reality to teach and assess psychomotor skills inlaparoscopic surgery: results from randomised controlled studiesusing the MIST VR laparoscopic simulator. In: Westwood J, Hoff-man HM, Stredney D, Weghorst SJ, eds. Medicine Meets VirtualReality. IOS Press, 1998:124–30.

28. Weghorst S, Airola C, Oppenheimer P, et al. Validation of the MadiganESS simulator. In: Westwood J, Hoffman HM, Stredney D, WeghorstSJ, eds. Medicine Meets Virtual Reality. IOS Press, 1998:399–405.

29. McCarthy A, Harley P, Smallwood R. Virtual arthroscopy training:do the “virtual skills” developed match the real skills required? In:Westwood J, Hoffman HM, Stredney D, Weghorst SJ, eds. Medi-cine Meets Virtual Reality. IOS Press, 1999:221–7.

30. Gorman P, Lieser JD, Marshall RL, Krummel TM. End user analy-sis of a force feedback virtual reality based surgical simulator. In:Westwood J, Hoffman HM, Stredney D, Weghorst SJ, eds. Medi-cine Meets Virtual Reality. IOS Press, 2000:102–4.

31. O’Toole RV, Playter RR, Krummel EM. Measuring and developingsuturing technique with a virtual reality surgical simulator [seecomments]. J Am Coll Surg 1999;189:114–27.

32. Prystowsky JB, Regehr G, Rogers DA, Loan JP, Hiemenz LL, SmithKM. A virtual reality module for intravenous catheter placement.Am J Surg 1999;177:171–5.

33. Beckmann CR, Lipscomb GH, Ling FW, Beckmann CA, JohnsonH, Barton L. Computer-assisted video evaluation of surgical skills.Obstet Gynecol 1995;85:1039–41.

34.

Rogers DA, Regehr G, Yeh KA, Howdieshell TR. Computer-assisted learning versus a lecture and feedback seminar for teachinga basic surgical technical skill. Am J Surg 1998;175:508–10.

35. Smith S, Wan A, Taffinder N, Read S, Emery R, Darzi A. Early ex-perience and validation work with Procedicus VA—the Prosolviavirtual reality shoulder arthroscopy trainer. In: Westwood J, Hoff-man HM, Stredney D, Weghorst SJ, eds. Medicine Meets VirtualReality. IOS Press, 1999:337–43.

36. Devitt JH, Kurrek MM, Cohen MM, et al. Testing internal consis-tency and construct validity during evaluation of performance in apatient simulator [see comments]. Anesth Analg 1998;86:1160–64.

37. Kapur PA, Steadman RH. Patient simulator competency testing: readyfor takeoff? [editorial; comment]. Anesth Analg 1998;86:1157–9.

38. Seki S. Accuracy of suture placement. Br J Surg 1987;74:195–7.39. Seki S. Suturing techniques of surgeons utilizing two different needle-

holder grips. Am J Surg 1988;155:250–52.40. Seki S. Accuracy of suture techniques of surgeons with different

surgical experience. Jpn J Surg 1987;17:465–9.41. Seki S. Techniques for better suturing. Br J Surg 1988;75:1181–4.42. Joice P, Hanna GB, Cuschieri A. Ergonomic evaluation of laparo-

scopic bowel suturing. Am J Surg 1998;176:373–8.43. Derossis AM, Fried GM, Abrahamowicz M, Sigman HH, Barkun

JS, Meakins JL. Development of a model for training and evalua-tion of laparoscopic skills. Am J Surg 1998;175:482–7.

44. Reznick R, Regehr G, McRae H, Martin J, McCulloch W. Testingtechnical skill via an innovative “bench station” examination. Am JSurg 1997;173:226–30.

45. Lossing A, Groetzsch G. A prospective controlled trial of teachingbasic surgical skills with 4th year medical students. Med Teach1992;14:49–52.