i

September 01, 2013

A NEW SYNTHETIC BRAIN SIMULATOR FOR ENDOSCOPIC THIRD VENTRICULOSTOMY

Author Gerben Eise Breimer

The Hospital for Sick Children, Department of Surgery, Division of Neurosurgery

Supervisor Groningen Dr. E.W. Hoving, University Medical Center Groningen

Supervisor Toronto Prof. Dr. J.M. Drake, The Hospital for Sick Children

ii

SUMMARY

Introduction

Endoscopic third ventriculostomy (ETV) is an effective but technically demanding procedure with significant risk. Current simulators include human cadavers, animal models and virtual

reality (VR) systems, however there are drawbacks. Human cadavers are expensive and relatively inaccessible; animal models do not represent the condition accurately; and VR systems

are expensive and lack realistic sensory feedback. We have constructed a realistic low cost, reusable brain simulator for ETV, evaluated it for fidelity, and used it to differentiate levels of surgical skill by means of an ETV assessment tool, which was constructed as part of this study.

Methods

A silicone-based brain simulator mimicking the normal mechanical properties of a four-month-old child with hydrocephalus was constructed, encased in the replicated skull and immersed in water. The thinned out third ventricle floor, which dissected realistically, was made to be

replaceable. Bleeding scenarios were also incorporated. The simulator was tested for fidelity by means of a questionnaire completed by sixteen neurosurgical trainees (PGY 1-6) and nine

pediatric and adult neurosurgeons. The Delphi method was used to reach consensus among seventeen international experts on a series of ETV assessment tools: a procedural checklist, a checklist of potential errors and a global rating scale. Performance of novices, senior residents

and neurosurgeons were compared using the validated ETV assessment tool.

Results

The simulator was portable, robust, and able to be set up in minutes. Over 95 % of participants agreed or strongly agreed that the simulator’s anatomical features, tissue properties, and bleeding

scenarios were a realistic representation of that seen during an ETV. Participants stated that the simulator helped develop the required hand-eye coordination and camera skills, and was a

valuable training exercise. Three assessment tools were created in order to evaluate surgical competence with ETV. Using the objective measures of the ETV assessment tool, neurosurgeons scored significantly higher than novices on the simulator.

Conclusion

A low-cost, reusable silicone-based ETV simulator realistically represents the surgical procedure to trainees and neurosurgeons. It can develop the technical and cognitive skills for ETV, including dealing with complications. In addition, an assessment tool with content validity was

designed as a standardized method to evaluate a trainee’s competence with ETV.

iii

SAMENVATTING

Introductie

Endoscopische derde ventriculocisternostomie (EDV) is een effectieve maar technisch complexe procedure met significante risico’s. Om EDV te leren worden nu menselijke kadavers,

diermodellen, en virtual reality (VR) simulators gebruikt. Aan al leermethoden kleven nadelen. Menselijke kadavers zijn kostbaar en schaars, diermodellen weerspiegelen het hydrocefaal beeld niet accuraat, en VR simulators zijn duur en missen realistische sensorische feedback. Wij

hebben een betaalbare, herbruikbare en realistische brein simulator ontwikkeld. Vervolgens hebben we het realisme en de bruikbaarheid ervan beoordeeld en hebben we de simulator

gebruikt om drie niveaus van chirurgische bekwaamheid te onderscheiden met behulp van een evaluatie instrument die als onderdeel van deze studie is ontwikkeld.

Methode

De eigenschappen van de brein simulator zijn gebaseerd op beeldvorming van een vier maanden

oud kind met hydrocefalie. De simulator is gemaakt van siliconen rubber. Een passende schedel werd gecreëerd waarin het brein kon worden geplaatst. Tijdens gebruik werd de simulator ondergedompeld in water. De vloer van de derde ventrikel, waarin een realistische

ventriculostomie kan worden aangebracht, kan snel worden vervangen. Er is een mogelijkheid de complexiteit van de procedure te verhogen met twee bloedingscenario’s. Zestien neurochirurgen

in opleiding en negen neurochirurgen hebben de procedure met de simulator getest. Met een vragenlijst hebben zij de simulator beoordeeld op realisme en bruikbaarheid. De Delphi methode is gebruikt om consensus te bereiken over een serie instrumenten waarmee de vaardigheid in

EDV gemeten wordt. Zeventien internationale experts hebben aan de Delphi studie deelgenomen. Beginners, ouderejaars neurochirurgen in opleiding, en neurochirurgen zijn

beoordeeld op het uitvoeren van een EVD met behulp van het evaluatie instrument en de uitkomsten werden tussen deze drie groepen vergeleken.

Resultaten

De simulator is gemakkelijk verplaatsbaar, stevig en kan in enkele minuten gebruiksklaar

worden gemaakt. Meer dan 95 % van de deelnemers hebben het model positief beoordeeld op realistische weergave van de anatomische kenmerken, weefsel eigenschappen en de bloedingscenario’s. Ze zijn van menig dat de simulator zeker bruikbaar kan zijn bij het

ontwikkelen van hand-oog coördinatie en camera vaardigheden die nodig zijn voor het uitvoeren van een EVD. Volgens de deelnemers is de simulator een waardevolle oefenmogelijkheid. Een

drietal evaluatie instrumenten is ontwikkeld om de bekwaamheid van neuroendoscopisten in opleiding in EVD te beoordelen. De neurochirurgen voerden de EVD procedure significant beter uit dan de beginners toen deze werden beoordeeld met behulp van het evaluatie instrument.

Conclusie

Deze studie heeft een betaalbare en herbruikbare synthetische EVD simulator die een ventriculostomie procedure realistisch nabootst, opgeleverd. De simulator kan bijdragen aan de ontwikkeling van technische en cognitieve vaardigheiden die nodig zijn voor het uitvoeren van

een EVD met inbegrip van omgaan met complicatie van de procedure door bloeding. Verder is een evaluatie instrument met inhoudsvaliditeit ontwikkeld en getoetst. Deze kan worden gebruikt

voor gestandaardiseerde beoordeling van de mate van bekwaamheid in EVD.

iv

TABLE OF CONTENTS

Summary ii

Samenvatting iii

1 Introduction 1 2 Methods 2

2.1 Phase 1. Development of brain simulator for ETV 3

2.1.1 Features of the simulator 3 2.1.2 Construction of the brain simulator 3

2.1.3 Construction of a replaceable third ventricle floor 4 2.1.4 Incorporating bleeding scenarios 4

2.2 Phase 2. Fidelity of the brain simulator 4

2.2.1 Design 4 2.2.2 Questionnaire 5

2.2.3 Statistical analysis 5 2.3 Phase 3. Construction of the ETV assessment tool 5

2.3.1 Design 5

2.3.2 First draft of items 6 2.3.3 Modified Delphi study 6

2.3.4 Statistical analysis 7 2.4 Phase 4. Preliminary construct validation study 7

2.4.1 Design 7

2.4.2 Data collection 8 2.4.3 Statistical analysis 8

3 Results 8 3.1 Phase 1 8 3.2 Phase 2 8

3.3 Phase 3 10 3.4 Phase 4 10

4 Discussion 11 4.1 Phase 1 11 4.2 Phase 2 11

4.3 Phase 3 12 4.4 Phase 4 13

5 Conclusion 16 6 Acknowledgements 17 7 References 18

Appendices:

Appendix 1. Three initial lists for online survey 23 Appendix 2. Checklist for preliminary construct validation study 26

Appendix 3. Images of the brain simulator 27

1

1. INTRODUCTION

This report documents the construction and validation of a new synthetic brain simulator for endoscopic third ventriculostomy (ETV). In addition, an assessment tool is designed that can be

used to evaluate a trainees’ ability to perform ETV. ETV is part of the basic curriculum of neurosurgical trainees since it is a commonly used and accepted method of treatment for obstructive hydrocephalus [1–12]. Hydrocephalus can be defined as being an active distension of

the ventricular system of the brain related to inadequate passage of cerebrospinal fluid (CSF) from its point of production within the ventricular system to its point of absorption into the

systemic circulation [13]. The two main treatment modalities in the management of hydrocephalus are ventriculoperitoneal shunt (VP shunt) implantation and ETV [14]. Studies comparing the two modalities have reached a variety of conclusions, however the absence of a

foreign body is a major advantage of ETV treatment over VP shunting. Evidence in favour of ETV [4,15–17], VP shunting [18], or neither one [10,19] have all been reached. The most

comprehensive randomized long-term outcome study is still ongoing [20,21]. In addition, ETV and shunting do not differ significantly when comparing cost-effectiveness [22].

During academic neurosurgical residency training, surgical dexterity is developed through practice. In learning surgical techniques, it is preferable to train on simulation models before

moving on to real patients [23–25], since a wide variety of complications can occur during surgical procedures. This is especially true for neurosurgical procedures, where minor mistakes can have dire outcomes. Complications for the ETV procedure include, but are not limited to,

both transient and sustained neurological and hormonal impairment and a potentially life-threatening basilar artery rupture [2,14,26–33]. Furthermore, several authors reported that a

majority of the complications during ETV occurred in the first period after introduction of the technique in their clinics [26,34–36]. Thus, simulators can create a risk-free learning environment, in which a novice surgeon can train before proceeding to patients, thereby allowing

the expertise to be acquired while reducing the risk of complications during the procedure.

There are various training models for ETV: human cadaveric and animal models, synthetic models and virtual reality (VR) simulators. Classical educational resources like human cadavers and animal models are far from ideal because they often lack resemblance to hydrocephalic

anatomical situation and live tissue properties, resulting in unrealistic haptic feedback [37,38]. Furthermore, there are ethical considerations in using animal models [37,38]. Attempts at

improving human cadaveric models are costly and the highly technical process of preparation is time consuming and requires technical support [39–41]. They also do not provide the possibility of training specific situations and are relatively inaccessible. As such, patient specific physical

models or VR simulators provide a more useful alternative.

Surgical simulators have been developed for craniotomy-based and minimally invasive surgical neuroendoscopy-based procedures [42–46]. The ImmersiveTouch VR platform was developed to simulate ETV [47]. While VR platforms will likely become a valuable part of surgical training in

the future, these simulators currently face many financial and functional challenges as they are expensive, lack convincing haptic feedback and do not provide training with real

neuroendoscopic instruments.

2

Training on a bench model transfers well to the human model [48]. In recent years, results for neuroendoscopy training with a phantom model have been satisfying [49]. Unfortunately, this

model can only be used once [50]. Therefore there is still a need for a inexpensive reusable alternative with realistic haptic feedback and usage of real neuroendoscopic instruments.

The purpose of this study is to develop and validate a reusable patient specific brain simulator that can be used to increase trainees’ familiarity with the camera skills, endoscopic instruments,

and hand-eye coordination required to successfully perform an ETV. After the initial development, the model will be tested for fidelity with a group of neurosurgery residents, fellows

and staff surgeons. In addition, a preliminary construct validation study will be performed by evaluating the simulator’s ability to differentiate levels of surgical skill by means of an ETV assessment tool, which will be constructed as part of this study.

The goal of the assessment tool is to determine the evaluation criteria necessary to assess

performance of trainees on ETV procedures. There are currently no standardized methods for teaching neuroendoscopy or evaluating competence in neuroendoscopic surgery [38]. Maintenance of a log of procedures performed by a resident and direct observation without

criteria seem to be unreliable tools for assessment of technical performance [51–53]. However, direct observation with a validated measurement instrument is more reliable [51,53–55] and thus

desirable for assessing neuroendoscopic performance in the operating room (OR) and on simulation models. It has also been shown that procedure-specific checklists can be used to assess surgical skills and to evaluate the transfer of skills from the skills lab to real life settings

[56].

For educational purposes, a procedure specific scale is most convenient as it enables evaluators to highlight relevant tasks that can be addressed for improvement [56–59]. This might stimulate the trainee to engage in deliberative practice with focus on specific elements, thereby hopefully

speeding up the learning process [25,58]. An additional purpose of this study is to develop an assessment tool with content validity for evaluating performance of neurosurgical trainees on the

ETV procedure by means of the modified Delphi method, which is an established methodology for generating consensus among an expert panel [56,60,61].

2. METHODS

This is a four phase study with specific objectives:

- Phase 1, create a synthetic brain simulator for ETV; - phase 2, test the brain simulator for fidelity based on feedback from residents, fellows

and staff; - phase 3, to design an assessment tool for evaluating performance of trainees on the ETV

procedure;

- phase 4, a preliminary construct validation study in which performance of novice surgeons, neurosurgical trainees and experts will be differentiated.

3

2.1 Phase 1: Development of brain simulator for ETV

2.1.1 Features of the simulator Based on a needs assessment for simulation training in neuroendoscopy conducted by Haji et al.

[62], the following features were found to be most important to simulate: selection of ventriculostomy site, navigation within the ventricles, and performance of the ventriculostomy. Thus, these features were incorporated into the simulator.

2.1.2 Construction of the brain simulator

A 3D model of the brain simulator and the skull was created using magnetic resonance imaging (MRI) and computed tomography (CT) images of a four-month-old hydrocephalic child, as per Cheung et al. [63]. The brain, brainstem, prepontine cistern, ventricles, and skull were segmented

to create a 3D surface model using the commercial image processing software Mimics (Materialise NV, Leuven, Belgium). Using Magics (Materialise NV, Leuven, Belgium), a

commercial modeling software, the principle of negative space was used to edit the 3D surface model and generate molds of each segmented brain anatomies. The molds were saved in the sterolithography (STL) file format and printed using a Spectrum Z510 3D printer (Z Corporation,

Burlington, MA). To create the brain simulator, liquid silicone (Dragon Skin®) was used to cast the mold of each segmented region in two stages. First, silicone was poured in one half of the

mold and given sixteen hours to cure. Then the second half was filled with silicone and the first half was positioned on top of the newly filled second half. After that the second half was given another 16 hours to cure; the two halves, now merged, could be removed as a whole from the

mold.

In the process of creating the brain molds, an extra five millimeter layer was cut out of both molds over the outer surface where the prepontine cistern model would later be positioned in order to create a layer of silicone. This layer recreated the clivus and resulted, after extraction of

the printed hard prepontine cistern model, in an open prepontine cistern with a wall. The mechanical properties of the silicone could be adjusted by adding more or less slacker. For the

brain, a ratio of one part A, one part B, and one and a half part slacker was used. A separate silicone based mold was created for the ventricle system. First, the ventricle system

was printed using the Spectrum Z510 3D printer. This ventricle system was submerged in a box filled with silicone and left to cure. The printed ventricle system was then removed from within

the silicone using a midline cut. Hot melted soft sculpting clay was poured into the two silicone molds containing exact copies of the negative space of both halves of the ventricle system. Once the clay cooled and hardened, the two halves were extracted and reconnected. Anatomical details

were hand sculpted into the clay ventricle model, and together with a 3D prepontine cistern model, were anatomically positioned between the two parts of the brain mold. Liquid silicone

was then poured into the brain mold in a two-stage process, as outlined previously. Once the total brain model could be extracted from the brain molds, the prepontine cistern was extracted so that it could be reused for creating other simulators. The clay ventricle system was also extracted

after making an incision using a superior midline approach and subsequently carving out the clay. Now the basic brain simulator was complete, with an open ventricle system and a

prepontine cistern that would later contain the brain stem and the basilar artery. The brain stem

4

was created in a similar way as previously described although a different ratio of silicone was used: one part A, one part B, and half part slacker.

Anatomical details were later added using coloured silicone for veins and arteries, mammillary

bodies and the infundibular recess. The basilar artery was created after the 3D printed basilar artery of the patient using iron wire as a spine structure and applying multiple layers of red coloured silicone. The basilar artery was then connected to the brain stem. Convoluted packaging

foam was coloured red using acrylic paint to artificially create choroid plexus. A skull was made from CT data of the four-month-old patient. The procedure involved segmenting the skull,

saving the data in STL format, and printing it in 3D using the Spectrum Z510 3D printer. The brain was placed in the skull stabilized in a custom-made skull holder that was constructed to fit in a planter (8” Tribeca Planter, Canadian Tire®) purchased in a local hardware store. This

planter was filled with water in order to simulate CSF in the ventricle system.

2.1.3 Construction of a replaceable third ventricle floor

The main purpose for the simulator was to train for ETV. The key element of the ETV procedure is creating a stoma in the third ventricle floor in the tuber cinereum, which is situated between

the mammillary bodies and the optic chiasm [64,65]. Since the brain simulator was designed to be reusable, the third ventricle floor had to be replaceable. In order to do so, the third ventricle

floor was made out of wax paper and attached to a ring. The ring, seated under the third ventricle, contained a small portion that was uncovered precisely where the stoma should be placed (i.e. in the midline, anterior to the mammillary bodies and posterior to the infundibular

recess). After the ventriculostomy, the perforated ring could easily be replaced with a non-perforated ring by removing the brain stem from the simulator.

2.1.4 Incorporating bleeding scenarios

The simulator could also be used to practice multiple bleeding scenarios. For this purpose, two

silicone tube end were positioned at two different points, the choroid plexus and the basilar artery, inside the brain. During the procedure, simulated blood (milk with red food coloring)

could be infused in the brain to allow trainees to learn how to cope with situations in which visual feedback is obscured by excessive bleeding.

2.2 Phase 2: Fidelity of the brain simulator

2.2.1 Design

A group of surgeons in various stages of their career were asked to perform an ETV procedure on the simulator. The simulator was included in some pre-set training courses for neurosurgery

residents – post-graduate year (PGY) 1, 3, 4 and 5, to reach as many residents as possible with maximal adaptation to their schedule. Staff neurosurgeons and fellows were asked to participate

in the study. The required neuroendoscopic tools (Endoscope, MINOP®, D:2.7mm,Viewing Angle 0°, 181mm; Forceps, Grasping and Dissecting, MINOP®, 255mm,10"; Trocar, MINOP®, Diam. 6.0 mm,4 Channels, 150mm,6") were provided by Trudell Medical Marketing Limited.

The endoscopic tower and other resources were available for use in the surgical skills centre in Mount Sinai Hospital in Toronto.

5

2.2.2 Questionnaire

The simulator was tested for fidelity using a questionnaire that was given to the group of

neurosurgical trainees and experienced neurosurgeons. They were asked to give their opinion on the simulator after performing the procedure. The aim of the questionnaire was to evaluate the

verisimilitude of the simulator (realism) and whether or not the participant believes the simulator could increase the competence of the trainee (efficacy). The questionnaire consisted of nine questions, the first four were on its realism and the last five were on its efficacy. The participants

were asked to rate their level of agreement with each item by means of a five-point Likert scale, according to the following descriptions: 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 =

agree, and 5 = strongly agree. The nine items were followed by a comments section in which the participant was asked to suggest areas of improvement. Only correctly completed items were eligible for analysis, and partial scores such as 3.5 or 4.5 were excluded. See table 1 for the full

questionnaire.

2.2.3 Statistical analysis Data collected from the questionnaire was processed using IBM SPSS Statistics for Windows, Version 20.0. Armonk, NY: IBM Corp. Since the feedback from participants was provided as a

categorical response variable, it was reported as median with interquartile range (IQR) or numbers with corresponding percentages.

The average total scores could be calculated by adding the score on each item divided by the number of items, because all items measure the appreciation of participants of the simulator.

Questionnaires partially completed incorrectly were still eligible for analysis as the total score was subsequently calculated by combining the scores of correctly rated items and dividing by the

number of those items. A one-way ANOVA was conducted to compare the means of the total scores of three groups (i.e. PGY 1 = novices, PGY 3 – 6 = senior residents, and fellows and staff = neurosurgeons). In addition, a one-way ANOVA was conducted to compare the means of the

average ratings on realism (items 1 – 4) and efficacy (items 5 – 9). A paired-samples t-test was conducted to compare the average rating on realism and efficacy. P values <0.05 were

considered as statistically significant. 2.3 Phase 3: Construction of the ETV assessment tool

2.3.1 Design

The assessment tool was a set of criteria that determined the skill level of a trainee. The first draft was developed internally and then modified through a content validity study. Content validity is defined as ‘the degree to which the content of a measurement instrument is an

adequate reflection of the construct to be measured’ [66]. Content validity of an assessment instrument should be assessed by an independent panel [67]. Therefore, a group of independent

experts were asked to contribute to the content of each criterion in terms of relevance and wording. Since the intended outcome of this study was to be used for the development of an ETV assessment tool applicable in a training setting, experts involved in teaching the procedure,

neurosurgeons with experience performing and/or have published on ETV, and additional surgeons recommended by Drs. Hoving and Drake were approached to participate in this study.

The experts received information about the study’s goal, importance, procedure, and expected time cost. The invited neurosurgeons that did not immediately respond received a reminder e-

6

mail after respectively one and two weeks. Experts who agreed to participate received an e-mail with a link to the online survey within a week.

The assessment tool criteria were finalized from the feedback collected using the Delphi

technique. As the Delphi method calls for progressive refinement of expert responses over a few rounds, participants were asked to complete the same online survey multiple times until consensus was achieved amongst all participants on which items to include in the assessment

scale.

2.3.2 First draft of lists

Prior to the first round of evaluation, three lists of potential items were generated: steps of the procedure, potential errors that might occur during the procedure, and a global rating scale of the

trainee’s performance.

The potential items lists were generated by means of a review of literature. Search strategies for PubMed were developed using natural language textwords and MeSH headings. For the steps of the ETV procedure, the following search strategy was used:

((ETV[tiab]) OR ("endoscopic third ventriculostomy"[tiab]) OR (ventriculostomy[mh]))

AND (methods[sh] OR standards[sh])

For the second list, the following strategy was used for ETV complications:

((ETV[tiab]) OR ("endoscopic third ventriculostomy"[tiab]) OR (ventriculostomy[mh])) AND ((postoperative complications[mh]) OR (adverse effects[sh]) OR (complication*[tiab]))

These search strategies resulted in 876 and 848 papers, respectively. Due to time limitations only

a portion of these articles were read and cited [2,26–34,62,68–72]. For the global rating scale, citations from two review papers by Ahmed et al. and van Hove et al. on observational tools for assessment of procedural skills were used [52,56,72,73]. Furthermore, Drs. Haji’s and Drake’s

firsthand experiences with ETV formed an important resource of potential items. For the first draft of the two checklists and the global rating scale, see appendix 1.

An online survey was created using http://www.surveymonkey.com (SurveyMonkey Inc., Palo Alto, California, USA) to facilitate data collection. Five-point Likert-type items were used to

allow participants to rate their level of agreement on inclusion of each item in the assessment instrument. All scales were anchored numerically and verbal descriptions were provided; 1 =

strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, and 5 = strongly agree. 2.3.3 Modified Delphi study

We anticipated that three rounds of surveys would be necessary for achieving consensus amongst the participants. In round 1, participants were asked to rate each item on the Likert-scale and

invited to add items, make suggestions for revisions (e.g. to change the wording or content of the individual items), or comment on the initial list of items. A reminder was sent to those who did

7

not respond within two weeks, after that weekly reminders were sent. Prior to round 2, participants received a summary of the results in round 1 to get an indication of where their

scores were placed in relation to the overall picture as per the Delphi method [74]. In round 2, the participants were offered the opportunity to change any of his or her scores that deviated

from the median, with an option to explain their reasoning if they did not change their score [74]. Again, a reminder was sent after two weeks followed by weekly reminders. Prior to round 3, medians and ranges were reported and participants were given an additional opportunity to

change scores that deviated from the median. Again, a reminder was sent to those who did not respond within two weeks, followed by weekly reminders. After the third round, the list of items

that remained was supported by the majority of participants. 2.3.4 Statistical analysis

Data was processed using the Survey-Monkey internet tool and IBM SPSS Statistics for Windows, Version 20.0. Armonk, NY: IBM Corp. Baseline characteristics of participants were

reported as mean and standard deviation (SD) or numbers with corresponding percentages. If variables were not normally distributed, values are reported as median with IQR. It was stated a-priori that all items endorsed by less than 20 % of participants would be deleted (i.e. < 20 %

rated either ‘agree’ or ‘strongly agree’). Consensus was arbitrarily considered to be reached when the rating of at least 80 % of the participants indicated ‘agree’ or ‘strongly agree’ on the

five-point Likert-scale. 2.4 Phase 4: Preliminary construct validation study

2.4.1 Design

In the final phase of the study, novices, senior residents, and neurosurgeons performed an ETV procedure on the brain simulator that was constructed in phase 1. These three groups were used to represent three points in the learning curve of mastering the ETV procedure, from the novice

stage to the expert stage as a neurosurgeon. The performance of each participant was evaluated using a selection from the previously constructed ETV assessment tool and the time it took to

complete the procedure. The measurements were compared between each group to assess the construct validity of the model. The expected outcome was that novice surgeons would need more time to complete the procedure and would make more errors during than the senior

residents. A similar, but probably less distinct, outcome was expected when comparing senior residents with experienced neurosurgeons. The expectation, based on previously mentioned

literature, was to find a steep learning curve that flattened as the novice progresses to being a senior resident [26,34–36,49].

All procedures were performed with neuroendoscopic tools provided by Trudell Medical Marketing Limited (Endoscope, MINOP®, D:2.7mm,Viewing Angle 0°, 181mm; Forceps,

Grasping and Dissecting, MINOP®, 255mm, 10"; Trocar, MINOP®, Diam. 6.0 mm, 4 Channels, 150mm, 6"). Participants were invited to perform the ETV procedure on the brain simulator in the surgical skills center at Mount Sinai Hospital. Participants were instructed to treat the brain

simulator as if it were a real patient. After the participant finished the procedure, they were asked to rate their level of agreement on a five-point Likert scale with the statement “I treated the ETV

simulator as I would treat a real patient”.

8

2.4.2 Data collection

Four novices (PGY 1), twelve senior residents (PGY 3 – 6), and seven neurosurgeons (fellows

and staff) participated in the study. All participants were recruited and tested in July and August 2013. For each participant, the total score was calculated using the checklist. Six of the nine

items in the checklist were worth one point, and three items had more weight (two points) as they had an average rating of more than 4.7 out of 5 as rated by the experts of phase 3. If a participant did not complete one or more of the steps of the checklist correctly, this would result in a lower

total score. For the checklist, see appendix 2. Only completed procedures were eligible for analysis.

Total time is calculated from the moment the instructor gave a start sign until the endoscope was retracted from the brain. Participants were not told the total time was recorded to reduce the

chances of errors due to time pressure.

2.4.3 Statistical analysis

Data was processed using IBM SPSS Statistics for Windows, Version 20.0. Armonk, NY: IBM Corp. As in previous phases, baseline characteristics of participants were reported as mean and

SD or numbers with corresponding percentages. If variables were not normally distributed, values are reported as median with interquartile range (IQR).

A Kruskal-Wallis test was performed to compare the total scores of the three groups. If outcome was significant, post-hoc testing was performed using Mann-Whitney U tests with Bonferroni

correction for group by group comparison (e.g. novices to senior residents, novices to neurosurgeons, and senior residents with neurosurgeons) to specify which groups differed

significantly from each other. The total mean time participants needed to complete the procedure was compared using an unpaired t test. The level of agreement with the “I treated the ETV simulator as I would treat a real patient” item between the three groups was compared with a

Kruskal-Wallis test, and again, if the outcome of this test was significant, post-hoc testing was performed with pair wise Mann-Whitney U tests with Bonferroni correction. We considered p

values <0.05 as statistically significant.

3. RESULTS

3.1 Phase 1

The silicone-based simulator was based on MRI images of a four-month-old child with hydrocephalus. Relevant landmarks include the choroid plexus, mammillary bodies, the

infundibular recess, and the basilar artery. The third ventricle floor is replaceable which makes the simulator reusable. Bleeding scenarios have been incorporated into the simulator. The brain

simulator, encased in the replicated skull and stabilized using the skull-holder is immersed in water. See appendix 3 for images of the brain simulator.

3.2 Phase 2

A total of twenty-five surgeons in various stages of their career participated in the questionnaire:

sixteen neurosurgical residents (four PGY 1, three PGY 3, four PGY 4, four PGY 5, and one

9

PGY 6), five fellows and four adult and pediatric staff neurosurgeons. The results from the questionnaire are shown in table 1.

Two participants indicated a score between two numbers for item 1 (camera view). One

indicated a 3 (neutral) – 4 (agree) and one indicated a 4 (agree) – 5 (strongly agree). One participant did not indicate his level of agreement with item 5 (camera skills). These three responses were excluded from analysis.

There were no statistically significant differences between average overall ratings of the three

group means as determined by one-way ANOVA, p= 0.19: novices (mean 4.3, SD 0.3); senior residents (mean 4.6, SD 0.3); neurosurgeons (mean 4.6, SD 0.2). Additionally, no statistically significant differences were found by conducting one-way ANOVA tests to compare the average

rating on realism (item 1 – 4) and efficacy (item 5 – 9) for each of the three groups (respectively, p= 0.46 and p= 0.16). There was a significant difference in the average ratings for realism (mean

4.3, SD 0.3) and efficacy (mean 4.8, SD 0.4); t(24)= -6.25, p< 0.001. Table 1. Fidelity of brain simulator (n= 25).

Strongly

disagree

Disagree Neutral Agree Strongly

agree

(1) (2) (3) (4) (5)

1. The camera view is comparable to what you would see in a real surgical

scene.

0 (0 %)

0 (0 %)

1 (4%)

14 (61%)

8 (35%)

2. Performing the ventriculostomy on

the floor of the 3rd ventricle of the

model feels like it does in real reality.

0 (0 %)

0 (0 %)

1 (4%)

16 (64%)

8 (32%)

3. The simulator matches actual tissue properties closely.

0 (0 %)

0 (0 %)

1 (4%)

19 (76%)

5 (20%)

4. The bleeding looks realistic.

0 (0 %)

0 (0 %)

0 (0 %)

12 (48%)

13 (52%)

5. This model helps to develop camera

skills needed for ETV.

0 (0 %)

0 (0 %)

0 (0 %)

7 (29%)

17 (71%)

6. This model helps to develop hand-eye coordination needed for ETV.

0 (0 %)

0 (0 %)

0 (0 %)

8 (32%)

17 (68%)

7. The ventriculostomy task is a valuable training exercise.

0 (0 %)

0 (0 %)

0 (0 %)

5 (20%)

20 (80%)

8. Use of this model will increase

resident competency when used to

train residents prior to their first ETV.

0 (0 %)

0 (0 %)

0 (0 %)

5 (20%)

20 (80%)

9. I would be interested in using this model to train residents.

0 (0 %)

0 (0 %)

0 (0 %)

6 (24%)

19 (76%)

10

3.3 Phase 3

Out of thirty-five experts invited, nineteen (54 %) agreed to participate and seventeen (49 %)

actually completed the first round of the online survey. The participants came from Australia (n = 1), Brazil (n = 1), Canada (n = 1), France (n = 2), Germany (n = 2), Italy (n = 1), Israel (n = 1),

the Netherlands (n = 2), Turkey (n = 1), United Kingdom (n = 1) and the United States of America (n = 4). One expert declined, fifteen did not respond to our request. All participants declared to be involved in teaching the ETV procedure to trainees. They indicated to have been

involved in teaching for a mean of 14.6 years (SD 5.9) and all have trained at least five trainees (median 28, IQR 16 – 145). They reported to have performed a median of 250 ETV procedures

(IQR 95 – 613). Ten participants were involved in treating both adult and pediatric populations, six experts were dedicated pediatric neurosurgeons, and one participant performed ETV’s on adult patients only.

In the first round, consensus was achieved for 6 out of 27 items from the checklist concerning

steps of the ETV procedure, and for 20 out of 27 items from the checklist concerning potential errors, and for 5 out of 9 items from the global rating scale. After the first round, 2 items were merged into 1 new item, and 23 items were revised based on comments made by panelists. These

items were included in the second round of the survey in spite of the fact that consensus was achieved for many of them. Eighteen new items were added based on comments made by

panelists in the first round. The second iteration is underway, with the expectation that consensus on all items will emerge

within three rounds.

3.4 Phase 4

Twenty-three surgeons in various stages of their career participated in this study. We divided the group in three categories: novices (PGY 1 residents, n= 4), senior residents (PGY 3 – 6 residents,

n= 12), and neurosurgeons (fellows and staff, n= 7).

During one of the sessions of testing the brain simulator there was a technical problem with the endoscope. The image became increasingly blurred and fuzzy during the procedures of the last two participants, most likely due to a leak in the scope. This led to a termination of the procedure

by the fourth participant due to this technical problem. This participant was excluded from the analysis due to the terminated procedure, thus bringing the total number of participants eligible

for analysis to twenty-two. None of the novices had ever performed an ETV procedure, the senior residents had performed a median of 8 (IQR 0 – 15) ETV procedures.

The results from the testing sessions are summarized in Table 2. The total points per participant were calculated using the checklist. The total time was not measured during the testing of

novices. Since this session was more education oriented, the novices were trained and tested simultaneously, which led to total times that were incomparable with the other groups.

11

Table 2. Total points and total time for three groups

Median total points (IQR) Mean total time in seconds (SD)

Novices 6 (5.3 – 7.5) -

Senior residents 8 (7 – 8) 385 (157)

Neurosurgeons 8 (8 – 8) 256 (122)

A significant difference was found between the total scores of the three groups with the Kruskal-Wallis test, p= 0.02. Post-hoc testing using Mann-Whitney U tests with Bonferroni correction

showed significant difference between novices and neurosurgeons (p= 0.02). No significant difference was found between novices and senior residents (p= 0.2) or between senior residents and neurosurgeons (p= 0.4).

The novice group made seven errors in total. Six (86 %) of these errors were due to insufficient

intraventricular anatomical knowledge (i.e. orientation and position in ipsilateral frontal horn couldn`t be confirmed using anatomical landmarks, and the anatomy of the third ventricular floor couldn`t be identified correctly). The three groups combined made twelve errors. The majority or

errors (n= 9, 75 %) were due to insufficient intraventricular anatomical knowledge. Other errors were advancing the trocar into the ipsilateral frontal horn to an inappropriate depth (n= 2, 17 %),

and selection of an inappropriate ventriculostomy site (n= 1, 8 %). The total time that was needed to complete the procedure for senior residents and neurosurgeons

was compared using an unpaired t test and did not differ significantly (p= 0.09). There were no statistically significant differences for the level of agreement with the “I treated the ETV

simulator as I would treat a real patient” item between the three groups, as determined by Kruskal-Wallis, p= 0.23.

4. DISCUSSION

4.1 Phase 1 The brain simulator can be set-up in minutes and the third ventricle floor can be easily replaced.

After usage, the simulator requires cleaning (only if simulated blood is used in the training). There are no additional maintenance or special storing requirements and that is a huge

improvement over animal or cadaver models. The simulator can be reproduced using the various molds. The only restriction with regard to

making exact copies of the simulator is the ventricle system since it is sculpted by hand.

4.2 Phase 2

ETV is an effective but technically demanding procedure with significant risk [14,29,31,68,71]. Current simulators including human cadavers, animal models and virtual reality systems are

expensive, relatively inaccessible, and can lack realistic sensory feedback [37–41,75]. A realistic, low cost, reusable silicone brain simulator that mimicks the normal mechanical

properties of a four-month-old child with hydrocephalus was constructed and evaluated for fidelity by means of questionnaires (five-point Likert-type items) with sixteen neurosurgical trainees (PGY 1-6) and nine pediatric and adult neurosurgeons. Realistic intraventricular

12

landmarks include the choroid plexus, veins, mammillary bodies, the infundibular recess, and the basilar artery. The thinned out third ventricle floor, which dissects appropriately, is quickly

replaceable. Standard neuroendoscopic equipment including irrigation is used. Bleeding scenarios are also incorporated.

One of the limitations of this study was that a number of the participants did not have previous experience with performing an ETV procedure. All participants were asked to rate their level of

agreement with each of the items on the questionnaire, therefore both novices and experienced participants rated the items concerning realistic feel of the third ventricle floor and of tissue

properties. Whereas the former group could only make a comparison based on what they know the ventriculostomy and brain tissue should feel like, the latter group could make a realistic comparison based on actual haptic perception during past ETV procedures. Between-group

comparisons (novices, senior residents and neurosurgeons) showed no significant difference in average ratings on realism or fidelity. These findings suggest that neurosurgeons in various

stages of their career appreciate the attributes of the simulator in a similar way. Mean scores on both realism and fidelity of the simulator exceed four out of five points on a 5-point Likert scale, however, the mean scores on usefulness were significantly higher than on realism of the

simulator.

What was referred to in this study as a test for fidelity is often called as face validity in literature. Face validity can be defined as the extent to which the assessment tool resembles the real-life situation [57]. Despite some criticism in literature on the meaning of ‘face validity’ [76], it

remains generally accepted that the property of ‘face validity’ is an important factor for determining the value of a simulator, since it contributes to its acceptability. This is especially

true if a simulator is to be used for training purposes, since a lack of face validity can be a strong argument for not using a tool [67]. Setting up an objective design which truly represented the value of the simulator was a significant challenge since we had to depend on the subjective

judgments of experts on the similarities between the simulator and real cases.

The simulator is portable, robust, and can set up in minutes. Over 95 % of participants agreed or strongly agreed that the simulator’s anatomical features, tissue properties and bleeding scenarios were a realistic representation of that experienced during an ETV. Participants stated that the

simulator helped develop the required hand-eye coordination and camera skills, and was a valuable training exercise.

4.3 Phase 3

In this three-round Delphi study conducted among seventeen experts in the field of ETV, three

lists for assessing trainees’ performance on the ETV procedure were constructed; one concerning the steps of the procedure, another concerning potential errors which might occur during the

procedure, and a global rating scale of the trainee’s performance. To our knowledge, a Delphi study has not been performed for this specific purpose before.

The first step in constructing a new assessment tool is content validation [56,67,77]. In literature, content validity is often not evaluated using an accepted scientific method like the Delphi

method used in this study [56]. In the validation process, different types of validation should be integrated to arrive at a conclusion about the validity of the assessment tool [67]. If there was a

13

gold standard for the assessment of one’s performance on the ETV procedure, then the next step in the process of validation of the assessment tool constructed in this study would be to assess

criterion validity [67]. Criterion validity is ‘the degree to which the scores of a measurement instrument are an adequate reflection of a gold standard’ [66]. But since a gold standard is

lacking, the next step in validating the assessment scales is to conduct a construct validation study [67]. Construct validity is defined as the degree to which the scores of a measurement instrument are consistent with hypotheses, e.g. with regard to internal relationships, relationships

with scores of other instruments or differences between relevant groups [66,67].

Another critical element is the acceptability of an assessment tool. Similar to the simulator, if the assessment tool is not accepted by staff or trainees it will not be implemented in the long term [67,77]. When the assessment tool is being used in clinical or training setting, the acceptability

can be measured by means of a questionnaire. After acceptability has been established, another critical element is feasibility (i.e. ease with which the tool can be implemented in a real-life

teaching setting) [67]. To incorporate feasibility into the tool constructed in this study, all the rated steps were chronological, allowing an examiner to follow the checklist intuitively and without having to rate multiple items simultaneously.

The Delphi procedure used in our study required high participant involvement and the limited

dropout rate of participants between each consecutive round may be an expression of the perceived need and value of a validated assessment tool for this procedure [60]. Although the subjective nature of a Delphi study should be taken into account [78], the outcome is useful

nevertheless. By using experts from twelve different countries, all of whom were involved in teaching the ETV procedure to neurosurgical trainees, the final list is expected to be acceptable

for worldwide usage for the purposes specified previously (i.e. assessing neurosurgical trainees on the ETV procedure).

Assessments should also be used as a learning exercise and that is achieved by providing information [77]. In our assessment tool, the trainee can receive individual feedback on various

aspects of the ETV procedure as the two checklists are subdivided in set-up, exposure, navigation, ventriculostomy, confirmation of adequate ventriculostomy and closure. This allows for personalized feedback to identify parts of the procedure that have been mastered and parts

that need further improvement. For example, in phase 4 of this study, 86 % of the errors made by the novices were associated with insufficient intraventricular anatomical knowledge. Part of the

feedback for the novices could be a suggestion to go through an intraventricular anatomy atlas (e.g. Rhoton’s paper on the lateral and third ventricles [65]).

In the future, providing test results from assessments like the one constructed in this study may improve quality of the teaching program and may be used for quality monitoring and control for

courses [77]. 4.4 Phase 4

A quantitative discrimination between the groups was possible due to the assessment tool, and thus it must be noted that the real target of this preliminary construct validation study was the

assessment tool and to a lesser extent the brain simulator. Nevertheless, a construct validation study like the one conducted by our group does demonstrate the educational value of the

14

simulator although the exact value is difficult to quantify. Before a simulator can be used for training purposes, it is of utmost importance to show that one can use the simulator to improve

the competence of the trainee. In literature, simulation training in health professions education is associated with positive effects on knowledge, skills and behaviours [75,79].

A single metric, for example, time, cannot confirm that the trainee has reached the competence level of an expert or give an indication of the quality of the procedure [80]. As such, we intended

to use the time metric in combination with the checklist derived score. However, the time to completion could potentially provide an explanation for outliers in terms of unexpectedly poor

performance. We hypothesized that when a participant made multiple errors in a very short period of time, he or she probably completed the procedure too hastily, and he or she should have then subsequently indicated so in the post-procedural question “I treated the ETV simulator

as I would treat a real patient”. It appeared that none of the participants actually completed the procedure in such a way. Although the total times from the novices are not recorded, and they

were in fact the lowest scoring group, it can be noted that they completed the procedure in longer periods of time than the mean times of the other two groups since they received simultaneous step-by-step explanation and instructions from a staff neurosurgeon.

In a 2011 paper by Filho et al., the authors incorrectly assumed that they assessed the simulators

precision using test-retest, and interrater reliability and subsequently assessed construct validity of the simulator [49]. The actual target of validation design, as previously mentioned, is the measurement tool and not the simulator. The tracking of the learning curve over a mean of six

procedures performed by inexperienced surgeons as described in the article, however, is relevant and illustrative. In the future, this study design can be used on our synthetic simulator. To

produce comparable results, it would be preferable that identical procedural and scoring protocols are used. For this purpose, a comprehensive method description must be provided to reproduce the study on the same or other simulators; this unfortunately was not provided in the

study conducted by Filho et al.

In the preliminary construct validation study, we couldn’t accurately differentiate between novices and senior residents. A possible explanation for this could be that PGY 1 residents were tested on the day of a neuroendoscopy course. They received extensive instructions with

emphasis on the same basic steps used on our scoring list. This led to relatively good scores for most PGY 1 residents. For some of the other, more senior residents, the most recent experience

with ETV may have been some time ago, and the scores were lower. In addition, we could not find a statistically significant difference between expert neurosurgeons and senior residents. The reason for this might be that the assessment tool included only basic steps which should be

mastered within the first years of training. Therefore, the greatest potential for this simulator seems to be in the beginning of the learning curve, with trainees. Trainees can become familiar

with the consecutive steps of the ETV procedure, and increase familiarity with camera skills, endoscopic instruments, and the hand-eye coordination required to perform a safe ETV procedure.

After a resident has mastered the basic steps, the simulator still provides a good platform for

task-specific training, also known as deliberate practice. The part of the procedure with potentially dire complications, the actual ventriculostomy, can be repeatedly practiced in a risk-

15

free environment. Furthermore, the simulator can be used for team training. This could be especially useful in emergency situations such as the basilar artery bleeding scenario which is

incorporated in the simulator.

During one of the testing sessions a technical problem with the neuroendoscopic equipment became apparent, which led to the termination of a procedure by one of the participants. The usage of real neurosurgical tools can be seen as both an advantage and a disadvantage. The

advantage is that the trainees may experience technical failures that can also occur in real life. The Delphi study experts put emphasis on the importance of verifying adequate function of

instruments before commencement of the procedure. The synthetic simulator allows the trainee to practice this crucial step, whereas this is not possible with VR simulators because they do not include technical failures in the simulations. The disadvantage is that the training may take

longer and cost more than expected due to the required replacement or repair in the case of a technical failure.

For our research study, the technical problem led to a modification of the procedural protocol halfway through the study. When we began testing, the participants were instructed to do the

procedure as they would on a real patient. In one part of the procedure, inserting the endoscope in the trocar, the fragile and expensive endoscope was particularly susceptible to technical

problems. To reduce the chances of repeated technical failure with our scope, we instructed the participants to insert the trocar and the scope as a whole to reduce the number of times the scope had to be inserted into the trocar. Due to this change of protocol, the checklist underwent an

adjustment as well and the time metric was not measured in a uniform way across all participants. These changes probably led to an unreliable between-group comparison of the time

metric. There were some additional limitations in this study. The brain simulator could only be used to

train for a specific part of the ETV procedure. The trainee did not have the opportunity to plan the procedure using MRI imaging, nor was there an option for draping and preparing the skin for

incision, actually making the incision and placing the burr hole, and closing the burr hole and watertight closure of the incision after the intraventricular part of the procedure was completed. In addition, there was a limited sample size for both phase 2 and phase 4 of this study. Phase 2

included mostly neurosurgical residents, fellows and staff who were affiliated with the senior researcher overseeing this study in some way. This might have led to a bias in the feedback as

many of the participants had a personal, and some a dependent, relationship with the research group. In phase 3, there was no plenary discussion among the experts. This could be seen as a limitation, but was mitigated by encouraging experts to provide comments, arguments for their

reasoning, and suggestions for revisions of particular items. These responses were included in the feedback that the experts received prior to the next round. It proved difficult to include more

novices in phase 4 of this research study as PGY 2 residents did not have a dedicated course during the time of testing. Due to the busy schedule of the target population (residents and neurosurgeons) we had to adapt our initial study design and reduce the number of times

participants performed the procedure from twice to once. This weakened the statistical power and conclusions on the construct validity. It also prevented measuring the degree of learning

between each attempt for the same individuals, and whether the degree of learning differed between novices and senior residents. Each participant´s session was used to gather data for both

16

the fidelity study and the preliminary construct study; there was no dedicated session for either. Since the need for feedback on the fidelity of the simulator was more pressing than the results of

the preliminary construct validation study, the majority of a senior participant’s session was spent providing feedback rather than focusing on the procedure as one would do in a real

situation. However this is only an observation from the author and is not reflected in the responses from participants as a great majority indicated that they did indeed treat the simulator as they would treat a real patient. A final limitation was that the three groups were supposed be

ordered in an ascending level of surgical experience, but the middle group, referred to here as ‘senior residents’, have a wide range of experience. In retrospect it would have been better to

stratify the cohort based on the actual experience of a participant with the ETV procedure, where the number of ETVs performed, assisted, and observed are taken into account.

Nevertheless a trend was observed when the three groups were compared, which confirmed our initial hypothesis that the results of the three groups could be reflected on a steep learning curve

that, as far as our assessment tool can evaluate, plateaus after the basic steps are mastered by the trainee. These findings are in line with the paper by Filho et al. who reported a decrease to zero errors after a mean of six procedures [49]. We observed a significant difference between the

novices and neurosurgeons, but a less compelling difference between novices and senior residents and between senior residents and neurosurgeons. This does not mean that there is no

difference between novices and senior residents and expert surgeons and senior residents, but merely that it cannot be accurately differentiated with the limited checklist that was used for this study.

5. CONCLUSION

We have created a realistic, low-cost, silicone-based ETV simulator to teach neurosurgical

trainees. This brain simulator may help increase familiarity with the camera skills, endoscopic instruments, and hand-eye coordination required to successfully perform an ETV. This might

reduce the operating room time in teaching hospitals and potentially reduce complication rates. An ETV assessment tool was also constructed as part of this study which, after subsequent thorough validation, can be used as a standardized method for evaluating competence in

performing ETV.

17

6. ACKNOWLEDGEMENTS

The author would like to thank Dr. E.W. Hoving for making the research internship at the Hospital for Sick Children possible and Prof. Dr. J.M. Drake for the great supervision and support during the project. Furthermore, Carling Cheung needs to be thanked for demonstrating

how to work with silicone and Dr. V. Bodani for helping with the gathering of data on the simulator. Dr. F. Haji guided the way in the process of conducting the Delphi study and M.

Darling helped with the PubMed search strategy for the first draft of the three item lists. Various lab members from the Centre for Image Guided Innovation and Therapeutic Intervention were of great help and were always available to give advice or suggestions. Special thanks to Fouzia

Khan, Jorik Booij, and Roxanne Leung who helped with proofreading this report. Prof. Dr. Ir. H.C.W. de Vet (Vrije Universiteit Amsterdam), Y.E.T. Reeuwijk (Universiteit Twente) and Dr.

M.G. Brusse-Keizer (MST Enschede) gave valuable advice concerning the design of the various parts of this study. It was great that Trudell Medical Marketing Limited provided us the neuroendoscopic equipment required for testing the simulator and thanks to L. Satterthwaite and

the technicians at the University of Toronto surgical skills centre in Mount Sinai Hospital for providing us the endoscopic tower and other necessary resources for testing the brain simulator.

Thanks to all the participating residents, fellows and neurosurgeons for offering their precious time and for the relevant feedback on the simulator and thanks to all the experts for their valuable contribution to the Delphi study: Dr. G. Cinalli, Italy; Dr. S. Constantini, Israel; Dr. P.

Decq, France; Dr. S. deRibaupierre, Canada; Dr. C. DiRocco, Germany; Dr. Due-Tønnessen, Norway; Dr. Y. Ersahin, Turkey; Dr. J. Grotenhuis, the Netherlands; Dr. N. Gupta, United States

of America; Dr. E. Hoving, the Netherlands; Dr. Lui, Taiwan; Dr. I. Pollack, United States of America; Dr. C. Sainte-Rose, France; Dr. S. Santoreneos, Australia; Dr. H. Schroeder, Germany; Dr. D. Thompson, United Kingdom; Dr. B. Warf, United States of America; Dr. J. Wellons,

United States of America; Dr. S. Zymberg, Brazil. And lastly, I would like to thank my parents for their unconditional support.

18

7. REFERENCES

1. Baldauf J, Oertel J, Gaab MR, Schroeder HWS. Endoscopic third ventriculostomy in children younger than 2 years of age. Childs Nerv Syst. 2007;23:623–626

2. Bouras T, Sgouros S. Complications of endoscopic third ventriculostomy. J Neurosurg Pediatr 2011;7:643–649

3. Breimer GE, Sival DA, Brusse-Keizer MGJ, Hoving EW. An external validation of the

ETVSS for both short-term and long-term predictive adequacy in 104 pediatric patients. Childs Nerv Syst. 2013;29:1305–11.

4. De Ribaupierre S, Rilliet B, Vernet O, Regli L, Villemure JG. Third ventriculostomy vs ventriculoperitoneal shunt in pediatric obstructive hydrocephalus: results from a Swiss series and literature review. Childs Nerv Syst 2007;23:527–533

5. Drake JM. Endoscopic third ventriculostomy in pediatric patients: the Canadian experience. Neurosurgery. 2007;60:816–881

6. Feng H, Huang G, Liao X, Fu K, Tan H, Pu H, et al. Endoscopic third ventriculostomy in the management of obstructive hydrocephalus: an outcome analysis. J Neurosurg. 2004;100:626-633

7. O’Brien DF, Javadpour M, Collins DR, Spennato P, Mallucci CL. Endoscopic third ventriculostomy: an outcome analysis of primary cases and procedures performed after

ventriculoperitoneal shunt malfunction. J Neurosurg. 2005;103:393–400 8. O'Brien DF, Seghedoni A, Collins DR, Hayhurst C, Mallucci CL. Is there an indication

for ETV in young infants in aetiologies other than isolated aqueduct stenosis? Childs Nerv

Syst 2006;22:1565–1572 9. Surash S, Chumas P, Bhargava D, Crimmins D, Straiton J, Tyagi A. A retrospective

analysis of revision endoscopic third ventriculostomy. Childs Nerv Syst. 2010;26:1693–1698

10. Tuli S, Alshail E, Drake J. Third ventriculostomy versus cerebrospinal fluid shunt as a

first procedure in pediatric hydrocephalus. Pediatr Neurosurg 1999;30:11–15 11. Sacko O, Boetto S, Lauwers-Cances V, Dupuy M, Roux F-E. Endoscopic third

ventriculostomy: outcome analysis in 368 procedures. J Neurosurg Pediatr. 2010;5:68–74 12. Sufianov AA, Sufianova GZ, Iakimov IA. Endoscopic third ventriculostomy in patients

younger than 2 years: outcome analysis of 41 hydrocephalus cases. J Neurosurg Pediatr.

2010;5:392–401 13. Rekate HL. A contemporary definition and classification of hydrocephalus. Semin Pediatr

Neurol. 2009;16:9-15 14. Drake JM. The surgical management of pediatric hydrocephalus. Neurosurgery. 2008;62

Suppl 2:633-640

15. Di Rocco C, Massimi L, Tamburrini G. Shunts vs endoscopic third ventriculostomy in infants: are there different types and/or rates of complications? A review. Childs Nerv

Syst. 2006;22:1573-1589 16. El-Ghandour NMF. Endoscopic third ventriculostomy versus ventriculoperitoneal shunt in

the treatment of obstructive hydrocephalus due to posterior fossa tumors in children.

Childs Nerv Syst. 2011;27:117-26 17. Kulkarni AV, Drake JM, Kestle JRW, Mallucci CL, Sgouros S, Constantini S. Endoscopic

third ventriculostomy vs cerebrospinal fluid shunt in the treatment of hydrocephalus in children: a propensity score-adjusted analysis. Neurosurgery. 2010;67:588-93

19

18. Takahashi Y. Long-term outcome and neurologic development after endoscopic third ventriculostomy versus shunting during infancy. Childs Nerv Syst. 2006;22:1591-602

19. Drake JM, Kulkarni AV, Kestle J. Endoscopic third ventriculostomy versus ventriculoperitoneal shunt in pediatric patients: a decision analysis. Childs Nerv Syst.

2009;25:467-72 20. Sgouros S, Kulkharni AV, Constantini S. The International Infant Hydrocephalus Study:

concept and rational. Childs Nerv Syst. 2006;22:338-345

21. Constantini S, Sgouros S, Kulkarni A. Neuroendoscopy in the youngest age group. World Neurosurg. 2013;79:S23.e1-11

22. Garton HJL, Kestle JRW, Cochrane DD, Steinbok P. A cost-effectiveness analysis of endoscopic third ventriculostomy. Neurosurgery. 2002;51:69-77; discussion 77-8

23. Reznick RK, MacRae H. Teaching surgical skills--changes in the wind. N Engl J Med.

2006;355:2664-9 24. Scott DJ, Cendan JC, Pugh CM, Minter RM, Dunnington GL, Kozar RA. The changing

face of surgical education: simulation as the new paradigm. J Surg Res. 2008;147:189-93 25. Haase J, Boisen E. Neurosurgical training: more hours needed or a new learning culture?

Surgical neurology [Internet]. 2009 Jul [cited 2013 Aug 23];72(1):89–95; discussion 95–7.

Available from: http://www.ncbi.nlm.nih.gov/pubmed/19559933 26. Navarro R, Gil-Parra R, Reitman AJ, Olavarria G, Grant J a, Tomita T. Endoscopic third

ventriculostomy in children: early and late complications and their avoidance. Childs Nerv Syst. 2006;22:506-513

27. Cinalli G, Sainte-Rose C, Maixner WJ (eds) Pediatric Hydrocephalus, 1st ed. Milano:

Springer Milan; 2005. 28. Brockmeyer D, Abtin K, Carey L, Walker ML. Endoscopic third ventriculostomy: an

outcome analysis. Pediatr Neurosurg. 1998;28:236-240 29. Jones RF, Stening WA, Brydon M. Endoscopic third ventriculostomy. Neurosurgery.

1990;26:86-91

30. Dusick JR, McArthur DL, Bergsneider M. Success and complication rates of endoscopic third ventriculostomy for adult hydrocephalus: a series of 108 patients. Surg Neurol.

2008;69:5-15 31. Beems T, Grotenhuis JA. Long-term complications and definition of failure of

neuroendoscopic procedures. Childs Nerv Syst. 2004;20:868-77

32. Chowdhry SA, Cohen AR. Intraventricular neuroendoscopy: complication avoidance and management. World Neurosurg. 2013;79:S15.e1-S15.e10

33. Erşahin Y, Arslan D. Complications of endoscopic third ventriculostomy. Childs Nerv Syst. 2008;24:943-948

34. Schroeder HWS, Niendorf WR, Gaab MR. Complications of endoscopic third

ventriculostomy. J Neurosurg. 2002;96:1032-40 35. Furlanetti LL, Santos MV, de Oliveira RS. The Success of Endoscopic Third

Ventriculostomy in Children: Analysis of Prognostic Factors. Pediatr Neurosurg. 2013 [Epub ahead of print]

36. Jones RF, Kwok BC, Stening WA, Vonau M. The current status of endoscopic third

ventriculostomy in the management of non-communicating hydrocephalus. Minim Invasive Neurosurg. 1994;37:28-36

37. Walker RL. Human and animal subjects of research: the moral significance of respect versus welfare. Theor Med Bioeth. 2006;27:305-31

20

38. Cohen AR, Lohani S, Manjila S, Natsupakpong S, Brown N, Cavusoglu MC. Virtual reality simulation: basic concepts and use in endoscopic neurosurgery training. Childs

Nerv Syst. 2013 [Epub ahead of print] 39. Aboud E, Al-Mefty O, Yaşargil MG. New laboratory model for neurosurgical training that

simulates live surgery. J Neurosurg. 2002;97:1367-72 40. Hayashi N, Kurimoto M, Hamada H, Kurosaki K, Endo S, Cohen AR. Preparation of a

simple and efficient laboratory model for training in neuroendoscopic procedures. Childs

Nerv Syst. 2008;24:749-51 41. Tubbs RS, Loukas M, Shoja MM, Wellons JC, Cohen-Gadol AA. Feasibility of

ventricular expansion postmortem: a novel laboratory model for neurosurgical training that simulates intraventricular endoscopic surgery. J Neurosurg 2009;111:1165-1167

42. Robison RA, Liu CY, Apuzzo MLJ. Man, mind, and machine: the past and future of

virtual reality simulation in neurologic surgery. World Neurosurg. 2011;76:419-30 43. Delorme S, Laroche D, DiRaddo R, Del Maestro RF. NeuroTouch: a physics-based virtual

simulator for cranial microneurosurgery training. Neurosurgery. 2002;71:32-42 44. Choudhury N, Gélinas-Phaneuf N, Delorme S, Del Maestro R. Fundamentals of

Neurosurgery: Virtual Reality Tasks for Training and Evaluation of Technical Skills.

World Neurosurg. 2012 [Epub ahead of print] 45. Neubauer A, Wolfsberger S. Virtual endoscopy in neurosurgery: a review. Neurosurgery.

2013;72 Suppl 1:97–106 46. Chan S, Conti F, Salisbury K, Blevins NH. Virtual reality simulation in neurosurgery:

technologies and evolution. Neurosurgery. 2013;72 Suppl 1:154–64

47. Alaraj A, Charbel FT, Birk D, Tobin M, Luciano C, Banerjee PP, et al. Role of cranial and spinal virtual and augmented reality simulation using immersive touch modules in

neurosurgical training. Neurosurgery. 2013;72 Suppl 1:A115-23 48. Anastakis DJ, Regehr G, Reznick RK, Cusimano M, Murnaghan J, Brown M, et al.

Assessment of technical skills transfer from the bench training model to the human model.

Am J Surg. 1999;177:167-70 49. Filho FVG, Coelho G, Cavalheiro S, Lyra M, Zymberg ST. Quality assessment of a new

surgical simulator for neuroendoscopic training. Neurosurg Focus. 2011;30:E17 50. Zymberg S, Vaz-Guimarães Filho F, Lyra M. Neuroendoscopic training: presentation of a

new real simulator. Minim Invasive Neurosurg 2010;53:44-46

51. Reznick RK. Teaching and testing technical skills. Am J Surg. 1993;165:358-61 52. Memon MA, Brigden D, Subramanya MS, Memon B. Assessing the surgeon’s technical

skills: analysis of the available tools. Acad Med. 2010;85:869-80 53. Beard JD. Assessment of surgical competence. Br J Surg. 2007;94:1315-6 54. Cristancho S, Moussa F, Dubrowski A. Simulation-augmented training program for off-

pump coronary artery bypass surgery: developing and validating performance assessments. Surgery. 2012;151:785-795

55. Scavone BM, Sproviero MT, McCarthy RJ, Wong CA, Sullivan JT, Siddall VJ, et al. Development of an objective scoring system for measurement of resident performance on the human patient simulator. Anesthesiology. 2006;105:260-266

56. Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB. Observational tools for assessment of procedural skills: a systematic review. Am J Surg. 2011;202:469-480.e6

57. Sugden C, Aggarwal R. Assessment and feedback in the skills laboratory and operating room. Surg Clin North Am. 2010;90:519-33

21

58. Sanfey H, Dunnington G. Verification of proficiency: a prerequisite for clinical experience. Surg Clin North Am. 2010;90:559-67

59. Eubanks TR, Clements RH, Pohl D, Williams N, Schaad DC, Horgan S, et al. An objective scoring system for laparoscopic cholecystectomy. J Am Coll Surg.

1999;189:566-74 60. Boogaard S, Heymans MW, Patijn J, de Vet HC, Faber CG, Peters ML, et al. Predictors

for persistent neuropathic pain--a Delphi survey. Pain Physician 2011;14:559-568

61. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs. 2000;32:1008-15

62. Haji FA, Dubrowski A, Drake J, de Ribaupierre S. Needs assessment for simulation training in neuroendoscopy: a Canadian national survey. J Neurosurg. 2013;118:250-257

63. Cheung CL, Looi T, Drake J, Kim PCW. Magnetic resonance imaging properties of

multimodality anthropomorphic silicone rubber phantoms for validating surgical robots and image guided therapy systems. Proc. SPIE 8316, Medical Imaging 2012: Image-

Guided Procedures, Robotic Interventions, and Modeling, 83161X doi:10.1117/12.908274

64. Romero ADCB, Aguiar PHP De, Borchartt TB, Conci A. Quantitative ventricular

neuroendoscopy performed on the third ventriculostomy: anatomic study. Neurosurgery. 2011;68(2 Suppl Operative):347–54; discussion 353–4

65. Rhoton AL. The lateral and third ventricles. Neurosurgery. 2002;51(4 Suppl):S207–71 66. Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The

COSMIN study reached international consensus on taxonomy, terminology, and

definitions of measurement properties for health-related patient-reported outcomes. J Clin Epidemiol. 2010;63:737-745

67. De Vet HCW, Terwee CB, Mokkink LB, Knol DL. Measurement in Medicine. 1st ed. New York: Cambridge University Press; 2011. p. 347.

68. Peretta P, Ragazzi P, Galarza M, Genitori L, Giordano F, Mussa F, et al. Complications

and pitfalls of neuroendoscopic surgery in children. J Neurosurg. 2006;105(3 Suppl):187-93

69. García LG, López BR, Botella GI, Páez MD, da Rosa SP, Rius F, et al. Endoscopic Third Ventriculostomy Success Score (ETVSS) predicting success in a series of 50 pediatric patients. Are the outcomes of our patients predictable? Childs Nerv Syst. 2012;28:1157-62

70. Hopf NJ, Grunert P, Fries G, Resch KD, Perneczky A. Endoscopic third ventriculostomy: outcome analysis of 100 consecutive procedures. Neurosurgery. 1999;44:795-804;

discussion 804-6 71. Bouramas D, Paidakakos N, Sotiriou F, Kouzounias K, Sklavounou M, Gekas N.

Endoscopic third ventriculostomy in obstructive hydrocephalus: surgical technique and

pitfalls. Acta Neurochir Suppl. 2012;113:135-139 72. Martin JA, Regehr G, Reznick R, MacRae H, Murnaghan J, Hutchison C, et al. Objective

structured assessment of technical skill (OSATS) for surgical residents. Br J Surg. 1997;84:273-8

73. Van Hove PD, Tuijthof GJM, Verdaasdonk EGG, Stassen LPS, Dankelman J. Objective

assessment of technical surgical skills. Br J Surg. 2010;97:972–87 74. Powell C. The Delphi technique: myths and realities. J Adv Nurs. 2003;41:376-82

22

75. Cook DA, Brydges R, Hamstra SJ, Zendejas B, Szostek JH, Wang AT, et al. Comparative effectiveness of technology-enhanced simulation versus other instructional methods: a

systematic review and meta-analysis. Simul Healthc. 2012;7:308-20 76. Downing SM. Face validity of assessments: faith-based interpretations or evidence-based

science? Med Educ. 2006;40:7-8 77. Van der Vleuten C. The assessment of professional competence: Developments, research

and practical implications. Adv Health Sci Educ Theory Pract 1996;1:41– 67

78. Pill J. The Delphi method: Substance, context, a critique and an annotated bibliography. Socio-Economic Planning Sciences 1971;5:57-71

79. Cook DA, Hatala R, Brydges R, Zendejas B, Szostek JH, Wang AT, et al. Technology-enhanced simulation for health professions education: a systematic review and meta-analysis. JAMA. 2011;306:978–88

80. Sturm LP, Windsor JA, Cosman PH, Cregan P, Hewett PJ, Maddern GJ. A systematic review of skills transfer after surgical simulation training. Ann Surg. 2008;248:166-79

23

Appendix 1. Three initial lists for online survey

First Checklist: Procedural Steps

A. Set-up of Endoscope and instruments

1. The camera was oriented to ensure the view was in the upright position before commencing the procedure 2. The camera was focused prior to commencing the procedure

3. The irrigation system was checked to ensure appropriate function 4. The appropriate irrigation solution was used (e.g. isotonic saline at body temperature)

5. Supportive equipment (e.g. articulating arm) was set up appropriately at the beginning of the procedure 6. The endoscopic instruments were checked to ensure appropriate function (e.g. instruments go down easily, full closure of alligator forceps, functioning monopolar cautery, etc.)

7. The endoscope was checked to ensure appropriate function (e.g. checked for rounded edges of scope, smooth walls of sheath, etc.)

B. Exposure

8. The image guidance was appropriately set-up and used to plan the cortical entry point, target and trajectory to reach the Foramen of Monro and floor of the 3rd ventricle (optional)

9. The position and size of the skin incision and burrhole were appropriate for the individual patient (i.e. at or anterior to the coronal suture and lateral to the midline) 10. The trajectory used to access the lateral ventricle was appropriate

11. The ventricle was tapped with a smaller brain needle before passing the larger sheath/trocar (optional) 12. The sheath/trocar was advanced into the ipsilateral frontal horn to an appropriate depth in the ventricle

13. Orientation and position in the ipsilateral frontal horn was confirmed using anatomical landmarks 14. Hemostasis was appropriately maintained throughout the procedure

C. Navigation

15. The endoscope was maintained in the selected trajectory 16. The endoscope was navigated through Foramen of Monro into 3rd ventricle

17. The anatomy of the third ventricular floor was correctly identified 18. An appropriate ventriculostomy site was selected

D. Ventriculostomy 19. An initial perforation at the ventriculostomy site was made using an accepted method (e.g. blunt perforation, etc.)

20. The ventriculostomy was widened to ensure patency using an accepted method (e.g. forceps or Fogarty balloon dilation)

E. Confirmation of adequate ventriculostomy

21. The adequacy of the ventriculostomy was assessed by visualizing bidirectional movement of 3rd ventricular floor

22. The endoscope was advanced to the ventriculostomy to visualize the pre-pontine cistern and confirm that no additional membranes were blocking CSF flow 23. If Lillequist membranes were present, they were perforated using an appropriate technique

F. Closure 24. The ventricle was refilled with irrigation solution to remove air

25. Upon removing the endoscope, the fornix was inspected to ensure no significant damage during procedure 26. The burrhole was appropriately covered (e.g. with cap, gelfoam, acrylic, bone dust etc.)

27. The skin was closed in water tight fashion

24

Second Checklist: Procedural Pitfalls

A. Improper set-up

1. Rotated camera

2. Unfocussed image 3. Image blurred by lens: debris / function

4. Wrong temperature or osmolality of irrigation solution 5. Inappropriate checking or set-up of endoscopic instruments or supports (e.g. poorly functioning alligator forceps, improper set-up of the articulating arm)

B. Improper entry / trajectory 6. Malposition of cortical entry (too far anterior/posterior or lateral/medial), resulting in abnormal orientation upon entry into ventricle 7. Endoscope inserted at improper trajectory (too far anterior/posterior or lateral/medial), resulting in an inability to advance into the lateral or 3rd ventricle without damaging adjacent neural structures

C. Poor exposure

8. Inability to insert endoscopic apparatus due to small skin exposure or inadequate hemostasis 9. Difficulty accessing cortical surface or ensuring adequate trajectory of trocar/sheath due to inadequate bony exposure

10. Unnecessary neural damage due to advancing trocar/sheath without establishing a track with a smaller instrument 11. Endoscope inserted to an inappropriate depth (e.g. too shallow or too deep)

12. Failure to establish orientation and position in lateral ventricle

D. Traction injuries, failure to identify anatomy

13. Tearing of ependymal vessels due to excessive endoscope movement

14. Damage to vascular structures (septal/thalamostriate veins, choroid plexus) as endoscope advanced through FOM (resulting in excessive bleeding or requiring cautery) 15. Excessive traction on fornix upon advancement into 3rd ventricle

16. Failure to appropriately identify anatomy of 3rd ventricular floor, resulting in inappropriate selection of ventriculostomy site 17. Obstruction of irrigation outflow causing raised ICP

E. Technically inadequate ventriculostomy

18. Failure to fenestrate the 3rd ventricular floor 19. Inadequate size of fenestration

20. Inappropriate placement of fenestration 21. Technically unsafe fenestration, e.g. excessive movement, rough handling of tissues, or failure to abort procedure when appropriate to do so

F. Failure to recognize technically inadequate fenestration

22. Did not recognize or attempt to correct lack of bidirectional flow 22. Did not check for membranes or other obstructions to CSF flow into pre-pontine cistern

23. Failed to open membranes of Lilequist if present

G. Improper closure

24. Collapse of ventricles after release of too much CSF

25. Traction injury to fornix upon removal of endoscope 26. CSF leak due to inadequate closure of burrhole or skin incision

25

Third list: Global Rating Scale

Item and Anchors

1. Preparation for the procedure (Rating Anchors: 1 = Did not organize or set-up equipment well; had to stop procedure frequently to prepare or fix equipment; 3 = Equipment generally organized; occasionally had to stop and prepare or fix equipment; 5 = All equipment neatly organized, prepared and ready for use)

2. Respect for tissue (Rating Anchors: 1 = Frequently used unnecessary force on tissue or caused damage; 3 = Careful handling of tissue but occasionally caused inadvertent damage; 5 = Consistently handled tissues appropriately with minimal damage) 3. Time and motion (Rating Anchors: 1 = Many unnecessary moves; 3 = Efficient time and motion, but some unnecessary moves; 5 = Clear economy of hand movement and maximum efficiency) 4. Instrument handling (Rating Anchors: 1 = Repeatedly makes tentative or awkward moves with instruments; 3 = Competent use of instruments but occasionally appeared stiff or awkward; 5 = Fluid movements with instruments and no stiffness or awkwardness)

5. Knowledge of instruments (Rating Anchors: 1 = Frequently asked for wrong instrument or used inappropriate instrument; 3 = Knew names of most instruments and used appropriate instrument; 5 = Obviously familiar with instruments and their names)

6. Flow of operation (Rating Anchors: 1 = Frequently stopped operating and seemed unsure of next step in the procedure; 3 = Demonstrated some forward planning with reasonable progression of procedure; 5 = Obviously planned course of operation with effortless flow from one step to the next)

7. Use of assistants (Rating Anchors: 1 = Consistently used assistants poorly or failed to use assistants; 3 = Appropriate use of assistants most of the time; 5 = Strategically used assistants to the best advantage at all times) 8. Knowledge of specific procedure (Rating Anchors: 1 = Deficient knowledge. Required specific instruction at most steps of operation; 3 = Knew all

important steps of operation; 5 = Demonstrated familiarity with all steps of the operation)

9. Overall performance (Rating Anchors: 1 = Very poor; 3 = Competent; 5 = Clearly superior)

26

Appendix 2. Checklist for preliminary construct validation Name ‘instructor’: Date:

Novice

Resident Year of residency:

Have you performed a ventriculostomybefore:

If yes, how many:

Fellow

Staff

First attempt or Second attempt

Yes No Points

1. The camera was oriented to ensure the view was in the upright before commencing

the procedure

2. The camera was focused prior to commencing the procedure

3. The sheath/trocar was advanced into the ipsilateral frontal horn to an appropriate

depth in the ventricle

4. Orientation and position in ipsilateral frontal horn was confirmed using anatomical

landmarks

5. The anatomy of the third ventricular floor was correctly identified

6. An appropriate ventriculostomy site was selected

7. An initial perforation at the ventriculostomy site was made using an accepted

method (e.g. blunt perforation, etc.)

8. The ventriculostomy was widened to ensure patency using an accepted method (e.g.

forceps or Fogarty balloon dilation)

9. The endoscope was advanced to the ventriculostomy to visualize the pre-pontine

cistern and confirm no additional membranes blocking CSF flow

Total time: Total points:

I treated the ETV simulator as I would treat a real patient

Strongly disagree

1

Disagree

2

Neutral

3

Agree

4

Strongly agree

5

27

Appendix 3. Images of the brain simulator

Figure 1. Brain simulator

1. Planter

2. Skull

3. Brain

4. Skull holder

5. Brain stem plus basilar artery

Figure 2. Set-up of testing brain simulator

1. Neurosurgical trainee

2. Neuroendoscope

3. Neuroendoscopic tower

4. Brain simulator in planter

5. Alligator forceps