is the script-concordance test a valid instrument for assessment of intra-operative decision-making...
TRANSCRIPT
Is the Script-Concordance Test a Valid Is the Script-Concordance Test a Valid Instrument for Assessment of Intra-Instrument for Assessment of Intra-operative Decision-making Skills?operative Decision-making Skills?
Brent Zabolotny1, Robert Gagnon2, Bernard Charlin2 and Sarkis Meterissian1
1Division of General Surgery and Center for Medical Education, McGill University
2Unit for Research and Development in Health Sciences Education, University of Montreal
The Basics of Medical ReasoningThe Basics of Medical Reasoning
Medical reasoning is a hypothetico-deductive process (Ellstein 1978, Barrows et al
1982) involving:
1. Early generation of hypotheses
2. Oriented data collection
3. Decision-making judgements
Problem-solving in Clinical MedicineProblem-solving in Clinical Medicine
Well-defined problems are rare:Clear data, clear goals, clear solutionsTechnical knowledge
Ill-defined problems are common:Uncertain data, unclear goals, different solutions
Professional knowledge
Donald A Schön, The Reflective Practitioner, 1983
What is an Expert?What is an Expert?
Experts can store, use and retrieve knowledge more efficiently than novices
Experts possess elaborated networks of knowledge fitted to the tasks they regularly do
These networks are called SCRIPTS(Feltovich 1983, Schmidt 1990, Charlin 2000)
SCRIPT TheorySCRIPT Theory
In clinical situations, clinicians fill their working memory with knowledge related to each relevant hypothesis
Then a deductive process is used to seek information to either accept or refute each respective hypothesis
A test has been developed that measures this deductive-reasoning process – The Script Concordance Test (SCT)
Script Concordance TestScript Concordance Test A challenging context
– Authentic clinical situation
– Not enough data or conflict
- Relevant options A Likert scale
– Capture opinion Scoring system
– Takes into account variability of experts
– Aggregate scoring method
The Qualities of a Good SCTThe Qualities of a Good SCT There must be at least 50-60 items to achieve a
reliability coefficient (Cronbach alpha) of 80 (r = .80)
Questions should not lead to a single answer (they must not be clearcut) otherwise the SCT would really be an MCQ
Conversely, the answers should not be too widely spread thus indicating a vague question
Evidence that this works?Evidence that this works?
PurposePurpose
To develop and validate a script-concordance test for assessment of clinical reasoning (ie: intra-operative decision-making) in General Surgery Residents.
Methods: SCT FormatMethods: SCT Format
Methods: Example (1)Methods: Example (1) A 50 year old male sun worshipper presents with a 2 cm. dark pigmented lesion on the R calf. Initial punch biopsy revealed a 2.3 mm. Breslow thickness superficial spreading melanoma. In your management:
If you were planning… and found… the planned management is:
1) Only a wide excision with 2 cm margins
A palpable inguinal node in the operating room
-2 -1 0 +1 +2
2) A wide local excision plus only a sentinel node biopsy of the groin nodes
The hot/blue node to be clearly black with tumor
-2 -1 0 +1 +2
3) A wide local excision plus a superficial node dissection for a palpable groin node
Involvement of Cloquet’s node at frozen section
-2 -1 0 +1 +2
Methods:Example (2)Methods:Example (2) A 35 year old female of Chilean descent presents with an upper GI bleed. Endoscopy reveals a gastric cancer and CT scan fails to reveal any distant spread. In your management:
If you were planning… and found… the planned management is:
4) A subtotal gastrectomy A single peritoneal implant biopsy proven adenocarcinoma
-2 -1 0 +1 +2
5) A subtotal gastrectomy Multiple palpable nodes along both the greater/lesser curvature
-2 -1 0 +1 +2
6) A total gastrectomy That a 5 cm. proximal margin with the esophagus was achievable with a subtotal resection
-2 -1 0 +1 +2
Methods: Scoring an SCTMethods: Scoring an SCT
The scoring grid of an SCT should be derived by administering it to 10 experts (aggregate scoring method)
Based on their responses some questions (if too vague or too clearcut) can be eliminated
Scoring grid takes into account variability due to the uncertainty of the clinical situation
The Scoring GridThe Scoring Grid
-2 -1 0 +1 + 2
Number of experts choosing answer 0 0 5 4 1
Score 0 0 5/10 4/10 1/10
Transformed score 0 0 5/5 4/5 1/5
Credit per item 0 0 1 0.8 0.2
MethodsMethods
Participants: 36 out of the 40 McGill surgical residents wrote the exam
Panel of experts: 10 board-certified general surgeons
MethodsMethods
Statistical Analysis:- Reliability: Cronbach alpha coefficient- Analysis of item-to-total correlation used to select
the best items for the final analysis- Construct validity tested with a one-way ANOVA
with post-hoc comparisons test and planned contrasts
- All p values at alpha<5% were considered significant
ResultsResults
Developed 196 questionsFace and content validity assessed by 3
independent Board-certified General Surgeons
Final exam consisted of the best 100 questions
ResultsResults
Initial test: 100 items Cronbach alpha : 0.70
Optimisation of the items (elimination of items with an inter-item correlation less than 10%) : final test used for analysis = 62 items
Cronbach alpha of the optimised test: 0.85
ResultsResultsExamination Scores (62-item test) by Resident Level
Resident Level
n Mean Score (%)
Std Deviation (%)
R1 8 52.5 9.96
R2 6 62.5 5.12
R3 9 68.3 9.19
R4 6 75.7 9.61
R5 7 68.0 6.44
ResultsResults
40.00
45.00
50.00
55.00
60.00
65.00
70.00
75.00
80.00
R1
R2
R3
R4
R5
DiscussionDiscussion
The SCT developed in this study demonstrated:
1. excellent internal reliability
2. Good construct validity
Discussion:Discussion:
Questions:1. Correlation with in-training evaluations?2. Correlation with oral exam scores?3. Ideal number of panel members?4. Makeup of the panel members?
subspecialists vs general surgeons5. Can the SCT identify residents having
difficulty with clinical reasoning?
More on More on
Web site:
http://www.cme.umontreal.ca/tcs
E-mail :
ResultsResults
Evaluation of the exam:
- 64% found the SCT tested their intra-operative decision-making skills
- 50% found the exam initially difficult but 79% found that the exam became easier as they better understood the format
- 71% found that the SCT exam was a valuable and educational examination tool