![Page 1: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/1.jpg)
The SlideTutor Project
Rebecca Crowley, MD, MS
University of Pittsburgh School of Medicine
Department of Biomedical Informatics
October 5, 2007
![Page 2: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/2.jpg)
2/33
Background
• Developing diagnostic expertise (for example inPathology) is difficult and time-consuming
• In domains outside of medicine, intelligentcomputer-based training is common– Aviation simulators
– Nuclear and power plant simulators
– Military
• Research in many domains has shown thatcomputer systems can simulate the well-knownbenefits of one-on-one teaching
![Page 3: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/3.jpg)
3/33
Outline
• What are intelligent tutoring systems (ITS)?
• How did we develop SlideTutor?
• How does SlideTutor work?
• How effective is the system?
• What are our plans for future research?
![Page 4: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/4.jpg)
Q: What are Intelligent Tutoring Systems?
![Page 5: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/5.jpg)
5/33
Intelligent Tutoring Systems
Adaptive, flexible, individually tailoredinstruction
Not ‘text and test’ but rather coached practiceenvironments
System able to ‘solve the problem’ on it’s ownand therefore able to provide feedback onstudent actions
Monitor student’s progress and change teachingbased on how the student is learning
![Page 6: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/6.jpg)
6/33
A Ballroom Dance Lesson
Student leads
Right step?…teacher follows
Wrong Step?…teacher corrects
Lost?…teacher leads
![Page 7: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/7.jpg)
7/33
Intelligent Tutoring Systems
Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter avionics troubleshooting, proven educational benefit over classroom
learning Very few Medical ITS have been developed – none
have been evaluated:• GUIDON and GUIDON II (Clancey)• Cardiac-Tutor (Woolf et al) – ACLS• Rad-Tutor (Azevedo, Lajoie)–Mammogram
Interpretation
![Page 8: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/8.jpg)
Student Model
PedagogicKnowledge
Interface
Expert Module
•Allow correct steps•Correct errors•Give hints on next step
•Collect data on what student does•Make predictions on what student knows•Provide data for pedagogic decision making
•Case sequence•When to intervene•How much to intervene•How to intervene
![Page 9: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/9.jpg)
9/33
Why are ITS so hard in Medical Domains?
• Expert Module
– Uncertainty and missing information
– Enormous amounts of declarative knowledge
– Knowledge changes over time, KB requires verification/maintenance
– Carefully selected and controlled case bases needed
• Interface
– No formal problem-solving notation
– Hard to reproduce environments in which medical PS occurs
• Student and Pedagogic Models
– Unclear how to model acquisition of skills which involve so much declarativeknowledge. How decomposable are these skills really?
– Very little research to guide pedagogic modeling outside of causal reasoningdomains (CircSim)
• Evaluation and Deployment
– Limited access to subjects for formative and summative evaluation
– No classrooms, potentially hard to deploy systems when they are done
![Page 10: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/10.jpg)
Q: How did we develop SlideTutor?
![Page 11: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/11.jpg)
11/33
Data Collection
![Page 12: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/12.jpg)
12/33
Methods
• Studied Novices, Intermediates, and Experts
• Think aloud protocols
• People can articulate the intermediate steps ofcognition that are available in Working Memory
• Collect detailed data and analyze
• Develop ideas about how expertise is acquired
![Page 13: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/13.jpg)
Crowley et al, JAMIA 2003
![Page 14: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/14.jpg)
Crowley and Medvedeva, AIM 2006
![Page 15: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/15.jpg)
15/33
Scaffolding skill acquisition• Searching skills
– Prevent students from interpreting non-diagnostic areas
– Require that students see the entire slide
– Monitor and provide feedback on specific search skills
• Feature identification
– Encourage identification and full specification of evidence
– Correct errors and give explanations
– Combine visual and symbolic aspects of training
• Hypothesis triggering and testing
– Provide feedback for forwards and backwards reasoning
– Support efforts to distinguish between hypotheses
• But allow enough flexibility so that system will support studentswho are in many stages of skill acquisition
![Page 16: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/16.jpg)
Q. How does SlideTutor work?
![Page 17: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/17.jpg)
17/33
Student has alreadyidentified some
evidence
Hints are hierarchicallystructured providing
increasing help
System provides hinton next best step
System moves viewer ormoves and annotates the area
for students
1. Student indicates new finding
2. Clicks on finding, creating ‘x’
3. Selects finding
Tutor permitscorrect answer
Student requests hintabout next best step
Later hint tells student theimportant quality and then opens
menu and displays choices
1. Student indicates new hypothesis
2. Selects hypothesis
1. Student indicates new refute link
2. Student draws betweenevidence and hypothesis
Objectflashes
Bug message explainserror
![Page 18: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/18.jpg)
18/33
IdentifiedEvidence
ClustersupportssameHypotheses
Moreevidence
Cluster nowdifferentiates
A hypothesis…reasonable but
wrong
“Backwards”reasoning
Cluster does notsupport Acute
Burn
![Page 19: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/19.jpg)
19/33
Student Model
![Page 20: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/20.jpg)
20/33
Analysis by hint and error
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
1 3 5 7 9 11 13 15 17 19
Problem
Hin
tR
ate
Case-FocusedKnowledge-Focused
0
0.1
0.2
0.3
0.4
0.5
0.6
1 3 5 7 9 11 13 15 17 19
Problem
Err
or
Rate
Case-FocusedKnowledge-Focused
![Page 21: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/21.jpg)
21/33
Analysis by learning curves
![Page 22: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/22.jpg)
Q. How effective is the system?
![Page 23: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/23.jpg)
23/33
Study Design
Crowley et al, JAMIA 2007
![Page 24: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/24.jpg)
M
F, IL
E
G
A
H,K
J
CD
B
![Page 25: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/25.jpg)
25/33
Subjects
• 21 pathology residents from 2 academic programs
• PGY1 – PGY5
![Page 26: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/26.jpg)
26/33
Learning gains on multiple-choice test
Multiple Choice Test Scores
0.0
20.0
40.0
60.0
80.0
100.0
Pretest Posttest Retention
Test
Sc
ore
*** ***
![Page 27: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/27.jpg)
27/33
Learning gains on case diagnosis test
Case Diagnosis: Tutored Patterns vs
Untutored Patterns
0
20
40
60
80
100
Pretest Posttest Retention
Test
Sc
ore
TutoredPatternsUntutoredPatterns
******
![Page 28: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/28.jpg)
28/33
0
0.2
0.4
0.6
0.8
1
1 3 5
Certainty
Dia
gn
os
tic
Te
st
Pe
rfo
rma
nc
e
0
0.2
0.4
0.6
0.8
1
1 3 5
CertaintyD
iag
no
sti
cT
es
t
Pe
rfo
rma
nc
e
Pretest
Post test
Retent ion Test
Optimum
Pretest
Post test
Retent ion Test
Optimum
Meta-cognitive gains –What do they know about what they know?
Crowley et al, AIED 2005
![Page 29: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/29.jpg)
29/33
Tutoring other cognitive skills
Saadawi et al, AHSE (in press)
![Page 30: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/30.jpg)
30/33
Learning Gains
Saadawi et al, AHSE (in press)
![Page 31: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/31.jpg)
31/33
Conclusions
• With some critical modifications, the ITSparadigm can be used effectively in medicine
• SlideTutor significantly improves performance indiagnostic problem solving and reporting, evenafter a single session, and learning gains areretained at one week (diagnostic)
![Page 32: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/32.jpg)
Q. What are our plans for future research?
![Page 33: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/33.jpg)
33/33
Ongoing and future Work
• Deployment– Develop case library and additional knowledge models
– Cancer Training Web – collaboration with University ofPennsylvania
• Evaluation– Summative evaluation of SlideTutor against “standard
practice”
– Can SlideTutor decrease medical errors? (Collaborationwith D. Grzybicki, AHRQ)
• Basic Research– Student Modeling and performance prediction
– Meta-cognitive Training
– Comparing pedagogic approaches (depth v. breadth)
![Page 34: The SlideTutor Project - University of Pittsburgh 2.pdf · Intelligent Tutoring Systems Many, many successful systems in diverse domains mathematics, programming, physics, F15 fighter](https://reader034.vdocuments.us/reader034/viewer/2022042402/5f13ab768d4b6b0a6210cfce/html5/thumbnails/34.jpg)
34/33
Acknowledgements• SlideTutor Project
– Collaborators:• Drazen Jukic, MD, PhD• Gilan Saadawi, MD, PhD• George Xu, MD, PhD (University of Pennsylvania)• Dana Grzybicki, MD, PhD
– Programmers:• Olga Medvedeva• Eugene Tseytlin• Girish Chavan
– Research Associates:• Elizabeth Legowski• Melissa Castine
– Students:• Michael Yudelson• Velma Payne
– Funding:• National Library of Medicine 1R01 LM007891• National Cancer Institute 1R25 CA101959-and 2R25 CA101959• Agency for Health Care Research and Quality I UL1 RR024153-
01(Grzybicki, PI)• National Library of Medicine Training Grant 5-T15-LM07059• University of Pittsburgh Competitive Medical Research Fund