image-based evaluation of video-acquired research skills unmil karadkar, marlo nordt richard furuta...
Post on 22-Dec-2015
216 views
TRANSCRIPT
Image-based Evaluation of Video-acquired Research Skills
Unmil Karadkar, Marlo Nordt
Richard Furuta
Cody Lee
Christopher Quick
Texas A&M University
Center for the Study of Digital Libraries
The Department of Computer Science
Michael E. DeBakey Institute
Veterinary Physiology & Pharmacology
What do these images have in common?
• Biomedical Images
• Cognitive & Perceptual Tasks
• Expertise Matters
Necessity for training and testing domain knowledge
Cardiovascular Research E Bouskela and CA Wiederhielm. Am J Physio 237(1): H59-H65, 1979.
Our Domain – Bat Lab
• Cardiovascular Research– Related to the blood
circulatory system
• Observe bat wings under a microscope– Pallid Bats have
transparent wings– Non-invasive, in-vivo
studies– Normal and modified
conditions
Learning curve of one semester…how can we train researchers faster?
Basic Research Skills
• Artery and Vein recognition
• Size differentiation– Vein, Venule, Capillary
• Lymphatic vessel identification
Our Question
Does the interface matter when testing?
Why might it matter?
How do we know what training is most effective?
And because we’re CSDL…
Testing
Layouts differ in context clues
Image Layout Contexts
Thumbnail
All
Images
2-D
Space
No
Temporal dim.
Scrolling
Subset
Images
1-D
Space
Yes
Temporal dim.
Montage
1
Image
0-D
Space
Yes
Temporal dim.
Experimental Design – 1
• 3 Layouts– Thumbnail– Scrolling– Montage– 16 images per set
• 3 Tasks– Artery/Vein recognition– Size Estimation– Lymphatic vessel wall
identification
A E L
Time/image 3s 4s 2s
Answers per image per image per set
Sets 2 2 3
Experimental Design – 2
• Balanced across subjects
Round 1 Round 2 Round 3
MTSMTSLymphatic
TSMTSMSize
SMTSMTArtery/Vein
MTS
MTSMTSArtery/Vein
TSMTSMLymphatic
SMTSMTSize
MTS
MTSMTSSize
TSMTSMArtery/Vein
SMTSMTLymphatic MTSVer
sio
n
Experimental Design – 3
• PowerPoint for presenting layouts
• Feedback– Verbal– PowerPoint’s Pen feature
• Validation of answers– Panel of experts– Accurate measurements
Subjects
• Trained researchers in the bat lab
• 9 experts– Research experience 2+ semesters
• 6 novices– Research experience 1 semester only
• Age group 18 to 35
• 4 female, 11 male
Artery/Vein Recognition Results
Experts
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
T M S
Image Layout
Su
bje
ct
Pe
rfo
rma
nc
e
Round 1
Round 2 Novices
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
T M S
Image Layout
Su
bje
ct P
erfo
rman
ce
Round 1
Round 2
Constrained by Layout Constrained by KnowledgeLearning (p=0.0004) No Learning
(p=0.005)
Size Estimation Results
General understanding of the order of blood vessels does not translate into accurate estimation of size
Experts performed better than novices
(p<0.002)
Experts
0%
10%
20%
30%
40%
50%
60%
70%
80%
M1 M2 S1 S2 T1 T2
Image Layout
Su
bje
ct P
erfo
rman
ce+/- 40%
+/- 30%
+/- 20%
+/- 10%
Lymphatic Identification Results
Experts and Novices use different strategies for applying contextual information
0%10%20%30%40%50%60%70%80%90%
100%
M S T
Image Layout
Su
bje
ct P
erfo
rman
ce
Expert
Novice(p=0.03)
Experts worked best with Montage (p=0.035)
Future Work
• Explore other image layouts– Static and dynamic collages
• Train subjects on new interfaces– Improve learning time– Increase productivity
• Investigate mechanisms for conveying size of objects within images
http://ebat.tamu.edu
Center for the Study of Digital Libraries
Michael E. DeBakey Institute
Further Information
Unmil Karadkar
Marlo Nordt
Richard Furuta
Christopher Quick [email protected]