haptics in medical simulation - cislcisl.stanford.edu/content/dam/sm/cisl/documents/about...in...
TRANSCRIPT
![Page 1: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/1.jpg)
Haptics in Medical Simulation
Allison M. Okamura
Associate ProfessorDepartment of Mechanical Engineering
Design and Biomechanics GroupsStanford University
Collaborative Haptics and Robotics in Medicine (CHARM) Labhttp://charm.stanford.edu
![Page 2: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/2.jpg)
Haptics Tissue modeling forhaptic simulation
Augmented reality Current Projects
Allison Okamura ◆ Stanford University ◆ 2012
![Page 3: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/3.jpg)
Haptics Tissue modeling forhaptic simulation
Augmented reality Current Projects
Allison Okamura ◆ Stanford University ◆ 2012
![Page 4: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/4.jpg)
hearing
sight
smell
taste
touch
which sense is most valuable
to you?
which would you relinquish
last?
Allison Okamura ◆ Stanford University ◆ 2012
![Page 5: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/5.jpg)
what would life be like without touch?
Ian Waterman
Allison Okamura ◆ Stanford University ◆ 2012
![Page 6: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/6.jpg)
why do we have brains?
Daniel Wolpert
Allison Okamura ◆ Stanford University ◆ 2012
![Page 7: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/7.jpg)
sightcentralized
broadpassive
cognitive
touchdistributednarrowactivephysical
Allison Okamura ◆ Stanford University ◆ 2012
![Page 8: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/8.jpg)
hap·tic ('hap-tik)adj. Of or relating to the sense of touch.
[Greek haptikos, from haptesthai, to grasp, touch. (1890)]
TemperatureTexture
SlipVibration
Force
Location/configurationMotionForce
ComplianceCutaneous
Kinesthesia
The haptic senses work together with the motor control system to:- Coordinate movement- Enable perception
Johansson and Westling
Allison Okamura ◆ Stanford University ◆ 2012
![Page 9: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/9.jpg)
haptics making inroadsEntertainment
Nintendo Stanford/JHU Immersion
Education
Human-Computer Interfaces
BMWSamsung Boeing
Allison Okamura ◆ Stanford University ◆ 2012
![Page 10: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/10.jpg)
Haptics
Augmented reality Current Projects
Allison Okamura ◆ Stanford University ◆ 2012
Tissue modeling forhaptic simulation
![Page 11: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/11.jpg)
Haptics for medical simulationMain Challenge:Users should be allowed to interact with the simulator in any manner, and the simulator should respond appropriately.
(Unanswered) Question:How realistic does the simulation need to be to have a significant impact?
This talk addresses the mechanical and haptic interaction components medical simulation.
Allison Okamura ◆ Stanford University ◆ 2012
![Page 12: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/12.jpg)
is the simulator is any good?• Face validity: does the system present an environment
resembling that which is encountered during a medical procedure?
• Content validity: is a skill measured by the system measured the specific skill desired, and not a different one?
• Construct validity: can the system capture the differences between experts and novices?
• Concurrent validity: to what extent does testing performance with the simulator yield the same results as other measures?
• Predictive validity: does performance/training with the simulator transfer to improvements in clinical practice?
Allison Okamura ◆ Stanford University ◆ 2012
![Page 13: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/13.jpg)
Reality-Based Modeling
UBC videos courtesy Simon DiMaio
Allison Okamura ◆ Stanford University ◆ 2012
![Page 14: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/14.jpg)
Cutting with Scissors
Blank Skin
Liver Tendon Forc
e
Scissor angle
Allison Okamura ◆ Stanford University ◆ 2012
![Page 15: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/15.jpg)
Needle Insertion
Time
Forc
e Ve
loci
ty
Allison Okamura ◆ Stanford University ◆ 2012
![Page 16: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/16.jpg)
Needle InsertionAllison Okamura ◆ Stanford University ◆ 2012
![Page 17: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/17.jpg)
Simulation
Accuracy
Needle InsertionAllison Okamura ◆ Stanford University ◆ 2012
![Page 18: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/18.jpg)
Good models also lead to new technology: Needle Steering
Alterovitz, et al. Webster, Reed, et al.
Allison Okamura ◆ Stanford University ◆ 2012
![Page 19: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/19.jpg)
Haptics Tissue modeling forhaptic simulation
Current Projects
Allison Okamura ◆ Stanford University ◆ 2012
Augmented reality
![Page 20: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/20.jpg)
Surgeon
PatientImage source: www.physicianphotos.com
Open SurgeryAllison Okamura ◆ Stanford University ◆ 2012
![Page 21: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/21.jpg)
Surgeon
Instrument/Camera
PatientImage source: www.womenssurgerygroup.com
Minimally Invasive SurgeryAllison Okamura ◆ Stanford University ◆ 2012
![Page 22: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/22.jpg)
© 2008 Intuitive Surgical, Inc.
Surgeon
Master Console
Teleoperated Robot-Assisted Minimally Invasive Surgery
Information-EnhancedRMIS (with Haptics)
Patient-Side Robot
Patient
Instrument/Camera
Allison Okamura ◆ Stanford University ◆ 2012
![Page 23: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/23.jpg)
da Vinci Surgical System(Intuitive Surgical)
surgeon’sconsole
patient-siderobot
Allison Okamura ◆ Stanford University ◆ 2012
![Page 24: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/24.jpg)
Experimental Platform
research version of the da Vinci surgical system
Allison Okamura ◆ Stanford University ◆ 2012
![Page 25: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/25.jpg)
Force feedback:haptics and graphics
Allison Okamura ◆ Stanford University ◆ 2012
![Page 26: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/26.jpg)
Effects of force feedback1. Haptic + Graphical2. Haptic Only3. Graphical Only
4. No Force Feedback
erro
r (d
eg)
feedback
Error (deg)
haptic feedbackreduces error
Allison Okamura ◆ Stanford University ◆ 2012
![Page 27: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/27.jpg)
Differentiating material propertiesAllison Okamura ◆ Stanford University ◆ 2012
![Page 28: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/28.jpg)
porcine kidney
arti!cial lumps
!ducial markers
!ducial markers
!ducialmarkers
!ducialmarkers
!ducial markers
!ducial markers
!ducialmarkers
!ducialmarkers
porcine kidney
X-ray imageCamera image
Lump detection taskAllison Okamura ◆ Stanford University ◆ 2012
![Page 29: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/29.jpg)
Lump detection Results1 2
3 4
Allison Okamura ◆ Stanford University ◆ 2012
![Page 30: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/30.jpg)
What about manipulation?
no overlay dot overlay
In collaboration with D. D. Yuh of JHMI Cardiac Surgery
Graphical force feedback results in lower peak forces, lower variability of forces, and fewer broken
sutures for novice robot-assisted surgeons
Allison Okamura ◆ Stanford University ◆ 2012
![Page 31: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/31.jpg)
Haptics Tissue modeling forhaptic simulation
Augmented reality
Allison Okamura ◆ Stanford University ◆ 2012
Current Projects
![Page 32: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/32.jpg)
In medical simulation today, haptic feedback is typically restricted to that provided by
• artificial materials (e.g., mannequins)
or
• resolved forces (e.g., force feedback haptic interfaces in virtual reality simulators)
Laerdal’s SimMan
Simon DiMaio, UBC
Allison Okamura ◆ Stanford University ◆ 2012
![Page 33: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/33.jpg)
Kinetic System for Medical Simulation (KineSys MedSim)
In collaboration with T. Judkins (IAI), J. E. Colgate (Tangible Haptics), D. Gaba (Stanford)
Allison Okamura ◆ Stanford University ◆ 2012
![Page 34: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/34.jpg)
Kinesthetic haptic display
Images courtesy T. Judkins (IAI)
Allison Okamura ◆ Stanford University ◆ 2012
![Page 35: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/35.jpg)
Electrostatic tactile display
Disney Research
Tangible Haptics
Allison Okamura ◆ Stanford University ◆ 2012
![Page 36: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/36.jpg)
Haptic Jamming: A Deformable Geometry, Variable Stiffness Tactile Display
using Pneumatics and Particle Jamming
Jim Gwilliam and Andrew Stanley
• Simultaneously controllable surface geometry and mechanical properties
• Uses pneumatics and “particle jamming”: a cell made from a flexible membrane filled with granular material solidifies when air is vacuumed out
Allison Okamura ◆ Stanford University ◆ 2012
![Page 37: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/37.jpg)
Haptic Jamming DisplayAllison Okamura ◆ Stanford University ◆ 2012
![Page 38: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/38.jpg)
Force deflection
curves before vacuuming
the cells
0 5 10 15 200
5
10
15
20
25
30
35
Forc
e (
N)
Deflection (mm)
Vacuumed After Inflation
No Inflation Max Inflation
0 5 10 15 200
5
10
15
20
25
30
35Inflated, not Vacuumed
Deflection (mm)
1.0" Cell
2.0" Cell
1.5" Cell
3.0" Cell
Force-deflection
curves after vacuuming
the cells
Allison Okamura ◆ Stanford University ◆ 2012
![Page 39: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/39.jpg)
Haptics Tissue modeling forhaptic simulation
Augmented reality Current projects
Allison Okamura ◆ Stanford University ◆ 2012
![Page 40: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/40.jpg)
JHU APL
Allison Okamura ◆ Stanford University ◆ 2012
![Page 41: Haptics in Medical Simulation - CISLcisl.stanford.edu/content/dam/sm/cisl/documents/About...In medical simulation today, haptic feedback is typically restricted to that provided by](https://reader034.vdocuments.us/reader034/viewer/2022043013/5faf880e6706d9029268410c/html5/thumbnails/41.jpg)
AcknowledgmentsPast PhD/Postdoc Lab MembersJake Abbott (Univ. Utah)Nasir Bhanpuri* (USC)Steven Charles*+ (BYU)Jessica Crouch+ (Old Dominion Univ.)David Grow (New Mexico Tech.)Paul Griffiths+ (Intuitive Surgical)Netta Gurari (IIT)Katherine Kuchenbecker+ (UPenn)Mohsen Mahvash+ (Boston Univ.)Panadda Marayong (Cal St Long Beach)Steven Marra+ (JHU)Sarthak Misra (U. Twente)Kyle Reed*+ (USF)Lawton Verner (Intuitive Surgical)Robert Webster* (Vanderbilt Univ.)Tomonori Yamamoto (EndoMaster) *co-advised, +postdoc
… And manytalentedundergraduateresearchers!
Current Lab MembersTroy Adebar (Stanford)Amy Blank* (JHU)Nick Colonnese (Stanford)Tricia Gibo* (JHU)Jim Gwilliam* (JHU)Ann Majewicz (Stanford)Kirk Nichols (Stanford)Ilana Nisky+ (Stanford)Zhan Fan Quek (Stanford)Michele Rotella (Stanford)Sam Schorr (Stanford)Andrew Stanley (Stanford)Tom Wedlick (JHU)
CollaboratorsAmy Bastian (KKI/JHU Neurosci.)Michael Choti (JHU Surgery)Noah Cowan (JHU Mech. Eng.)David Gaba (Stanford Medicine)Ken Goldberg (UCB IEOR)Gregory Hager (JHU Comp. Sci.)Steven Hsiao (JHU Neuroscience)Michael Hsieh (Stanford Urology)Peter Kazanzides (JHU Comp. Sci.)KT Ramesh (JHU Mech Eng.)Danny Song (JHU Radiology)Russell Taylor (JHU Comp. Sci.)Louis Whitcomb (JHU Mech. Eng.)David Yuh (JHU Surgery)Intuitive Surgical, Inc.
SupportNational Science FoundationNational Institutes of Health (NIBIB, NCI, NINDS) Whitaker FoundationJHU Brain Science InstituteDARPA
Allison Okamura ◆ Stanford University ◆ 2012