medical robotics and computer- integrated therapy delivery ...cis/cista/445/lectures/foo.pdf ·...

1

Copyright © CISST ERC, 2000, 2001 NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology

Medical Robotics and Computer-Integrated Therapy Delivery: Coupling Information to Action in 21’st Century Surgery

Russell H. TaylorDept. of Computer Science/Radiology/Mechanical EngineeringDirector, Center for Computer-Integrated Surgical Systems and

TechnologyThe Johns Hopkins University3400 N. Charles Street; Baltimore, Md. [email protected]


Thanks to many people …

• J. Anderson• W. Bargar• A. Bzostek• B. Eldridge• G. Fichtinger• J. Funda• F. Frassica• R. Goldberg• S. Gomory• G. Hager• L. Joskowicz• Y. Kim• R. Kumar• D. Shen• C. Riviere• D. Yousem• A. Kalvin• C. Cutting• A. Morris

• J. Lazarus• D. Long• K. Masamune• K. Murphy• D. Yousem• B. Jaramaz• A. DiGioia• A. Lahmer• M. Borner• A. Bauer• A. Barnes• P. Jensen• J. Yao• B. Mittelstadt• M. Li• D. Sheng• D. Rothbaum• A. Gueziec• M. Talamini

• K. Chinzei• N. Hata• R. Kikinis• D. Stoianovici• A. Patriciu• L. Whitcomb• A. Okamura• C. Davatzikos• E. Atalar• E. Zerhouni• L. Kavoussi• P. Kazanzides• H. Paul• J. Wenz• E. DeJuan• J. Roy

… and many more

2


Prediction

The impact of computer-integrated surgical systems and technology on medical care in the next 20 years will be as great as the impact of computer-integrated manufacturing systems and technology on industrial production over the past 20 years.

Engineering Research Center for Computer Integrated Surgical Systems and Technology


Basic means for fulfilling prediction: systems that integrate information to action

• Provide new capabilities that transcend human limitations in surgery

• Increase consistency and quality of surgical treatments

• Promote better outcomesand more cost-effectiveprocesses in surgical practice

3


Basic means for fulfilling prediction: systems that integrate information to action





Patient images & treatment

plans

Interface devices

Computers and networks

patient surgeon

Images & other sensor data

Command & control

Decision support &

commandsTreatment records &

outcome data

4


Computer-assisted planning

Patient-specific Model

Update Model

Computer-Assisted Execution

Update Plan

Computer-Assisted

Assessment

Preoperative Intraoperative

Atlas

Postoperative

Patient




Update Model


Update Plan

Computer-Assisted

Assessment


Atlas

PostoperativePatient

Surgical “CAD”

Surgical “TQM”

Surgical “CAM”

5




Update Model


Update Plan

Computer-Assisted

Assessment


Atlas

Postoperative

Patient

Surgical

Assistants


NSF Engineering Research Center for Computer-Integrated Surgical

Systems and Technology

• Multi-institution, multi-disciplinary center– Johns Hopkins University + Medical Institutions– MIT + Brigham & Women’s Hospital– CMU + Shadyside Hospital

• University researchers, clinicians, industry• Research, Systems, Education, Outreach

6


Medical Applications

Engineering Research

Novel algorithms& technology

Requirements/ContextValidation of research

Enabling systems, Validation & evaluation

Requirements/ContextNovel therapy ideas

Engineered CIS Systems


Medical robotics and CIS Systems

• Surgical CAD/CAM Systems

• Surgical Assistant Systems

• Concluding remarksCopyright © CISST ERC, 2000 Engineering Research Center for Computer Integrated Surgical Systems and Technology



Update Model

Computer-Assisted

Execution

Update Plan

Computer -Assisted

Assessment


Atlas

Postoperative

Patient

Surgical CAD/CAM

7


Surgical CAD/CAM Advantages

• Optimized planning• Accurate and

consistent execution• Safety• Verification• Information

managementCopyright © CISST ERC, 2000 Engineering Research Center for Computer Integrated Surgical Systems and Technology



Update Model

Computer-Assisted

Execution

Update Plan

Computer -Assisted

Assessment


Atlas

Postoperative

Patient

Surgical CAD/CAM


Orthopaedic Surgical CAD/CAM Example: Joint Replacement Surgery

Photo: courtesy Brian Davies

8


Urologic Surgical CAD/CAM Example: Robotic TURP

Photo: courtesy Brian Davies


Optimized Local Therapy• Deliver optimized patterns of local

treatments (e.g., radiation seeds, injections) with an image-guided robot

• Applications

– Prostate, brain, spine, liver, bone, kidney ….

• Advantages

– Accuracy of dose– Cost and time of procedure– Lower morbidity– Avoid complications– Enable new treatments

Photo: D. Stoianovici, L. Kavoussi

9


Example: Stereotactic Neurosurgery

IGOR (Lavallee, Troccaz, et al)

Neuromate (ISS)


Example: Radiosurgery

Varian Clinac®

Accuray Cyberknife®

10


Current ERC focus applications

Spine$120 billion cost70% in lifetime

LiverMets from colorectal130,000 new /year60,000 death /year

Prostate~210,000 new/year25% men in lifetime

~30-40% prostatectomy~20% radiation seeds

~30-40% extracapsular


•Follow up•Statistics•Process learning

Pre-op Screening

Image patient and plan treatment pattern

In-scanner robotically assisted treatment

Post-op imaging

Reimagevolume

MaintainMaintainTreatmentTreatment

LogLog

One endpoint: 1-stop shopping

11


Key Enablers

• Imaging – Image-based modeling & analysis– Integration of image sensors with delivery

devices for guidance and monitoring– Design points set for interventions


Example: integrated MRI imaging and therapy delivery

Coaxial transmission line

LCR ScannerSurface coilRF electrodes

Combined endoscopic MRI coil & RF ablator

78 µm resolution !

Canine urethra

Beagle prostate

Photo: E. Atalar, JHU

E. Atalar, JHU

12


• Labeled images* of 20 subjects for cancer and normal tissue

• Deformably registered images to average prostate (red)

• Computed spatial distribution of cancer (green)

• Computed 5 needle positions to maximize probability of detection

• Use statistical correlation of cancer spatial distributions to optimize pattern

Statistical atlas of prostate cancerChristos Davatzikos & Dinggang Shen

* Images provided by Jianchao Zeng, Georgetown U.


Key Enablers

• Imaging • Robots & delivery aids

– Modular designs – Compatible with imaging

device– End effectors

Sketch: Kirby Vosberg, Ferenc Jolesz

13


Remote Center of Motion (RCM)

• 2 DOF

• Mechanically constrained motion center

• Maintains entry site

R. Taylor, D. Stoianovici, L. Whitcomb, A. Barnes


Remote Center of Motion (RCM)

• 2 DOF

• Mechanically constrained motion center

• Maintains entry site

R. Taylor, D. Stoianovici, L. Whitcomb, A. Barnes

14


Alternate RCM implementations:Parallel linkages

Goniometer arcsDavies, et al.

Copyright © CISST ERC, 2000, 2001 NSF Engineering Research Center for Computer Integrated Surgical Systems and TechnologyEngineering Research Center for Computer Integrated Surgical Systems and Technology

Clinical example• Percutaneous access to kidney• Radiolucent needle driver• Robot aligns needle under x-ray

fluoroscopy guidance• Has been done both locally and

remotely

Photos: D. Stoianovici, L. Kavoussi

D. Stoianivici

15

Copyright © CISST ERC, 2000, 2001 NSF Engineering Research Center for Computer Integrated Surgical Systems and TechnologyPhoto: D. Stoianovici, JHU Urology Dept


X-ray/CT compatible robots

R. Susil, K. Masamune, R. Taylor, D. Stoianovici, G. Fichtinger

16


CT-Compatible End-Effector

R. Susil, R. Taylor, J. Anderson, D. Stoianovici


Phantom Experiment• Standard prostate phantom• Demonstrated ~1mm accuracy

with robot from single CT image targeting

R. Susil, G. Fichtinger, K. Masamune, A.Patriciu, D. Stoianovici, E. DeWeese, R. Taylor

17


Clinical example• Kidney biopsy• Robot registered to CT

from single image using markers on end-effector

Photos: D. Stoianovici, L. Kavoussi, A. Patriciu, S. Solomon (JHU Bayview)

Other contributors: R. Susil, G. Fichtinger, K. Masamune, R. Taylor (JHU WSE)


Clinical example• Lung biopsy• Robot registered to CT manually

using scanner alignment laser

Credit: D. Stoianovici, L. Kavoussi , A. Patriciu, S. Solomon, JHU Bayview

18


Burdette Medical

CISST ERC

Ultrasound-Guided Robotically-Assisted Prostate Brachytherapy

G. Fichtinger, D. Stoianovici, C. Burdette


• Throughput• Accurate and

consistent dose delivery

• Verification• Data capture• Improved

documentation

Advantages

19


• Safely performs all routine tasks– Imaging– Volume definition– Dose calculation &

visualization– Robotically-assisted

seed placement– Data logging

• Leaves clinician free to focus on therapy

Goal: An “Ideal” Brachytherapy System


Photos: Harald Fischer FZ Karlsruhe

In-MRI Biopsy (FZK)

20


Key Enablers

• Imaging • Robots & delivery aids• Integrated systems


Integration Example: Open-MR Brachytherapy System (BWH/MEL)

• MR-compatible mechanical manipulator

• Pneumatic and ultrasound motors

• 5 DOF holder

Engineering Research Center for Computer Integrated Surgical Systems and Technology

Credit: K. Chinzei & N. Hata

21


Laserpointer

Provided by Hata, Peters, Chinzei et al.

Copyright © CISST ERC, 2000, 2001 NSF Engineering Research Center for Computer Integrated Surgical Systems and TechnologyTempanyet al .

22


Visualization Workstation

Intraoperative MRI System Planning System

Robot control PC

High-B

andwidth N

etwork

MR Compatible Robot

Security monitoring host

Open MRI robot system overview

Chinzei, Hata,Bzostek, Fichtinger, et al.



Intraoperative CT System Planning System

Robot control PC

High-B

andwidth N

etwork


Open MRI robot system overview

Chinzei, Hata, Stoianovici, Fichtinger, Susil, MasamunePatriciu, Taylor, et al.

CT Compatible Robot

CT-guided

23



Intraoperative ultrasound Planning System

Robot control PC

High-B

andwidth N

etwork


Open MRI robot system

Chinzei, Hata, Stoianovici, Fichtinger, Susil, MasamunePatriciu, Taylor, et al.

Ultrasound Robot

Ultrasound-guided


Surgical CAD/CAM Examples

• Robot-assisted execution– Joint replacement– Percutaneous therapies

• Navigation-assisted execution– Navigation systems for

brain, spine, ENT– Evolution: Information-

enhanced minimally-invasive surgery

Copyright © CISST ERC, 2000 Engineering Research Center for Computer Integrated Surgical Systems and Technology



Update Model

Computer-Assisted

Execution

Update Plan

Computer -Assisted

Assessment


Atlas

Postoperative

Patient

Surgical CAD/CAM

24




Update Model


Update Plan

Computer-Assisted

Assessment


Atlas

Postoperative

Patient


Cutting, Bookstein, Taylor, et al.

25


Navigation Systems

Photo: VTI Inc.


Example: “Virtual Fluoroscopy”

Source: Sofamor Danek, Inc.

26


Example: Pedicle Screw Placement

Photos: Orthosoft, Inc.


Example: Cup Placement in THR

• Navigation system to set angle of cup component in hip replacement

• Potential to reduce dislocations

• Used clinically at Shadyside Hospital Source: Center for Orthopaedic

Research, Shadyside Hospital

27


Example: Cup Placement in THR

Source: Center for Orthopaedic Research, Shadyside Hospital


Example: “Virtual Fluoroscopy”

• Computer tracks patient, instruments, and x-ray c-arm

• Computer overlays real-time graphics onto x-ray images

• Advantages– Reduced x-ray exposure– Easy to learn & use

• Limitations– 2D projections– Relies on accurate

tracking

Source: Sofamor Danek, Inc.

28



Example: Advanced Image Overlay

Photo: CMU and Shadyside Hospital

29





• Some concluding remarks

Copyright © CISST ERC, 2000Engineering Research Center for Computer Integrated Surgical Systems and Technology


Patient -specific Model

Update Model

Computer-Assisted

Execution

Update Plan

Computer-Assisted

Assessment


Atlas

Postoperative

Patient

Surgical

Assistants


Surgical Assistant Examples

• Navigation-assisted execution– May be viewed as

“information assistants” for surgery

• Robotic systems– Endoscopic surgery– Microsurgery

• Evolution: a grand fusion

Copyright © CISST ERC, 2000Engineering Research Center for Computer Integrated Surgical Systems and Technology


Patient -specific Model

Update Model

Computer-Assisted

Execution

Update Plan

Computer-Assisted

Assessment


Atlas

Postoperative

Patient

Surgical

Assistants

30


Skilled Tasks

• Situation assessment• Task strategy & decisions• Sensory-motor coordination

R. Taylor


Skilled Task Augmentation


Augmentation System

• Sensor processing• Model interpretation• Display

atlases

• Manipulation enhancement

• Online references & decision support

• Cooperative control and “macros”

atlases

librariesR. Taylor

31


Research & Technology Barriers

Augmentation System

atlases

libraries

Performance assessment• Metrics & measurements • Human baseline data• System evaluation

Robotics & cooperative control

Human-machine interfaces• Displays & visualization • Haptic input/output• Voice

Sensor processing, modeling, and anatomical atlases

Task & task element characterization

System infrastructure, integration, and end user experience

R. Taylor


• Robot holds endoscope• Surgeon controls view

through various interfaces– Joystick or foot pedal– Voice recognition– Head or eye tracking

• Main current justifications– Labor cost saving– Somewhat steadier view

Example: Robotic endoscope Holders

Photo: Computer Motion, Inc

32


R. Taylor, K. Gruben, J. Funda


R. Taylor

33


Example: Robotic retraction assistant


Ultrasound Manipulation Robots

Salcludean ultrasound system Hippocrate Robot

E.Degoulange, et al

34

Copyright © CISST ERC, 2000, 2001 NSF Engineering Research Center for Computer Integrated Surgical Systems and TechnologyR. Goldberg

Credit: Randy Goldberg

High Dexterity End-Effector


High Dexterity End-Effector for Ultrasound


35



Volume Scan Demonstration


36


Volume Scan Demonstration - Rotation



Volume Scan Demonstration - Fan


37


Volume Scan Demonstration -Translation



Applications/Future Development

• Image volume reconstruction

• Real time tracking & registration

• Ultrasonography skill enhancement

• “Third hand” for ultrasound-guided interventions

Photo: Randy Goldberg

38


Robotic Assistants for Microsurgery


Microsurgical Assistant

• Develop human augmentation systems to assist surgeons in delicate microsurgical procedures

• Applications

– Eye, ENT, Neurosurgery, Vascular

• Advantages

– Tremor, precision, force feedback– Information integration– New procedures (e.g., injections

into retinal vessels)

Photo: D Rothbaumr

39


Limitations of microsurgerySurgical microscopes– Resolution (~ 30 mm)– Visible light

Human sensory-motor performance– Precision (~ 15 – 20 mm)– Dexterity– Force/tactile sensitivity


Reducing limitations will

• Increase performance

• Make new surgical procedures possible

• Make surgery better, safer, faster, cheaper

40


Current focus: mostly Eye & ENT

Eye:– Retinal vein cannulation– Retinal translocation– …

ENT:– Stapes prosthesis– Cochlear implant– …

MicrovascularNeurosurgery….

Retina with occluded retinal vein

2 mm


Example Procedure

Retinal Vein Cannulation: the insertion of a needle into the lumen (interior) of a retinal vein or artery in order to introduce theraputic drugs.

Retina with occluded retinal vein

2 mm

100 um

Cross Section of Retina

Retinal Vein

Micropipette

0.5 µm

41


Stapedotomy drilling

• Have established surgical simulation station for evaluating stapedotomies

• Clinical problem is making very small hole in stapes (bone in middle ear) without damaging inner ear

• Quantitative outcome measures– Compare robot-assisted

versus unaided procedure

D. Rothbaum


Cooperative force guiding of surgical robots

Photo: JHU

Photo: BGU

42


Handle force

Kv

Steady Hand Guiding for Microsurgery

R. Taylor

Copyright © CISST ERC, 2000, 2001 NSF Engineering Research Center for Computer Integrated Surgical Systems and TechnologyR. Taylor, R. Kumar, P. Jensen, L. Whitcomb, J. Roy

43


Handle force

Tip force

Khandle

Ktip

Kv

+

-

Steady Hand Force Scaling Concept

R. Taylor

Copyright © CISST ERC, 2000, 2001 NSF Engineering Research Center for Computer Integrated Surgical Systems and TechnologyR. Kumar

44


High-resolution visualization sensors

1 mm 500 µm

100 µm250 µm

GRIN Lens Endoscope

Operating Microscope

Next-generation end effector

GRIN Lens

Force sensors

MovieP. Jensen


Free hand motion Steady-hand motion

Robot as stable platform for intraoperative sensing

Movies: CISST ERC, MADLAB, JHU

R. Kumar, P. Jensen, G. Hager

45


positions

images

R. Kumar


GRIN lens endoscopy; OCT;

tissue sensors

Real time image fusion & overlay

Steady-hand robotic systems

Tissue & task characterization

and modeling

Current Practice

Surgical knowledge /

action enhancement

Information-Enhanced Microsurgery

Microsurgery Augmentation

Microsurgical Assistant

46


Evolution: Integrate imaging and robots

• In-scanner visualization and guidance• Steady-hand robots for ultrasound & instrument manipulation• Applications include biopsies, ablations, MIS

R. Taylor


Evolution: Integration with imaging

• Combine robotic system with intraoperative imagers and navigation systems

• Applications include ENT, Brain, Spine, Bone, Perc.

************Misc. percutaneous

******ENT

******Spine

********Bone

*****BrainUSX-rayCTMRI

47


Telerobotic Surgical Augmentation

• Surgeon manipulates “master” robot

• “Slave” robot mimics motions

• Typically systems have:– Force & position

scaling– Remote visualization

via video• Can be expensive but

provide improved capability for minimally invasive procedures

Photo: Intuitive Surgical


48


Evolution: Augmented Teleoperation


Augmentation System

• Sensor processing• Model interpretation• Display

atlases

• Manipulation enhancement

• Online references & decision support

• Cooperative control and “macros”

atlases

libraries

R. Taylor


Example: Robotic Knot Tying

John Wen, et al., RPI

49





• Concluding remarks


• Robots are simply information-driven surgical tools

• Their real potential will come from their use within the context of computer-integrated surgical systems.


• There is much research to be done.

50


Research and technology barriers

Modeling & analysis• Segmentation• Registration• Atlases• Optimization• Visualization• Task characterization• etc.

Interface Technology• Sensing• Robotics• Human-machine

interfacesSystems• Safety & verifiability• Usability & maintainability• Performance and validation


How can we get there?

Strong and committed teams

– Surgeons– Engineers– Industry

Focus on systems that address important needs

Rapid iteration with measurable goals

Have fun!

51


The real bottom line: patient care




medical robotics and computer- integrated therapy delivery ...cis/cista/445/lectures/foo.pdf ·...

Documents