automatic detection and segmentation of robot-assisted surgical motions presented by henry c. lin...
TRANSCRIPT
![Page 1: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/1.jpg)
Automatic Detection and Segmentation of Robot-Assisted Surgical Motions
presented by
Henry C. Lin
Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy,
Dr. David D. Yuh, Dr. Allison M. Okamura,
Dr Gregory D. Hager
![Page 2: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/2.jpg)
2MICCAI 2005
ERC-CISST Johns Hopkins University
AuthorsHenry LinPhD Student
Computer Science
Izhak ShafranResearch Scientist
ECE
Todd MurphyMS, 2004
Mechanical Engineering
David YuhSurgeon
Cardiac Surgery
Allison OkamuraAssistant Professor
Mechanical Engineering
Gregory HagerProfessor
Computer Science
![Page 3: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/3.jpg)
3MICCAI 2005
ERC-CISST Johns Hopkins University
Can we automatically detect and segment the surgical motions common in both videos?
Can we quantitatively and objectively determine which video is an expert surgeon and which is an intermediate surgeon?
Motivation
QuickTime™ and aCinepak decompressor
are needed to see this picture.
QuickTime™ and aCinepak decompressor
are needed to see this picture.
QuickTime™ and aCinepak decompressor
are needed to see this picture.
QuickTime™ and aCinepak decompressor
are needed to see this picture.
Expert Intermediate
![Page 4: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/4.jpg)
4MICCAI 2005
ERC-CISST Johns Hopkins University
Cartesian Position Plots - Left Manipulator
--- Pull suture with left hand
--- Move to middle with needle
Expert Surgeon - trial 4 Intermediate Surgeon - trial 22
![Page 5: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/5.jpg)
5MICCAI 2005
ERC-CISST Johns Hopkins University
Previous Work
• Darzi, et al. Imperial College Surgical Assessment Device
(ICSAD) quantified motion information by tracking electromagnetic markers on a trainee’s hands.
• Rosen, et al.Used force/torque data from laparoscopic
trainers to create a hidden Markov model task decomposition specific to each surgeon.
![Page 6: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/6.jpg)
6MICCAI 2005
ERC-CISST Johns Hopkins University
Goals
• Train LDA-based statistical models with labeled motion data of an expert surgeon and an intermediate surgeon.
• Be able to accurately parse unlabeled raw motion data into a labeled sequence of surgical gestures in an automatic and efficient way.
• Ultimately create evaluation metrics to benchmark surgical skill.
![Page 7: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/7.jpg)
7MICCAI 2005
ERC-CISST Johns Hopkins University
Corpus
78 motion variables acquired at 10Hz(we use 72 of them)
4-throw suturing task
15 expert trials12 intermediate trials
each trial roughly 60 seconds in length
![Page 8: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/8.jpg)
8MICCAI 2005
ERC-CISST Johns Hopkins University
Gesture Vocabulary1. Reach for needle
2. Position needle
3. Insert and push needle through tissue
6. Pull suture with left hand
4. Move to middle with needle(left hand)
8. Orient needle with both hands
7. Pull suture with right hand
5. Move to middle with needle(right hand)
![Page 9: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/9.jpg)
9MICCAI 2005
ERC-CISST Johns Hopkins University
APIsignals
X(1,t)
X(78,t)
X(t)Local
FeatureExtraction
L(t) FeatureNormalization
N(t)Linear
DiscriminantAnalysis
Y(t)Probabilistic
(Bayes)Classifier
P(Y(t)|C)
ProbabilisticModels for
Surgical Motions
C(t)
System Approach
APIsignals
X(1,t)
X(72,t)
X(t)Local
FeatureExtraction
![Page 10: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/10.jpg)
10MICCAI 2005
ERC-CISST Johns Hopkins University
Local Feature Extractions
X(kt)
+ + + +
X(kt-m+1) X(kt+m-1)
+ +
X(kt-m) X(kt+m)
€
L(kt ) = [X(kt−m ) | X(kt−m+1) | ... | X(kt ) | ... | X(kt+m−1) | X(kt+m )]
|L(kt)| = (2m+1)|X(kt)| Example: m=5, |L| = 792
![Page 11: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/11.jpg)
11MICCAI 2005
ERC-CISST Johns Hopkins University
APIsignals
X(1,t)
X(78,t)
X(t)Local
FeatureExtraction
L(t) FeatureNormalization
N(t)Linear
DiscriminantAnalysis
Y(t)Probabilistic
(Bayes)Classifier
P(Y(t)|C)
ProbabilisticModels for
Surgical Motions
C(t)
System Approach
APIsignals
X(1,t)
X(72,t)
X(t)Local
FeatureExtraction
L(t) FeatureNormalization
N(t)Linear
DiscriminantAnalysis
![Page 12: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/12.jpg)
12MICCAI 2005
ERC-CISST Johns Hopkins University
Linear Discriminant Analysis
x1
x2
The objective of LDA is to perform dimensionality reduction while preserving as much of the class discriminatory information as possible.
![Page 13: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/13.jpg)
13MICCAI 2005
ERC-CISST Johns Hopkins University
where the linear transformation matrix W is estimated by maximizing the Fisher discriminant.
Linear Discriminant Analysis
€
Y (k) =W (N(k))
Fisher discriminant - ratio of distance between the classes and the average variance of each class
LDA
class-labeledmotion data
expected reducedoutput dimension
reduceddimension
motion data
![Page 14: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/14.jpg)
14MICCAI 2005
ERC-CISST Johns Hopkins University
LDA Reduction (6 Labeled Classes, 3 Dimensions)Expert Surgeon
![Page 15: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/15.jpg)
15MICCAI 2005
ERC-CISST Johns Hopkins University
LDA Reduction (6 Labeled Classes, 3 Dimensions)Intermediate Surgeon
![Page 16: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/16.jpg)
16MICCAI 2005
ERC-CISST Johns Hopkins University
Storage Savings of LDA
Method Temporal Neighbors (m) Space required (values)
Raw no - 432,000
Raw + temporal
yes 5 4,752,000
LDA-based yes 5 18,000
For a 10 minute procedure (6000 input samples)
![Page 17: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/17.jpg)
17MICCAI 2005
ERC-CISST Johns Hopkins University
APIsignals
X(1,t)
X(78,t)
X(t)Local
FeatureExtraction
L(t) FeatureNormalization
N(t)Linear
DiscriminantAnalysis
Y(t)Probabilistic
(Bayes)Classifier
P(Y(t)|C)
ProbabilisticModels for
Surgical Motions
C(t)
System Approach
APIsignals
X(1,t)
X(78,t)
X(t)Local
FeatureExtraction
L(t) FeatureNormalization
N(t)Linear
DiscriminantAnalysis
Y(t)Probabilistic
(Bayes)Classifier
P(Y(t)|C)
ProbabilisticModels for
Surgical Motions
C(t)
![Page 18: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/18.jpg)
18MICCAI 2005
ERC-CISST Johns Hopkins University
Results
• ‘Leave 2 out’ cross-validation paradigm used. 15 expert trials, 15 rounds
• Output of 2 test trials were compared against the manually labeled data.
• The average across the 15 tests was used to measure performance.
Training set (13) Test set (2)
![Page 19: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/19.jpg)
19MICCAI 2005
ERC-CISST Johns Hopkins University
Results
![Page 20: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/20.jpg)
20MICCAI 2005
ERC-CISST Johns Hopkins University
Resultsn Number of
labeled classesLDA output dimensions
% correct
1 6 3 91.26
2 6 4 91.46
3 6 5 91.14
4 5 3 91.06
5 5 4 91.34
6 5 3 92.09
7 5 4 91.92
8 4 3 91.88
![Page 21: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/21.jpg)
21MICCAI 2005
ERC-CISST Johns Hopkins University
Contributions• An automated and space efficient method to
accurately parse raw motion-data into a labeled sequence of surgical motions.
• Results support previous work that there exist quantitative differences in the varying skill levels of surgeons.
• Linear discriminant analysis is a useful tool for separating surgical motions.
![Page 22: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/22.jpg)
22MICCAI 2005
ERC-CISST Johns Hopkins University
Future Work
• Currently getting synchronized stereo video and API data. Will allow vision-based segmentation methods to complement our statistical methods.
• Apply to a larger set of expert surgeons to other representative surgical tasks.
• Create performance metrics to be used as benchmarks for surgical skill evaluation.
![Page 23: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/23.jpg)
23MICCAI 2005
ERC-CISST Johns Hopkins University
Acknowledgements
• Minimally Invasive Surgical Training Center at the Johns Hopkins Medical School (MISTC-JHU)
- Dr. Randy Brown, Sue Eller
• Intuitive Surgical Inc.- Chris Hasser, Rajesh Kumar
• National Science Foundation
![Page 24: Automatic Detection and Segmentation of Robot-Assisted Surgical Motions presented by Henry C. Lin Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy, Dr](https://reader035.vdocuments.us/reader035/viewer/2022062409/56649ec65503460f94bd1829/html5/thumbnails/24.jpg)
Automatic Detection and Segmentation of Robot-Assisted Surgical Motions
Henry C. Lin, Dr. Izhak Shafran, Todd E. Murphy,
Dr. David D. Yuh, Dr. Allison M. Okamura,
Dr Gregory D. Hager
Thank you! Any questions?