operation room supervision for safe robotic surgery … · operation room supervision for safe...

KIT – University of the State of Baden-Wuerttemberg and

National Research Center of the Helmholtz Association

Institute for Process Control and Robotics (IPR)

www.kit.edu

Operation Room Supervision for Safe Robotic Surgery with a multi 3D-Camera Setup

P. Nicolai, J. Raczkowsky

Dipl.-Inform. Philip Nicolai


2 Operation Room Supervision for Safe Robotic Surgery with a multi 3D-Camera

Setup – IROS 2011 Workshop on Safer Surgical Robotics

Karlsruhe Institute of Technology

The Merger of Forschungszentrum

Karlsruhe and Universität Karlsruhe

Mission of a state university with

research and teaching

Mission of a research institution of

the Helmholtz Association with

programoriented provident research

29.09.2011






29.09.2011

Employees

373 Professors

Students

20,771 8,893

Teaching

Research

Innovation






30.09.2011

Institute for Process

Control and Robotics

EU projects

AccuRobAs, RoboCast

SAFROS, ACTIVE

EuroSurge





Outline

Introduction

OP:Sense

Multi 3D-camera setup

Outlook

28.09.2011





Safety by Sensing

Intervention analysis

Extract information about actions

Workflow modelling and analysis

Supervision of intervention execution

Checking against surgical workflow

Alert personal of deviations from plan

Scene knowledge for active moving actuators

Guarantee safe interactions

29.09.2011

pre-operative

intra-operative

intra-operative





Sensing parameters

Supervision area

Whole operation theatre

Single regions of interest

Robots workspace

Instrument table

…

Sensing equipment

Optical sensors 2D, Stereo, 3D

Medical devices pulse, blood pressure, oxygen levels, …

Robots sensors force-torque, skin / distance sensors

28.09.2011





OP:Sense

Scenario

Shared workspace between surgeon and

multi-purpose robot

Robot can be handled in different modes

teleoperated

autonomous

Safety implications: 3D scene representation of robots surroundings is

needed in order to

Altogether avoid robot collisions (with humans or objects)

Lessen impact of collisions

Redundantly control pose and position of the robot

29.09.2011





Components: robotic system

28.09.2011

Lightweight robots by KUKA

High payload, small footprint

Force-torque sensors in all joints

Seven degrees of freedom

Haptic devices by Force Dimension

Seven degrees of freedom

Three / four of the DOF active

Standard endoscopic instruments

Fitted with marker spheres

Easily changable by quick coupling

RX90 holding an endoscopic camera

28.09.2011





Time of Flight cameras

Measurement principle

Short infrared pulses

Time measurement until reflection

Distance calculation

Accuracy

Dependens upon distance and reflectivity

Error in the cm range

Advantages

No moving mechanical parts

Detection of objects without markers

No expensive calculations

29.09.2011

Real-time sensing of

markerless objects





Hybrid sensing I: PMD[vision] ToF cameras

PMD[vision] S3

Range: up to seven meters

Frame rate: 20 fps

Resolution: 64 x 48

Connection: Ethernet

PMD[vision] CamCube 2.0

Range: up to seven meters

Frame rate: 30 fps

Resolution: 204 x 204

Connection: USB

29.09.2011





Hybrid sensing I: PMD[vision] ToF cameras

29.09.2011





Hybrid Sensing II: ARTtrack2

Properties

Marker based tracking system

Very high accuracy

Update rate: 60 Hz

Advantages

Precise pose detection and tracking

Detailled object representation

ARTtrack 2

Commercially available

Range up to four meters

Used in a six camera configuration

30.09.2011

Real-time sensing of

objects with markers





Hybrid Sensing: Microsoft Kinect

Measurement principle: Light Coding

Constant projection of a static pattern

by infrared laser

Scene capture by infrared camera

Depth calculation by triangulation

Properties

Low-cost device

RGB-D

Game console „toy“

SDK available

No further

configuration

htt

p:/

/ww

w.f

lickr.

com

/photo

s/a

udre

ypenven/5

198670323/s

izes/o

/in/s

et-

72157625454305998/

htt

p:/

/ww

w.r

os.o

rg/w

iki/kin

ect_

calib

ration/technic

al?

action=

AttachF

ile&

do=

get&

targ

et=

kin

ect_

ir_pattern

.jpg

30.09.2011





Problem: crosstalk between IR devices?

Evaluation of PMD[vision] cameras, Microsoft Kinect and ARTtrack2

Evaluation of camera performance in different configurations

Standalone

Multi-camera with same type

Multi-camera with different types

30.09.2011





Multiple PMD[vision] S3

30.09.2011





PMD cameras: synchronisation

Possible solutions

Frequency multiplexing: usage of different modulation frequencies

Time multiplexing: alternative triggering of cameras

Implemented solution: split S3 cameras

into two groups

Frequency multiplexing in each group

Time multiplexing between groups

Update rates

CamCube : 20 fps

S3-groups: 10 fps each

„interlaced“ scene update rate: 20 fps

30.09.2011





Multiple PMD[vision] S3

30.09.2011





Results: sensing system

28.09.2011





Outlook

Data interpretation

Detection and interpretation of humans and their gestures

Collision avoidance

Built a 3D occupancy grid

Subtract robot based on its

current pose

Use environment information

for path-planning and 7 DOF

pose optimizations

General setup

Optimization of spatial configuration of cameras

Introduce redundancy

30.09.2011

SAFROS – Patient Safety in Robot Surgery

ACTIVE – Active Constraints Technologies for ill-defined or volatile Environments





28.09.2011

This work is supported in the seventh framework program of the European

Union in the projects „SAFROS – Patient Safety in Robotic Surgery“ under

Grant No. 248960 and „ACTIVE – Active Constraints Technologies for ill-

defined or volatile Environments“ under Grant No. 270460

operation room supervision for safe robotic surgery … · operation room supervision for safe...

Documents