development of open source software for c omputer- a ssisted i ntervention systems

Copyright © CISST ERC, 2005NSF Engineering Research Center for Computer Integrated Surgical Systems and Technology

Development of Open Source Software for Computer-Assisted Intervention

Systems

Peter Kazanzides, Anton Deguet, Ankur Kapoor, Ofri Sadowsky, Andy LaMora, Russell Taylor

Johns Hopkins University

www.cisst.org

October 30, 2005


Outline

• Background and Motivation

• Where are we going?

• CISST Libraries

– Foundation libraries

– Real time support

• Development process and tools


NSF Engineering Research Center for Computer-Integrated Surgical

Systems and Technology

• Multi-institution, multi-disciplinary center– Johns Hopkins University + Medical Institutions– MIT + Brigham & Women’s Hospital– CMU + UPMC– Others: Morgan State, Georgetown, Harvard, Penn, Columbia

• Funding– Year 7: Core NSF Grant = $3.9M; Total = ~$8.3M– Years 1-10: Core NSF Grant = $30M; Total = ~$55M

• University researchers, clinicians, industry• Research, Systems, Education, Outreach


Historical Background

• ERC-developed software was captured in:– CIS library

• Common tools, such as logging• Vectors, matrices, transformations• Interface to tracking systems• Numerical methods, registration, …

– MRC library• Common interface to different robots• Essentially a “wrapper” around API for hardware

that provides low-level control


Robot Controller Architecture

Supervisory/Trajectory Control (~100 Hz)

Application(non-real-time)

Ha

rdw

are

Read Sensors

Compute Joint Goals

Compute Goal on Trajectory

Interpolate Setpoint

Compute Control

Read Sensors

Servo Control(~1000 Hz)

Application

API

MRCcisstMRC


Motivation for CISST Package

• Improve process, design, testing, portability, maintainability for open source release and to facilitate clinical use:– Programming standards– Design reviews– Portable build tools– Automated testing– User documentation

• Enable the development of real-time software for physical devices such as robots and trackers


Motivation for Real Time Support

• Motivated by transition from motion controller boards (with processor and vendor’s software) to I/O boards (no processor) and research software

Motion Controller Boards

(intelligent hardware)

I/O Boards (non-intelligent)


Example: Teleoperation of Snake Robot

LoPoMoCo

Control PC (RTAI/Linux) I/O and Amps

I/O and Amps

LoPoMoCo

Read Sensors

Compute Joint Goals



Compute Control

Read SensorsApplication

API

Read Sensors

Compute Joint Goals



Compute Control


API

Control PC (RTAI/Linux)

Read Sensors

Compute Joint Goals



Compute Control


API

Read Sensors

Compute Joint Goals



Compute Control


API

Master Robots

Slave Robots


Example: Image-Guided Robot for Rodent Research

PC (Windows) Servo Control and Amps (Galil)

DMC-2143 RobotApplication

API

3D Slicer


Where are we going?

C++ Software Libraries• cisst libraries

• other libraries

Frameworks• Based on system complexity

• Component of larger system

Binary components• hardware interfaces

• research algorithms

stat

ic li

nkin

g

frozen spots

hot spots

Distributed Interfacedy

nam

ic

load

ing


CISST Libraries

• (Mostly) Stable– Open source* Sept 2005

• cisstInteractive ~Nov 2005

• Beta version– Open source ~Jan 2006

• Work in process– cisstTracker ~Jan 2006

– cisstMRC ~June 2006

Foundation libraries

cisstCommoncisstVectorcisstNumericalcisstInteractive

Real Time Support

cisstOSAbstraction

cisstDeviceInterfacecisstRealtime

Interventional Devices

cisstTrackercisstMRC…

*www.cisst.org, current license based on Slicer 2.x, goal is Slicer 3.0


cisstVector

• Vectors, Matrices and Transformations• Extensive use of C++ templates (metaprogramming)• Fixed size and dynamic

– Fixed size especially suited for real-time code• Operations on slices and sub-regions• References to vectors and matrices

– Improves interoperability with other libraries• Matrices in row-major or column-major format• C++ wrapping of NetLib (numerical methods,

including CLAPACK, MINPACK)


Fixed Size Vectors and Matrices

• Templated by:– Element type (int, double, etc.)– Dimension (number of rows, columns, etc.)

• Efficiency considerations (for templated dimension):– Loop is easy, but not efficient for small vectors:

int Sum() { sum = 0 for (i=0; i < _size; i++) sum += data[i]; return sum;}

– Recursive function also not efficient:int RecursiveSum(int size){ return (size == 1) ? data[0] : RecursiveSum(size-1) + data[size-1];}int Sum() { return RecursiveSum(_size); }

– Recursive template (template metaprogramming) is efficient:• Compiler “unwraps” recursive template into straight-line code


Fixed Size Vectors and Matrices

• Operations provided by “recursive engines”:– Classify operations by:

• Number of operands

• Type of operands

• Type of result

• Storage location for result

– Small number of engines handle all operations• Example: same engine for addition, subtraction, …


Python Interface

• Automatic wrapping of C++ for Python (Swig)• Object registry to share objects between languages• Can load cisst libraries into Python shell• Can start Python shell from C++ program• GUI features provided by cisstInteractive (using

wxWidgets)

C++ Robot Control Library

C++ Numerical Library

C++ Vector/ Matrix Libraries

Application Software

(Complex executable with

GUI tools)

SWIG Class Wrapper

Python Workspace (IRE)

SWIG Class Wrapper

SWIG Class Wrapper

C++ Object

SWIG Wrapper

Registry(Shared Object)

WxWidgets

Simple dialogs, frames, etc.

WxWidgets

Simple dialogs, frames, etc.Custom Python Tools

Error Log ControllerConfig File EditorExperiment Scripts


Device Interface

• Device Hierarchies:

Tracker Tool

GetPosition

Robot Tool

GetPosition

MoveToPosition

Should be able to use robot in place of a tracker

Can lead to complexity!


Device Interface

• Our solution: query with string to obtain command object (Command Pattern)

Tracker Tool

Provides

Robot Tool

Provides

ddiDeviceInterface

Configure

Provides

GetMethodByName

…

Initialize (e.g., from XML)

Return list of implemented operations (strings)

Return command object for specified operation (string)

Abstract Base Class

{ “GetPosition” } { “GetPosition”, “MoveToPosition” }


Device Interface

· · · dev = ptr to devicedevConfigure(…);cmd = devGetMethodByName(“GetPosition”);

· · ·

public:

void Startup();

void Run();

};

· · ·cmdExecute(data);

· · ·

class myTask : public rtsTask {

private: ddiDeviceInterface* dev; ddiCommand* cmd;

Non-real-time

Real-time


Real Time Support

1. Devices and Tasks should be interchangeable:– Example: Servo control via an intelligent device or via a

software task– Solution: Task class derived from device class, but also

includes a device

ddiDeviceInterface

rtsTask


Real Time Support

2. Maintain time history of important state data– State Data Table (SDT), indexed by time and data id

3. Task communication with Command Pattern:– Read from Task SDT or Device– Write to Task Mailbox or Device– Command object can handle remote communications

High-level task (low frequency)

Low-level task (high frequency)

Device Interfacemailbox

SDT

mailbox

SDTHardwareCommand

objects

Command objects


rtsTask

ddiDeviceInterface

osaThread

osaThreadBuddy

rtsStateDataTable

vector <osaMailBox >

map <string, ddiCommandBase *>

StateDataTableMailBoxes

DeviceThread

ThreadBuddy

vector < rtsStateDataArrayBase *>

vector < rtsTimeTicks >

rtsTimeIndex

vector <string>

Ticks

StateVector

StateVectorDataNames

rtsStateDataArrayBase

template<_type>

rtsStateDataArray vector <_type>Data

ddiCommandBase

template<_device>

ddiCommand

_device* Device

bool (_device::*)() Action

Operations

PositionOutput Voltage

Desired Position

[0] [1] [2]

IndexReaderIndexWriter

Legend:Member ofInherits fromInstance of (snapshot)

cisstRealTime


Development Tools

CTest

Build Environment

LaTeX

Doxygen

SWIG

Compile

Compile

Compile

Library Binaries(static & dynamic, e.g.,

cisstVector, cisstCommon)

Test Programs

Applications

Optional Interpreter (IRE)

Wrapped Source

Formatted Documentation(e.g., pdf, html)

(e.g., VC++, gcc/make)

Test Results

Link

CVSTrac (bug/feature requests)

CMake

CVSRepository(source control)

CMake Build Instructions

Documentation

Libraries

Test App

Applications

Scripts

CppUnitDart2

(dashboard)PyUnit

Link


Development Summary

• Tools adequately manage implementation, (unit) testing and maintenance phases of development– Automated testing will be hard for physical devices

• Much documentation is manually created (not enforced by the process)– Requirements, risk analysis, (high-level) design, validation– User guide, tutorial, quickstart

• We needed Dart sooner than we expected!– gcc is getting pickier!– Windows static/shared libraries (dll export)

development of open source software for c omputer- a ssisted i ntervention systems

Documents