phase recognition during surgical procedures using embedded and body-worn sensors

Jakob Bardram, Afsaneh Doryab, Rune M.Jensen, Poul M. Lange, Kristian L. G.Nielsen, and Soeren T. Petersen

•Motivation•Field studies•Sensor platform•The experiment•The results•Discussion and Conclusion

Outline

Motivation

• Existing work has mainly focused on daily routines of an individual person in home or outdoor

MOTIVATIONHospital work is collaborative which involves constant coordination and communication

MOTIVATIONAutomatic recognition of surgical procedure can be used for:

•Coordination, communication and planning

•Patient safety

•Context-aware information management and retrieval

Motivation

How can we automatically recognize the phases of a surgery using sensors?

Motivation

How can we automatically recognize the phases of a surgery using sensors?

What should be sensed?

Motivation

How can we automatically recognize the phases of a surgery using sensors?

What should be sensed?Which sensors provide more significant input ?

What clinicians do in a surgery?

How do they move around?

Field study

4 important activity zones

Who is involved in a surgery?

Field study

Which tools and instruments they use?Where they use them?

Field study

Field study

• 97% of tasks involved at least one physical instrument and 78% involved several instruments

• Direct relation between physical instruments and physical tasks

Field study

• 97% of tasks involved at least one physical instrument and 78% involved several instruments

• Direct relation between physical instruments and physical tasks

Temporal and sequential procedure

•Based on our detailed study, these parameters seemed important to track:

•Based on our detailed study, these parameters seemed important to track:

•The location of clinicians and the patient

•Based on our detailed study, these parameters seemed important to track:

•The location of clinicians and the patient

•The location of objects on different tables

•Based on our detailed study, these parameters seemed important to track:

•The location of clinicians and the patient

•The location of objects on different tables

•The use of objects and instruments by clinicians

Sensor platform - Hardware

Ubisense server

Platform server

Arduino hub

ArduinoArm-wrist object tracking

UbisensePerson-tracking

AlienShort-range- object tracking

TCP Serial

TCP

TCP

USB

XBee

IcodeShort-range- object tracking

•Synchronize input from different sensors•store raw observations (one instance per second)

Sensor platform- Software

Sensor platform – Feature vector

Sensor Platform – Feature vector

The Experiment

The Experiment

The Experiment

The Experiment

Analysis

Test the feasibility of the sensor platform

Analysis

Test the feasibility of the sensor platformHow accurate phase recognition could be using sensed

data

Analysis

Test the feasibility of the sensor platformHow accurate phase recognition could be using sensed

dataIdentify the impact of each sensor in achieving high

accuracy

4 simulated operations – (4 datasets)Leave-one-out cross validationUsing decision trees for classification

Analysis - Phase Recognition

Analysis - Decision tree

Analysis – Initial Results

Analysis – Feature Set

Laryngoscope in anesthesia table zone

Anesthesia nurse in anesthesia machine zone

Tube in anesthesia table zone

… time

0 1 0 … t1

0 1 1 … t2

0 1 1 … t3

. . . . .

. . . . .

1 1 0 … tn

Features

Feature Instances at time t1-tn

Results – Phase RecognitionLaryngoscope in anesthesia table zone

Anesthesia nurse in anesthesia machine zone

Tube in anesthesia table zone

… time

0 1 0 … t1

0 1 1 … t2

0 1 1 … t3

. . . . .

. . . . .

1 1 0 … tn

Features

Feature Instances at time t1-tn

Laryngoscope in anesthesia table zone

Laryngoscope in anesthesia table zone- acc

Anesthesia nurse in anesthesia machine zone

Anesthesia nurse in anesthesia machine zone- acc

Tube in anesthesia table zone

Tube in anesthesia table zone- acc

… time

0 0 1 1 0 0 … t1

0 0 1 2 1 1 … t2

0 0 1 3 1 2 … t3

. . . . . .

. . . . . .

1 10 1 13 0 2 … tn

Feature Processing

Laryngoscope in anesthesia table

zone

Anesthesia nurse in anesthesia machine zone

Tube in anesthesia table zone

… time

0 1 0 … t1

0 1 1 … t2

0 1 1 … t3

. . . . .

. . . . .

1 1 0 … tn

Feature Processing

Laryngoscope in anesthesia table zone

laryngoscope in anesthesia table zone- accumulated

time

0 0 t1

1 1 t2

1 2 t3

0 2 t4

0 2 t5

1 3 t6

Laryngoscope in anesthesia table

zone

Anesthesia nurse in anesthesia machine zone

Tube in anesthesia table zone

… time

0 1 0 … t1

0 1 1 … t2

0 1 1 … t3

. . . . .

. . . . .

1 1 0 … tn

Results – Phase Recognition

With historical features

Without historical features

Sensor Significance

Sensor Significance

Ubisense only

Ubisense only

Tables+

wristband

Sensor Significance

Sensor Significance

Ubisense only

Tables+

Wristband

Wristbandsonly

Location important or not?Using palm based sensors in the OR?Using sensors or images?Classical machine learning enough?

Discussion

Conclusion

• Possible to achieve relatively high classification accuracy in phase recognition during surgical procedures using machine learning techniques

Conclusion

• Possible to achieve relatively high classification accuracy in phase recognition during surgical procedures using machine learning techniques

• The experiment hepled to analyze the weight and hence the importance of different sensors

Conclusion

• Possible to achieve relatively high classification accuracy in phase recognition during surgical procedures using machine learning techniques

• The experiment hepled to analyze the weight and hence the importance of different sensors

• Our study gives important input for further research in design of suitable sensors for the OR

Thank you