nri large: collaborative research: complementary...

1
NRI Large: Collaborative Research: Complementary Situational Awareness for Human-Robot Partnerships Nabil Simaan 1 , Howie Choset 2 , Russell H. Taylor 3 1 ARMA Lab, Dept. of Mech. Engineering, Vanderbilt University, Nashville, TN 37235 2 Biorobotics Laboratory, Carnegie Mellon University, Pittsburgh, PA 15213 3 LCSR Lab, Dept. of Computer Science, Johns Hopkins University, Baltimore MD 21218 Motivation Preliminary Results Publications: 1. Bajo A. & Simaan, N., “KinematicsBased Detection and Localization of Contacts Along Multisegment Continuum Robots”, IEEE Transactions on Robotics, Vol 28, No. 2, pp. 291302 2. S. Tully, A. Bajo, G. Kantor, H. Choset, and N. Simaan, “Constrained filtering with contact detection data for the localization and registration of continuum robots in flexible environments,” in 2012 IEEE International Conference on Robotics and Automation, 2012, pp. 3388–3394. 3. S. Tully, A. Bajo, N. Simaan and H. Choset, and, “A Filtering Approach for Surgical Registration with Unknown Stiffness”, in 2013 proceedings of the Hamlyn Symposium on Medical Robotics. Robots are currently used to extend the reach of humans and to augment their physiological skills during manipulation. This research aims to introduce a new paradigm of complementary situational awareness (CSA), which is the simultaneous perception and use of the environment and operational constraints for task execution. Using this new paradigm robots can act as our partners, not only in manipulation, but in perception and control. Broader Impact To demonstrate the flexibility of our approach, we will use both standard and prototype high degreeoffreedom (DoF) robots for experimental validation. The DaVinci Classic platform is present across all collaborating institutions as a standard platform for benchmarking and clinical translation. Illustrative Examples A search and rescue telemanipulation robot that uses its own perception to form an understanding of task constraints. For example the robot could use LIDAR information to augment the perception of the user who typically sees only the video feed from a forwardlooking camera. An intelligent excavator that uses sensory information about the ground and potentially hidden pipes to assist the user in safe excavation An intelligent surgical robot that uses invivo sensory information to characterize its operational constraints and to assist the surgeon in avoiding sensitive anatomy Research Goals Realtime Sensing during Task Execution: Design algorithms for acquisition of invivo sensory information including methods for assessing the interaction with the environment. Preliminary results Planned research activity Constraint stiffness ellipsoid Shape and stiffness map Situational Awareness Modeling: extend simultaneous localization and mapping (SLAM) to develop algorithms that synthesize invivo, intraoperative and preoperative data to augment human situational awareness. Telemanipulation based on CSA: Create a telemanipulation framework that uses CSA to enable the use of assistive virtual fixtures based on the updated understanding of environmental characteristics to semiautomate surgical tasks. Environment Situational awareness modeling Real-time sensing during task execution Telemanipulation based on complementary situational awareness Human Robot Intuitive Surgical DaVinci Classic VU IREP surgical robot CMU surgical robot Camera Visible to user Perceived by robot Intrinsic force sensing allows for the detection of contact in continuum robots. The robot can restrict movement to avoid collisions outside the surgeon’s view. A combination of contact detection and constrained filtering has demonstrated the ability to register the robotic system to an unknown, flexible environment. Key Challenges In-vivo Sensing Situational Awareness Modeling Control and Telemanipulation Calculate robot state, estimate wrench and contact, estimate tool constraints, stiffness, and cancel flexibility and friction effects Update pre-operative environmental models by fusing visual feature estimates with tracker data. Annotate surface models with stiffness features. Register a tool to a flexible environment. Hybrid force/motion control, telemanipulation with force feedback, Adapt virtual fixtures in real-time based on annotated surface models from situational awareness modeling Intelligent Higher-Level Control • Task tracking • Task-specific virtual fixtures • Exploration of shape/stiffness Task and control mode specification Master manipulator impedance controller Slave Robot Environment Sensory substitution Commanded motion and local compliance Master local impedance map Prior anatomic model SLAM corrected environment model Robot state, force, stiffness Environment and advanced state estimation Master manipulator state Multi-mode low level controller Slave robot state Master Automated forceguided tissue exploration has allowed for the generation of stiffness maps. This data may be used to identify tissue abnormalities such as tumors. Stiffness information allows for constrained motion between tissue clefts. Telemanipulation based on CSA includes the intelligent high level controller (HLC), the low level controller (LLC) of the master robot, LLC of the slave robot, and SLAMbased modeling block to correct the environment model. ERCCISST

Upload: others

Post on 23-Aug-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: NRI Large: Collaborative Research: Complementary ...arma.vuse.vanderbilt.edu/images/stories/research/... · NRI Large: Collaborative Research: Complementary Situational Awareness

NRI Large: Collaborative Research: Complementary Situational Awareness for Human-Robot PartnershipsNabil Simaan1, Howie Choset2, Russell H. Taylor3

1ARMA Lab, Dept. of Mech. Engineering, Vanderbilt University, Nashville, TN 372352 Biorobotics Laboratory, Carnegie Mellon University, Pittsburgh, PA 15213

3 LCSR Lab, Dept. of Computer Science, Johns Hopkins University, Baltimore MD 21218

Motivation

Preliminary Results Publications:1. Bajo A. & Simaan, N., “Kinematics‐Based Detection and Localization of Contacts Along 

Multisegment Continuum Robots”, IEEE Transactions on Robotics, Vol 28, No. 2, pp. 291‐3022. S. Tully, A. Bajo, G. Kantor, H. Choset, and N. Simaan, “Constrained filtering with contact 

detection data for the localization and registration of continuum robots in flexible environments,” in 2012 IEEE International Conference on Robotics and Automation, 2012, pp. 3388–3394.

3. S. Tully, A. Bajo, N. Simaan  and H. Choset, and, “A Filtering Approach for Surgical Registration with Unknown Stiffness”, in 2013 proceedings of the Hamlyn Symposium on Medical Robotics. 

• Robots are currently used to extend the reach of humans and to augment their physiological skills during manipulation. 

• This research aims to introduce a new paradigm of  complementary situational awareness (CSA),  which is the simultaneous perception and use of the environment and operational constraints for task execution. 

• Using this new paradigm robots can act as our partners, not only in manipulation, but in perception and control.

Broader ImpactTo demonstrate the flexibility of our approach, we will use both 

standard and prototype high degree‐of‐freedom (DoF) robots for 

experimental validation.  The DaVinci Classic platform is present across 

all collaborating institutions as a standard platform for benchmarking 

and clinical translation.

Illustrative Examples• A search and rescue telemanipulation robot that uses its own perception to form an understanding of task constraints. For example the robot could use LIDAR information to augment the perception of the user who typically sees only the video feed from a forward‐looking camera. 

• An intelligent excavator that uses sensory information about the ground and potentially hidden pipes to assist the user in safe excavation

• An intelligent surgical robot that uses in‐vivo sensory information to characterize its operational constraints and to assist the surgeon in avoiding sensitive anatomy

Research Goals

Real‐time Sensing during Task Execution: Design algorithms for acquisition of in‐vivo sensory information including methods for assessing the interaction with the environment.

Preliminary results

Planned research activity

Constraint stiffness ellipsoid

Shape and stiffness map

Situational Awareness Modeling:extend simultaneous localization andmapping (SLAM) to develop algorithmsthat synthesize in‐vivo, intraoperativeand pre‐operative data to augmenthuman situational awareness.Telemanipulation based on CSA:Create a telemanipulation frameworkthat uses CSA to enable the use ofassistive virtual fixtures based on theupdated understanding ofenvironmental characteristics to semi‐automate surgical tasks.

Environment

Situational awareness modeling

Real-time sensing during task execution

Telemanipulation based on complementary situational awareness

Human Robot

Intuitive Surgical DaVinci Classic VU IREP surgical robot CMU surgical robot

Camera

Visible to user

Perceived by robot

Intrinsic force sensing 

allows for the detection 

of contact in continuum 

robots.  The robot can 

restrict movement to 

avoid collisions outside 

the surgeon’s view.

A combination of contact 

detection and constrained 

filtering has demonstrated the 

ability to register the robotic 

system to an unknown, flexible 

environment.

Key ChallengesIn-vivo Sensing Situational Awareness

ModelingControl and

TelemanipulationCalculate robot state, estimate wrench and contact, estimate tool constraints, stiffness, and cancel flexibility and friction effects

Update pre-operativeenvironmental models byfusing visual featureestimates with tracker data.Annotate surface modelswith stiffness features.Register a tool to a flexibleenvironment.

Hybrid force/motion control, telemanipulationwith force feedback, Adapt virtual fixtures in real-time based on annotated surface models from situational awareness modeling

Intelligent Higher-Level Control • Task tracking• Task-specific virtual fixtures• Exploration of shape/stiffness• Task and control mode

specification

Master manipulator impedance controller

Slave Robot

Environment

Sensory substitution

Commanded motionand local

compliance

Master local impedance map

Prior anatomic model

SLAMcorrectedenvironmentmodel

Robot state, force, stiffness

Environment and advanced state

estimation

Master manipulatorstate

Multi-mode low level controller

Slave robotstate

Master

Automated force‐guided tissue 

exploration has allowed for the 

generation of stiffness maps.  

This data may be used to 

identify tissue abnormalities 

such as tumors.

Stiffness information allows for 

constrained motion between 

tissue clefts.

Telemanipulation based on CSA includes the intelligent high level controller (HLC), the low level controller (LLC) of the master robot, LLC of the slave robot, and SLAM‐based modeling block to correct the environment model.

ERC‐CISST