Technologies for Mobile Manufacturing

Sanjiv Singh/Reid SimmonsRobotics InstituteCarnegie Mellon University

February 2008

OutlineOutline

Motivation Objectives A Simple Example

Autonomous Assembly Tele-operated Operation “Sliding” Autonomy

More complicated examples Key Technologies

Relative Position Estimation Coordinated Control of Mobile Manipulation Task Control Architecture

Terrestrial ConstructionTerrestrial Construction

• Many different tasks• Complimentary entities• Big plan that is constantly refined

ObjectivesObjectives

Enable heterogeneous multiple robots to coordinate complex assembly tasks

Emphasis on tasks that can not be done by single robots

Enable flexible human-robot interaction during assembly Deal with unanticipated contingencies Reduce need for operator

Candidate Tasks: Assemble multi-element, compliant structure Brace structure for strength

Cable structure

Previous Work: Distributed Previous Work: Distributed ArchitecturesArchitectures

ExecutiveExecutive

BehavioralControl

PlanningPlanning

ExecutiveExecutive

BehavioralControl

ExecutiveExecutive

BehavioralControl

PlanningPlanning PlanningPlanning

Previous Work: Multi-Robot Previous Work: Multi-Robot SynchronizationSynchronization

Enable agents to allocate and synchronize tasks; detect and handle each others exceptions

Robot 1

Robot 2

Robot 3

Execute Sequentially

Execute Concurrently

Execute Sequentially

Task A

Task C

Task B

Coordinated AssemblyCoordinated Assembly

Three heterogeneous robots

Crane has large workspace, high weight capacity

Manipulator has fine control

Roving eye provides high degree of resolution

Independent robot operation without accurate inter-robot calibration

Multi-Robot TestbedMulti-Robot Testbed

A Simple Example: Fully A Simple Example: Fully AutonomousAutonomous

Dock a single beam into two upright connectors with mm tolerance

QuickTime™ and aCinepak decompressor

are needed to see this picture.

Combined State-Machine for Dual-End Combined State-Machine for Dual-End DockDock

Lower beam

Align beamover far

stanchion

Turn far endinto view

Align beamover near stanchion

Lower beaminto far

stanchion

Watch crane

WatchMM

Move to far stanchion

Watchdock

Watch MM

Watchcrane

Watchpush

Move awayFrom MM

Align MM at near

stanchion

Dock beam innear stanchion

Grasp beam

Dock beam

Turn 180˚Align MM atfar stanchion

Push beam overfar stanchion

Contact beam

Push beam

Stow armStow arm

CR

AN

EM

OB

ILE

M

AN

IPU

LAT

OR

RO

VIN

G E

YE

*Sequential connections for watch tasks not shown for clarity.

Dual End Dock - Percent Completed

0%

20%

40%

60%

80%

100%

Startup failure.

Bad turn.

Beam caught on grove.

Startup failure.Near end failure.Near end failure.Near end failure.Beam misaligned.Beam misaligned.Base misaligned.Base misaligned.

Successful.

Near end failure.Fiducial blocked.

Successful.

Near end failure.Beam misaligned.

Bad turn.

Startup failure.

Beam caught on grove.

Base misaligned.Base misaligned.Xavier problem.Xavier problem.Fiducial blocked.Base misaligned.

MM problem.MM problem.Successful.

Fiducial blocked.Base misaligned.Xavier problem.

Base misaligned.

Beam caught on grove.

Base misaligned.Fiducial blocked.Base misaligned.Fiducial blocked.

MM problem.Xavier problem.

Bad turn.Bad turn.

MM problem.MM problem.

Fiducial blocked.Fiducial blocked.Fiducial blocked.Fiducial blocked.

1 2 3 4 5 6 7 8 9 1011 121314151617181920212223242526272829303132333435363738394041424344454647484950

Trials

Near End Dock

Swap Ends

Far End Dock

First ResultsFirst Results

FailuresFailures

First dock succeeds 70% of the time Complete second dock succeed only 6% of the time 20% of the time, initial conditions are not set

Common errors: Mobile Manipulator might over or underturn Beam gets caught on groove Arm gets caught on beam Fiducials are blocked Actuator deadband causes infinite loop Visual servoing fails Crane actuator slip causes offset error

Mature Autonomous SystemMature Autonomous System

Setup Completely autonomous 50 trials

Typical Failures Electrical failure on MM Software crash Near collision due to errors

in visual perception Obscured fiducial MM lost grip on beam Assembled portion broke

apart Speed

µ=9.9min, =1.6min

Teleoperated SystemTeleoperated System Setup

50 trials (total) with four robot-experienced users

Several robot-specific GUIs Teleoperation using visual

feedback from Roving Eye

Typical Failures Visual feedback created

“tunnel” vision Stereo vision did not provide

users with good depth perception

Experienced one network and one electrical failure

Speed µ=12.5min, =4.0min

Sliding Autonomy: Adding an Sliding Autonomy: Adding an OperatorOperator

Fully autonomous operation has many failure modes Not enough bang for the buck to automate some

operations Would like seamless method to switch between

operator and system Operator should be able to take over either control or

monitoring of task Three modes of human interaction:

Pre-assigned task Intervention Exception Handling

Sliding TasksSliding Tasks

Mobile Manipulator First dock Turn Second dock*

Roving Eye Visual search Turn

Crane Second dock*

Results w. Sliding AutonomyResults w. Sliding Autonomy

Setup Several task-specific GUIs Limited adjustable tasks Feedback available from

any autonomous task 50 trials

Results Discretionary-Intervention

Successes Mandatory-Intervention

Successes Failures due to damaged

hardware & network failure

Successes

Completion Times

Mean Std-dev

Fully Autonomous 64% 9.9m 1.6m

Tele-operated 96% 12.5m 4.0m

Sliding Autonomy 94% 9.9m 1.9m

Discretionary Only 68% 9.5m 1.5m

Mandatory Only 26% 11.1m 2.6m

OutlineOutline

Motivation Objectives A Simple Example

Autonomous Assembly Tele-operated Operation “Sliding” Autonomy

More complicated examples Key Technologies

Relative Position Estimation Coordinated Control of Mobile Manipulation Task Control Architecture

More complicated example #1More complicated example #1

ModesModes

Teleoperated: User controls each robot in turn through keyboard and mouse

Autonomous: Hit start and step back System Initiative: System asks for help when needed Mixed Initiative: System initiative + Operator intervention

Completion Time

Success Rate TLX

Workload

Teleop

Autonomous

System Initiative

Mixed Initiative

ModesModes

Teleoperated: User controls each robot in turn through keyboard and mouse

Autonomous: Hit start and step back System Initiative: System asks for help when needed Mixed Initiative: System initiative + Operator intervention

Completion Time [Std]

Success Rate (Total Exp)

TLX

Workload [Std]

Teleop 732 [227] sec 94% (16) 52 [16]

Autonomous 516 [125] sec 89% (35) 0

System Initiative 500 [182] sec 100% (16) 27 [21]

Mixed Initiative 529 [148] sec 94% (16) 29 [13]

Tele-Op

Mixed

System Initiative

Autonomous

516[125]

89% (35)

0

500[182]

100% (16)

27 [21]

529 [148]

94% (16)

29 [13]

732 [227]

94% (16)

52 [16]

More complicated example # 2More complicated example # 2

QuickTime™ and aAnimation decompressor

are needed to see this picture.

Extended scenario involves planning because

• constraints make it difficult to script a plan

• Recovery from failure might require many steps

University of Maryland Space Systems LabUniversity of Maryland Space Systems LabNeutral Buoyancy TankNeutral Buoyancy Tank

EASE Truss Assembly

Ranger

Space Shuttle Cargo Bay

“Roving Eye”

“Crane”

“Mobile Manipulator”

Trestle - U Maryland SSL Trestle - U Maryland SSL CooperationCooperation

Operator at CMU

Ranger Arms at U Maryland

internet

OutlineOutline

Motivation Objectives A Simple Example

Autonomous Assembly Tele-operated Operation “Sliding” Autonomy

More complicated examples Key Technologies

Relative Position Estimation Coordinated Control of Mobile Manipulation Task Control Architecture

Sensing LocationSensing Location

Need to localize parts wrt to robot during operation so robot can plan motion and adapt to any variations in starting conditions & performance

Complicated because the robot base is not stationary Method 1: No global reference frame. Relative position

(between parts & between robot and part) is determined via fiducials

Advantages: flexible, low infrastructure Disadvantages: accuracy can be low unless high fiducials are sensed

with high resolution, computationally expensive

Method 2: Establish global reference frame. Parts and Robots are all registered in common frame.

Advantages: high accuracy, low computation requirements Disadvantages: high infrastructure costs, must guarantee line of sight

from fixed infrastructure

Sensing Relative PositionSensing Relative Position

Visual Fiducial allows determination of ID & 6 DOF displacement between camera and fiducial.

Fiducials have some redundancy, can work even if the fiducial is partly obscured.

Main computational expense is in detecting fiducial in the scene.

Accuracy increases as camera gets closer to fiducial

Tracking Fiducials (with occlusion) Tracking Fiducials (with occlusion)

QuickTime™ and a decompressor

are needed to see this picture.

Other kinds of FiducialsOther kinds of Fiducials

Active Fiducials can be used.

QuickTime™ and aMotion JPEG OpenDML decompressor

are needed to see this picture.

Sensing in a Global Reference Sensing in a Global Reference FrameFrame

Transmitters fixed to infrastructure Receivers on items that move Requires synchronization between receivers

QuickTime™ and aTIFF (Uncompressed) decompressor

are needed to see this picture.

http://www.indoorgps.com/

Mobile ManipulationMobile Manipulation

Want to place the robot end effector accurately in a large workspace. Could do this by coupling manipulator & mobile base.

Coordination of base and arm motions of Mobile manipulators is complicated because of redundant degrees of freedom degrees of freedom.

Further considerations: Want to keep the arm from getting close to singularities. Want to control end-effector but want to ensure that the

base meets the constraints. Arm and base have very different response

Mobile ManipulationMobile Manipulation

Resolved motion control withArm motion only-- SMALLER WORKSPACE

Resolved motion control withCoordinated Arm and Base-- LARGER WORKSPACE

QuickTime™ and aCinepak decompressor

are needed to see this picture.

QuickTime™ and aCinepak decompressor

are needed to see this picture.

Implementation on CMU MMImplementation on CMU MM

QuickTime™ and a decompressor

are needed to see this picture.

Projecting into the Null Space Projecting into the Null Space (Example1)(Example1)

QuickTime™ and aCinepak decompressor

are needed to see this picture.

Projecting into the Null Space Projecting into the Null Space (Example2)(Example2)

QuickTime™ and aCinepak decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

QuickTime™ and a decompressor

are needed to see this picture.

Offline Planning to decouple Base & Offline Planning to decouple Base & Arm MotionArm Motion

Each grid cell gets a score based on how much of the path and how well the it can be covered with the base at that point.

Seam Following Seam Following

Motion sped up by 4x

QuickTime™ and aCinepak decompressor

are needed to see this picture.