Synchronous vs. Synchronous vs. Asynchronous Video in Asynchronous Video in

Multi-Robot SearchMulti-Robot SearchPrasanna Velagapudi, Jijun Wang, Huadong Prasanna Velagapudi, Jijun Wang, Huadong

Wang,Wang,Paul Scerri, Michael Lewis, Katia SycaraPaul Scerri, Michael Lewis, Katia Sycara

University of PittsburghCarnegie Mellon University

Urban Search and Rescue Urban Search and Rescue (USAR)(USAR)

► Location and rescue Location and rescue of people in a of people in a structural collapse structural collapse

► Urban disastersUrban disasters LandslidesLandslides EarthquakesEarthquakes TerrorismTerrorism

Credit: NIST

USAR RobotsUSAR Robots

► Robots can helpRobots can help Unstable voidsUnstable voids Mapping/clearingMapping/clearing

► Want them to be: Want them to be: SmallSmall CheapCheap PlentifulPlentiful

Credit: NIST

Urban Search and Rescue Urban Search and Rescue (USAR)(USAR)

►Now: One operator Now: One operator one robot one robot Directly teleoperatedDirectly teleoperated Victim detection through synchronous videoVictim detection through synchronous video

►Future: One operator Future: One operator many robots many robots Manufacturing robots is easyManufacturing robots is easy Training operators is hardTraining operators is hard

►Need to scale navigation and Need to scale navigation and searchsearch

Synchronous VideoSynchronous Video

►Most common form of camera Most common form of camera teleoperationteleoperation High bandwidthHigh bandwidth Low latencyLow latency

►ApplicationsApplications SurveillanceSurveillance Bomb disposalBomb disposal InspectionInspection

iRobot PackBot

Synchronous VideoSynchronous Video

►Does not scale with team sizeDoes not scale with team size

Synchronous VideoSynchronous Video

►Does not scale with team sizeDoes not scale with team size

Synchronous VideoSynchronous Video

►Does not scale with team sizeDoes not scale with team size

Asynchronous ImageryAsynchronous Imagery

► Inspired by planetary robotic solutionsInspired by planetary robotic solutions Limited bandwidthLimited bandwidth High latencyHigh latency

► Multiple photographs from single locationMultiple photographs from single location Maximizes coverageMaximizes coverage Can be mapped to virtual pan-tilt-zoom camera Can be mapped to virtual pan-tilt-zoom camera

HypothesisHypothesis

►Asynchronicity may improve performanceAsynchronicity may improve performance Helps guarantee coverageHelps guarantee coverage Can review images multiple timesCan review images multiple times

►Asynchronicity may reduce mental Asynchronicity may reduce mental workloadworkload Only navigation must be done in real-timeOnly navigation must be done in real-time Search becomes self-pacedSearch becomes self-paced

USARSimUSARSim

[http://www.sourceforge.net/projects/usarsim]

► Based on Based on UnrealEngine2UnrealEngine2

► High-fidelity physicsHigh-fidelity physics► ““Realistic” Realistic”

renderingrendering CameraCamera Laser scanner Laser scanner

(LIDAR)(LIDAR)

MrCSMrCSMMulti-ulti-rrobot obot CControl ontrol SSystemystem

MrCSMrCSMMulti-ulti-rrobot obot CControl ontrol SSystemystem

Waypoint Waypoint NavigationNavigation

TeleoperationTeleoperation

Video/ Video/ Image Image ViewerViewer

Status Status WindowWindow

Map Map OverviewOverview

Experimental ConditionsExperimental Conditions

► Objective:Objective: Find victims Find victims Mark victims on map Mark victims on map

► Control 4 robotsControl 4 robots Waypoint control (primary)Waypoint control (primary) Direct teleoperationDirect teleoperation

► Explore the mapExplore the map Map generated online w/ Occupancy Grid SLAMMap generated online w/ Occupancy Grid SLAM Simulated laser scannersSimulated laser scanners

Experimental ConditionsExperimental Conditions

10 Victims10 Victims

Experimental ConditionsExperimental Conditions

► Streaming ModeStreaming Mode ► Panorama ModePanorama Mode

Panoramas stored for later viewingPanoramas stored for later viewingStreaming live videoStreaming live video

Experimental ConditionsExperimental Conditions(Streaming Mode)(Streaming Mode)

Experimental ConditionsExperimental Conditions(Panorama Mode)(Panorama Mode)

SubjectsSubjects

►21 paid participants21 paid participants 9 male, 12 female9 male, 12 female No prior experience with robot controlNo prior experience with robot control Frequent computer users: 71%Frequent computer users: 71% Played computers games > 1hr/week: Played computers games > 1hr/week:

28%28%

MethodMethod

►Written instructionsWritten instructions►15-20 min. training session15-20 min. training session

Both streaming and panoramas enabledBoth streaming and panoramas enabled Encouraged to find and mark a victimEncouraged to find and mark a victim

►20 min. testing session20 min. testing session►20 min. testing session20 min. testing session

MetricsMetrics

►Switching timesSwitching times►Number of victimsNumber of victims

Thresholded accuracyThresholded accuracy

Victims FoundVictims Found

Average Average # of # of

victims victims foundfound

Accuracy ThresholdAccuracy Threshold

11

22

33

44

55

66

Within Within 0.75m0.75m

Within 1mWithin 1m Within 1.5mWithin 1.5m Within 2mWithin 2m00

PanoramaStreaming

11

22

33

44

55

66

77

First SessionFirst Session Second SessionSecond Session

Panorama First

Streaming First

< 1.5m< 2m

< 1.5m< 2m

00

Trial Order InteractionTrial Order Interaction

AveragAverage # of e # of victimvictim

s s foundfound

Switching TimeSwitching Time (Streaming Mode) (Streaming Mode)

AveragAverage # of e # of reportreport

ed ed victimvictim

ss

1122

2020

Number of SwitchesNumber of Switches

1100

88

66

44

22

4040 6060 8080 100100 12012000

00

Switching TimeSwitching Time (Panorama Mode) (Panorama Mode)

AveragAverage # of e # of reportreport

ed ed victimvictim

ss

1122

2020

Number of SwitchesNumber of Switches

1100

88

66

44

22

4040 6060 8080 100100 12012000

00

ConclusionsConclusions

►Streaming is better than panoramicStreaming is better than panoramic Perhaps not by as much as expectedPerhaps not by as much as expected Conditions favorable to streaming videoConditions favorable to streaming video

►Similar asynchronous performance is Similar asynchronous performance is goodgood May avoid forced pace switchingMay avoid forced pace switching May scale with team sizeMay scale with team size

# of robots

Ope

rato

r sw

itch

time

Operator-induced latencyOperator-induced latency

Switch Time >>

Comm. Latency

Victims FoundVictims Found

►Repeated Measures ANOVARepeated Measures ANOVA 1.5m radius1.5m radius

►F(1,19) = 8.038F(1,19) = 8.038►p = 0.01p = 0.01

2.0m radius2.0m radius►F(1,19) = 9.54F(1,19) = 9.54►p = 0.006p = 0.006

Trial Order InteractionTrial Order Interaction

►Repeated Measures ANOVARepeated Measures ANOVA 1.5m radius1.5m radius

►F(1,19) = 7.34F(1,19) = 7.34►p = 0.014p = 0.014

2.0m radius2.0m radius►F(1,19) = 8.77F(1,19) = 8.77►p = 0.008p = 0.008

Switching TimeSwitching Time

►Streaming modeStreaming mode Repeated Measures ANOVARepeated Measures ANOVA

►F(1,19) = 3.86F(1,19) = 3.86►p = 0.064p = 0.064

►Panorama modePanorama mode No relation foundNo relation found