Potential Scaling Potential Scaling Effects for Effects for
Asynchronous Video Asynchronous Video in Multirobot Searchin Multirobot Search
Prasanna VelagapudiPrasanna Velagapudi11, Huadong Wang, Huadong Wang22, Paul , Paul ScerriScerri11, Michael Lewis, Michael Lewis22 and Katia Sycara and Katia Sycara11
11Carnegie Mellon University, USACarnegie Mellon University, USA22University of Pittsburgh, USAUniversity of Pittsburgh, USA
Urban Search and Rescue Urban Search and Rescue (USAR)(USAR)
► Location and rescue Location and rescue of people in a of people in a structural collapse structural collapse
► Urban disastersUrban disasters LandslidesLandslides EarthquakesEarthquakes TerrorismTerrorism
Credit: NIST
USAR RobotsUSAR Robots
► Robots can helpRobots can help Unstable voidsUnstable voids Mapping/clearingMapping/clearing
► Want them to be: Want them to be: SmallSmall CheapCheap PlentifulPlentiful
Credit: NIST
Urban Search and Rescue Urban Search and Rescue (USAR)(USAR)
►Now: One operator Now: One operator one robot one robot Directly teleoperatedDirectly teleoperated Victim detection through synchronous videoVictim detection through synchronous video
►Future: One operator Future: One operator many robots many robots Manufacturing robots is easyManufacturing robots is easy Training operators is hardTraining operators is hard
►Need to scale navigation and Need to scale navigation and searchsearch
Synchronous VideoSynchronous Video
►Most common form of camera Most common form of camera teleoperationteleoperation High bandwidthHigh bandwidth Low latencyLow latency
►ApplicationsApplications SurveillanceSurveillance Bomb disposalBomb disposal InspectionInspection
Credit: iRobot
Synchronous VideoSynchronous Video
►Does not scale with team sizeDoes not scale with team size
Synchronous VideoSynchronous Video
►Does not scale with team sizeDoes not scale with team size
Synchronous VideoSynchronous Video
►Does not scale with team sizeDoes not scale with team size
Asynchronous ImageryAsynchronous Imagery
► Inspired by planetary robotic solutionsInspired by planetary robotic solutions Limited bandwidthLimited bandwidth High latencyHigh latency
► Multiple photographs from single locationMultiple photographs from single location Maximizes coverageMaximizes coverage Can be mapped to virtual pan-tilt-zoom camera Can be mapped to virtual pan-tilt-zoom camera
HypothesisHypothesis
►Asynchronicity may improve performanceAsynchronicity may improve performance Helps guarantee coverageHelps guarantee coverage Can review imagery on demandCan review imagery on demand
►Asynchronicity may reduce mental Asynchronicity may reduce mental workloadworkload Only navigation must be done in real-timeOnly navigation must be done in real-time Search becomes self-pacedSearch becomes self-paced
USARSimUSARSim
[http://www.sourceforge.net/projects/usarsim]
► Based on Based on UnrealEngine2UnrealEngine2
► High-fidelity physicsHigh-fidelity physics► ““Realistic” Realistic”
renderingrendering CameraCamera Laser scanner Laser scanner
(LIDAR)(LIDAR)
MrCSMrCSMMulti-ulti-rrobot obot CControl ontrol SSystemystem
MrCSMrCSMMulti-ulti-rrobot obot CControl ontrol SSystemystem
Waypoint Waypoint NavigationNavigation
TeleoperationTeleoperation
Video/ Video/ Image Image ViewerViewer
Status Status WindowWindow
Map Map OverviewOverview
Pilot StudyPilot Study
► Objective:Objective: Find victims Find victims Mark victims on map Mark victims on map
► Control 4 robotsControl 4 robots Waypoint control (primary)Waypoint control (primary) Direct teleoperationDirect teleoperation
► Explore the mapExplore the map Map generated online w/ Occupancy Grid SLAMMap generated online w/ Occupancy Grid SLAM Simulated laser scannersSimulated laser scanners
Experimental ConditionsExperimental Conditions
10 Victims10 Victims
Arena 2
Arena 1
Experimental ConditionsExperimental Conditions
► Streaming ModeStreaming Mode ► Panorama ModePanorama Mode
Panoramas stored for later viewingPanoramas stored for later viewingStreaming live videoStreaming live video
Experimental ConditionsExperimental Conditions(Streaming Mode)(Streaming Mode)
Experimental ConditionsExperimental Conditions(Panorama Mode)(Panorama Mode)
SubjectsSubjects
►21 paid participants21 paid participants 9 male, 12 female9 male, 12 female No prior experience with robot controlNo prior experience with robot control Frequent computer users: 71%Frequent computer users: 71% Played computers games > 1hr/week: Played computers games > 1hr/week:
28%28%
MethodMethod
►Written instructionsWritten instructions►20 min. training session20 min. training session
Both streaming and panoramas enabledBoth streaming and panoramas enabled Encouraged to find and mark at least one Encouraged to find and mark at least one
victimvictim
►20 min. testing session (Arena 1)20 min. testing session (Arena 1)►20 min. testing session (Arena 2)20 min. testing session (Arena 2)
MetricsMetrics
►Switching timesSwitching times►Number of victimsNumber of victims
Thresholded accuracyThresholded accuracy
Victims FoundVictims Found
Average Average # of # of
victims victims foundfound
Accuracy ThresholdAccuracy Threshold
11
22
33
44
55
66
Within Within 0.75m0.75m
Within 1mWithin 1m Within 1.5mWithin 1.5m Within 2mWithin 2m00
PanoramaStreaming
11
22
33
44
55
66
77
First SessionFirst Session Second SessionSecond Session
Panorama First
Streaming First
< 1.5m< 2m
< 1.5m< 2m
00
Trial Order InteractionTrial Order Interaction
AveragAverage # of e # of victimvictim
s s foundfound
Switching TimeSwitching Time (Streaming Mode) (Streaming Mode)
AveragAverage # of e # of reportreport
ed ed victimvictim
ss
1122
2020
Number of SwitchesNumber of Switches
1100
88
66
44
22
4040 6060 8080 100100 12012000
00
p=0.064
Switching TimeSwitching Time (Panorama Mode) (Panorama Mode)
AveragAverage # of e # of reportreport
ed ed victimvictim
ss
1122
2020
Number of SwitchesNumber of Switches
1100
88
66
44
22
4040 6060 8080 100100 12012000
00
SummarySummary
►Streaming is better than panoramicStreaming is better than panoramic Perhaps not by as much as expectedPerhaps not by as much as expected Conditions favorable to streaming videoConditions favorable to streaming video
►Asynchronous performance has Asynchronous performance has potentialpotential May avoid forced pace switchingMay avoid forced pace switching May scale with team sizeMay scale with team size
Synchronous ScalingSynchronous Scaling
► Objective:Objective: Find victims Find victims Mark victims on map Mark victims on map
► Control 4, 8, 12 robotsControl 4, 8, 12 robots Waypoint control (primary)Waypoint control (primary) Direct teleoperationDirect teleoperation
► Explore the mapExplore the map Map generated online w/ Occupancy Grid SLAMMap generated online w/ Occupancy Grid SLAM Simulated laser scannersSimulated laser scanners
Experimental ConditionsExperimental Conditions
8
4
12
Experimental ConditionsExperimental Conditions(Streaming Mode)(Streaming Mode)
SubjectsSubjects
►15 paid participants15 paid participants 8 male, 7 female8 male, 7 female No prior experience with robot controlNo prior experience with robot control Most were frequent computer usersMost were frequent computer users
MethodMethod
►Written instructionsWritten instructions►20 min. training session20 min. training session
Encouraged to find and mark at least one Encouraged to find and mark at least one victimvictim
►20 min. testing session (4 robots)20 min. testing session (4 robots)►20 min. testing session (8 robots)20 min. testing session (8 robots)►20 min. testing session (12 robots)20 min. testing session (12 robots)
MetricsMetrics
►Explored regionsExplored regions►Number of victimsNumber of victims►Neglect toleranceNeglect tolerance►Switching timesSwitching times►Number of missionsNumber of missions►NASA-TLX workloadNASA-TLX workload
Explored RegionExplored Region
Area explore
d
Victims FoundVictims Found
Number of
Victims
Victims Found per RobotVictims Found per Robot
Number of
Victims
Neglected RobotsNeglected Robots
Number of Robots
Totally
Initial Move
Switch TimesSwitch Times
Number of Switches
Mission NumbersMission Numbers
Number of Missions
NASA-TLX WorkloadNASA-TLX Workload
Workload
(Interaction Time)
(Neglect Tolerance)
Fan-outFan-out
SummarySummary
►Bounded number of directly Bounded number of directly controllable robots between 8 and 12controllable robots between 8 and 12 Diminishing returns as robots are addedDiminishing returns as robots are added Performance drops above 8 robotsPerformance drops above 8 robots
►Fan-out parallels the number of robots Fan-out parallels the number of robots operator controlsoperator controls Operators using satisficing strategyOperators using satisficing strategy
Asynchronous Scaling Asynchronous Scaling (Proposed)(Proposed)
► Objective:Objective: Find victims Find victims Mark victims on map Mark victims on map
► Control 4, 8, 12 robotsControl 4, 8, 12 robots Waypoint control (primary)Waypoint control (primary) Direct teleoperationDirect teleoperation
► Explore the mapExplore the map Map generated online w/ Occupancy Grid SLAMMap generated online w/ Occupancy Grid SLAM Simulated laser scannersSimulated laser scanners
Experimental ConditionsExperimental Conditions
8
4
12
Experimental ConditionsExperimental Conditions(Panorama Mode)(Panorama Mode)
MethodMethod
►Written instructionsWritten instructions►20 min. training session20 min. training session
Both streaming and panoramas enabledBoth streaming and panoramas enabled Encouraged to find and mark at least one Encouraged to find and mark at least one
victimvictim
►20 min. testing session (4 robots)20 min. testing session (4 robots)►20 min. testing session (8 robots)20 min. testing session (8 robots)►20 min. testing session (12 robots)20 min. testing session (12 robots)
MetricsMetrics
►Explored regionsExplored regions►Number of victimsNumber of victims►Neglect toleranceNeglect tolerance►Switching timesSwitching times►Number of missionsNumber of missions►NASA-TLX workloadNASA-TLX workload
Expected ContributionsExpected Contributions
►Determine when asynchronicity is Determine when asynchronicity is usefuluseful Advantages for larger team sizesAdvantages for larger team sizes Simultaneous search is not viableSimultaneous search is not viable
►Establish performance baselines for Establish performance baselines for asynchronous searchasynchronous search