USARsim for Robocup

Jijun Wang & Michael Lewis

Background..

USARsim was developed as a research tool for an NSF project to study Robot, Agent, Person Teams in Urban Search & Rescue

Katia Sycara CMU- Multi agent systemsIllah Nourbakhsh CMU/NASA- RobotsMike Lewis Pitt- PeopleJijun Wang- primary developer

We wanted it to:

Study Teams of Robots controlled by People

The Idea for USARsim in Robocup

High fidelity simulation environment that provides:

Detailed models of USAR environmentsDetailed models of robots & sensors

The user bringsIndividual robot & team control codeUser interface

Strategy vs. Dropped Leads:Why we need something between Robocup Rescue & the NIST arenas

The Devil is in the details.. USAR missions fail because the operator can’t figure out:Why the robot is stuck (all of us)That the robot has flipped (Robin Murphy)That the camera never pointed at the victimThat the camera is pointing at a victim but we didn’t see ….Etc!

We can be strategically perfect & still failThere needs to be a level of planning & execution

that mixes strategy & sensing & control issues

NIST’s Existing Transportable ArenasFORMING A CONTINUUM OF CHALLENGES(from Jacoff et al. 2003)

Yellow RegionSimple to traverse, no agility requirementsPlanar (2-D) mazeIsolates sensors with obstacles/targetsReconfigurable in real time to test mapping

Orange RegionMore difficult to traverse, variable flooringsSpatial (3-D) maze, stairs, ramp, holesSimilarly reconfigurable

Red RegionDifficult to traverse, unstructured environmentSimulated rubble piles, shifting floorsProblematic junk (rebar, plastic bags, pipes…)

YELLOW COURSE

ORANGE COURSERED COURSE

60'

4'

4'

4'

4'20' 28'

52'

24'

24'

60'

[START/FINISH]

STAIRS (5'/8')

SHALLOW RAMP (5'/15')

Human Factors ChallengesWorld through a straw (restricted FOV)Camera control for search/navigation Survey knowledge (mapping environment) from restricted FOV & impeded movementVisual “smearing” from close surfacesunfamiliar ground level perspectiveDifficult distance judgments from degraded 2D imageDifficult orientation judgments from visual cues in disorderly environmentDifficult locomotion due to out-of-view & negative obstacles

Multi-Robot Problems

Common mapping & awarenessPerceptual cooperation (you see what I’m stuck on)Exploiting heterogeneous capabilitiesDaisy chaining & communication modelingTeam planning in failure-prone environmentsHuman interaction/control of teamsEtc.

Simulation Desiderata

Expense and availability of simulation hardware and software to USAR robotics communityEase of programming to reflect targeted aspects of designFidelity of simulation w.r.t. aspects of design to be tested

Simulation Requirements

Video feed for teleoperation and visual search and identification

Sensor simulation- for autonomous control and fused displays

Simulated robot dynamics- for teleoperation and autonomous control

Multiple entity simulation- to allow interaction and cooperation among teams of robots

Why a Game Engine?(Lewis & Jacobson, CACM 2002)

HardwareBest available graphics now on Nvidia/ATI gpu’s for commodity priced pc’s

SoftwareCheapTakes advantage of current gpu featuresSophisticated IDE’s for both behaviors (mods) and environments (levels)Client-server architecture to support multiple robotsRanging commands used by ‘bots’ provide hooks needed for sensor simulationSophisticated physics engines such as the Karma engine allow high fidelity kinematical modeling

Why the Unreal Engine?

Lewis, M. & Jacobson, J. (2002) Game Engines in Research Communications of the Association for Computing Machinery, NY: ACM 45(1)

Good graphics, good physics, object orientation, good tools, & good documentation

Unreal is the defacto game engine used in research:Pitt & PARC- multiscreen (CAVE-like) displaysJohn Laird/ U Mich- AI test environmentJ. Anderson & C. Leibiere- Act-R & charactersK. Sycara, J. Hodgins, & G. Sukanthar- synthetic charactersArchitectural modeling (Notre Dame), etc.Mike Zyda & NPS- America’s Army, etc.ITC center at USC, Alterne project University of Teeside & others (using

our cave set-up)(crystal space, garage games, etc.)

Alternatives

Excellent 3D graphics & drivers from VR community like UCF toolsGood physics engines like ODE (being used for 3D soccer simulation)

But hard to put them together..With ODE you have to work with OpenGL in Xwindows &

lose the standard formats that let you use modeling tools like 3D studio max or Maya

With VR tools you have to program physical behaviors yourself

USARsim Architecture

Unreal Engine

Map Models (Robots model, Sensor model, victim model etc.)

Gamebots

Network

Control Interface

Unreal Client (Attached spectator)

Middle Level Control

High Level Control

Controller

Video Feedback

Controller

Unreal Client (Attached spectator)

Video Feedback

……

Team Cooperation

Unreal Data

Control Data

Control Interface

Middle Level Control

High Level Control

USARsim Architecture

Unreal Engine

Map Models (Robots model, Sensor model, victim model etc.)

Gamebots

Network

Control Interface

Unreal Client (Attached spectator)

Middle Level Control

High Level Control

Controller

Video Feedback

Controller

Unreal Client (Attached spectator)

Video Feedback

……

Team Cooperation

Unreal Data

Control Data

Control Interface

Middle Level Control

High Level Control

Gamebots Shot

Robot control architecture (simulation)

AttachedUnrealSpectator

Control software

GAMEBOTS

UnrealServer

Karma Physics enginerigid multi- body simulation

Vehicle class lets us characterize robot kinematics precisely..

Sensor hierarchyThe trace command gives ground truth for range informationSound is attenuated according to distanceHuman motion (pyroelectric) uses line of sight & magnitude of movement

Sensor

Range Sensor Sound Sensor HumanMotion Sensor

RangeScanner Sensor

Sensor Class

HiddenSensor This Boolean value is used to indicate whether the sensor will be visually showed in the simulator.

MaxRange It is the maximum distance that can be detected.

Noise It is the relative random noise amplitude. With the noise, the data will be data = data + random(noise)*data

OutputCurve It’s the distortion curve. It is constructed by a series of points that describe the curve.

Sensor visualizations (courtesy of Player)

Can be controlled through:

Native Gamebots interfacePyro middleware (including a hack for Windows)Player.. (USARsim plays the role of Stage)

The Arenas

Arena Simulations

ProEngineer solid model converted to Unreal formatDigital photographs used to create textures to be applied to the modelGlass, mirrors, orange safety fencing, and other “special effects” addedRubble, debris, and victim models added to simulationRobot characteristics adapted from Karma vehicle class

Data Collection at NIST ArenasGaithersburg, MD

Illumination levels in Lux

Yellow Arena

Yellow Arena Simulation

Fisheye view of Orange Arena(from Jacoff et al. 2003)

Simulation of Orange Arena

Red Arena Simulation

Orange Arena Platform: photo & simulation

Parts & materials to “build your own”

The Robots

Activmedia Pioneer P2AT

P2AT has:•Four wheels•Skid-steer•Size: 50cm x 49cm x 26cm•Wheel diam: 21.5cm

equipped with:•PTZ camera•Front sonar ring•Rear sonar ring

Pioneer P2DX

iRobot ATVJr

CMU experimental: Personal Explorer Rover (PER)

CMU experimental: Corky

First generation interface(runs with both Corky & simulation)

Observations

Both Corky and the simulation are very difficult to control with same problems:

Nonvisible obstaclesSurface blindingDisorientationCamera controlDifficulty with stairs or rubble

Validation

We are currently planning validation studies comparing real & simulated PER & Pioneer robots in Orange Arena

Hardware: Sensors, KinematicsBehavior: Sensors x KinematicsAutomation: scripted behaviorsHRI: human x automation

Stefano Carpin has a student doing a validation for laser rangefinder model

SimpleUI sample interface

Pyro Controls

Jijun’s USARsim Practicals

Today (Saturday) after lunchBasic- architecture & installation

Late afternoon- ControllersPyroPlayerVideo feedback

Sunday afternoon- Programming in Unrealbuild a sensor, robot

Hardware Issues

Requires current (2000 +) pentium 4 pcFor Linux requires Nvidia video cardServer uses few resourcesClients (attached spectators) are resource hogsWithout modification server handles 32 spectator clients & unlimited robots without spectatorsTo add robot

Create robot in simulationConnect socket to control robotAdd ‘spectator’ to provide camera view

Near term Mods

Rewrite vehicle classes to use downloadable Unreal Runtime (warfare) EngineExtend manual instructions for

Assembling new arenas from partsMore pointers to available Unreal tutorials & FAQs

Extend video control demo’d in SimpleUI to other releases of DirectXAdd Aibo (courtesy of Sheila Tejada)

Send us questions so we can write a better manual & FAQ

USARsim & Robocup

Who is the community?Our (Mike, Katia, Illah, & Jijun) interests are in:

Cooperating teamsRobots with adjustable autonomyHuman-Robot Interaction

So we would like to see USARsim in Robocupevolve from a simulation analgue of the USAR league into something in between the USAR physical robot league & the Robocup Rescue simulation league

USARsim & Robocup

So we would favor rules that:Encourage multiple robotsEncourage heterogeneous robotsEncourage Human-Robot Interaction

BUTFor now the IMPORTANT thing is to AGREE ON RULES HERE AT THIS CAMP

Modest Proposal

Accept USAR rules(but maybe with just a nod to the previous slide?)

Issues

Fixing or scoring platforms & sensorsa) Everybody runs the same robot & sensor

suiteb) Use weighting factor to reward performance

using low capability platform or sensorsc) As robots are added allow higher capability

platforms

Issues

Arenas & Nike Silo orNovel Environments

Who is going to make themWhat should be modeled

Issues USAR/USARsim Differences

Size of venueCould add dynamic events such as collapsesHazards such as fire, water, etc.Radio drop outSmoke, etc.

Not this year but 2006?

USARsim contol & maintenance

Maintain informally within USAR communityTransfer to NIST to administer like physical arenasMove to Sourceforge or similar opensourcerepositoryIdeas?

Download simulator & docs athttp://usl.sis.pitt.edu/ulab