1

CCC/Robotics‐Roadmapf.ServiceRobotics1.Introduction........................................................................................................................ 2

2.StrategicFindings................................................................................................................ 32.1PrincipalMarketsandDrivers..................................................................................................... 42.2NearTermOpportunitiesandFactorsEffectingCommercialization ............................................ 52.3ScientificandTechnicalChallenges ............................................................................................. 6

Mobility ................................................................................................................................................6Manipulation ........................................................................................................................................7Planning ................................................................................................................................................8SensingandPerception ........................................................................................................................9Architectures,Cognition,andProgrammingParadigms.......................................................................9HumanRobotInteraction(HRI) ..........................................................................................................10ResearchInfrastructure ......................................................................................................................10MechanicalHardware.........................................................................................................................11

3.KeyChallenges/Capabilities ............................................................................................ 123.1MotivatingScenarios................................................................................................................ 123.2CapabilitiesRoadmap............................................................................................................... 15

Human‐likeDexterousManipulation .................................................................................................16Real‐World3DPlanningandNavigation ............................................................................................17Cognition ............................................................................................................................................18RobustPerception ..............................................................................................................................19Physical,intuitiveHRIandinterfaces .................................................................................................19SkillAcquisition...................................................................................................................................20Saferobots..........................................................................................................................................20

4.BasicResearchandTechnologies ...................................................................................... 21ArchitectureandRepresentations.................................................................................................. 21ControlandPlanning...................................................................................................................... 21Perception ..................................................................................................................................... 22Robust,High‐fidelitySensors.......................................................................................................... 22NovelMechanismsandHigh‐PerformanceActuators ..................................................................... 22LearningandAdaptation................................................................................................................ 22PhysicalHuman‐RobotInteraction ................................................................................................. 23SociallyInteractiveRobots ............................................................................................................. 23

5.Contributors ..................................................................................................................... 23

2

1.IntroductionServiceRoboticsisdefinedasthoseroboticssystemsthatassistpeopleintheirdailylivesatwork,intheirhouses,forleisure,andaspartofassistancetohandicappedandelderly.Inindustrialroboticsthetaskistypicallytoautomatetaskstoachieveahomogenousqualityofproductionorahighspeedofexecution.In

contrast,serviceroboticstasksareperformedinspacesoccupiedbyhumansandtypicallyindirectcollaborationwithpeople.Serviceroboticsisnormallydividedintoprofessionalandpersonalservices.

Professionalserviceroboticsincludesagriculture,emergencyresponse,pipelinesandthenationalinfrastructure,forestry,transportation,professionalcleaning,andvariousotherdisciplines.

[Professionalservicerobotsarealsousedformilitarypurposesbuttheirapplicationinthisareaisnotincludedinthisreport.]Thesesystemstypicallyaugmentpeopleforexecutionoftasksintheworkplace.AccordingtotheIFR/VDMAWorldRoboticsmorethan38,000professionalrobotsareinusetodayand

themarketisgrowingrapidlyeveryyear.Severaltypicalprofessionalrobotsareshowninfigure1.

Figure1:Typicalservicerobotsforprofessionalapplications.

Personalservicerobotsontheotherhandaredeployedforassistancetopeopleintheirdailylivesin

theirhomesorasassistantstothemforcompensationformentalandphysicallimitations.Thebyfarlargestgroupofpersonalservicerobotsconsistsofdomesticvacuumcleaners;over3millioniRobotRoomba’salonehavebeensoldworldwideandthemarketisgrowing60%+/year.Inaddition,alarge

numberofrobotshavebeendeployedforleisureapplicationssuchasartificialpets(AIBO),dolls,etc.Withmorethan2millionunitssoldoverthelast5years,themarketforsuchleisurerobotsisexperiencingexponentialgrowthandisexpectedtoremainoneofthemostpromisinginrobotics.A

numberoftypicalpersonalservicerobotsystemsareshowninfigure2.

Figure2:Typicalservicerobotsforpersonalapplications.

TALON®Hazmatrobot

EnvirobotTMPaintStrippingRobotPackBotTMTacticalRobot ResponderTMPipelineRobot

RoombaTMVacuumCleaningRobot ATRSTMRoboticWheelchairSystemStrippingRobot

VerroTMPoolCleaningRobot LEGO®MindstormsTMEducationalRobot

3

Theservicerobotspanelincludedbothprofessionalandpersonalservicesandassuchcoveredahighlydiversesetofapplicationsandproblems.

2.StrategicFindingsAftermuchdiscussion,therewasgeneralagreementamongthosepresentatthemeetingthatwearestill

10to15yearsawayfromawidevarietyofapplicationsandsolutionsincorporatingfull‐scale,generalautonomousfunctionality.Someofthekeytechnologyissuesthatneedtobeaddressedtoreachthatpointarediscussedinalatersectionofthisreport.Therewasfurtheragreementamongthosepresent,

however,thatthetechnologyhassufficientlyprogressedtoenableanincreasingnumberoflimitedscaleand/orsemi‐autonomoussolutionsthatarepragmatic,affordable,andproviderealvalue.Commercialproductsandapplicationsbasedonexistingtechnologyhavealreadybeguntoemergeandmoreare

expectedasentrepreneursandinvestorsrealizetheirpotential.Theparticipantsidentifiedseveralmarketswheretheseearlycommercialsolutionsareappearingandwhereserviceroboticsislikelytohavethegreatestimpact.Amongtheareasidentifiedarehealthcare,nationalinfrastructureandresource

management,energyandtheenvironment,security,transportationandlogistics,andeducationandentertainment.

Figure3.ThechangesindemographicsinUSAandJapanrespectively.

4

Oneofthekeyfactorscontributingtotheidentifiedtrendsisouragingpopulation.Thisimpactsserviceroboticsbothintermsoftheneedtoaddressashrinkingworkforceaswellastheopportunitytodevelop

solutionsthatwillmeettheirhealthcareneeds.Asshowninfigure3,theUnitedStatesisonthethresholdofa20yeartrendthatwillseeaneardoublingofthenumberofretireworkersasapercentageofthecurrentworkforce;fromjustover2retireesforevery10workerstodaytojustover4retireesforevery10

workersin2030.InJapanthesituationisevenworseandhasfueledamajornationalinitiativetodeveloptheroboticstechnologyneededtohelpcarefortheirrapidlyagingpopulation.Generallyspeaking,professionalserviceroboticsisexpectedtoserveasaworkforcemultiplierforincreasedeconomicgrowth,

whiledomesticserviceroboticsisexpectedtoenablesustainedpersonalautonomy.

Whileincreasingproductivityandreducingcostsarethecommondenominatorofservicerobotics,eachsystemisexpectedtouniquelyprovideacompellingsolutiontocertain,criticalmarketspecificissuesorneeds.Forexample,akey,primarydriverinusingroboticstechnologytoautomatetheautomobile

factorieswasthedesiretoobtainconsistent,day‐to‐dayqualityandavoidthe“builtonMonday”syndrome.

2.1PrincipalMarketsandDriversHealthcare&QualityofLife–thecurrentapplicationofroboticstechnologytoprovidetele‐operatedsolutionssuchasIntuitiveSurgical’sdaVincisurgicalsystemrepresentsthetipofthe

iceberg.Roboticstechnologyholdsenormouspotentialtohelpcontrolcosts,empowerhealthcareworkers,andenableagingcitizenstolivelongerintheirhomes.

Energy&Environment–theattendeesidentifiedthesetwocloselylinkedissuesasbothcriticalto

thefutureofourcountryandripefortheemergenceofroboticstechnologyapplications,especiallyintheareasofautomatingtheacquisitionofenergyandmonitoringtheenvironment.

Manufacturing&Logistics–beyondthetraditionalapplicationofroboticstechnologytoautomatecertainassemblylinefunctions,themeetingparticipantsagreedthatthereistremendouspotential

tofurtherautomatethemanufactureandmovementofgoods;asfullyexploredintheparallelroadmappingeffortinthisarea.Inparticular,roboticstechnologypromisestotransformsmallscale,or“micro”,manufacturingoperationsandintheprocesshelpacceleratethetransitionof

manufacturingbacktoAmerica.Thisbeliefhassincebeensubstantiatedbytheformationofanewstart‐uproboticscompany,HeartlandRobotics,organizedspecificallyforthatpurpose.

Automotive&Transportation–althoughwearestilldecadesawayfromthefullyautonomousautomobile,roboticstechnologyisalreadyappearingintheformofadvanceddriverassistanceand

collisionavoidancesystems.Publictransportationisanotherareathatisexpectedtobecomeincreasinglyautomated.Asroboticstechnologycontinuestoimproveandmature,unmannedtransportationsystemsandsolutionsdevelopedforlimitedscaleenvironmentssuchasairportswill

beadaptedforimplementationinurbancentersandothergeneralpurposeenvironments.

HomelandSecurity&InfrastructureProtection–participantsinthemeetingagreedthatroboticstechnologyofferstremendouspotentialforapplicationsinborderprotection,searchandrescue,portinspectionandsecurity,andotherrelatedareas.Inaddition,roboticstechnologyisexpectedto

5

beincreasinglyusedtoautomatetheinspection,maintenance,andsafeguardingofournation’sbridges,highways,waterandsewersystems,energypipelinesandfacilities,andothercritical

componentsofournation’sinfrastructure.

Entertainment&Education–thisarea,perhapsmorethananyotherhasseentheearlyemergenceofroboticstechnologyenabledproducts.Inparticular,roboticshasthepotentialtosignificantlyaddressthescience,technology,engineering,andmath(“STEM”)crisisfacingthenationandto

becometheveritable“fourthr”ofeducation.ThisisevidencedbythetremendoussuccessofFIRST,anon‐profitorganizationfoundedin1999thatrunsnationalroboticscompetitionstoinspireyoungpeopletobescienceandtechnologyleaders,andotherroboticsinspirededucationalinitiatives.

Roboticsprovideskidswithacompellingandtactileavenuetolearnandapplyboththeunderlyingkeymathematicsandsciencefundamentalsandtheengineeringandsystemintegrationprinciplesrequiredtoproduceintelligentmachinestoaccomplishcertainmissions.

2.2NearTermOpportunitiesandFactorsEffectingCommercializationSignificantinvestmentisrequiredforexpandedresearchanddevelopmentofroboticstechnologyifthefull

promiseofwhatcanbeachievedineachoftheaboveareasistoberealized.Asnotedabove,wearestillalongwayfromthefullyautonomousroboticstechnologyrequiredtoautomateprocessestotheextentthatnohumanattentionorinterventionisrequired.Thatsaid,itwasthecollectiveopinionofthosein

attendancethatenoughprogressinroboticstechnologyhasbeenmadetoenablethedevelopmentandmarketingofawidevarietyofinitialapplicationsandproductsineachoftheseareastoachievesignificant

levelsof“humanaugmentation”.

Suchsolutionswillbecapabletovaryingdegreesofautomaticallyperformingthefollowingtypesoffunctions:monitoringdefined,yetdynamicphysicalenvironments,identifyingobjects,detectingchanges,orotherwiseperceivingthestatusoftheirassignedenvironments,analyzingandrecommendingactionsthat

shouldbetakeninresponsetodetectedconditions,takingsuchactionsinresponsetohumancommands,and/orautomaticallyperformingsuchactionswithincertainpre‐authorizedboundariesnotover‐riddenbyhumanoperators.

Examplesofsuchroboticssolutionstodayincludetele‐operatedsystemssuchasthedaVincisurgicalsystem

andautonomous,specializedproductivitytoolssuchastheRoomba.AstheInternetcontinuestoevolve,itwillinspireanaturalprogressionfromsensingatadistancetotakingactionatadistance.ThisextensionoftheInternetintothephysicalworldwillservetofurtherblurtheboundariesamongcommunity,

communication,computing,andservicesandinspirenewdimensionsintelecommutingandtelepresenceapplications.Hybridsolutionsarelikelytoemergethatenabledistributedhumancognitionandenabletheefficientuseofhumanintelligence.Suchsolutionswillcombinetherobotics‐enabledcapabilitytoremotely

andautonomouslyperceivesituationsrequiringinterventionwiththeInternet‐enabledcapabilityforhumanoperatorstotakeactionfromadistanceonanas‐neededonlybasis.

6

Asreferencedabove,ouragingpopulationwillresultinafuturelaborshortage.Asworkersseektomoveupthejobhierarchy,therewillbeagrowingneedtoaugmentandincreasinglyautomatejobsatthebottom

becausetheworkerstoperformthemmaynotbereadilyavailableandeventuallymaynotexist.Whilethechallengeofachievingfullyautonomoussolutionsinthelongrunremainsprimarilytechnological,thechallengeintheneartermisoneofinvestinginthescienceofdevelopingrequirementsandotherwise

determininghowtobest“crossthechasm”;itisoneofidentifyingtherightvaluepropositions,drivingdowncosts,developingefficient,effectivesystemsengineeringprocesses,determininghowtobestintegratesuchsolutionsintocurrentoradaptedprocesses,andotherwiseaddressingtheknow‐howgapof

transitioningtechnologyintoproducts.

2.3ScientificandTechnicalChallengesWorkshopparticipantsworkedinthreebreak‐outgroupstoidentifytechnicalandscientificchallengespertinenttotheapplicationsandbusinessdriversdescribedintheprevioussection.Thefirstbreak‐outgroupfocusedonapplicationandsystemsdesign;thesecondgroupdiscussedaction,cognition,

planning,andotherelementsofroboticintelligence;andthefinalgroupidentifiedchallengesinhumanrobotinteraction.Thissectionsummarizestheirfindings.Becausethechallengesidentifiedbythethreegroupsspantheboundariesbetweentherespectivetopicareas,wewillpresentthetechnicaland

scientificchallengesidentifiedbythebreak‐outgroupsinanintegratedmanner.Theemphasisofthissectionisondescribingthechallenges,notonlayingoutaroadmaptowardsaddressingthesechallenges—sucharoadmapwillbeoutlinedinthenextsection.

MobilityMobilityhasbeenoneofthesuccessstoriesofroboticsresearch.Thissuccessisexemplifiedbya

numberofsystemswithdemonstratedperformanceinrealworldenvironments,includingmuseumtourguidesandautonomouslydrivingcars,asintheDARPAGrandChallengeandUrbanChallenge.Nevertheless,workshopparticipantsagreedthatanumberofimportantopenproblemsremain.Finding

solutionstotheseproblemsintheareaofmobilitywillbenecessarytoachievethelevelofautonomyandversatilityrequiredfortheidentifiedapplicationareas.

Participantsidentified3Dnavigationasoneofthemostimportantchallengesintheareaofmobility.Currently,mostmapping,localization,andnavigationsystemsrelyontwo‐dimensionalrepresentations

oftheworld,suchasstreetmapsorfloorplans.Asroboticapplicationsincreaseincomplexityandaredeployedineveryday,populatedenvironmentsthataremoreunstructuredandlesscontrolled,however,these2Drepresentationswillnotbesufficienttocaptureallaspectsoftheworldnecessaryfor

commontasks.Ifwillthereforebeimportanttoenabletheacquisitionofthree‐dimensionalworldmodelsinsupportofnavigationandmanipulation(seenextsection).These3Drepresentationsshouldnotonlycontainthegeometrylayoutoftheworld;instead,mapsmustcontaintask‐relevantsemantic

informationaboutobjectsandfeaturesoftheenvironment.Currentrobotsaregoodatunderstandingwherethingsareintheworld,buttheyhavelittleornounderstandingofwhatthingsare.Whenmobilityisperformedinservicetomanipulation,environmentalrepresentationsshouldalsoinclude

objectaffordances,i.e.knowledgeofwhattherobotcanuseanobjectforAchievingsemantic3Dnavigationwillrequirenovelmethodsforsensing,perception,mapping,localization,objectrecognition,

7

affordancerecognition,andplanning.Someoftheserequirementsarediscussedinmoredetaillaterinthissection.

Oneofthepromisingtechnologiestowardssemantic3Dmapping,asidentifiedbytheparticipants,is

usingdifferentkindsofsensorsforbuildingmaps.Currently,robotsrelyonveryhighprecisionlaser‐basedmeasurementsystemsforlearningabouttheirenvironment,usingmappingalgorithmsknownas"SLAM"algorithms.Theparticipantsidentifiedadesiretomoveawayfromlaserstocameras,todevelop

anewfieldof"visualSLAM"(VSLAM).Thistechnologyreliesoncameras,whicharerobust,cheap,andreadilyavailablesensors,tomapandlocalizeinathree‐dimensionalworld.Alreadytoday,VSLAMsystemsexhibitimpressivereal‐timeperformance.ParticipantsthereforebelievedthatVSLAMwill

likelyplayaroleinthedevelopmentofadequateandmoreaffordable3Dnavigationcapabilities.

Participantsidentifiedadditionalrequirementsfor3Dnavigationthatwillbecriticaltomeettherequirementsoftargetedapplications.Outdoor3Dnavigationposesanumberofimportantchallengesthathavetobeaddressedexplicitly.Amongthemisthefactthatcurrent2Denvironmental

representationscannotcapturethecomplexityofoutdoorenvironmentsnorthechanginglightingconditionsthatcausesubstantialvariabilityintheperformanceofsensormodalities.Participantsalsoidentifiedrobustnavigationincrowdsasanimportantmobilitychallenge.

ManipulationSubstantialprogressinmanipulationisneededforalmostalloftheserviceroboticsapplications

identifiedintheprevioussection.Theseapplicationsrequirearobottointeractphysicallywithitsenvironmentbyopeningdoors,pickingupobjects,operatingmachinesanddevices,etc.Currently,autonomousmanipulationsystemsfunctionwellincarefullyengineeredandhighlycontrolled

environments,suchasfactoryfloorsandassemblycells,butcannothandletheenvironmentalvariabilityanduncertaintyassociatedwithopen,dynamic,andunstructuredenvironments.Asaresult,participantsfromallthreebreak‐outgroupsidentifiedautonomousmanipulationasacriticalareaof

scientificinvestigation.Whilenospecificdirectionsforprogresswereidentified,thediscussionsrevealedthatthebasicassumptionsofmostexistingmanipulationalgorithmswouldnotbesatisfiedintheapplicationareastargetedbythiseffort.Graspingandmanipulationsuitableforapplicationsin

open,dynamic,andunstructuredenvironmentsshouldleveragepriorknowledgeandmodelsoftheenvironmentwheneverpossible,butshouldnotfailcatastrophicallywhensuchpriorknowledgeisnotavailable.Asacorollary,trulyautonomousmanipulationwilldependontherobot’sabilitytoacquire

adequate,task‐relevantenvironmentalmodelswhentheyarenotavailable.Thisimpliesthat—incontrasttomostexistingmethodswhichemphasizeplanningandcontrol—perceptionbecomesanimportantcomponentoftheresearchagendatowardsautonomousmanipulation.

Participantsidentifiednovelrobotichands(discussedinthesubsectiononHardware),tactilesensing(seeSensingandPerception),andhighly‐accurate,physicallyrealisticsimulatorsasimportantenablersforautonomousmanipulation.Participantsuggestedthatcompetent“pickandplace”operationsmayprovideasufficientfunctional

basisforthemanipulationrequirementsofamanyofthetargetedapplications.Itwastherefore

8

suggestedthatpickandplaceoperationsofincreasingcomplexityandgeneralitycouldprovidearoadmapandbenchmarkforresearcheffortsinautonomousmanipulation.

PlanningResearchintheareaofmotionplanninghasmadenotableprogressoverthelastdecade.Theresultingalgorithmsandtechniqueshaveimpactedmanydifferentapplicationareas.Nevertheless,participants

agreedthatrobustdynamic3Dpathplanningremainsanopenproblem.Animportantaspectofthisproblemisthenotionofarobot’ssituationalawareness,i.e.therobot’sabilitytoautonomously

combine,interleave,andintegratetheplanningofactionswithappropriatesensingandmodelingoftheenvironment.Theterm“appropriate”alludestothefactthatcompleteandexactmodelsoftheenvironmentcannotbeacquiredbytherobotinrealtime.Instead,itwillbenecessarytoreasonabout

theobjectives,theenvironment,andtheavailablesensingandmotoractionsavailabletotherobot.Asaresult,theboundarybetweenplanningandmotionplanningisblurred.Toplanamotion,theplannerhastocoordinatesensingandmotionundertheconstraintsimposedbythetask.Toachievetask

objectivesrobustlyandreliably,planninghastoconsiderenvironmentalaffordances.Thismeansthattheplannerhastoconsiderinteractionswiththeenvironmentandobjectsinitaspartoftheplanningprocess.Forexample:topickupanobject,itmaybecomenecessarytoopenadoortomoveintoa

differentroom,topushawayachairtobeabletoreachtoacabinet,toopenthecabinetdoor,andtopushanobstructingobjectoutoftheway.Inthisnewparadigmofplanning,thetaskandconstraintsimposedbythetaskandtheenvironmentarethefocus;the“motion”of“motionplanning”isameans

toanend.Constraintsconsideredduringplanningcanarisefromobjectmanipulation,locomotion(e.g.footstepplanning),kinematicanddynamicconstraintsofthemechanism,postureconstraints,orobstacleavoidance.Planningundertheseconstraintsmustoccurinrealtime.

Someoftheconstraintsontherobot’smotionaremosteasilyenforcedbyleveragingsensorfeedback.

Obviousexamplesarecontactconstraintsandobstacleavoidance.Theareaoffeedbackplanningandtheintegrationofcontrolandplanningarethereforeimportantareasofresearchtowardssatisfyingtheplanningrequirementsidentifiedbytheparticipants.Afeedbackplannergeneratesapolicythat

directlymapsstatestoactions,ratherthangeneratingaspecificpathortrajectory.Thisensuresthatsensor,actuation,andmodelinguncertaintiescanadequatelybeaddressedusingsensoryfeedback.

Theincreasedcomplexityofplanninginthiscontextwillalsorequirenovelwaysofcapturingtaskdescriptions.Whileinclassicalmotionplanningthespecificationoftwoconfigurationsfullyspecifieda

planningtask,theviewofplanningdescribedherehastohandlemuchrichertaskrepresentationstoaddresstherichnessofmanipulationtasksandintermediateinteractionswiththeenvironment.

Participantsalsoperceivedtheneedforformalmethodstoperformverificationandvalidationoftheresultsofplanners.Suchguaranteesmayberequiredtoensuresafeoperationofrobotsin

environmentspopulatedwithhumans.

9

SensingandPerceptionSensingandperceptionareofcentralimportancetoallaspectsofrobotics,includingmobility,manipulation,andhuman‐robotinteraction.Participantswereconvincedthatinnovationinsensingandperceptionwillhaveprofoundimpactontherateofprogressinrobotics.

Participantsbelievedthatnewsensingmodalitiesaswellasmoreadvanced,higher‐resolution,lower‐

costversionsofexistingmodalitieswouldbeareasofimportantprogress.Forexample,participantsexpectimportantadvancesinmanipulationandmobilityalikefromdense3Drangesensing,possiblyby

LIDAR.Advancesindexterousmanipulationarelikelytorequireskin‐liketactilesensorsforrobotichands.Butalsospecializedsensors,forexampleforsafety,termedsafetysensors,werediscussedbytheparticipants.Thesesensorscouldtakevariousforms,suchasrangeorheatsensingtodetectthe

presenceofhumans,orcouldbeimplementedbyspecialtorquesensorsaspartoftheactuationmechanism,capableofdetectingunexpectedcontactbetweentherobotanditsenvironment.Skin‐likesensorsfortheentireroboticmechanismwouldalsofallintothiscategory.

Thedatadeliveredbysensormodalitiesmustbeprocessedandanalyzedbyalgorithmsforperceptionin

complexandhighlydynamicenvironmentsundervaryingconditions,includingdifferencesbetweendayandnightandobscurantslikefog,haze,brightsunlight,andthelike.Participantsidentifiedtheneedforprogressinhigh‐levelobjectmodeling,detection,andrecognition,inimprovedsceneunderstanding,

andintheimprovedabilitytodetectactivitiesandintent.Novelalgorithmsforaffordancerecognitionarerequiredtosupportthetypeofplanningdescribedintheprevioussubsection.Participantsalsodiscussedtheneedforaccuratesensormodelsinsupportofperceptualalgorithms.

Architectures,Cognition,andProgrammingParadigmsThediscussionsonthetopicsofmobility,manipulation,planning,andperceptionrevealedthattheseissuescannotbeviewedinisolationbutareintricatelylinkedtoeachother.Thequestionofhowto

engineerasystemtoeffectivelyintegratespecificskillsfromthoseareastoachievesafe,robust,task‐directed,orevenintelligentbehaviorremainsanopenquestionoffundamentalimportanceinrobotics.Researchtowardsthisobjectivehasbeenconductedunderthenameofarchitectures,cognition,and

programmingparadigms.Thisdiversityinapproachesorevenphilosophicalviewpointsmayreflectthelackofunderstandinginthecommunityonhowtoadequatelytacklethischallenge.Thisdiversityofviewpointsisalsoreflectedinthediversityoftoolscurrentlybroughttobearonthisissue:theyrange

fromimitationlearningtoexplicitprogrammingofso‐calledcognitivearchitectures.Someparticipantsfeltthatamixtureofthesewouldprobablyberequiredtoachievethedesiredoutcome.

Oneoftheclassicalapproachestowardstheoverarchingissueofgeneratingrobust,autonomousbehavioristhesense/plan/actloopusuallyemployedbymoderncontrolsystems.While

sense/plan/acthasbeenaconstantinroboticsresearchoverthelastseveraldecades,someparticipantsfeltthatnovelapproacheswouldlikelydeviatefromthisapproachinitssimplestform.Possible

alternativesaremultiplenestedorhierarchicalloops,thebehavior‐basedapproach,combinationsofthetwo,orpossiblyevencompletelynovelapproaches.

10

Allparticipantsagreedthatthisareaofinvestigationwillrequiresubstantialattentionandprogressonthepathtowardsautonomousroboticsystems.

HumanRobotInteraction(HRI)Giventheultimategoalofdeployingmobileanddexterousrobotsinhumanenvironmentstoenablecoexistenceandcooperation,substantialprogresswillberequiredintheareaofhumanrobotinteraction.Theseinteractionscouldalsobecomeanimportantcomponentinanoverarchingapproach

torobustrobotbehavior,asdiscussedintheprevioussubsection.Robotmightlearnnovelskillsfromtheirinteractionswithhumansbutunderallcircumstancesshouldbecognizantofthecharacteristicsandrequirementsoftheircommunicationwithhumans.

Inadditiontothemodesofcommunication(verbal,nonverbal,gesture,facialexpression,etc.),

participantsidentifiedanumberofimportantresearchtopics,includingsocialrelationships,emotions(recognition,presentation,socialemotionalcognition/modeling),engagement,andtrust.Anunderstandingoftheseaspectsofhumanrobotcommunicationshouldleadtoanautomaticstructuring

oftheinteractionsbetweenhumansandrobotswhereroboticsystems’abilitytooperateindependentlyrisesorfallsautomaticallyasboththetaskandthehumansupervisor'sinteractionwiththesystemchange.

Progresstowardstheseobjectiveswilldependoneffectiveinputdevicesandintuitiveuserinterfaces.

ParticipantsalsoadvocatedthedevelopmentofavarietyofplatformstostudyHRI,includinghumanoidrobots,mobilemanipulationplatforms,wheelchairs,exoskeletons,andvehicles.Participantsidentifiedadesign/build/deploycycleinwhichHRIresearchshouldprogress.Thedesignprocessshouldconsider

inputfromanumberofrelevantcommunities,includingthebasicresearchcommunityandendusers.Thebuildprocessintegratesnumerouscomponentsandresearchthreadsintoasinglesystem;herethereisanopportunityforindustrycollaborationsandtechnologytransfer.Finally,theintegrated

systemisdeployedinareal‐worldcontext.ParticipantssuggestedthenotionofaRobotCity(seenextsubsection)asapromisingideatoevaluateHRIinareal‐worldcontext.Thecycleisclosedbyincorporatingenduserfeedbackintotheexperimentaldesignofthenextiterationofthe

design/build/deploycycle.

ResearchInfrastructureWorkshopparticipantsfeltstronglythatrapidprogresstowardstheidentifiedscientificobjectiveswillcriticallydependonthebroadavailabilityofadequateresearchinfrastructure,includinghardwareandsoftware.Toaddresstheresearchchallengesgivenabove,itwillbenecessarytoconstructrobotic

platformsthatcombinemanyadvancedandinteractingmechanicalcomponents,providingadequatecapabilitiesformobility,manipulation,andsensing.Thesesplatformswillbecontrolledbyamultitudeofindependentlydeveloped,yetinterdependentlyoperatingsoftwarecomponents.Asaresult,these

integratedroboticplatformsexhibitadegreeofcomplexitythatisbeyondwhatcaneasilybedesigned,developed,tested,andmaintainedbymanyindependentlyoperatingresearchgroups.Thelackofstandardizationofhardwareandsoftwareplatformsmayalsoresultinafragmentationoftheresearch

11

community,difficultiesinassessingthevalidityandgeneralityofpublishedresults,andthereplicationofmuchunnecessaryengineeringandintegrationeffort.

Toovercomethesechallenges,workshopparticipantsadvocatedcoordinatedcommunityeffortsforthe

developmentofhardwareandsoftwaresystems.Theseeffortsshouldincludethedevelopmentofanopenexperimentalplatformthatwould—preferablyatlowcost—supportabroadrangeofresearcheffortsontheonehand,whileenablingtechnologyandsoftwarereuseacrossresearchgroupsonthe

otherhand.OneexampleofsuchanopenplatformisROS,arobotoperatingsystembeingdevelopedbyWillowGaragethatenablescodereuseandprovidestheservicesonewouldexpectfromanoperatingsystem,suchaslow‐leveldevicecontrol,implementationofcommonly‐usedfunctionality,and

message‐passingbetweenprocesses.Ideally,suchplatformswouldbecomplementedbyphysicalsimulationsoftwaretosupportearlydevelopmentandtestingofalgorithmswithoutcompromisingthesafetyofresearchersandhardware.Developmenteffortscouldalsobenefitfromroboticintegrated

developmentenvironments(IDEs);theseIDEsenforcedmodularityinsoftwaredevelopmenttherebyfacilitatingreuseanddocumentation.

Participantsnotedthatresearchinroboticsisrarelythoroughlyevaluatedandtestedinwell‐defined,repeatableexperiments.Otherfields,suchascomputervision,havegreatlybenefitedfrompublicly

availabledatasets,whichenabledanobjectivecomparisonbetweenmultiplealgorithmsandsystems.Theparticipantsthereforesuggestedthecreationandexpansionofrepositoriesofexperimentaldata,whichcouldthenserveascommunity‐widebenchmarks.However,asmuchoftheresearchinrobotics

isfocusedonthephysicalinteractionbetweentherobotanditsenvironment,electronicdatasetsarenotsufficient.Theyshouldbecomplementedbyskill‐specificbenchmarksconsistingofphysicalobjects.Forexample,anumberofreadilyavailableobjectscanbeselectedasabenchmarkforgrasping

research.Furthermore,entirebenchmarkenvironmentsweresuggestedtodevelop,evaluate,andcomparetheperformancewithrespecttoaparticularapplicationorimplementation.Such

environmentscouldrangeinsizeandcomplexityfromasimpleworkspace(anofficedeskorakitchencounter)toanentireroom,ahouse,oranentirecityblock.Inthiscontext,thenotionofaRobotCitywasmentioned:aregularurbanenvironmentinwhichallinhabitantsarepartoftheexperimentand

helpintheevaluationprocessaswellaswiththedefinitionofadequaterequirementsforeverydayapplicationenvironments.

Manyoftheproposedefforts—andinparticularhardwareorsoftwareintegrationefforts—falloutsideofthescopeofexistingfundingprograms.Participantsnotedthatapolicychangeinthisregardwould

benecessarytoensurethattheavailabilityofresearchinfrastructuredoesnotrepresentabottleneckintheprogresstowardsautonomousroboticsystemsineverydayenvironments.

MechanicalHardwareSafetyisacriticalfactorforthedeploymentofroboticsystemsinhumanenvironments.Inherentlysaferobotswouldalsoenablemodesofhumanrobotinteractionthatcanincreaseacceptanceofrobotictechnologyineverydaylife.Participantsthereforefeltthatinherentlysafermotorsandmechanisms

withincreasedstrengthtoweightratiowouldrepresentanimportantenablingtechnology.Insuchmechanismsvariablecompliancewouldbeadesirableproperty.Theconceptofvariablecompliance

12

referstoamechanismsabilitytoadjustitsbehaviortoreactionforceswhencontactingtheenvironment.Thesereactionforcescanbevariedfordifferenttasks.Suchmechanismsenablesafe

operation,especiallywheninteractingwithhumans,aswellasflexible,robust,andcompetentmotionwhenincontactwiththeenvironment.Furthermore,energyefficiencywasidentifiedasacriticalconcernformanyapplications,asrobotswillhavetooperatewithouttethersforextendedperiodsof

time.Finally,novelorimprovedmodesoflocomotionbeyondwheelsareneededtoenablesafeandreliableoperationinindoorandoutdoorenvironments.Outdoorenvironmentsoftentimesexhibithighlyvariableterrainpropertieswhileoutdoormaycontainstairs,ladders,ramps,escalators,or

elevators.

Participantsidentifiedhighlydexterousandeasilycontrollablerobotichandsasanimportantareaforresearch.Progressinroboticgraspingandmanipulationverylikelywillgohandinhandwiththedevelopmentofnovelhandmechanisms.Atthesametime,participantsfeltthatthepotentialof

currenthandtechnologywerenotfullyleveragedbyexistinggraspingandmanipulationalgorithms.Itisthereforeconceivablethatmanyinterestingandrelevantapplicationscanbeaddressedwithavailablegraspingandmanipulationhardware.

3.KeyChallenges/Capabilities

3.1MotivatingScenarios

QualityofLifeRoboticstechnologyisexpectedtomakeatremendouscontributiontothelivesoftheelderlyanddisabled.Onesuchexampleofanexistingapplicationisarevolutionarytransportationmobilitysolutionthatenables

thosewithlimitedmobilitywhousewheelchairstoindependentlygetintoandoutoftheirvehiclesandremotelyloadandunloadtheirwheelchairsfromawiderangeofvehicles.Thissystemmakesitpossibleforthosedependent

onwheelchairstotransporttheirwheelchairusinganordinarypassengervanandtoaccessitwheneverneededwithoutassistancefromothersofferingthemadegreeoffreedomandindependenceheretoforeunavailable.Thissystemprovidessignificant

benefitsoverexistingtransportationmobilitysolutions,includinglowercostofownership,abilitytousestandardcrash‐testedautomotiveseats,greaterchoiceofvehicles,norequiredstructuralmodifications,andabilitytore‐installonsubsequentvehicles.

Agriculture

Roboticstechnologyisexpectedtoimpactamyriadofapplicationsinagricultureandaddressfarmers’constantstruggletokeepcostsdownandproductivityup.Mechanicalharvestersandmanyotheragriculturalmachinesrequireexpertdriverstoworkeffectively,whilefactors

suchaslaborcostsandoperatorfatigueincreaseexpensesandlimittheproductivityofthesemachines.Automatingoperationssuchascropspraying,harvesting,andpickingofferthepromiseofreducedcosts,increasedsafety,

greateryields,increasedoperationalflexibility,includingnighttime

ATRSTMRoboticWheelchair

AutonomousTractor

13

operations,andreduceduseofchemicals.Anumberofsuchprototypesystemsandapplications,includingautomatedfruitcropsprayingandfieldcropharvesting,havebeendevelopedandthetechnologyhasnow

maturedtothepointwhereitisreadytobetransitionedforfurthercommercializationandfielddeploymentwithinthenextfewyears.

InfrastructureRoboticstechnologyhastremendouspotentialtoautomatetheinspectionandmaintenanceofournation’sbridges,highways,pipelines,andotherinfrastructure.Already,thetechnologyhasbeenadaptedtodevelopautomatedpipelineinspectionsystemsthatreducemaintenanceandrehabilitationcostsbyproviding

accurate,detailedpipeconditioninformation.Suchsystems,basedonadvancedmulti‐sensorandotherroboticstechnology,aredesignedforundergroundstructuresandconditionsthatareotherwisedifficulttoinspect,includinglargediameterpipes,longhaulstretches,inverts,crowns,culverts,andmanholes,andin‐serviceinspections.Theseroboticplatformsnavigatethiscriticalwastewaterinfrastructuretoinspectsewerpipeunreachablebytraditionalmeansandproduceveryaccurate3Dimagesofthepipeinsidesurface.The

inspectioninformation,capturedindigitalform,servesasabaselineforfutureinspectionsandasaresultcanautomaticallycalculatedefectfeaturechangesovertime.

MiningRoboticstechnologyisalreadystartingtohaveadramaticimpactonboththeundergroundandsurfaceminingindustries.Aninnovativebeltinspectionsystemthatusesahigh‐speed"machinevision"systemandsoftwarealgorithmstomonitortheconditionofconveyorbeltsandhelpoperatorsdetectdefects,forexample,isineverydayuseatseveralundergroundcoalmines.Thepatentedsystemisdesignedtoreducecostlydowntimecausedbythedegradationandeventualruptureofconveyorbeltsplices.Onalargerscaleroboticstechnologyisbeingusedtodevelopautonomousversionsoflargehaultrucksusedinminingoperations.Caterpillarrecentlyannouncedthatitisdevelopinganautonomousmininghaulagesystemwithplanstointegrateautonomoushaultrucks,eachwithpayloadcapacitiesof240tonsormore,intosomeminesitesby2010.Theautonomoustechnologyisdesignedtoprovideproductivitygainsthroughmoreconsistencyinprocessesandminimizeenvironmentalimpactbybothimprovedefficiencyandoverallminesafety.

TransportationRoboticstechnologywillsignificantlyaffecteveryaspectofhowwetransportpeopleandgoodsinthecomingdecades;frompersonaltransportationsystemstointelligenthighwaystoautonomouspublictransportationsystems.CompaniessuchasSegwayandToyotahaveintroducedpersonaltransportationrobotsthatareriddeninstandingpositionandcontrolledbyinternalsensorsthatconstantlymonitortherider’spositionandautomaticallymaketheaccordingadjustments.Meanwhile,carmakersanddevicemanufacturersarecreating“smartcars”byinstallingmorepowerfulcomputersandsensors,givingdriversabetterideaoftheirenvironmentandcarperformance.

ResponderTMPipelineRobot

AutonomousHaulTruck

14

AlthoughAmericandriverslognearlytwiceasmanymiles(1.33trillionperyear)astheydid25yearsago,theroadstheyaredrivingonhaveincreasedincapacitybyonly5percent,resultingin3.7billionhoursofdriverdelaysand2.3billiongallonsofwastedfuel.Toaddressthisissuehighwayagenciesareattemptingtocreate“smartroads”byinstallingsensors,camerasandautomatictollreadersandapublic‐privatenationalinitiativecalledVehicleInfrastructureIntegration(VII)hasbeenlaunchedtomergesmartcarsandsmartroadstocreateavirtualtrafficinformationnetworkandbustupgridlock.Masstransportationsystemsarealsoexpectedtoadoptroboticstechnologytoprovideoperatorswithgreatersituationalawarenessandnavigationassistanceincrowdedurbancorridorstherebyhelpingtocontrolcostsandincreasesafety.

EducationRoboticshasalreadycommencedtransformingtheAmericanclassroom.RoboticsputsacademicconceptsincontextandisbeingusedatalllevelsinK‐12andcollegeeducation.Roboticsprovidesstudentswitha

tactileandintegratedmeanstoinvestigatebasicconceptsinmath,physics,computerscienceandotherSTEMdisciplines,whileenablingteachersatthesametimetointroduceconceptsaboutdesign,innovation,problemsolving,andteamwork.Roboticscurriculumshavebeendeveloped,teachershavebeentrained,andscoresofcompetitionsareheldeveryyearacrossthecountry.PerhapsthebestknownroboticscompetitionprogramsareoperatedbyFIRST,anon‐profitorganizationfoundedin1999toinspireyoungpeopletobescienceandtechnologyleaders.Asameasureofthe

growingpopularityofroboticscompetitions,FIRSTisexpectingover195,000studentstoparticipateinitscompetitionsinthecomingyear.Evenmoresignificantly,arecentBrandeisUniversitysurveyfoundthatFIRSTparticipantsaremorethantwiceaslikelytopursueacareerinscienceandtechnologyasnon‐FIRSTstudentswithsimilarbackgroundsandacademicexperiences.Althoughmuchprogresshasbeenmade,thesurfacehasonlybeenscratchedintermsofthepotentialimpactofroboticsineducation.Tomorefullyrealizethispotential,robotsneedtobemademoreaccessible,affordableandeasytouseforbothstudentsandteachers.

HomelandSecurityandDefenseTheuseofroboticstechnologyforhomelandsecurityanddefensecontinuestogrowasinnovativetechnologyhasimprovedthefunctionalityandviabilityofsearchandrescueefforts,surveillance,explosivescountermeasures,firedetection,andotherapplications.Unmannedsurveillance,detection

andresponsesystemswillbeabletomakeuseofroboticplatforms,fixedsensors,andcommandandcontrolnetworkstopotentiallymonitorandpatrolhundredsofmilesofroughborder

TopThreeFinishersinthe2008DARPAUrbanGrandChallenge

FIRSTLegoLeagueTMParticipants

DisasterSiteApplication

15

terrain,tosniffoutandlocatechemical/biological/radioactive/nuclear/explosivethreats,andsurveylargeperimetersassociatedwithborders,powerplantsorairports.Suchsystemswillenablesecuritypersonneltoautomaticallydetectpotentialthreats,totakeaclose‐infirstlookfromasafedistance,andtoprovideinitialdisruptionandinterdictionatthepointofintrusionifnecessary.Whileother“man‐packable”robotsequippedwithinstrumentsincludinginfraredcameras,nightvisionsensorsandmillimeter‐waveradarhavebeenusedatdisastersites,includingtheWorldTradeCenter,tosearchforvictims.

3.2CapabilitiesRoadmapInthefollowing,weidentifythekeychallengesthathavetobemetandthekeycapabilitiesthathaveto

bedevelopedinordertodeliverservicerobotscapableofaddressingtheaforementionedmotivatingscenarios.Figure4providesanoverviewoftheproposedroadmapandtheremainderofthisdocument.Therightcolumninthefigurelaysouttheapplicationareas,manyofwhicharedescribedinthe

motivatingexamplescenariosabove.High‐impactadvancesintheseapplicationareascanonlybeenabledifanumberofcapabilitiesforautonomousservicerobotsbecomeavailable.ThesecapabilitiesarelistedinthemiddleofthefigureanddescribedinmoredetailinSection3.Toachievetherequired

levelofcompetencyinthoseareas,sustainedinvestmentinresearchanddevelopmentsinanumberofbasicresearchareasandtechnologiesisrequired.Figure4showstheseresearchareasandtechnologiesintheleftcolumn;theyaredescribedinmoredetailinSection4.

16

Figure4.Overviewoftheroadmapfordomesticandindustrialservicerobotics:Sustainedresearchanddevelopmentinthebasicresearchareasintherightmostcolumnofthefigurewillenableanumberofelementarycapabilities,showninthemiddlecolumnofthefigure.Thesecapabilitiesinturnenable

progressintheapplicationareasontheright.

Human­likeDexterousManipulationEvensimpletasks,suchaspickingupunknownobjects,stillrepresentmajorresearchchallenges.The

levelofdexterityandcapabilitiesinphysicalreasoningrequiredforautonomousmanipulationinthecontextofprofessionalanddomesticserviceroboticsseemsfaroutofreach.Pressingproblemsinthisareaincludeadequatesensorsandassociatedperceptualcapabilities,dexteroushandsandsafe

manipulators,planningunderuncertainty,advancedcontrol,skilllearningandtransfer,andmodelingandsimulation.

Someparticipantsbelievedthattherequiredcompetencyinmanipulationcanonlybeachievedwhenthesedifferentareasareadvancedinacoordinatedfashionratherthaninisolation.Forexample,novel,

skin‐liketactilesensorsholdgreatpromisefordexterousin‐handmanipulation.However,welackthealgorithmstoprocessthedatafromsuchsensors.Itisconceivablethattechniquesfromcomputer

17

visioncouldinterpretthetactileinformationasanimageandthereforeareabletocomputeusefulabstractionsofthehigh‐dimensionaltactiledata.Atthesametime,inspirationfromcomputervision

algorithmsmayenablethedesignofsimplertactilesensorsthatcontainsimplelocalpre‐processingtailoredtothespecificalgorithmstheysupport.

In5,10,and15yearsthefollowinggoalsarepossiblewithsustainedresearchanddevelopment:

• 5years:Robotsperformlimitedpickandplacetaskinthehomeandinindustrialsettings;robotsareabletoreliablyopendoorsandcabinets.Thesemanipulationtasksareaccomplished

partiallybyengineeringtheenvironment,partiallybyequippingrobotswithspecialized(oratleastnotverygeneralpurpose)end‐effectors,andbymakingsimplifyingassumptionsregardingtheenvironment.

• 10years:Robotsrobustlymanipulatelarge,graspable,rigid,possiblyarticulatedobjectsandtoolswithoutpossessingapriorimodels.Robotsimprovetherobustnessandapplicabilityofmanipulationandgraspingskillswithexperience.Robotsacquiregeneralizedmanipulation

knowledgetogivetheminformationabouttheuseofobjectsandtools,eveniftheyhavenotencounteredthembefore.

• 15years:Robotspossesshandswithnearlyhumanlevelsofmechanicaldexterity.Handsare

coveredwithhigh‐resolutiontactileskin.Robotsareabletoperformrobust,sensor‐based,prehensileandnon‐prehensilemanipulationofobjects.Theypossessrudimentarycapabilitiesofmanipulatingflexibleobjects.

Real­World3DPlanningandNavigationAutonomousservicerobotsaccomplishtasksbymovingabouttheirenvironmentandbyinteractingwith

theirenvironment.Thesemotionsandinteractionsneedtoachieveagiventaskbychangingtherobot’sposeandbymovingobjectsintheenvironment.Theaccomplishmentofataskmayrequirecomplexsequencesofmotionsandinteractions;therobotmayhavetomovefromoneroomtoanotheroritmay

havetoopendoors,clearobstaclesoutofitspath,removeobstructions,orusetools.Toachievethislevelofcompetency,substantialadvancesattheintersectionofmotionplanning,taskplanning,andcontrolhavetobemade.Historically,theseareashaveprogressedinisolation.Theproblemsposedby

servicerobotics,however,canonlybeaddressedthroughatightintegrationofthesetechniques.

Considerthetaskofpickingupacuptowhichaccessisobstructedbyabox.Toreasonaboutpushingtheboxtothesidetopickupthecup,therobothastoreasonaboutitsowncapabilities,thegeometryofthescene,constraintsimposedbyactuationandjointlimits,thecontactdynamicsandfrictionthat

arisewhenpushingthebox,etc.

Toreasonabouttheworldinsuchawaythattheappropriatesequenceofactionsandmotionscanbedetermined,therobothastobeawareofitsenvironment.Notalloftherequiredinformationcanbeprovidedtotherobotbeforehand,asservicerobotsoperateinunstructuredanddynamic

environments.Therobotthereforehastopossesscapabilitiestoperceiveandmapitsenvironment.“Semanticmapping”providestherobotwithinformationabouttheenvironmentthatisrequiredto

18

achieveatask.Objectdetectionandrecognitionandrelatedperceptualskillsprovideinformationforsemanticmappingandforobjectmanipulation.

In5,10,and15yearsthefollowinggoalsarepossiblewithsustainedresearchanddevelopment:

• 5years:Robotsinresearchlaboratoriescannavigatesafelyandrobustlyinunstructured2D

environmentsandperformsimplepickandplacetasks.Relevantobjectsareeitherfromaverylimitedsetorpossessspecificproperties.Robotslearnsemanticmapsabouttheirenvironmentthroughexplorationandinteractionbutalsothroughinstructionfromhumans.Theyareableto

reasonabouttasksofmoderatecomplexity,suchasremovingobstructions,openingcabinets,etc.toobtainaccesstootherobjects.

• 10years:Givenanapproximateandpossiblyincompletemodelofthestaticpartofthe

environment(possiblygivenaprioriorobtainedfromdatabasesvietheInternet,etc.),servicerobotsareabletoreliablyplanandexecuteatask‐directedmotioninserviceofamobilityormanipulationtask.Therobotbuildsadeepunderstandingoftheenvironmentfromperception,

interaction,andinstruction.Therobotmodifiesitsenvironmenttoincreasethechancesofachievingitstask(removeobstructions,clearobstacles,turnonlights),anditcandetectandrecoverfromsomefailures.

• 15years:Servicerobotscanperformhigh‐speed,collision‐free,mobilemanipulationincompletelynovel,unstructured,dynamicenvironments.Theyperceivetheirenvironment,translatetheirperceptionsintoappropriate,possiblytask‐specificlocalandglobal/short‐and

long‐termenvironmentalrepresentations(semanticmaps)andusethemtocontinuouslyplanfortheachievementofglobaltaskobjectives.Theyrespondtodynamicchangesintheenvironmentinawaythatisconsistentwiththeglobalobjective.Theyareabletointerleave

exploratorybehaviorwhennecessarywithtask‐directedbehavior.Theyinteractwiththeirenvironmentandareabletomodifyitinintelligentwayssoastoensureandfacilitatetask

completion.Thisincludesreasoningaboutphysicalpropertiesofinteractionsbetweenobjectsandtheenvironments(sliding,pushing,throwing,etc.)andtheuseoftoolsandotherobjects.

CognitionInserviceroboticsthereisaneedtooperateinnon‐engineeredenvironments,toacquirenewskills

fromdemonstrationbyusers,andtointeractwithusersfortaskingandstatusreporting.Cognitivesystemsenableacquisitionofnewmodelsoftheenvironmentandtrainingofnewskillsthatcanbeusedforfutureactions.Cognitionisessentialforfluentinteractionwithusersanddeploymentindomains

wherethereislimitedopportunitiesforusertraining.Inadditionanaddeddegreeofintelligenceforcopingwithnon‐engineeredenvironmentisessentialtoensuresystemrobustness.

In5,10,and15yearsthefollowinggoalsarepossiblewithsustainedresearchanddevelopment:

• 5years:Demonstrationofarobotthatcanlearnskillsfromapersonthroughgestureand

speechinteraction.Inadditionacquisitionofmodelsofanon‐modeledin‐doorenvironment.

19

• 10years:Arobotthatinteractswithuserstoacquiresequencesofnewskillstoperformcomplexassemblyoractions.Therobothasfacilitiesforrecoveryfromsimpleerrors

encountered.• 15years:Acompanionrobotthatcanassistinavarietyofservicetasksthroughadaptationof

skillstoassisttheuser.Theinteractionisbasedonrecognitionofhumanintentandre‐planning

toassisttheoperator.

RobustPerceptionServicerobotsoperateinrelativeunconstrainedenvironmentsandassuchthereisaneedtoproviderobustperceptualfunctionalitytocopewiththeenvironmentalvariation.Perceptioniscriticaltonavigationandinteractionwiththeenvironmentandforinteractionwithusersandobjectsintheproximityofthesystem.Todayperceptionistypicallyusedforrecognizingandinteractingwithsingle,knownobjects.Toenablescalabilitythereisaneedtohavefacilitiesforcategorizationofperceptsandgeneralizationacrossscenes,eventandactivities.Alreadytodaytherearemethodsformappingandinterpretationofscenesandactivitiesandthemainchallengeisinscalabilityandrobustnessforoperationinunconstrainedenvironments.In5,10,and15yearsthefollowinggoalsarepossiblewithsustainedresearchanddevelopment:

• 5years:Demonstrationofarobotsystemthatcancategorizespacesandautomaticallyassociatesemanticswithparticularplaces.Thesensingwillbeintegratedovertimeforrobust

operationinlargescalescalessuchasmallorabuildingstructure.Therobotwillbeabletorecognizehundredsofobjects.

• 10years:Demonstrationofarobotsystemthatcanperceiveeventandactivitiesinthe

environmenttoenableittooperateoverextendedperiodsoftime.• 15years:DemonstrationofarobotthatintegratesmultiplesensorymodalitiessuchasGPS,

visionandinertialtoacquiremodelsoftheenvironmentandusethemodelsfornavigationand

interactionwithnovelobjectsandevents.

Physical,intuitiveHRIandinterfacesDeploymentofservicerobotsbothinprofessionalanddomesticsettingsrequirestheuseofinterfacesthatmakesthesystemseasilyaccessiblefortheusers.Diffusionofroboticstoabroadercommunityrequires

interfacesthatcanbeusedwithnoorminimaltraining.Therearetwoaspectstointerfaces:physicalinteractionwithusersandpeopleinthevicinityandthecommandinterfacefortaskingandcontroloftherobot.Thephysicalinteractionincludesbodymotiontomove/nudgeobjectsandpeopleandnon‐contact

interactionsuchaschangeofmotionbehaviortocommunicateintentorstate.Theinterfaceaspectisessentialtotaskingandstatusreportingforoperatorstounderstandtheactionsoftherobot.

In5,10,and15yearsthefollowinggoalsarepossiblewithsustainedresearchanddevelopment:

• 5years:Demonstrationofarobotwheretaskinstructionisfacilitatedbymulti‐modaldialogfor

simpleactions/missionsandrobotsthatcancommunicateintentofactionsbythebodylanguage.• 10years:Demonstrationofarobotwhereprogrammingbydemonstrationcanbeusedfor

complextasklearningsuchasmealpreparationinaregularhome.

20

• 15years:Demonstrationofarobotthatcanbeprogrammedbyanoperatorforcomplexmissionatatimescalesimilartotheactualtaskduration.

SkillAcquisitionServicerobotsmustpossesstheabilitytosolvenoveltaskswithcontinuouslyimprovingperformance.Thisrequiresthatservicerobotsbeabletoacquirenovelskillsautonomously.Skillscanbeacquiredin

manyways:theycanbeobtainedfromskilllibrariesthatcontainskillsacquiredbyotherrobots;skillscanbelearnedfromscratchorbycomposingotherskillsthroughtrialanderror;skillscanalsobe

learnedthroughobservationofotherrobotsorhumans;furthermore,theycanbetaughttoarobotbyahumanorroboticinstructor.Butskillacquisitionalsorequirestherobottoidentifythosesituationsinwhichaskillcanbebroughttobearsuccessfully.Skillscanbeparameterized;learningandselecting

appropriateparametersforavarietyofsituationsisalsoincludedinthecapabilityofskillacquisition.Theabilitytotransferskillsfromonedomaintoanotherortotransferexperienceacquiredwithoneskilltoanotherskillcanbeexpectedtoprovidesubstantialadvancesinskillacquisition.Adequate

capabilitiesinskilllearningwillbeenabledbyadvancesinperception,representation,machinelearning,cognition,planning,control,activityrecognition,andotherrelatedareas.

In5,10,and15yearsthefollowinggoalsarepossiblewithsustainedresearchanddevelopment:

• 5years:Robotscanlearnavarietyofbasicskillsthroughobservation,trialanderror,andfrom

demonstration.Theseskillscanbeappliedsuccessfullyunderconditionsthatvaryslightlyfromtheonesunderwhichtheskillwaslearned.Robotscanautonomouslyperformminoradaptationsofacquiredskillstoadaptthemtoperceiveddifferencefromtheoriginalsetting.

• 10years:Asperceptualcapabilitiesimprove,robotscanacquiremorecomplexskillsanddifferentiatespecificsituationsinwhichskillsareappropriate.Multipleskillscanbecombinedintomorecomplexskillsautonomously.Therobotisabletoidentifyandreasonaboutthetype

ofsituationinwhichskillsmaybeappliedsuccessfully.Therobothasasufficientunderstandingofthefactorsthataffectthesuccesssoastodirecttheplanningprocessinsuchawaythatchancesofsuccessaremaximized.

• 15years:Therobotcontinuouslyacquiresnewskillsandimprovestheeffectivenessofknownskills.Itcanacquireskill‐independentknowledgethatpermitsthetransferofsingleskillsacrossdifferenttasksanddifferentsituationsandthetransferofskillstonoveltasks.Therobotisable

toidentifypatternsofgeneralizationfortheparameterizationofsingleskillsandacrossskills.

SaferobotsTodaysafetyforrobotsisachievedthroughaclearseparationoftheworkspacesforhumansandrobotsor

throughoperationatspeedsthatdonotrepresentarisktohumansintheproximityofthesystem.Astheoperationofhumansandrobotsbecomemoreandmoreintertwinedtherewillbeaneedtoexplicitly

consideroperationathigherspeedswhileoperatingindirectproximitytopeople.Thereisaneedtoconsiderstandardsforsafetytoenablecertification.Whiletechnologically,safetyinvolvesseveralaspectsincludingtheneedfor:advancedperceptioncapabilitiestodetectobjectsandpersonsandpredict

possiblesafetyhazards,controlsystemsthatreacttopossibledangeroussituations,andinherentlysafeactuationmechanismstoensurethatcontactwithapersonorobjectscauseslittleornodamage.

21

In5,10,and15yearsthefollowinggoalsarepossiblewithsustainedresearchanddevelopment:

• 5years:Asafetystandardforserviceroboticshasbeendefinedandacceptedworldwide,which

specifiesallowimpactsandenergytransfers.Basicmanipulationsystemshavefirstversionsofsafetystandardimplemented.

• 10years:Aninherentlysaferobotforoperationinproximityofhumansisdemonstratedfor

industrialapplicationscenarios.• 15years:Arobotsystemthatdoesmobilemanipulationincooperationwithhumansis

demonstratedandthesafetyisdemonstratedbothforhardwareandsoftwarecomponents.

4.BasicResearchandTechnologies

ArchitectureandRepresentationsOverthelast20yearsanumberofestablishedmodelsforsystemorganizationhaveemerged.Characteristically,however,noagreementoroverallframeworkforsystemorganizationhas

materialized.Forautonomousnavigation,mobility,andmanipulationtherearesomeestablishedmethodssuchas4D/RCSandHybridDeliberativeArchitectures,butonceinteractioncomponentsareaddedsuchasHuman‐RobotInteraction(HRI)thereislittleagreementonacommonmodel.Overthe

lastfewyearstheareaofcognitivesystemshasattemptedtostudythisproblem,butsofarwithoutaunifiedmodel.Forwideradoptionofrobotsystemsitwillbeessentialtoestablisharchitecturalframeworksthatfacilitatesystemsintegration,componentmodeling,andformaldesign.Appropriate

architecturalframeworksmayinitiallyorinherentlydependonthetask,theapplicationdomain,therobot,oravarietyofotherfactors.Nevertheless,adeeperunderstandingoftheconceptsunderlyingcognitioncanbeexpectedfromanincrementalunificationofmultipleframeworksintomoreless

problem‐orrobot‐specificarchitectures.Anyoftheaforementionedarchitecturalframeworkswillbeintricatelylinkedtoasetofappropriaterepresentationsthatcaptureaspectsoftheenvironmentand

theobjectscontainedinit,therobot’scapabilities,domaininformation,aswellasadescriptionoftherobot’stask

ControlandPlanningAsservicerobotsaddressreal‐worldproblemsindynamic,unstructured,andopenenvironments,novelchallengesariseintheareasofrobotcontrolalgorithmsandmotionplanning.Thesechallengesstem

fromanincreasedneedforautonomyandflexibilityinrobotmotionandtaskexecution.Adequatealgorithmsforcontrolandmotionplanningwillhavetocapturehigh‐levelmotionstrategiesthatadapttosensorfeedback.Researchchallengesincludetheconsiderationofsensingmodalitiesanduncertainty

inplanningandcontrolalgorithms;thedevelopmentofrepresentationsandmotionstrategiescapableofincorporatingfeedbacksignals;motionsubjecttoconstraints,arisingfromkinematics,dynamics,andnonholonomicsystems;addressingthecharacteristicsofdynamicenvironments;developingcontrol

andplanningalgorithmsforhybridsystems;andunderstandingthecomplexityofthesealgorithmicproblemsincontrolandmotionplanning.

22

PerceptionOverthelastfewdecadestremendousprogresshasbeenachievedinperceptionandsensoryprocessingasisseenforexampleinwebbasedsearchessuchasGoogleimagesandfacerecognitioninsecurity

applications.Mappingandlocalizationinnaturalenvironmentsisalsopossibleforengineeredenvironments.OverthelastdecadeinparticularuseoflaserscannersandGPShaschangedhownavigationsystemsaredesignedandenabledanewgenerationofsolutions.Nonetheless,localization

andplanninginGPS‐deniedenvironmentswhicharequitecommonremainsaveryimportantresearcharea.Inadditiontherehasbeentremendousprogressonimagerecognitionwithscalingtolargedatabases.Inthefuturealargenumberofrobotswillrelyonsensoryfeedbackfortheiroperationand

theapplicationdomainwillgobeyondpriormodeledsettings.Thereisthereforeaneedforrelianceonmultiplesensorsandfusionofsensoryinformationtoproviderobustness.Itisexpectedthattheuseofimage‐basedinformationinparticularwillplayamajorrole.Visionwillplayacrucialroleinnew

mappingmethods,infacilitatingthegraspingofnovelobjects,inthecategorizationofobjectsandplacesbeyondinstancebasedrecognition,andinthedesignofflexibleuserinterfaces.

Robust,High­fidelitySensorsAdvancesinmicroelectronicsandpackaginghaveresultedinarevolutioninsensorysystemsoverthe

lastdecade.Imagesensorshavemovedbeyondbroadcastqualitytoprovidemega‐pixelimages.MEMStechnologyhasenabledanewgenerationofinertialsensorpackagesandRFIDhasenabledmoreefficienttrackingofpackagesandpeople.Sensorshaveenabledsolidprogressindomainswithgood

signalquality.Asthedomainsofoperationarewidenedtherewillbetheneedfornewtypesofsensorsthatallowrobustoperation.Thisrequiresbothnewmethodsinrobustcontrol,butmoreimportantlysensorsthatproviderobustdatainthepresenceofsignificantdynamicvariationsandadomainwith

poordataresolution.NewmethodsinsiliconmanufacturingandMEMSopenopportunitiesforanewgenerationofsensorsthatwillbeakeyaspectoffutureprogressinrobotics.

NovelMechanismsandHigh­PerformanceActuatorsThereisanintricateinterplaybetweenprogressinmechanicaldevicesandactuationandthealgorithmiccomplexityrequiredtousetheminaccordancewiththeirfunction.Somealgorithmicproblemscanbe

solvedortheirsolutiongreatlyfacilitatedbyintelligentmechanicaldesign.Advancesinmechanismdesignandhigh‐performanceactuatorscouldthereforecriticallyenableground‐breakinginnovationsinotherbasicresearchareasaswellasenableseveralofthecapabilitieslistedintheroadmap.Importantresearch

areasincludethedesignanddevelopmentofmechanismswithcomplianceandvariablecompliance,highlydexteroushands,inherentlycomplianthands,energy‐efficient,safe,high‐performanceactuators,energy‐efficientdynamicwalkers,andmanymore.Ofparticularinterestare“intelligent”mechanical

designsthatcansubsume—throughtheirdesign—afunctionthatotherwisehadtobeaccomplishedthroughexplicitcontrol.Examplesincludeself‐stabilizingmechanismsorhandswithspecialprovisionstoachieveformclosurewithoutexplicitcontrol.

LearningandAdaptationManyofthebasicresearchareasdescribedinthissectioncanbenefitfromadvancesinandapplicationof

learningandadaptation.Servicerobotsoccupycomplexenvironmentandliveinhigh‐dimensionalstate

23

spaces.Knowledgeoftheenvironmentandoftherobot’sstateisinherentlyuncertain.Therobot’sactionsmostoftenarestochasticinnatureandtheirresultcanbestbedescribedbyadistribution.Many

ofthephenomenathatdeterminetheoutcomeofanactionaredifficultorevenimpossibletomodel.Techniquesfrommachinelearningprovideapromisingtooltoaddresstheseaforementioneddifficulties.Thesetechniquescanbeusefulforlearningmodelsofrobots,taskorenvironments;learningdeep

hierarchiesorlevelsofrepresentationsfromsensorandmotorrepresentationstotaskabstractions;learningofplansandcontrolpoliciesbyimitationandreinforcementlearning;integratinglearningwithcontrolarchitectures;methodsforprobabilisticinferencefrommulti‐modalsensoryinformation(e.g.,

proprioceptive,tactile,vision);structuredspatio‐temporalrepresentationsdesignedforrobotlearningsuchaslow‐dimensionalembeddingofmovements.

PhysicalHuman­RobotInteractionGraduallythesafetybarriersthathavebeencommoninindustrialroboticsareremovedandrobotswilltoalargerdegreeengagewithpeopleforcooperativetaskexecutionandforprogrammingby

demonstration.Aspartofthis,robotswillhavedirectphysicalcontactwiththeuser.Thisrequiresfirstofallcarefulconsiderationofsafetyaspects.Inadditionthereisaneedtoconsiderhowtheserobotscanbedesignedtoprovideinteractionpatternsthatareperceivedasnaturalbyusers.Thisspansallaspectsof

interactionfromphysicalmotionoftherobottodirectphysicalinteractionwithaperceptionofminimuminertiaandfluidcontrol.Inadditionthereisaneedheretoconsidertheinteractionbetweendesignandcontroltooptimizefunctionality.

SociallyInteractiveRobotsAsrobotsengagewithpeoplethereisaneedtoendowthesystemswithfacilitiesforcooperativeinteractionwithhumans.Thisinteractionisneededfortaskingofasystem,forteachingofnewskillsandtasksandforcooperativetaskexecution.Thecurrentmodelsforsocialinteractionincludegestures,

speech/sound,bodymotion/pose,andphysicalposition.Thereishereaneedtointegrateskillandtaskmodelswithinterpretationofhumanintenttoenableinterpretationofnewandexistingactivities.Inserviceroboticsthereisabroadneedforsocialinteractionfromencounterswithnoviceusersto

cooperativetaskingwithanexpertoperator.Thefullspanofcapabilitiesisrequiredtoprovideengagingandlong‐termadoptionofrobotics.

5.ContributorsThisreportdocumentstheresultofbrainstormingsessionthattookplace7‐8August2008inSanFrancisco,CA.ThereportispartoftheCCCstudyonRobotics.TheComputingCommunityConsortium

(CCC)isaprojectmanagedbytheComputingResearchAssociation(CRA)andissponsoredbytheNationalScienceFoundation(NSF).ThepresentreporthasbeenauthoredbytheworkshoporganizersanddoesnotreflecttheoptionofCRA,CCCorNSF.Theresponsibilityofthereportliesentirelywiththe

authors.

TheCCCworkshoponserviceroboticswasorganizedbyOliverBrock,UniversityofMassachusetts,BillThomasmeyer,TheTechnologyCollaborative,Inc,andHenrikIChristensen,GeorgiaInstituteofTechnology.Theworkshopwasattendedbythefollowingpeoplefromacademiaandindustry:

24

ChadJenkins BrownNicholasRoy MITAaronDollar MITStefanoCarpin UCMercedJanaKosecka GeorgeMasonAndrewNg StanfordAndreaThomaz GeorgiaTechJingXiao UNCCharlotteCharlesRich WPICandaceSidner WPIStewartTansley MicrosoftResearchJoshuaSmith IntelEricBerger WillowGarageMartinBuehler iRobotPaoloPirjanian EvolutionRoboticsBillTownsend BarrettTechnologyScottThayer RedZoneChrisUrmson CMU/GMCynthiaBreazeal MITMichaelO’Connor NovariantPaulJames AdeptEricWhinnem Boeing‐MfrTechCharlieKemp GeorgiaTechTrevorBlackwell AnybotsDanMiller AnybotsBrianCarlisle PreciseAutomationParagBatavia Foster‐MillerAndreasHoffman VecnaJamesKuffner CMUAlexFoessel DeereOliverBrock UMassAmherstBillThomasmeyer TechCollaborativeHenrikChristensen GeorgiaTechJakeHuckaby GeorgiaTech