reactive reaching and grasping on a humanoid: towards closing the action-perception loop on the icub...
DESCRIPTION
My presentation at the ICINCO 2014 (the 11th International Conference on Informatics in Control, Automation and Robotics) Abstract: We propose a system incorporating a tight integration between computer vision and robot control modules on a complex, high-DOF humanoid robot. Its functionality is showcased by having our iCub humanoid robot pick-up objects from a table in front of it. An important feature is that the system can avoid obstacles – other objects detected in the visual stream – while reaching for the intended target object. Our integration also allows for non-static environments, i.e. the reaching is adapted on-the-fly from the visual feedback received, e.g. when an obstacle is moved into the trajectory. Furthermore we show that this system can be used both in autonomous and tele-operation scenarios.TRANSCRIPT
![Page 1: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/1.jpg)
Jürgen ‘Juxi’ Leitner
Reactive Reaching and Grasping on a Humanoid Robot
Dalle Molle Institute for AI (IDSIA)
#ICINCO 2014
![Page 2: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/2.jpg)
humanoidour iCub
![Page 6: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/6.jpg)
!
perceptionvisual
thanks to G. Metta and IIT for this picture
![Page 7: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/7.jpg)
!
objectsdetecting
Harding et al., 2013Leitner et al., ICDL 2012, ARS 2012, BICA 2012, CEC 2013
![Page 8: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/8.jpg)
cartesian genetic programming
+ min dilate avg INP INP INP
[Leitner et al, 2012a/b, Harding et al., 2013]
![Page 9: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/9.jpg)
learningapproach
![Page 10: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/10.jpg)
detection
! icImage GreenTeaBoxDetector::runFilter() { icImage node0 = InputImages[6]; icImage node1 = InputImages[1]; icImage node2 = node0.absdiff(node1); icImage node5 = node2.SmoothBilateral(11); icImage node12 = InputImages[0]; icImage node16 = node12.Sqrt(); icImage node33 = node16.erode(6); icImage node34 = node33.log(); icImage node36 = node34.min(node5); icImage node49 = node36.Normalize(); ! //cleanup ... icImage out = node49.threshold(230.7218f); return out; }
![Page 11: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/11.jpg)
detect
![Page 12: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/12.jpg)
detect
![Page 13: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/13.jpg)
detection
![Page 16: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/16.jpg)
handsdetecting
![Page 17: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/17.jpg)
approachsupervised learning
BUT
![Page 18: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/18.jpg)
segmentationfeature
saliencymap
collaboration FIAS
![Page 19: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/19.jpg)
presegmentation
![Page 20: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/20.jpg)
approachcombined
![Page 21: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/21.jpg)
!
transferringspatial perception
![Page 22: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/22.jpg)
setuplearning
![Page 23: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/23.jpg)
trainingset
9DOF
iCubbounding box
6 per eye Carte
sian
Coor
dinate
s
.
.
.
![Page 24: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/24.jpg)
spatial perception neural network
. . .
9DO
F iC
ubbo
undi
ng b
ox
6 pe
r eye
Cart
esian
Coor
dina
tes
!fu
lly c
onne
cted
!fu
lly c
onne
cted
. . .
![Page 26: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/26.jpg)
MoBeEframework Frank et al., ICINCO, 2012.
![Page 27: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/27.jpg)
MoBeEFrank et al., ICINCO, 2012.
![Page 28: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/28.jpg)
Frank et al., ICINCO, 2012.
![Page 29: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/29.jpg)
![Page 31: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/31.jpg)
generationmotionStollenga et al, 2013
Shak
ey 2
013
Win
ner
![Page 32: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/32.jpg)
MoBeEv2[Frank et al., 2011,2012, 2013]
![Page 33: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/33.jpg)
hand/armop-space forcing
CSWorld
CSHand
CSR/CSL
[Leitner et al, in prep]
-
![Page 35: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/35.jpg)
coordinationhand-eye
![Page 36: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/36.jpg)
model
![Page 38: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/38.jpg)
manipulation for improved perception
![Page 39: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/39.jpg)
manipulation actions
![Page 40: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/40.jpg)
extracting information
![Page 41: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/41.jpg)
improveddetection
![Page 42: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/42.jpg)
detection
! icImage* BlueCupFilter::runFilter() { icImage* node43 = InputImages[4]; icImage* node49 = node43->LocalAvg(15); ! icImage* out = node49->threshold(81.532f); return out; }
![Page 43: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/43.jpg)
detection
! icImage* BlueCupFilter::runFilter() { icImage* node0 = InputImages[4].Exp(); icImage* node5 = InputImages[0]; icImage* node16 = node0->Gabor(-8,14,1,13); icImage* node17 = InputImages[4]->LocalAvg(6); icImage* node18 = node16->Laplace(5); icImage* node19 = node5->Sobel(13,9); icImage* node24 = node17->Erode(5); icImage* node28 = node19->Min(node18); icImage* node29 = node28->Min(node24); icImage* node41 = node29->LocalAvg(7); icImage* node49 = node41->LocalMax(7); ! icImage* out = node49->threshold(68.032f); return out; }
resulting
![Page 44: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/44.jpg)
operationtele
![Page 45: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/45.jpg)
teleoperation
![Page 46: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/46.jpg)
!
workfuture
improved eye-hand coordination
different object representations
online, continuous learning
vSLAM aerial robotics & for world model
![Page 47: Reactive Reaching and Grasping on a Humanoid: Towards Closing the Action-Perception Loop on the iCub #Icinco 2014](https://reader033.vdocuments.us/reader033/viewer/2022052410/555da27ed8b42a257b8b56db/html5/thumbnails/47.jpg)
for listeningthanks
[email protected] http://Juxi.net/projects
http://dilbert.com/strips/comic/2013-10-24/