solving perception uncertainty problems in...
TRANSCRIPT
![Page 1: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/1.jpg)
Prof. Alberto Sanfeliu
Director
Institut de Robòtica i Informàtica Industrial
(UPC-CSIC)
http://www.iri.upc.edu
Barcelona, 15 February 2013
Solving Perception Uncertainty
Problems in Robotics
![Page 2: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/2.jpg)
Overview
Problem statement
Solving perception uncertainty using probabilistic models:
Robot localization and navigation
Grasping deformable objects
Taken care of plants
Assembly with aerial robotics
Solving perception uncertainty using probabilistic and un/supervised learning models in the loop:
Online Human-Assisted Learning using Random Ferns
Human Motion Prediction to Accompany People
![Page 3: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/3.jpg)
Problem Statement
Robot environment are inherently unpredictable. The uncertainty is particularly high for robot operating in the proximity of people
Sensors are limited in what they can perceive and they are subject to noise.
Robot actuation involves actuators (motors,…) and motion elements (wheels, legs, …) which produces sometimes unpredictable movements (for example due to dead reckoning)
Robot tasks with people are inherently unpredictable, due to the aforementioned issues and the people task behavior
![Page 4: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/4.jpg)
Some Robotic Challenges
Robot localization and navigation (FP6 URUS project)
The perception uncertainty is due to sensors and data association
![Page 5: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/5.jpg)
Some Robotic Challenges
Manipulation and grasping of objects (FP7 GARNICS and IntellAct projects)
Taken care of plants
(FP7 GARNICS project)
Manipulating deformable objects
(FP7 IntellAct project)
The perception uncertainty is due to sensor information and robot manipulation
![Page 6: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/6.jpg)
Some Robotic Challenges
Aerial assembly by robots (FP7 ARCAS project)
Assembly structures
Transporting structures
The perception uncertainty is due to sensor, robot actuation and robot task
![Page 7: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/7.jpg)
Some Robotic Challenges
Human Robot Interaction (FP6 URUS and RobTaskCOOP projects)
The uncertainty is due to sensor, robot actuation, person behavior
![Page 8: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/8.jpg)
General Multimodal Scheme
Robotic
Multimodal
Interactive
System
M On-line
training
Off-line
training
x
x
x h
h
h’
h’
Input
stimulus
x1,h1
x2,h2
….
feedback
Output or
hypothesis
Model
Environment, task, …
h’’
h’’
![Page 9: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/9.jpg)
Solving Perception Uncertainty using Probabilistic Models
![Page 10: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/10.jpg)
General Multimodal Scheme
Robotic
Multimodal
Interactive
System
M On-line
training
Off-line
training
x
x
h
Input
stimulus
x1,h1
x2,h2
….
Output or
hypothesis
Model
Environment, task, …
h’’
h’’
![Page 11: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/11.jpg)
Solving Perception Uncertainty in Robot Localization and Navigation
Tibi and Dabo
![Page 12: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/12.jpg)
Tibi & Dabo Robots
Front Vertical Hokuyo Laser Scanner
Back Horizontal Leuze Laser Scanner
Front Horizontal Leuze Laser Scanner
Wheel encoders (2D odometry)
Bumblebee Stereo Camera
Touch Screen
HRI sensors Navigation Sensors
[Sanfeliu et. al., 2010] [Trulls et al., 2011]
[Corominas, Mirats, Sanfeliu, 2008]
![Page 13: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/13.jpg)
acquisition Particle Filter map-based Localization
WayPoint Path
Execution
RRT Local Planning
Path Planning
Traversability Inference
DWA Motion Control
horizontalBack LeuzeRS4
horizontalFront LeuzeRS4
Vertical Hokuyo URGX
Wheel encoders
acquisition
acquisition
acquisition
driver Wheel Motors
Segway RMP200
Perc
eptio
n &
Estim
atio
n
Pla
nnin
g
& C
ontro
l
UR
US
in
terfa
ce
Reactive Loop, 10Hz (local coord. frame)
Deliberative Loop, 3Hz (map coord. frame)
Autonomous Navigation Framework
![Page 14: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/14.jpg)
Obstacle Avoidance Diagram
Front horizontal Laser
Goal position in
local frame
Odometry data
Platform
commands
Front vertical Laser
Inputs: Outputs:
FREE Goal position in
local frame
![Page 15: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/15.jpg)
Obstacle Avoidance: Traversability Inference
Two situations where Traversability
Inference is required (ramp zones)
Extraction of vertical regression
line from vertical laser data to
detect ramps
Vert. laser Hori. laser
![Page 16: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/16.jpg)
Videos
Videos_pruebas\Tibi_Navegando_BRL_2010_WMV V9_002.wmv
![Page 17: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/17.jpg)
Solving Perception Uncertainty in Deformable Object Grasping
![Page 18: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/18.jpg)
Objective
Objective
Using depth and appearance
features for informed robot
grasping of highly wrinkled
clothes
[Ramisa, Alenya, Moreno, Torras, 2012]
[Monso, Alenya, Torras, 2012]
![Page 19: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/19.jpg)
Method
Method
In order to handle the large variability a deformed cloth may have, we build
a Bag of Features based detector that combines appearance and 3D
geometry features. An image is scanned using a sliding window with a
linear classifier, and the candidate windows are refined using a non-linear
SVM and a “grasp goodness” criterion to select the best grasping point. For
this work we have used polo shirt collars as test cloth part.
![Page 20: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/20.jpg)
Video
![Page 21: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/21.jpg)
Solving Perception Uncertainty in Taken Care of Plants
![Page 22: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/22.jpg)
Objective
Objective
Using ToF and color
cameras to segment plant
images into their composite
surface patches by
combining hierarchical
color segmentation with
quadratic surface fitting
using ToF depth data.
[Alenya, Dellen, Foix, Torras, 2012]
[Alenya, Dellen, Torras, 2011]
![Page 23: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/23.jpg)
WARN Robot with a ToF camera
Color image
Point cloud
European project GARNICS
(Gardening with a Cognitive System)
![Page 24: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/24.jpg)
The process consists of three stages.
First, color and depth images are acquired and combined to obtain a colored point cloud.
Second, the different leaves are segmented from the point cloud, and a plane or a quadratic surface is fitted to each of them. The surface model provides the position and orientation of each leaf. This first segmentation may contain some errors, e.g., several superimposed leaves may fall in the same region, and regions including few points may lead to a relatively large fitting error.
Third, using the position and orientation of the best leaf candidate, the robot moves the camera system closer to it to obtain a more detailed view, which is used to obtain a better model and eventually separate different leaves.
![Page 25: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/25.jpg)
Experimental results for different views of plants. A Color image first view (upper left panel). Sparse depth with segment boundaries (middle upper panel). Fitted depth (right upper panel). Color image close view (lower left panel). Sparse depth with segment boundaries of close view (middle lower panel). Fitted depth of selected segment (right lower panel). The selected segment is marked in red in the color images. A schematic showing the robot base position (0), the initial camera position (1), the leaf position (2) and the computed target position of the camera to capture the second viewpoint (3) are shown in the left panel. Distances are given in meters.
![Page 26: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/26.jpg)
Video
![Page 27: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/27.jpg)
Video
![Page 28: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/28.jpg)
Solving Perception Uncertainty in Aerial Robot Applications
![Page 29: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/29.jpg)
Aerial Robotics Cooperative Assembly System
(ARCAS)
ARCAS Objectives:
Development and experimental
validation of the first
cooperative free-flying robot
system for assembly and
structure construction
![Page 30: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/30.jpg)
Application Scenarios
Flying + Manipulation + Perception + Multi-robot Cooperation
![Page 31: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/31.jpg)
Video
![Page 32: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/32.jpg)
Objetives • The 3D object tracking component involves the identification
and tracking of objects using on-board cameras in aerial robots.
• FeatureDetection: This component localizes and identifies relevant features in the input image for object detection and tracking.
• 3DPoseEstimation-Tracking: Using the information concerning to image features, provided by the FeatureDetection component, this component estimates the pose of assembly objects and tracks their image features during the assembly task and transportation.
• 3DVisualServo: This component is used to perform visual servoing in flying robots using the estimated object pose and the tracked features given by the 3DPoseEstimation-Tracking component.
3D Object Tracking
![Page 33: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/33.jpg)
• Environment conditions:
• Wind
• Lighting changes (due to clouds,
the hour of the day, etc.)
• Shadows (cast shadows, etc.)
• Object/scene appearance:
• 3D rotations and translations
• Scale variations
• Perspective projections
• Cluttered background
• Surface reflections and color
variations
• Aerial robot platform:
• Camera pose
• Camera lens
• Vibrations
• Real time processing
• Cooperative delay of the
information
• Task:
• Partial occlusions
• Relative pose between components
to be assembly
Factors that Provoke Uncertainties
![Page 34: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/34.jpg)
Onboard Camera
Bar Detection During Assembly Operations
![Page 35: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/35.jpg)
Visual marks: Outdoor challenges
Vibrations, noise outdoor and cluttered background
![Page 36: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/36.jpg)
Video: Visual Marks in Indoor
![Page 37: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/37.jpg)
Focal length unknown
Given: 2D/3D correspondences
We want: Compute camera pose + focal length
World Coordinate
System
Rotation,
Translation,
Focal length ?
The (uncalibrated) PnP Problem
[Peñate, Andrade, Moreno, 2013]
![Page 38: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/38.jpg)
Unknowns:
Camera coord. of the 4 control points and
the focal length
World Coord. System
c 4
world
c 3
world
c 2
world
c 1
world
c 3
camera
c 4
camera
c 2
camera
c 1
camera
p i world =
j=1
4
a ij c j world
p i camera =
j=1
4
a ij c j camera
Number of unknowns = 12
3D points expressed as a weighted sum of 4 control points
Known
Introducing Control Points
![Page 39: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/39.jpg)
Independent detection in every frame.
3D Pose Estimation: the UPnP
![Page 40: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/40.jpg)
3D Pose Estimation: the UPnP
![Page 41: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/41.jpg)
Solving Perception Uncertainty using Probabilistic Models and
un/supervised learning models in the loop
![Page 42: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/42.jpg)
General Multimodal Scheme
Robotic
Multimodal
Interactive
System
M On-line
training
Off-line
training
x
x
x h
h
h’
h’
Input
stimulus
x1,h1
x2,h2
….
feedback
Output or
hypothesis
Model
Environment, task, …
h’’
h’’
![Page 43: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/43.jpg)
Solving Perception Uncertainty in Online Human-Assisted Learning
using Random Ferns
![Page 44: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/44.jpg)
Motivation
Robot TIBI learns and improves its visual perception
capabilities by means of interactions with humans
Robot TIBI Robot TIBI
[Villamizar, Andrade, Sanfeliu, Moreno, 2012]
[Villamizar, Garrell, Sanfeliu, Moreno, 2012]
[Villamizar, Moreno, Andrade, Sanfeliu, 2010]
![Page 45: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/45.jpg)
Objective
Robot TIBI learns to recognize faces and
objects using human assistance
![Page 46: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/46.jpg)
Objective
Face Recognition
Faces
Robot TIBI learns to recognize faces and
objects using human assistance
![Page 47: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/47.jpg)
Objective
Face Recognition Object Recognition
Faces 3D Objects
Robot TIBI learns to recognize faces and
objects using human assistance
![Page 48: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/48.jpg)
Objective
Face Recognition Object Recognition
Faces 3D Objects
Robot TIBI learns to recognize faces and
objects using human assistance
The interaction takes a couple of
minutes (~ 5 min.)
![Page 49: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/49.jpg)
Approach
Online Human-Assisted Learning
Human-Robot Interaction
![Page 50: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/50.jpg)
Approach
Online Human-Assisted Learning
Human-Robot Interaction
Recognition Results
Robot Camera
Online Learning: The visual system is updated continuously
using its own detection hypotheses
hypothesis
![Page 51: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/51.jpg)
Approach
Online Human-Assisted Learning
Human-Robot Interaction
Difficult Cases
Human-Assisted Learning: The visual system
requires the human intervetion
![Page 52: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/52.jpg)
Approach
Online Human-Assisted Learning
Human-Robot Interaction
Camera Touch Screen wii mote Keyboard
Difficult Cases
![Page 53: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/53.jpg)
Approach
Online Human-Assisted Learning
Human-Robot Interaction
TIBI : Can you tell me if the
detection is correct? Difficult Cases
Camera Touch Screen wii mote Keyboard
![Page 54: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/54.jpg)
Approach
Online Human-Assisted Learning
Human-Robot Interaction
TIBI : Can you tell me if the
detection is correct? Difficult Cases
Camera Touch Screen wii mote Keyboard
![Page 55: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/55.jpg)
Approach
Online Human-Assisted Learning using Random Ferns
![Page 56: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/56.jpg)
Approach
Online Classifier
Online Human-Assisted Learning using Random Ferns
![Page 57: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/57.jpg)
Approach
Online Classifier
Online Classifier:
• Fast classifier: RFs
• Updated continuously
Human Assistance:
• Interactive object detection
• Reduce drifting
Human Assistance
Online Human-Assisted Learning using Random Ferns
![Page 58: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/58.jpg)
Approach
• Object hypotheses: detections given by the classifier
Object Hypothesis
![Page 59: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/59.jpg)
Approach
• Object candidate: highest-confidence hypothesis (detection)
Object Hypothesis
![Page 60: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/60.jpg)
Approach
• New samples: positive and negative samples
Object Hypothesis
![Page 61: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/61.jpg)
Approach
Online Human-Assisted Learning using Random Ferns
Online Classifier
![Page 62: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/62.jpg)
Approach
Drifting
Online Human-Assisted Learning using Random Ferns
Online Classifier
Self-learning
![Page 63: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/63.jpg)
Approach
Online Classifier
Human Assistance:
• Interactive object detection
• Reduce drifting
Human Assistance
Online Human-Assisted Learning using Random Ferns
![Page 64: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/64.jpg)
Training Step
Description Human
Assistance
Recognition
Scores
RFs: Offline Random Ferns
ORFs: Online Random Ferns
A-ORFs: Online Human-Assisted Random Ferns
Detections
![Page 65: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/65.jpg)
Training Step
http://www.youtube.com/watch?feature=player_embedded&v=hdc6PreOuMM
![Page 66: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/66.jpg)
Testing Step
![Page 67: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/67.jpg)
Testing Step
The classifiers are not updated!
![Page 68: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/68.jpg)
Testing Step
http://www.youtube.com/watch?feature=player_embedded&v=hdc6PreOuMM
![Page 69: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/69.jpg)
Video: Robot Tibi Interacting with People for
Face Learning
![Page 70: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/70.jpg)
Solving Perception Uncertainty in Human Motion Prediction to
Accompany People
![Page 71: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/71.jpg)
Robot Companion
Objective:
To accompany people in urban areas maintaining a specific distance and an angle.
Perception uncertainty problems:
Detection of the person to be accompany
Tracking the person
Tracking other people and predicting their motion
[Garrell and Sanfeliu, 2012]
![Page 72: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/72.jpg)
Robot Accompanying a Person
![Page 73: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/73.jpg)
Robot Accompanying a Person in Dense Urban Area
![Page 74: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/74.jpg)
Conclusions
Robots must deal with uncertainty in perception and robot actuation problems in real life tasks
There is not a unique way to solve uncertainty perception problems, but a multimodal scheme allows to solve a great diversity of problems
Human in the loop allows to improve perception and robot behavior results
![Page 75: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/75.jpg)
References
A. Peñate, J. Andrade-Cetto and F. Moreno-Noguer. Exhaustive linearization for robust camera pose and focal length estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, to appear.
M. Villamizar, J. Andrade-Cetto, A. Sanfeliu and F. Moreno-Noguer. Bootstrapping boosted random Ferns for discriminative and efficient object classification. Pattern Recognition, 45(9): 3141-3153, 2012.
A. Garrell Zulueta and A. Sanfeliu. Cooperative social robots to accompany groups of people. The International Journal of Robotics Research , 31(13): 1675-1701, 2012.
A. Ramisa, G. Alenyà, F. Moreno-Noguer and C. Torras. Using depth and appearance features for informed robot grasping of highly wrinkled clothes, 2012 IEEE International Conference on Robotics and Automation, 2012, St. Paul, Minnesota, USA, pp. 1703-1708, IEEE.
![Page 76: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/76.jpg)
References
M. Villamizar, A. Garrell Zulueta, A. Sanfeliu and F. Moreno-Noguer. Online human-assisted learning using random ferns, 21st International Conference on Pattern Recognition, 2012, Tsukuba, Japan, pp. 2821-2824, IEEE Computer Society.
P. Monsó, G. Alenyà and C. Torras. POMDP approach to robotized clothes separation, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, 2012, Vilamoura, Portugal, pp. 1324-1329, IEEE.
G. Alenyà, B. Dellen, S. Foix and C. Torras. Robotic leaf probing via segmentation of range data into surface patches, 2012 IROS Workshop on Agricultural Robotics: Enabling Safe, Efficient, Affordable Robots for Food Production, 2012, Vilamoura, Portugal, pp. 1-6.
E. Trulls Fortuny, A. Corominas Murtra, J. Perez, G. Ferrer, D. Vasquez, J. M. Mirats Tur and A. Sanfeliu. Autonomous navigation for mobile service robots in urban pedestrian environments. Journal of Field Robotics, 28(3): 329-354, 2011.
![Page 77: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/77.jpg)
References
G. Alenyà, B. Dellen and C. Torras. 3D modelling of leaves from color and ToF data for robotized plant measuring, 2011 IEEE International Conference on Robotics and Automation, 2011, Shanghai, pp. 3408-3414, IEEE.
M. Villamizar, F. Moreno-Noguer, J. Andrade-Cetto and A. Sanfeliu. Efficient rotation invariant object detection using boosted random Ferns, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2010, San Francisco, pp. 1038-104
A. Sanfeliu, J. Andrade-Cetto, M. Barbosa, R. Bowden, J. Capitán, A. Corominas Murtra, A. Gilbert, J. Illingworth, L. Merino, J. M. Mirats Tur, P. Moreno, A. Ollero, J. Sequeira and M.T. Spaan. Decentralized sensor fusion for ubiquitous networking robotics in urban areas. Sensors, 10(3): 2274-2314, 2010.
A. Corominas Murtra, J. M. Mirats Tur and A. Sanfeliu. Action evaluation for mobile robot global localization in cooperative environments. Robotics and Autonomous Systems, 56(10): 807-818, 2008.
![Page 78: Solving Perception Uncertainty Problems in Roboticsdigital.csic.es/bitstream/10261/99105/1/ICPRAM2013.pdf · Solving Perception Uncertainty Problems in Robotics . Overview Problem](https://reader035.vdocuments.us/reader035/viewer/2022081513/5e17416da2200642460f6ef5/html5/thumbnails/78.jpg)
Thank You