CSE 4392/5369
Robotic Vision:Robotic Vision: Sensing, Sensing, Localization and ControlLocalization and Control
University of Texas at Arlington
Dr. Gian Luca Mariottini, Ph.D.Department of Computer Science and Engineering
University of Texas at Arlington
WEB : http://ranger.uta.edu/~gianluca
Gian Luca Mariottini, Univ. Texas at Arlington
“ The most beautiful thing that we can experience is the Mystery. It is the true source of all art and all science.”
- Albert Einstein
Gian Luca Mariottini, Univ. Texas at Arlington
Robotic Vision: Sensing...Robotic Vision: Sensing...
No. of objects; Area; Center;Object Orientation;
Gian Luca Mariottini, Univ. Texas at Arlington
... Localization ...... Localization ...
No. of objects; Area; Center;Object Orientation;
Camera orientation w.r.t. Table;Distance to each object;
Gian Luca Mariottini, Univ. Texas at Arlington
and Controland Control
FANUC robot vision system
Gian Luca Mariottini, Univ. Texas at Arlington
ApplicationsApplicationsStair-Climbing & Descending for Rescue Operations
Gian Luca Mariottini, Univ. Texas at Arlington
ApplicationsApplicationsPersonal and Health-care Robotics
Omnivision sensing 3-D object locations Robot control
Courtesy from Charlie Kemp, director of the Healthcare Robotics Center at Georgia Tech;
Gian Luca Mariottini, Univ. Texas at Arlington
ApplicationsApplicationsMulti-robot Navigation
Team of Robots
Formation Control
Gian Luca Mariottini, Univ. Texas at Arlington
ApplicationsApplicationsLaparoscopic SurgeryCourtesy from Univ. Strasbourg
Gian Luca Mariottini, Univ. Texas at Arlington
ApplicationsApplications
Blue Team (U.Berkley, TA&M) Carnegie Mellon
Autonomous Navigation
Gian Luca Mariottini, Univ. Texas at Arlington
ApplicationsApplications
NAO (Aldebaran Robotics) http://www.youtube.com/watch?v=HS2QfpGAfRU
Sensing, Cognition and Humanoid-Robot Control
Gian Luca Mariottini, Univ. Texas at Arlington
ApplicationsApplicationsSpace Exploration
Gian Luca Mariottini, Univ. Texas at Arlington
ApplicationsApplications
MOVAID Project to aid disabled people (Scuola Superiore St.Anna, Pisa, ITALY)
PAM-AID (Trinity College, Ireland)
Health-care RObotics
Gian Luca Mariottini, Univ. Texas at Arlington
ApplicationsApplicationsVisual ServoingCourtesy from SIRS Lab, University of Siena
Gian Luca Mariottini, Univ. Texas at Arlington
ApplicationsApplicationsAutonomous Aerial-Vehicle Sensing and Navigation
http://www.youtube.com/watch?v=WoMrPKCpWXwhttp://www.youtube.com/watch?v=WoMrPKCpWXw
Gian Luca Mariottini, Univ. Texas at Arlington
ApplicationsApplications3-D Dense Reconstruction with Commodity Cameras
Video Kinect Fusion
ApplicationsApplications
• Determine user’s location• Identify surroundings (parks, bench, trees, etc.)
• Represent objects with unique sounds• Audio waypoints (e.g. metasounds) to destination
path3D (top)
...
•MAP localization (RANSAC/LM)•N-views Structure-from-Motion
Goal:
System for Wearable and Audio Navigation @ GaTech
Gian Luca Mariottini, Univ. Texas at Arlington
Gian Luca Mariottini, Univ. Texas at Arlington
Am I a robot ? (Yes/No)Am I a robot ? (Yes/No)
A B C
D E F
Gian Luca Mariottini, Univ. Texas at Arlington
Course TopicsCourse TopicsRobotics and Vision-sensing: Basics
Vision-based Localization
Probabilistic Robotics
- Introduction to Robotics;- The human eye and image formation;- Camera models;
– Pinhole, Stereo and Omnidirectional- Feature detection
– Edge, Corner, Color, SIFT- Feature tracking
- Reference frames and rigid body transformations- Full pinhole and stereo camera models- Camera Calibration- Camera Pose Estimation- Two-view geometry
- Position- and Image-based Visual Servoing- Epipole-based Visual Servoing
Vision-based Control
- Basics of probability reasoning in robotics- Bayesian filtering for localization
Gian Luca Mariottini, Univ. Texas at Arlington
Class Schedule and Office HoursClass Schedule and Office Hours
Class Schedule: Mon. & Wed., 5:30-6:50pm in COBA 141
Office Hours: Mon. & Wed., 1:30-3:30pm
I am generally available before or after class and by appointment, as well as at the office hours scheduled above.
Since this is a special-topic course in robotics, and since each presented topic contains many exciting sub-fields, the interested students who want to know more about a specific problem are encouraged to schedule an appointment with Dr. Mariottini for additional material
Gian Luca Mariottini, Univ. Texas at Arlington
Course PrerequisitesCourse Prerequisites
CSE 4392/5369 is self-contained and does not need special prerequisites.
MATLAB programming experience is required. If you are unfamiliar, please check this: http://www.math.ucsd.edu/~bdriver/21d-s99/matlab-primer.html.
During the course we will also make use of the MATLAB's Epipolar Geometry Toolbox, which can be efficiently used for creating vision-based robot simulations.
Gian Luca Mariottini, Univ. Texas at Arlington
Course TextCourse Text
There is no required textbook for this course.
Suggested references:
B. Siciliano, L. Sciavicco, L. Villani, G. Oriolo "Robotics: Modelling, Planning and Control", 3rd Edition, Springer, 2009
E. Trucco, A. Verri "Introductory Techniques for 3-D Computer Vision", Prentice Hall, 1998
R. Hartley, A. Zisserman, "Multiple View Geometry", 2nd Edition, Cambridge Univ. Press 2004
Y. Ma, S. Soatto, J. Kosecka, S. Sastry "An Invitation to 3-D Vision. From Images to Geometric Models", Springer 2000
Roberto Cipolla, “Robot and Computer Vision Course”, Cambridge University, UK
Gian Luca Mariottini, Univ. Texas at Arlington
Homeworks and Final Project (1)Homeworks and Final Project (1)Students will be graded based on homework assignments and a major course project.
Homeworks: theoretical questions, programming assignments (in MATLAB), etc. related to the
topics of the course. Dates will be announced in class.
Tentatively, course grades will based on the following:
Assignments % of final grade Percent Grade LetterHomeworks (total) 60 % 90 %-100 % ACourse Project 30 % 80 %-89.9 % B - Proj. Presentation 10 % 70 %-79.9 % C
60 %-69.9 % D<60 % F
Note 1: For students enrolled in CSE 5369 the homeworks will contain additional problems (not required for CSE 4392).
Note 2: All work turned in for grading must be the student's own work. (see also the course's webpage)
Note 3: MATLAB (with some toolboxes) is available also in NH 231
Gian Luca Mariottini, Univ. Texas at Arlington
Homeworks and Final Project (2)Homeworks and Final Project (2)Course project: designing and implementing a particular algorithm from a list provided in class (check the webpage!)
Additional projects proposed by the students and related to the course topics can be added to the list.
The course project can be done either individually or in a team (~max 2 students).
Students will report their findings in a 8-10 page research report, and an in-class oral presentation (with Power Point slides) during last day of class.
I recommend using OpenCV (a C/C++ library by Intel) for the Image Processing part (LK tracker, Color tracker, Optical Flow, etc.)
Epipolar Geometry ToolboxEpipolar Geometry Toolboxhttp://egt.dii.unisi.it
Structure-from-Motion Multiple View Geometryfor Omnidirectional Cameras
epipole
• Texture mapping• Structure from motion from lines• Detailed manual • Many examples• and many others...
Gian Luca Mariottini, Univ. Texas at Arlington
Intel's OpenCVIntel's OpenCV
Gian Luca Mariottini, Univ. Texas at Arlington
Gian Luca Mariottini, Univ. Texas at Arlington
List of Projects (1) List of Projects (1) Note that this list is temporary (e.g., you can propose new projects)
Project 1: Design a robotic cane that guides visually impaired in an indoor environment (e.g., indoor a corridor).
Project 2: Design an algorithm that allows a camera mounted on the robot to detect a person and follow him.
Project 3: Design a control algorithm to move a robot with your gestures
Gian Luca Mariottini, Univ. Texas at Arlington
List of Projects (2)List of Projects (2)Note that this list is temporary (e.g., you can propose new projects)
Project 4: Create an omnidirectional camera and design an obstacle avoidance algorithm that uses it.
Project 5: Stair detection, localization and approach
Project 6: Vision-Based Robot Localization Sytems
Gian Luca Mariottini, Univ. Texas at Arlington
List of Projects (3)List of Projects (3)Note that this list is temporary (e.g., you can propose new projects)
Project 7: Augmented reality system for surgery
Project 8: Visual Hull Reconstruction system
Project 9: Building recognition with your cellphone
Gian Luca Mariottini, Univ. Texas at Arlington
List of Projects (4)List of Projects (4)Note that this list is temporary (e.g., you can propose new projects)
Project 10: Multi-camera calibration
Project 11:Time-of-Flight camera system
I will show more applications in the next classes...you'll find many fascinating topics!
Project 12: Cane Localization from on/off-the-shelf camera
Gian Luca Mariottini, Univ. Texas at Arlington
What is a Robot ?What is a Robot ?
Gian Luca Mariottini, Univ. Texas at Arlington
A more formal definition of RobotA more formal definition of RobotHuman being always looked for substitutes that would mimic their behaviours.
Great ambition: give life to human's artifacts (e.g.,Talus, the bronze giant forged by Hephaestus, protected the island of Crete)
“Robot” (from the czech “robota”=slave) was coined in 1920 by Karel Capek in the play Rossum's Universal Robots.
RIA Definition: A robot is a reprogrammable, multifunctional manipulator designed to move material, parts, tools, or specialized devices through variable programmable motions for the performance of a variety of tasks. [Robot Institute of America]
A more recent definition, defines robotics as the science studying the intelligent connection between perception and action
= PERCEPTION + COGNITION + CONTROL
Gian Luca Mariottini, Univ. Texas at Arlington
Intelligent...action...perceptionIntelligent...action...perception
Actuation system: – Provides the capability to exert an action (locomotion or manipulation).
Sensory system: – “Proprioceptive sensors” = acquire data on the status of the robot.
(e.g., position, joint angles, etc.)– “Exteroceptive sensors” = acquire data on the external status of environment.
Control system: – It commands the execution of the action with respect to the goals.
Components of a robotic system:Mechanical system:
– Locomotion apparatus (wheels, tracks, legs,...)– Manipulation apparatus (mechanical arm, end-effector, artificial hand).
Gian Luca Mariottini, Univ. Texas at Arlington
Am I a robot ? (Solutions)Am I a robot ? (Solutions)
A=Yes B=No C = No --> Yes
D=No E=Yes F=No