visual servoing - georgia institute of technologyhic/4632b-07/lecture... · a tutorial on visual...
TRANSCRIPT
![Page 1: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/1.jpg)
Visual Servoing
Charlie Kemp
4632B/8803Mobile Manipulation
Lecture 8
![Page 2: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/2.jpg)
From: http://www.hsi.gatech.edu/visitors/maps/
![Page 3: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/3.jpg)
4th floor4100QM Building167
First office on HSI side
From: http://www.hsi.gatech.edu/visitors/maps/
![Page 4: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/4.jpg)
Find the KUKA cup.
![Page 5: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/5.jpg)
Manipulate the KUKA cup.
![Page 6: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/6.jpg)
Overview
● Manipulation● Global approach● Local approach● Visual Servoing
![Page 7: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/7.jpg)
Manipulating a rigid body
● Grasping it● Transporting it● Placing it
![Page 8: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/8.jpg)
Rigid body transformations
Initial pose Final pose
![Page 9: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/9.jpg)
We have a generative model!
![Page 10: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/10.jpg)
Motion planning(search for an entire trajectory)
From: James Kuffner's home page (CMU) http://www.kuffner.org/james/humanoid/pics/mainWindowSnap.gif
![Page 11: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/11.jpg)
Motion planning
● Global solutions● Drawbacks
– Difficult to move from simulation to the real world
– Sensing rarely included– Usually assume known state– Usually assume well-modeled
transitions between states– May not meet real-time
constraints
From: Steve LaValle's home page at UIUC (http://msl.cs.uiuc.edu/~lavalle/)
![Page 12: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/12.jpg)
Can we find an efficient and robust local solution?
![Page 13: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/13.jpg)
Rigid body transformations
Initial pose Final pose
● Define an error function● Locally minimize this error function● Simple feedback control (PID)● Not sensor specific
– e.g., Rod Grupen's group at UMass Amherst
![Page 14: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/14.jpg)
From: wikipedia article on PID control http://en.wikipedia.org/wiki/Image:Pid-feedback-nct-int-correct.png
proportional-integral-derivative controller (PID controller)
![Page 15: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/15.jpg)
What poses?Relative to what?
● Grasping it● Transporting it● Releasing it
![Page 16: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/16.jpg)
Visual Servoing
From: Hutchinson et. al., A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.
![Page 17: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/17.jpg)
Visual servoing
S. Hutchinson, G. Hager, and P. Corke, A tutorial on visual servo control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.
![Page 18: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/18.jpg)
Examples
http://people.csail.mit.edu/cckemp/publications.shtml
![Page 19: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/19.jpg)
Examples
http://people.csail.mit.edu/cckemp/publications.shtml
![Page 20: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/20.jpg)
Major options● Camera placement
– End-effector mounted ● Eye-in-hand● EOL (endpoint open-loop)
– Fixed in the workspace ● Might be PTZ● ECL (endpoint closed-loop)
● Error function– 3D pose– Image features
● Control hierarchy
![Page 21: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/21.jpg)
Error function in 3D
From: Hutchinson et. al., A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.
![Page 22: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/22.jpg)
Generative Model (Basic Image Formation)
Pinhole camera model
Perspective projectionFrom: Hutchinson et. al., A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.
![Page 23: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/23.jpg)
Camera Calibration● http://www.vision.caltech.edu/bouguetj/calib_doc/● Estimates parameters
– focal length, principal point, skew coefficient, distortions (radial and tangential)
● Rectify the images
![Page 24: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/24.jpg)
Examplefixed camera, point to point
From: Hutchinson et. al., A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.
![Page 25: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/25.jpg)
Exampleeye-in-hand, EOL
From: Hutchinson et. al., A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.
![Page 26: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/26.jpg)
servo without reconstruction(error as a function of image features)
Error measurements imply 3D
Multi-cameraSingle camera
![Page 27: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/27.jpg)
feature based servoing
From: Hutchinson et. al., A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.
![Page 28: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/28.jpg)
Image Jacobian
From: Hutchinson et. al., A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.
![Page 29: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/29.jpg)
Example Image Jacobian
From: Hutchinson et. al., A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.
![Page 30: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/30.jpg)
Inverting the Image Jacobian
Overconstrained Underconstrained
From: Hutchinson et. al., A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.
![Page 31: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/31.jpg)
Resolved rate motion control
From: Hutchinson et. al., A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.
![Page 32: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/32.jpg)
servo without calibration● Estimate the jacobian online
From: Jägersand M. Nelson R. Visual Space Task Specification, Planning and Control In Proc. of IEEE Int. Symp. on Computer Vision 95, p 521-526, 1995.
![Page 33: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/33.jpg)
“A short description of higher level aspects of uncalibrated visual control. Many experiments solving complex manipulation tasks in unstructured environments.” - Martin Jägersand
Jägersand M. and Nelson R., Visual Space Task Specification, Planning and Control. In Proc. of IEEE Int. Symp. on Computer Vision 95, p 521-526, 1995.
Martin Jägersand University of Rochester -> Yale -> University of Alberta
From: http://www.cs.ualberta.ca/~jag/
![Page 34: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/34.jpg)
From: Jägersand M. Nelson R. Visual Space Task Specification, Planning and Control In Proc. of IEEE Int. Symp. on Computer Vision 95, p 521-526, 1995.
![Page 35: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/35.jpg)
From: Jägersand M. Nelson R. Visual Space Task Specification, Planning and Control In Proc. of IEEE Int. Symp. on Computer Vision 95, p 521-526, 1995.
Non-rigid Manipulation
![Page 36: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/36.jpg)
From: Jägersand M. Nelson R. Visual Space Task Specification, Planning and Control In Proc. of IEEE Int. Symp. on Computer Vision 95, p 521-526, 1995.
![Page 37: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/37.jpg)
Discussion questions?
● When is visual servoing a good idea?● When is visual servoing a bad idea?
![Page 38: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/38.jpg)
What will you do?
![Page 39: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/39.jpg)
Next week
● You're the presenters!
![Page 40: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/40.jpg)
Extra
![Page 41: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/41.jpg)
Velocity of a Rigid Object
Velocity screw
From: Hutchinson et. al., A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.
![Page 42: Visual Servoing - Georgia Institute of Technologyhic/4632B-07/lecture... · A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670,](https://reader030.vdocuments.us/reader030/viewer/2022040918/5e942ada5f05675cfb7466a7/html5/thumbnails/42.jpg)
Velocity of a Rigid Object (continued)
From: Hutchinson et. al., A Tutorial on Visual Servo Control, IEEE Transactions on Robotics and Automation, vol. 12, pp. 651–670, October 1996.