automatic calibration of a robot manipulator and multi 3d camera system
TRANSCRIPT
![Page 1: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/1.jpg)
Automatic Calibration of a Robot Manipulator and Multi 3D Camera System
Justinas Mišeikis, Kyrre Glette, Ole Jakob Elle and Jim Torresen
![Page 2: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/2.jpg)
Another Calibration Paper…
• Standard calibration methods: Zhang and Bouguet
• Calibrate multiple RGB-D cameras at once
• Robot does all the hard work automatically
• Cameras guide the robot movements
![Page 3: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/3.jpg)
Internal Camera Calibration
![Page 4: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/4.jpg)
RGB to IR Camera Calibration
Uncalibrated CalibratedErrorOffset
![Page 5: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/5.jpg)
External: Eye-to-Hand
![Page 6: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/6.jpg)
Motivation
Image from: http://upenncis581finaltiantianhaoxiang.blogspot.jp/2011/12/camera-calibration-with-help-of-matlab.html
![Page 7: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/7.jpg)
Setup
Kinect V2 Kinect V2
Kinect V1UR5
Robot
![Page 8: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/8.jpg)
Calibration Pattern
![Page 9: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/9.jpg)
Cam Drivers
Cam Drivers
Cam Drivers
Detect Checkerboard
Calibration Move Planner
Robot Trajectory Planner
Camera Internal Calibration
Eye-to-Hand Calibration
Robot Drivers
Store Calibration
Images
color and depth images
color and depth images
color and depth images
checker board pos
pos in image
coordinatespos in robot coordinates
target pos in robot coordinates
current robot pos
calibration images
calibration images
camera images
robot move completion
![Page 10: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/10.jpg)
Cam Drivers
Cam Drivers
Cam Drivers
Detect Checkerboard
Calibration Move Planner
Robot Trajectory Planner
Camera Internal Calibration
Eye-to-Hand Calibration
Robot Drivers
Store Calibration
Images
color and depth images
color and depth images
color and depth images
checker board pos
pos in image
coordinatespos in robot coordinates
target pos in robot coordinates
current robot pos
calibration images
calibration images
camera images
robot move completion
![Page 11: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/11.jpg)
Cam Drivers
Cam Drivers
Cam Drivers
Detect Checkerboard
Calibration Move Planner
Robot Trajectory Planner
Camera Internal Calibration
Eye-to-Hand Calibration
Robot Drivers
Store Calibration
Images
color and depth images
color and depth images
color and depth images
checker board pos
pos in image
coordinatespos in robot coordinates
target pos in robot coordinates
current robot pos
calibration images
calibration images
camera images
robot move completion
![Page 12: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/12.jpg)
Cam Drivers
Cam Drivers
Cam Drivers
Detect Checkerboard
Calibration Move Planner
Robot Trajectory Planner
Camera Internal Calibration
Eye-to-Hand Calibration
Robot Drivers
Store Calibration
Images
color and depth images
color and depth images
color and depth images
checker board pos
pos in image
coordinatespos in robot coordinates
target pos in robot coordinates
current robot pos
calibration images
calibration images
camera images
robot move completion
![Page 13: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/13.jpg)
Detect Checkerboard
![Page 14: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/14.jpg)
Cam Drivers
Cam Drivers
Cam Drivers
Detect Checkerboard
Calibration Move Planner
Robot Trajectory Planner
Camera Internal Calibration
Eye-to-Hand Calibration
Robot Drivers
Store Calibration
Images
color and depth images
color and depth images
color and depth images
checker board pos
pos in image
coordinatespos in robot coordinates
target pos in robot coordinates
current robot pos
calibration images
calibration images
camera images
robot move completion
Step 3a
Initialisation Movement
![Page 15: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/15.jpg)
![Page 16: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/16.jpg)
Cam Drivers
Cam Drivers
Cam Drivers
Detect Checkerboard
Calibration Move Planner
Robot Trajectory Planner
Camera Internal Calibration
Eye-to-Hand Calibration
Robot Drivers
Store Calibration
Images
color and depth images
color and depth images
color and depth images
checker board pos
pos in image
coordinatespos in robot coordinates
target pos in robot coordinates
current robot pos
calibration images
calibration images
camera images
robot move completion
![Page 17: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/17.jpg)
Camera Frame
First Stage Second Stage
Initial Checkerboard Position
![Page 18: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/18.jpg)
Camera Frame
First Stage Second Stage
Initial Checkerboard Position
First Stage
![Page 19: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/19.jpg)
Camera Frame
First Stage Second Stage
Initial Checkerboard Position
First Stage
1. Rotate 2. Pan 3. Tilt
![Page 20: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/20.jpg)
Camera Frame
First Stage Second Stage
Initial Checkerboard Position
First Stage
![Page 21: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/21.jpg)
Camera Frame
First Stage Second Stage
Initial Checkerboard Position
First Stage
![Page 22: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/22.jpg)
Camera Frame
First Stage Second Stage
Initial Checkerboard Position
First Stage
![Page 23: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/23.jpg)
Camera Frame
First Stage Second Stage
Initial Checkerboard Position
Second Stage
![Page 24: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/24.jpg)
Cam Drivers
Cam Drivers
Cam Drivers
Detect Checkerboard
Calibration Move Planner
Robot Trajectory Planner
Camera Internal Calibration
Eye-to-Hand Calibration
Robot Drivers
Store Calibration
Images
color and depth images
color and depth images
color and depth images
checker board pos
pos in image
coordinatespos in robot coordinates
target pos in robot coordinates
current robot pos
calibration images
calibration images
camera images
robot move completion
![Page 25: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/25.jpg)
Cam Drivers
Cam Drivers
Cam Drivers
Detect Checkerboard
Calibration Move Planner
Robot Trajectory Planner
Camera Internal Calibration
Eye-to-Hand Calibration
Robot Drivers
Store Calibration
Images
color and depth images
color and depth images
color and depth images
checker board pos
pos in image
coordinatespos in robot coordinates
target pos in robot coordinates
current robot pos
calibration images
calibration images
camera images
robot move completion
![Page 26: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/26.jpg)
![Page 27: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/27.jpg)
The process is repeated for each of the cameras to get the full system calibration
![Page 28: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/28.jpg)
![Page 29: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/29.jpg)
Results: Internal Calibration
![Page 30: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/30.jpg)
Results: Eye-to-Hand Calibration
![Page 31: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/31.jpg)
Conclusions and Future Work✓ Good for quick camera calibration and recalibration
✓ Useful for systems with changing camera setups
✓ Works with ROS supported cameras and robots
‣ Calibrate only for a part of the image area
‣ Calibrate multiple cameras at once covering areas of overlapping field of view
‣ Use ANNs to learn the best positions of calibration pattern
![Page 32: Automatic Calibration of a Robot Manipulator and Multi 3D Camera System](https://reader031.vdocuments.us/reader031/viewer/2022022415/5a6485ce7f8b9a36568b4c5b/html5/thumbnails/32.jpg)
Automatic Calibration of a Robot Manipulator and Multi 3D Camera SystemJustinas Mišeikis, Kyrre Glette, Ole Jakob Elle and Jim Torresen
Thank You!