National Research Council CanadaConseil national de recherches CanadaNational Research Council CanadaConseil national de recherches Canada
Institute for Information TechnologyVisual Information Technology GroupInstitute for Information TechnologyVisual Information Technology Group
CanadaCanada
Dr. Dmitry O. GorodnichyComputational Video Group
Institute for Information Technology National Research Council Canada
http://www.cv.iit.nrc.ca/~dmitry
Adding safety to autonomous robot Adding safety to autonomous robot manipulation using video-camerasmanipulation using video-cameras
((Monitoring of robot motions Monitoring of robot motions ))
ROSA final workshop. April 17, 2002ROSA final workshop. April 17, 2002
Adding safety to autonomous robot Adding safety to autonomous robot manipulation using video-camerasmanipulation using video-cameras
((Monitoring of robot motions Monitoring of robot motions ))
ROSA final workshop. April 17, 2002ROSA final workshop. April 17, 2002
2. ROSA: Adding safety using video. (Dmitry Gorodnichy)
Setup and Driving DesireSetup and Driving Desire
- There are many video-cameras on the station- 14 cameras on ISS- 4 cameras on SSRMS
- Can they be used to make operations on the station- easier?- safer?- in particular, the manipulation of SSRMS?
- autonomous - manual
3. ROSA: Adding safety using video. (Dmitry Gorodnichy)
Cameras and what they can seeCameras and what they can see
4. ROSA: Adding safety using video. (Dmitry Gorodnichy)
Application #1Application #1
1. Automatically detect the malfunctioning of arm joint encoders, by observing the arm with video-cameras.
“Compare what is seen with what should be seen”
- Requires- Knowledge and agreement of knowledge of both
SSRMS kinematics and camera configuration- Somebody to operate the cameras
5. ROSA: Adding safety using video. (Dmitry Gorodnichy)
Application #2Application #2
2. Help operator to manipulate the arm, by automatically detecting it in the view of video-cameras
“Is there SSRMS in the image? If yes, where it is?”
- Requires - Images only- SSRMS description
This can be a part of any SSRMS monitoring system
6. ROSA: Adding safety using video. (Dmitry Gorodnichy)
Two problems to be resolvedTwo problems to be resolved
Kinematics problem
- to find out where the arm should be seen
-> FastKin, Cosmos
Vision problem
- to find out where the arm is actually seen
-> R.A.C.E.Major challenge – Recognition Problem
Extract the image of SSRMS =
Recognize SSRMS in the image
7. ROSA: Adding safety using video. (Dmitry Gorodnichy)
Don’t forget: Vision is very ill-posed problemDon’t forget: Vision is very ill-posed problem
i.e. everything (approaches,complexity,results,success…) depends on the images used
8. ROSA: Adding safety using video. (Dmitry Gorodnichy)
R.A.C.E. softwareR.A.C.E. software
- For testing the applicability of different approaches
- Uses custom made
kinematics & image processing
- Integrates: FastKin (MDR)
Cosmos (NRC)
- Features:
joints setup selection
camera selection
9. ROSA: Adding safety using video. (Dmitry Gorodnichy)
R.A.C.E. allows …R.A.C.E. allows …
Allows one to detect parts of the arm which are possible to see by a camera and - to remove background- to show it by colouring- to obtain the usefulness of the image- to calculate the discrepancy level between what is seen
and what should be seen (according to the kinematics)
- With different image processing techniques
10. ROSA: Adding safety using video. (Dmitry Gorodnichy)
Approaches testedApproaches tested
- Intensity discontinuity based (+ gradient filters + Hough transform): assumes edges can be seen and many of them are parallel…………………………………...… good
- Texture based segmentation (+ region-growing + shape-from-shading): assumes colour histogram of SSRMS is known ….good
- Geometry (Cylinder) based: assumes SSRMS is a collection of cylinders of a known width ……………… very good
- Morphology: skeletonization, AND/NOR/NOT of the above approaches …………………………….……. best
- Contour-based (snakes): assumes there’s a complete contour….bad
12. ROSA: Adding safety using video. (Dmitry Gorodnichy)
Results and ConclusionsResults and Conclusions
• Tests conducted on images generated by Cosmos showed that: • while it is not always possible to detect completely the arm
position, R.A.C.E can significantly facilitate understanding of image context (such as “where SSRMS is”), which can be used
• 1) for automatic detection of the best views• 2) for operator-based manipulation of the arm.
• Most efficient recognition techniques are determined• Possibility of automatic detection of joint malfunctioning is
shown (provided that there is the agreement between the kinematics equations and the camera parameters)
• Ready for experiments with real images.