National Research Council CanadaConseil national de recherches CanadaNational Research Council CanadaConseil national de recherches Canada

Institute for Information TechnologyVisual Information Technology GroupInstitute for Information TechnologyVisual Information Technology Group

CanadaCanada

Dr. Dmitry O. GorodnichyComputational Video Group

Institute for Information Technology National Research Council Canada

http://www.cv.iit.nrc.ca/~dmitry

Adding safety to autonomous robot Adding safety to autonomous robot manipulation using video-camerasmanipulation using video-cameras

((Monitoring of robot motions Monitoring of robot motions ))

ROSA final workshop. April 17, 2002ROSA final workshop. April 17, 2002

Adding safety to autonomous robot Adding safety to autonomous robot manipulation using video-camerasmanipulation using video-cameras

((Monitoring of robot motions Monitoring of robot motions ))

ROSA final workshop. April 17, 2002ROSA final workshop. April 17, 2002

2. ROSA: Adding safety using video. (Dmitry Gorodnichy)

Setup and Driving DesireSetup and Driving Desire

- There are many video-cameras on the station- 14 cameras on ISS- 4 cameras on SSRMS

- Can they be used to make operations on the station- easier?- safer?- in particular, the manipulation of SSRMS?

- autonomous - manual

3. ROSA: Adding safety using video. (Dmitry Gorodnichy)

Cameras and what they can seeCameras and what they can see

4. ROSA: Adding safety using video. (Dmitry Gorodnichy)

Application #1Application #1

1. Automatically detect the malfunctioning of arm joint encoders, by observing the arm with video-cameras.

“Compare what is seen with what should be seen”

- Requires- Knowledge and agreement of knowledge of both

SSRMS kinematics and camera configuration- Somebody to operate the cameras

5. ROSA: Adding safety using video. (Dmitry Gorodnichy)

Application #2Application #2

2. Help operator to manipulate the arm, by automatically detecting it in the view of video-cameras

“Is there SSRMS in the image? If yes, where it is?”

- Requires - Images only- SSRMS description

This can be a part of any SSRMS monitoring system

6. ROSA: Adding safety using video. (Dmitry Gorodnichy)

Two problems to be resolvedTwo problems to be resolved

Kinematics problem

- to find out where the arm should be seen

-> FastKin, Cosmos

Vision problem

- to find out where the arm is actually seen

-> R.A.C.E.Major challenge – Recognition Problem

Extract the image of SSRMS =

Recognize SSRMS in the image

7. ROSA: Adding safety using video. (Dmitry Gorodnichy)

Don’t forget: Vision is very ill-posed problemDon’t forget: Vision is very ill-posed problem

i.e. everything (approaches,complexity,results,success…) depends on the images used

8. ROSA: Adding safety using video. (Dmitry Gorodnichy)

R.A.C.E. softwareR.A.C.E. software

- For testing the applicability of different approaches

- Uses custom made

kinematics & image processing

- Integrates: FastKin (MDR)

Cosmos (NRC)

- Features:

joints setup selection

camera selection

9. ROSA: Adding safety using video. (Dmitry Gorodnichy)

R.A.C.E. allows …R.A.C.E. allows …

Allows one to detect parts of the arm which are possible to see by a camera and - to remove background- to show it by colouring- to obtain the usefulness of the image- to calculate the discrepancy level between what is seen

and what should be seen (according to the kinematics)

- With different image processing techniques

10. ROSA: Adding safety using video. (Dmitry Gorodnichy)

Approaches testedApproaches tested

- Intensity discontinuity based (+ gradient filters + Hough transform): assumes edges can be seen and many of them are parallel…………………………………...… good

- Texture based segmentation (+ region-growing + shape-from-shading): assumes colour histogram of SSRMS is known ….good

- Geometry (Cylinder) based: assumes SSRMS is a collection of cylinders of a known width ……………… very good

- Morphology: skeletonization, AND/NOR/NOT of the above approaches …………………………….……. best

- Contour-based (snakes): assumes there’s a complete contour….bad

11. ROSA: Adding safety using video. (Dmitry Gorodnichy)

12. ROSA: Adding safety using video. (Dmitry Gorodnichy)

Results and ConclusionsResults and Conclusions

• Tests conducted on images generated by Cosmos showed that: • while it is not always possible to detect completely the arm

position, R.A.C.E can significantly facilitate understanding of image context (such as “where SSRMS is”), which can be used

• 1) for automatic detection of the best views• 2) for operator-based manipulation of the arm.

• Most efficient recognition techniques are determined• Possibility of automatic detection of joint malfunctioning is

shown (provided that there is the agreement between the kinematics equations and the camera parameters)

• Ready for experiments with real images.