A new approach for terrain description in mobile robots for humanitarian demining missions

C. Salinas, M. Armada, P. Gonzalez de Santos

Department of Automatic Control

Industrial Automation Institute – CSIC Ctra. Campo Real Km. 0.2

28500 Madrid, Spain ABSTRACT The humanitarian demining missions require the use of robust systems such as efficient mobile robots and improved sensors. This application domain involves performing tasks in non structured scenarios and dynamically changing environments. This paper is focused on a new approach based on omnidirectional vision systems for terrain description. Computer vision systems are widely used in similar applications. However, conventional video cameras have limited fields of view which make them restricted for several applications in robotics. For example, mobile robots often require a full 360º view of their environment, in order to perform navigational tasks such localizing within the environment, identifying landmarks and determining free paths in which to move. The omnidirectional sensors allow the capture of much wider field of view; they can provide panoramic images of 360º around the robot. Certain techniques have been developed for acquiring panoramic images. This work outlines the fundamental principles of three of these techniques, and shows a direct application of a low-cost catadioptric omnidirectional vision sensor on-board a six-legged robot intended for antipersonnel landmine localization.

1. Introduction

Millions of landmines currently are still buried all over the world, causing threats to the economy and particularly to lives of the nations affected for these pyrotechnic instruments. Detection and removal of antipersonnel landmines has become a global issue. According to recent computes landmines kill/injury more than 2000 civilians per month. Demining has still been in progress by a manual method, since it was proposed several decades ago, this procedure is risky and slow [1]. Relying only on manual work, to dispose all these landmines it would be necessary a hundreds of years.

Solutions are being explored in different engineering fields. The best course for releasing human operators from this risky task is to apply fully automatic systems; nevertheless, this solution is still far from successful, due to the complexity of applying fully autonomous systems in such unstructured environments. However, there are some aspects of the job that robots can do quite well, like scanning the ground to detect and locate buried mines [2, 7].

This is where robots, carrying efficient sensor, can play an important role. The automation of an application such as the detection and removal of antipersonnel mines implies the use of autonomous or teleoperated mobile robots. These robots follow a predefined path, send the recorded data to their expert-system, when a mine is detected its position is marked/ saved with a probability of predefined level and possibly remove the detected mine.

Several sensors have been adopted over the last years; the work on rough terrain navigation has been broadly implemented with high-cost solutions. In general, vision systems are mounted

high in the frame to look down towards the ground. The field of view covers the area between the metal detector and the scanning sensors, the scanning of an area can be obtained by moving the robot body itself.

Stereovision systems based on image processing are used to provide 3-dimensional maps for the detection. The target trajectory is generated from the depth information previously acquired from several images of the minefield. A large volume of data is produced, that is inconvenient for the planning trajectory and the calibration of stereo systems must be carefully performed [3, 4]. These methods usually need an offline planning trajectory and are hardly limited by the image sensor.

Others configuration generally used are 3D-raster image consisting of a number of laser spots located on several concentric scan lines [6], combined with/or ultrasonic sensor, rangefinder sensors and visual servoing systems based on a pan-and-tilt color camera. The consistency of remote sensing is improved by fusing synergistic measurements of different types of detectors [5], which meant implementation of high-cost systems.

In these work, we present a new approach for terrain description based on low-cost omnidireccional vision system, on-board a six-legged robot intended for antipersonnel landmine localization, to improve efficiency of involved tasks in automated detection and removal operations.

The outline of this paper is as follows. Section 2 introduces the DYLEMA project and explains the main features of the mobile system Silo6. Section 3 details important issues of omnidirectional vision techniques. In section 4, catadioptric omnivision techniques and camera central projection model are explained. Section 5 report some results accomplish with the ongoing experimental prototype of catadioptric omnivision system implemented in DYLEMA project. Finally, section 6 presents the main conclusions and future research.

2. The DYLEMA project

The DYLEMA project is devoted to the configuration of a humanitarian de-mining system that consists of a sensor head, a scanning manipulator and a mobile platform based on a hexapod walking robot (see Figure 1) [8]. The sensor head consists on a commercial mine-detecting set customised with a ground-tracking set to adapt the head to ground irregularities. This ground-tracking set provides adequate information for keeping the manipulator’s end-effector at a given height above the ground. To achieve this objective, the manipulator has to track the surface whilst moving the sensor head. In addition, the manipulator has to avoid obstacles in the way of the sensor head, such as big stones, bushes, trees and so on. The sensor head information is also used by the system controller to steer the mobile robot during mine detection missions.

It is commonly agreed that an efficient detection system should merge different technologies. The DYLEMA project, however, focuses only on the development of robotic techniques. The development of mine-detecting sensors does not fall within the scope of the project. Therefore, the simplest mine sensor (a metal detector) is considered in this work, just to help detect and locate potential alarms. After a suspect object is detected, its location must be marked in the system database for further analysis and possible deactivation. Additionally, DYLEMA project includes research field in methods of complete coverage of unstructured environments for mobile robot navigation [9] and sensor integration and control for scanning activities [10].

Figure 1: DYLEMA configuration.

3. Omnidirectional vision Systems

During the last decades, researchers in several engineering fields as applied optics, computer vision, and robotics, has presented a remarkable work related to omnidirectional cameras and their applications [11]. The standard cameras typically have a constrained field of view (~30–60�) and are therefore adequate for observing small local areas. However, there are many applications that require or benefit from observing wider areas than is not possible with a TV camera. For example, mobile robots often require a full 360� view of their environment in order to perform navigational tasks such identifying landmarks, localizing within the environment, and determining free paths in which to move. For these reasons many techniques related to omnidirectional vision systems have been developed. For mobile robots, several specific tasks are essential for navigating in either structured or unstructured environments. The robot must be able to sense its environment and construct a local representation that is sufficient in detail and accuracy to allow the robot to find free paths in which to move. It also must be able to perform localization, i.e., determine its position and orientation within the environment and register this information with its local representation of its surroundings, combined in this case with detection of landmine task.

These vision systems consist of omnidirectional sensors, which came in many varieties; whatever it is, by definition the essential idea of omnidirectional sensor is afford a wide field of view. According to their structure, the sensors are classified in 3 groups: Dioptric cameras, which can acquire wide angles view of as much as hemispherical view, e.g. the commercial cameras named “fisheye”. Polidiotric cameras, these systems are able to provide ~360 degrees of FOV, the typical configuration is composed by multiples overlapping cameras. And the third group, Catadioptric cameras, that are able to acquire more than 180 degrees of field of view and normally are composed of perspective cameras and convex mirror (see Figure 2).

The main application of the system (i.e. autonomous mobile robots, surveillance, teleopereance) defines the he solution within dioptric, polidiotric or catadioptric systems, one or two fish-eye or synchronized cameras. The polidiotric sensors have high resolution per viewing angle, the

Figure 1 - DYLEMA Configuration

Mobile platform

Manipulator

GPS antenna

Magnetic compass

Sensor head

cameras can be cheaper if they are “homemade”, but they must be calibrated and synchronized, and commercial cameras usually are expensive. The principal disadvantage is the amount of bandwidth required to simultaneous acquisition of numerous cameras. In addition, the complexly to manufacture compact systems and mechanical problems, it is an important issue to calibrate multiple synchronized cameras. In the case of dioptric cameras, only is required the acquisition of a single image per time, it means a low acquisition rate. They are difficult to made, normally these cameras not satisfy the constraint of single effective viewpoint and central projection, causing a complicated computing of three-dimensional information. The resolution through the omnidirectional image is not constant, been poor in the peripheral area. Catadioptric sensor also required a single image capture, the 3-dimensional data processing it could be workable due the possibility to satisfying the single effective viewpoint constraint. The resolution is lower than the original image, however the wide view angle benefit the estimation algorithms stabilizing for ego-motion, the rotation and translation can be easily distinguished. Starting with two panoramic images is possible to carry out a surrounding scene area, by means of 360 view angle around the camera.

Figure 2: Omnidirectional vision sensors: (a) Dioptric, (b) Polidioptric and (c) Catadioptric.

4. Catadioptric omnidirectional vision systems

Catadioptric is referred to sciences of refracting and reflecting elements, such as lenses and mirrors. The central projection of a convex mirror should be aligned with an optical axis of the camera lens, and also must be placed into focal point of the mirror, in this way; the intersection of all reflected rays at the focal point of the mirror is assured. This means that the whole catadioptric sensor has a single effective viewpoint (central catadioptric camera). The remarkable work in designing panoramic catadioptric cameras were presented by Baker [12], Yagi [13], Svodoba [14]. Several mirrors have been used with perspective and orthographic cameras, spherical, hyperbolic and parabolic mirrors (Figure 3).

(a) (b) (c)

Figure 3: Projection of (a) spherical, (b) hyperbolic and (c) parabolic mirror and their corresponding reflected rays.

Considering the information in Figure 3, the reflected ray of spherical mirrors have similar properties as fish-eye lenses, they have good resolution in central region but the peripheral resolution is poor. Parabolic mirrors (Figure 3(c)) where proposed in late 90’s, they work as a parabolic antenna, their reflected ray are parallel to the rotation axis of the mirror, they are modelled by coupling and orthographic camera, in order to accomplish the single effective viewpoint. Finally, hyperbolic mirrors reflect 3 dimensional rays in the space through its second focal point located in its central axis (Figure 3(b)).

The projection of hyperbolic mirror is represented in detail in Figure 4, where a point P in the space is reflected by the hyperboloid surface and projected on the image plane. From point P(X, Y, Z) a ray goes toward the focal point of the mirror Focus1 and is reflected by the mirror, then is conduced through the other focal point Focus2, intersecting into the image plane at point p(-x,-y). This relation is satisfied for any point in the space in the field of view (360 degrees around the Z axis) of the hyperbolical projection. Consequently, we can obtain an omnidirectional image of the scene on the image plane with a single center of projection within Focus1 and Focus 2.

The solution of fixed viewpoint constraint and geometry of convex mirror are widely explained in [12]. The hyperboloid surfaces can be obtained by revolving the hyperbola around the Z axis and having two focal points as shown in Figure 4. Using the world coordinates system (X, Y, Z) the hyperboloid surface can be represented as equation (1) and (2):

12

2

2

22

−=−+bZ

aYX

(1)

22 bac += (2)

Where a and b define the shape of a hyperboloid surface. If the system consists of a CCD camera and a hyperboloid mirror; notice that the focal point of the hyperbolic mirror Focus1

(a) (b) (c)

Image plane Image

plane

and the lens center of the camera Focus2, are fixed at the focal points of the hyperboloid surfaces (0, 0, c) and (0, 0,2c), respectively. The axis of the camera and mirror are aligned. The image plane should also be placed at a distance f (focal length of camera) from the lens center of camera Focus2, and be parallel to the XY plane.

Figure 4: Hyperbolic projection

In order to accomplish the image modeling for this catadioptric sensor is necessary to obtain the homogenous transformation amount the world, the mirror and the camera frames. Defining carefully three references, the world reference system denoted Ow which a corresponding 3D point is Xw; the mirror coordinates system centered at the focus Focus1 whose vector is Xm; and the camera coordinates system centered at Focus2, according to what Xc is its vector The projection between the pinhole camera and mirror frame, is obtained by equation (1) and (2), and is given by:

( )( )( )mc

mww

mTw

mc ttXRR

cKm +−= λ

21

(3)

Where m is the projection to be calculated, is a non linear function of Xm, K is the internal calibration matrix of the camera looking at the mirror. m

ct is the center of the mirror expressed in the camera frame, corresponding to (0,0,2c). The rotation between camera and mirror frames is represented by matrix m

cR . And finally, the configuration of the mirror with respect to the

world frame, rotation and orientation are represented by mwt and m

wR respectably.

In this way, it is possible to determine the position of 3 dimensional points in image plane (panoramic image) [15]. Moreover, this feature is useful to design the shape and dimensions of the mirror, in order to improve and maximize resolution and field of view of our system. Some experiments are shown in follow figures, where 3 point in the world frame are acquire for 2 hyperbolic mirror whit different configurations.

�

(a) (b)

(c)

Figure 5: Simulation of catadioptric sensor image acquisition, configuration 1

Figure 6: Simulation of catadioptric sensor image acquisition, configuration 2.

In Figure 5, the graphic (a) shows the representation of the catadioptric system and the 3 dimensional points attached in the world frame, (b) shows the top view of the system, and finally (c) represents the position of these points in a 2D acquired image. Figure 6, as Figure 5, show the representation of the catadioptric sensor acquisition, corresponding to a different configuration, where the maximum angle between a reflected ray and the center axis is 140°, which mean that the highest resolution of the image is placed in peripheral area.

5. Experimentation and results

If a standard camera is mounted on a mobile robot and it is aligned with its forward direction of motion. This is sufficient for vehicles that can move in a constrained set of directions (e.g., a car), where the primary vision tasks typically consist of obstacle detection and avoidance. Nevertheless, a rigid camera with a limited field of view is not ideally suited to robots with omnidirectional motion capability, and navigation on unstructured environments, and when other vision tasks must be performed such as building a local representation or localizing within its environment, detection or scanning rough terrains. For these tasks it would be much better to use a vision sensor that can provide panoramic images, i.e., 360� images around the robot; these are also referred to as omnidirectional images.

In previous section we present several techniques to obtain omnidirectional images, and according to the features required for mobile robot system, such us the case of Silo6 six-legged robot (section 2), a catadioptric hyperbolic vision system is adequate and will benefit the tasks performed by the robot.

Several simulation where done in section 4, as a result a low-cost catadioptric system is been developed in Department of Automation Control. An omnidirectional image acquired by our system is presented as follow, and its corresponding panoramic image. It is easily to detect that both images are distorting from our point of view; however for omnidirectional theory it is straightforward to calculate this deformation angle and introduce it in the system.

Figure 7: (a) Omnidirectional and (b) panoramic images.

The angle in the image can be calculated as y/x, showing the azimuth angle of point P in space (section 4). Also, it can be easily understood that all points with the same azimuth in space appear on a radial line through the image center. This useful feature with a hyperboloid projection, allow the vertical edges in the environment appear radially in the image (represented by blue and red arrows). By simple geometrical analysis, equations relating the point in space P(X, Y, Z) and its image point on the image plane p(x, y), can be derived as follows:

yx

YX ==θtan (4)

cYXZ ++= αtan22 (5)

( )( ) γ

γαcos

2sintan

22

221

cbbccb

−−+= − (6)

���

����

� += −

fyx 22

1tanγ (7)

Where � denotes the tilt angle of point P from the horizontal plane, f is the focal length of the camera lens. From equations (4), (5) and (6), the azimuth angle � and the tilt angle � of the line connecting the focal point of the mirror Focus1 and the point in space P can be obtained from the position of the image point p(x, y) (see Figure 4). This means that the equation of the line connecting Focus1 and P can be determined uniquely from the coordinates of the image point p(x, y), regardless of the location of the point P in space.

The prototype was tested on-board of six-legged robot Silo6, a sequence acquired by the system is presented in Figure 8, where corresponding entities do not vanish due to limited field of view (enclosed by green). The displacements of such entities considerably vary with different kind of motion. With this system is possible to detect object around the robot only acquiring a singles image per time. The terrain area around the robot can be easily separated form the peripheral area, such as trees, bushes stones, and moving objects.

-

Figure 8: Sequence of catadioptric system onboard six-legged robot Silo6

The system obtains the capability to detect several obstacles, fixed or moving object. It is possible to apply image processing techniques as optical flow for segmented these objects and avoid them. Also is able to tracking the trajectory of the manipulator (Figure 9) and use the vision to correct it movement, especially in situation where two objects are close and the distance between then is smaller that the sensor head diameter.

Figure 9: Sequence of manipulator movement detection.

Other important issue is the capability to observe with a single image the environment and the robot itself. The area covered by the image includes the space between the scanning system manipulator and the robot. This system also provides the feature to observe the 360 °view for teleoperated applications, saving the security of the operator and the robot itself.

6. Conclusions

The removal of antipersonnel landmine is a global issue. In this work we presented the possibility of designing a low-cost system based on omnidirectional sensors, to improve the efficiency of humanitarian de-mining tasks. Because these tasks require devices that can automate the location of unexploded ordnance, it is proposed that they can be accomplished by using robotic systems capable of carrying scanning sensors over infested fields

The ongoing prototype has very usefully features, because they can benefit several tasks involved in humanitarian demining missions. The robotic system can be able to respond in advance, i.e. obstacle situated in a distance larger than the manipulator range. The system will be capable to make online corrections of its trajectory. Other important point is the benefit of the efficiency of a complete coverage of a minefield wider area, since the system has a previous knowledge of the terrain and its obstacles.

The next step in this research will be the test of the catadioptric vision system in specific online tasks on hexapod mobile platform and to study particular image processing algorithms for panoramic images to terrain description.

Acknowledgements DYLEMA project is funded by the Spanish Ministry of Education and Science through grant DIP2004-05824. This work was supported in part by Consejería de Educación of Comunidad de Madrid under grant RoboCity2030 S-0505/DPI/0176. This work is funded by Autonomous Community of Madrid through fellowship in Research Professional in Training (FPI).

References

[1] J.-D. Nicoud, Vehicles and robots for humanitarian demining, Industrial Robot 24 (2) (1997) 164 – 168.

[2] J. Trevelyan, Robots and landmines, Industrial Robot 24 (2) (1997) 114–125. [3] P. Bellutta, R. Manduchi, L. Matthies, K. Owens, and A. Rankin. Terrain Perception for

DEMO III. In Procs. IEEE Intelligent Vehicles Symposium 2000, 326 -331, Detroit, USA, Oct. 2000.

[4] Seiji Masunaga and Kenzo Nonami, Controlled Metal Detector Mounted on Mine Detection RobotInternational Journal of Advanced Robotic Systems, Vol. 4, No. 2 (2007).

[5] G. A. Clark, “Computer Vision and Sensor Fusion for Detecting Buried Objects”, Annual Asilomar Conference on Signal, Systems, and Computers (26th), October 1992.

[6] H. Najjaran, A. Goldenberg, Landmine detection using an autonomous terrain-scanning robot, Industrial Robot: An International Journal 32 (3) (2005) 240–247.

[7] E. Colon, G. De Cubber, H. Ping, J-C Habumuremyi, H. Sahli and Y. Baudoin. Integrated robotic systems for Humanitarian Demining, International Journal of Advanced Robotic Systems, Vol. 4, No. 2 (2007).

[8] P. Gonzalez de Santos, J.A. Cobano, E. Garcia, J. Estremera, M.A. Armada. A sex-legged robot-based system for humanitarian demining missions. Mecatronics 17 (2007) 417-430.

[9] E. Garcia, P. Gonzalez de Santos, Mobile robot navigation with complete coverage of unstructured environments, Robotics and Autonomous Systems 46 (4) (2004) 195–204.

[10] E. Garcia, P. Gonzalez de Santos, Hybrid deliberative/reactive control of a scanning system for landmine detection, Robotics and Autonomous Systems 55 (6) (2007) 490-497.

[11] Y. Yagi. Omnidireccional sensing and its applications. IEICE Trans. Inf. Syst. E82-D(3) (1999) 568–579.

[12] S. Baker, S.KNayar. A theory of single-viewpoint catadioptric image formation. Int. J. of Computer Vision 35(2) (1999) 175–196.

[13] Y. Yagi, M. Yachida. Real-Time Omnidirectional Image Sensors. International Journal of Computer Vision 58(3)(2004) 173–207.

[14] T. Svodoba, T. Pajdla. Epipolar geometry for central catadioptric camera. Int. J. of Computer Vision, 49(1)(2002) 23-37.

[15] G.L. Mariottini,D. Prattichizzo. The epipolar geometry toolbox. IEEE Robotics and Automation Magazine, 12(3)(2005).