* Ecole des Mines de Paris, Centre de CAO-Robotique, 60 ED Saint-Michel, 75006 Paris, France** Ecole des Mines de Paris, Centre de Morphologie Mathemalique. 35 rue St-Honore. 77305 Fontainebleau, France

Abstra<:L The paper presents the recognitionof vehicles considered as obstacles in the road, inorder to avoid collision. An inboard video cam-era is used as an optical sensor to supply exteriorscene information. The algorithm of vehicle de-tection from video image, built by morphologicaltransformations (Serra 1982), is proposed. Theadvantage of the proposed approach is the reliablerecognition for various models and colors ofvehi-cle, and the good efficiency for difficult case likedistant vehicles and lateral sun shadows. The ex-perimentation has been done on a large amount ofdata. Some of results is given in the article.

Key Words: traffic control, vehicle, obstacleavoiding, sensor, image processing

This paper describes the algorithms for vehicles detec-tion as obstacles in a road. The research has been doneat the Center of Mathematical Morphology -(C.M.M.)in France, in the framework of the PROMETHEUS Eu-ropean project (Beucher 1990, Yu 1992, Yu 1993). Un-der the supervision of French automobile industries RE-NAULT, PEUGEOT and CITROEN, the project aimsat improving driving security by means of a percep-tion system based on computer vision. Vehicles arethe most frequent obstacles in a road for daily driv-ing. Their recognition should depend on their relativeposition with respect to the optical sensor. The paperdeals with the case of the view ahead or behind forvehicles in a highway or country road. Two kinds ofinboard sensors can be used to receive the vehicle in-formation: telemeters and video cameras. The use ofvideo camera allows the road edges tracking and theobstacles recognition to be coupled by only one sensor.

The existing algorithms for vehicles recognition ina video image can be classified into two maingroups: The first group is based on optical flow field.It exploits the information of the vehicle's motion bycalculating velocity field from the optical observation.

The apparent size of the vehicle should be quite largefor reliable recognition. Furthermore, the motion ofthe camera, i.e. the velocity of the vehicle, should notbe significant, otherwise, the other objects in the scenewill cause false detection due to their relative motionwith respect to the camera. The rejection of the falsedetection is very difficult, due to the poor precision ofobject's shape archived by the approach. The secondgroup consists in contour analysis of the candidate re-gions. The shape of a vehicle can be represented bysome geometrical characteristics, like coins, segments,and closed regions, etc. These criteria are valuable onlyif the apparent size of the vehicle is large. For smallvehicles, these geometrical characteristics will be con-fused with noise and become no significant. Moreover,both of the approaches are based on the observation ofthe vehicle's shell, and depend thus on their grey tones.For those who have a grey tone almost equal to that ofthe road, the mentioned approaches will fail because ofthe lack of contrast between the object to detect and thebackground. To overcome the above difficulties, a newapproach is proposed in this paper. It is independent ofvehicle's model and color, and efficient for a large rangeof distance. The recognition process can be describedby the detection of candidates and the application ofsome criteria to reject the false ones. The details ofalgorithm are given in the following sections.

As the vehicle's shell may have poor contrast with re-spect to the road background, other parts of the vehicleshould be studied. In fact, the low part of the front orback represents a good marker for any models of vehi-cles. The examination of numerous video images hasshown that these parts are always darker than the roadbackground (see figure 1). This is true for all weatherconditions, traffic situations, and vehicle types. The de-pendence on vehicle color is then avoided. The obser-vation done above remains valuable for distant vehicleshaving very small apparent size.

The detection of candidates can be realized by search-ing for the regions darker than the road. Dark regions

are detected by a morphological transformation callednumerical reconstruction. The relation of grey tonebetween different objects in a typical road scene is il-lustrated in the figure 2a. For easier comprehension, thesignal is inversed in the figure 2b, where the white re-gions should be considered as vehicles. The numericalreconstruction to extract a white regions is expressedas follows. Firstly, the initial signal f is subtracted bya constant h, giving a signal f - h in the figure 2b. Thenumerical reconstruction g of signal f - h in the maskf is the iterative morphology dilations (Serra 1982) off - h in the mask f until idempotent (figure 2c).

~,

f-g,.J"'-J\,(d)

The residue (figure 2d) of the reconstruction f - !J rep-resents the detected white regions. or dark regions ininitial image. In this transformation. the parameter hhas to be determined. It characterizes the dynamic ofluminance variation, and is determined by an adaptivemanner. A constant value of the parameter can not sat-isfy the unpredictable luminance variation of the roadscene. In fact. for each image the value of h can be de-duced easily from histogram information. which shouldbe calculated in a window including road pixels. Sucha window is shown in the figure 3 by a rectangle placedat the bottom of the image.

The variance (J of this histogram can then be consideredas that of the road luminance. because almost all pix-

Figure 3: Binary dark regions detected by numerical recon-struction

els used to calculate the histogram belong to the roadsurface. The value of h is then deduced from (7. Some-times the small window in the bottom of the imagecan include some other pixels than those of road, likelane markers, spots, and trace of vehicle wheels. Thepresence of these objects causes the variance of the his-togram to be different from that of the road luminance.One alternative method to reduce the influence of otherobjects consists in enlarging the windows, but this willinclude vehicle objects. Finally, a very simple methodis suggested to resolve this conflict: adding histogramsin the same windows but at different instants. As themotion of the camera, the small windows represent dif-ferent road sections in real world. The percentage ofthe number of no-road pixels with respect to that ofthe road pixels is reduced, because the number of roadpixels is increased due to the addition of some otherroad sections. The influence of unexpected pixels isthus reduced.

The dark regions got by numerical reconstruction isgiven in the figure 3. In order to minimize unnecessaryoperation, the search of dark regions is only done inthe window at the middle of image. The shapes of thedetected binary regions in the road (see figure 3) areregular, which are equivalent to that of vehicle. Butthis is not always the case. Sometimes the dark regionattached to a vehicle can be much smaller than it. Toget good contour shape of vehicle's front or back, amorphological transformation watershed is used. It is anon parameter transformation (Beucher 1982) appliedto the gradient module of an initial image, but a markeris necessary to be put into each object. The dark regionsdetected by numerical reconstruction are used here asthe markers for watershed transformation. The figure 4shows the vehicle contours determined by the water-shed transformation. In this special example, there isno evident difference between the contours of water-

Among all candidate regions, some of them are not thevehicles, and should be rejected by a certain criterion.In the example of figure 4, there is no false detectionin the road, but there could be if some dark spots arepresent in the road. Several false detections are pro-duced outside the road. In fact, those false detectionsoutside the road can be easily eliminated by consideringonly the candidates inside the road. if the result of theroad edge detection is supported (BED CHER 1994). Toexamine the effect of the criterion of rejection on un-significant irregular objects, the false detections in thisexample are not eliminated by road edge information.

The first proposed criterion is derived from a geometri-cal characteristic, i.e., the back or the front of a vehicleis always composed of some horizontal and vertical seg-ments. This geometrical description of vehicle is moreuniversal and pertinent than the description of coins andshapes. The last one can not be valuable for all modelsof vehicle and becomes unsignificant for distant vehi-cle. The segment lines description can be formed by thedirectional gradient criterion. Let go and g90 be respec-tively directional gradient module of the initial imagein the direction 0 and 90, and [/ = .\LLY (go, [/90)

the combination of the gradient in two directions. Foreach candidate region, the volume of gradient modulenormalized by the area is calculated. This measure-ment gives high values for the regions including thehorizontal and vertical segments. The false candidatesdue to the spots have irregular forms, and they are notcomposed of significant segment lines. They will bethen eliminated for their low values of directional gra-dient volume. The figure 5 gives the result of candidateselection by this criterion. where only the baselines of

retained candidates are drawn in the figure. The heightsof the vehicles are not indicated because the contrastof the high parts of vehicles depends upon their colorand is not always good. Any way, the information ofheight is not important. The useful parameters for driv-ing security are the distance of the obstacles ahead andbehind, and their horizontal extensions (width).

As the figure 5 shows, the traffic sign at the left side ofthe road is also retained by the criterion of gradient vol-ume, because it includes some horizontal and verticalsegments. That means that the approach developed inthis paper can be used to recognize other manufacturedobjects, since they are often composed of segment lines.

To make the decision of the rejection of false candi-dates, a parameter should be introduced as the thresh-old of gradient volume. The value of this thresholdshould not be determined seriously, in order to avoidrejection of real candidates. On the other hand, thismay cause a insufficient rejection, and false candidateswill be retained after the application of this criterion.Therefore, another criterion, geometrical symmetry, isintroduced here for second rejection. Obviously, all ve-hicles have a symmetric form. But this criterion shouldnot be exploited simply upon the form of the binaryregion. Under the influence of noise, the shapes of thefront or back of vehicle are sometimes not regular; es-pecially for distant vehicles, the apparent dimension ofa vehicle may be only in order of several pixels, andthe perturbation of the noise may also be in the sameorder. The symmetry for far vehicles will then looseany signification.

To avoid this disadvantage, the criterion of symmetryproposed here does not depend directly on the contour

shape, but exploits the information of grey tones in theinitial image. The figure 6 explains the principle of thiscriterion. The figure 6a represents an initial image orsignal. Let Xo be an axis of mirror. By reflecting the leftside of the signal to the right side of the axis, the twosignal are superposed at the right side in the figure 6c.The module of the difference of two signals is given inthe figure 6e. The signal in (e) represents the residue ofthe difference of two superposed signals. The surfaces( xo) of the residue is then calculated in the interval of[xo, Xo + D], with D as the half-width of the region inexamination. s( xo) has a small value if the axis Xo isthe same as that of vehicle, and a large value if it is notthe case (the figures 6b,d,f).

R- /1~e,....r-'~

!b) ;:,

I(x)l(x-2(x-x,))

Considering each line y of initial image as one signal,a family of functions Sy (£0) is available. Each func-tion for a given y has its form like the figure 7a withsome irregularity. To reduce this irregularity, Sy (xo) isintegrated for y in a vertical interval, and normalizedby the height of the vertical integration interval, giving8(xo) as result.

r~dissvrnelric obiect ~- \ . '-e

"§0-0or

I "I ~x, ! xo

(b)

In the symmetry calculation. the horizontal and verticalinterval of integration have to be known, they are justlythe width and height of the binary region in examina-tion. The figure 7b shows that the integrated symmetryfunction S( xo) has a much more regular form. For sym-metric objects. the function has the form of an inversetriangle, and it has an irregular form for dissymmetri-cal objects. That means that the form of the symmetryfunction can be used as a criterion of symmetry. But ascalar value should be introduced to describe its form.This value can be calculated from certain characteristicpoints of the function. For example, it can be the ratioof minimum value and the value at the extreme sides ofthe function. The minimum point of the function has

a significance, it is the median axis of the vehicle. Itshould be pointed om that median axis of the vehicle arenot always the same as that of binary region attached toit. This remark implies a very important utility, whichwill be demonstrated in the next section.

The figure 8 gives the median axis and baseline ofobjects retained after symmetry criterion. The trafficsign at the left side of the road is retained, but this wiilnot cause false detection since it is outside the road.

The algorithm has been tested on numerous images.Some typical results are given in the figure 9. Thefigure 9a represents a very difficult case, where the ve-hicles produce their shadows in the left side due to sunlight from the right side. These shadows are connectedto the dark regions of vehicles. After the segmentationby watersheds. the dark regions overflow by the leftof the real width of the vehicle. A solution is neededto resolve this problem. The proposed solution in thispaper is very simple and efficient. The method uses theinformation of the median axis of vehicle to correct theoverflowed part of width: The two longer horizontalbars in the figure 9b indicate the baseline of detectedvehicles. They are longer than the real width of vehi-cle. TIle two vertical lines represent the median axeof vehicles. which cut the longer horizontal line in twoparts. The median axis is correctly determined in thiscase. because the symmetry calculation is done in ini-tial image, instead of binary regions. The shorter partsare C{juals to the half size of the real width, in spiteof the presence of lateral shadows. By reflecting theshorter parts to opposite side of the median axe, thecomplete size of rca1 width can be known correctly (theshorter horizontal baselines). The operation does notneed any supplemental calculation, since the median

As mentioned in the introduction, the approach shouldhave a good perfonnance of recognition for distant ve-hicles. Some examples of distant vehicles are shown inthe figure 9c,d,e,f. From the figure 9c to figure 9d, thevehicle carrying the camera moves from the left lane tothe right lane. The vehicle in front of the camera andat nonnal distance are correctly recognized. A vehicleappeared in the horizon is also recognized. The dimen-sion of this vehicle is equal to several pixels, and allother existing approaches depending contours analysiswill fail in this case, because the shape of small vehi-cles will loose any signification. In the figure ge,f, thevehicles are recognized in more difficult cases, i.e., thevehicles in the left road have very small apparent size,their base part are partially hidden by a central sepa-ration bar. The examples prove that the recognitioncan give satisfactory results even in these extremelydifficult cases.

The paper proposed an efficient approach to recognizethe vehicles as obstacles in a road. The particular ad-vantage of the developed algorithm is that the recogni-tion does not depend on the color or model of vehicle,and has a good efficiency for the extremely difficultcases as vehicles at far distance and the problem of lat-eral shadows. The algorithm has been tested on a largeamount of data representing different weather condi-tions and traffic situations, giving satisfactory results.The algorithm is now implemented in a specially de-voted hardware for real time application, in the frame-work of the French testbed of intelligent vehicle PRO-LAB2.

Beucher, S. (1982) Watersheds of functions and picturesegmentation. In Proc. IEEE Int. Conference onAcoustics, Speech and Signal Processing, pages 1928-1931, Paris.

Beucher, S., ~1. Bilodeau. and X. Yu (1990). Road seg-mentation by watersheds algorithms. In 3th WorkshopPROMETHEUS, Antibes. April 1990.

Beucher, S .. X. Yu, and M. Bilodeau (1990). Road seg-mentation and tracking by mathematical morphology.In 4th Workshop PROMETHEUS, Compiegne. France,Oct. 1990.

Beucher, S. and X. Yu (1994). Road recognition incomplex traffic situations. IFAC TS' 94, Symposium.Tianjin, China.

Yu. X.. S. Beucher, and M. Bilodeau (992). Roadtracking, lane segmentation and obstacle recognitionby mathematical morphology. In Proceedings of theIntelligent Vehicle '92, pages 166-172, Detroit,USA,June-July 1992, IEEE Industrial Electronics Society.

Yu, X. (1993). Dynamic vision and mathemQlicalmorphology (in French) Ph.D. dissertation of mineralschool of Paris, May 1993, France.

Serra, J. (1982). Image analyses and mathematicalmorphology. London, Academic Press.