Pergamon Pattern Recognition. Vol. 30, No. 12, pp. 2043-2052, 1997

© 1997 Pattern Recognition Society. Published by Elsevier Science Ltd Printed in Great Britain. All rights reserved

0031-3203/97 $17.00+,00

P H : S0031-3203(97)00015-0

A MULTISCALE GRADIENT ALGORITHM FOR IMAGE SEGMENTATION USING WATERSHEDS

DEMIN WANG* Communications Research Centre, 3701 Carling Avenue, Ottawa, Ontario K2H 8S2, Canada

(Received 6 August 1996; in revised form 3 January 1997)

Abstract--Watershed transformation is a powerful tool for image segmentation. However, the effectiveness of the image segmentation methods based on watershed transformation is limited by the quality of the gradient image used in the methods. In this paper we present a multiscale algorithm for computing gradient images, with effective handling of both step and blurred edges. We also present an algorithm for eliminating irrelevant minima in the resulting gradient images. Experimental results indicate that watershed transformation with the algorithms proposed in this paper produces meaningful segmentations, even without a region merging step. The proposed algorithms can efficiently improve segmentation accuracy and significantly reduce the computational cost of watershed-based image segmentation methods. © 1997 Pattern Recognition Society. Published by Elsevier Science Ltd.

Gradient operator Image segmentation Watersheds Edge detection

Mathematical morphology

1. INTRODUCTION

Image segmentation is an essential step for many image analysis tasks, such as object recognition, com- puter vision and image compression. The goal of image segmentation is to partition an image into homogeneous regions and locate the contours of the regions as accu- rately as possible. A large number of techniques and algorithms have been proposed for image segmenta- tion. (1"2) Among them, those based on watershed trans- formation (3"4) can potentially provide accurate segmentation with very low computational cost.

For image segmentation, watershed transformation starts with the gradient of the image to be segmented. It views the gradient image as a three-dimensional (3-D) surface where gradient values act as surface heights. Intensity edges in the image to be segmented generally have high gradient values which appear as watershed lines (also known as mountain ridges) on the 3-D surface, while the interior of each region usually has a low gradient value which is considered as a catchment basin (3"4) on the 3-D surface. The watershed lines parti- tion the gradient image into different catchment basins which correspond to homogenous regions of the image to be segmented. Watershed transformation involves a search for watershed lines in the gradient image. There- fore, the performance of a watershed-based image seg- mentation method depends largely on the algorithm used to compute the gradient.

Conventional gradient algorithms exhibit a serious weakness for watershed-based image segmentation. A conventional gradient operator, such as the first partial

* Tel.: 1-613-991-5621; Fax: 1-613-990-6488; e-mail: dentin.wang @ crc.doc.ca

derivative of Gaussian filter C5) and morphological gra- dient operators, (4'6'7) produces too many local minima because of noise and quantization error within homo- geneous regions. Each minimum of the gradient intro- duces a catchment basin with the watershed transformation. Hence, these gradient operators result in over-segmentation, e.g. a homogeneous region may be partitioned into a large number of regions and proper contours are lost in a multitude of false ones. A straight- forward method to deal with this problem is to threshold the gradientJ s'6) However, conventional gradient opera- tors produce low gradient values at blurred edges, even though the intensity change between the two sides of an edge may be high. By thresholding, one cannot eliminate the local minima caused by noise and quantization error while preserving those produced by blurred edges. Another solution for this problem is to extract markers and impose them on the gradient image, (4'6"7~ which may require prior information about the objects and background to be segmented. After water- shed transformation, region merging or relaxation labeling is usually performed to further remove false contours. (4"5"g) This process may be much more eompu- rationally expensive than watershed transformation because too many catchment basins have to be merged. This greatly decreases the speed of the entire segmenta- tion method.

In this paper, we propose a multiscale gradient algo- rithm based on morphological operators for watershed- based image segmentation. This algorithm efficiently enhances blurred edges while being very robust to multi- edge interactions. This enhancement increases the gradient value for blurred edges above those caused by noise and quantization error. We then present an algorithm to eliminate the local minima produced by

2043

2044 D. WANG

noise and quantization error. Finally, we show how to exploit the multiscale gradient algorithm for region merging.

2. T H E M U L T I S C A L E G R A D I E N T A L G O R I T H M

Many gradient operators and edge detection algo- rithms have been based on the step edge model. (9-n) However, ideal step edges do not exist in natural images since every edge is blurred to some extent. A blurred edge can be modelled by a ramp and the intensity change between two sides of the edge is referred to as edge height. For a ramp edge, the output of a conventional gradient operator, such as Prewitt gradient, (9) is the slope of the edge. Hence, the ramp edge cannot be separated from noise and quantization error by thresholding if the slope of the edge is small. Fig. 1 shows a 1-D example where a ramp edge and a step edge are digitized in order to illustrate the effect of quantization error. The ideal gradient operator for watershed transformation is the one whose output is equal to the input edge height, but not the edge slope.

The morphological gradient operators used in refer- ences (4,6) can be described as

~(f ) - (f • B) - ( foB) , (1)

where • and O, respectively, denote dilation and ero- sion, (12'13) and B is called a structuring element. This gradient operator is referred to as a mono-scale morpho- logical gradient operator. Its performance depends on the size of structuring element B. If B is large, the output of this gradient operator for a ramp edge is equal to the edge height. Unfortunately, large structuring elements result in serious interaction among edges which may lead to gradient maxima not coinciding with edges. However, if the structuring element is very small, this gradient operator has a high spatial resolution, but produces a low output value for ramp edges.

In order to exploit the advantages of both small and large structuring elements, we propose a multiscale morphological gradient algorithm. Let Bi, for 0 < i < n, denote a group of square structuring elements. The size of Bi is ( 2 i + 1 )× ( 2 i + 1) pixels, i.e. Bo contains only one pixel and B1 is a 3 × 3 square and

T r step edge

x y

so on. The multiscale gradient is defined by

MG(f) = _1 ~ [((f • Bi) - (fOBi))OBi_l]. (2) n

i = l

For a step edge, the operation ((feBi) - (fOBi))OBi_~ produces a line of two pixels wide which coincides with the edge. The intensity (height) of the line is equal to the edge height. Hence, the multiscale gradient algorithm is equivalent to the mono-scale morphological gradient operator in this case. In practice, it is more robust to noise due to the averaging operation used in the algo- rithm.

For a ramp edge, we denote respectively the edge width and height by w and h, as shown in Fig. 2. The operation ((f ® Bi) - (fOBi))OBi 1 produces a line co- inciding with the edge. The cross section of the line appears as a trapezoid if i < (w + 2) /4 and as a triangle otherwise. The width of the bottom side of the trapezoids or triangles is always equal to w + 2 pixels. The height of the trapezoid is 2ih/w and that of the triangle is h(w + 2)/(2w), which are greater than the edge slope h/w. The value of MG(r3 approaches to h(w + 2)/(2w) if n is large enough. Therefore, the multiscale algorithm responds effectively to ramp edges without enlarging edges.

Y T

j (x)

I I

x

((f~Bi)-(l~Bi))OBi_l when i<(w+2)14

X

/ \

( (figBi)-(fOBi) )OBi_l when i>(w+2)/4

X

y

gradient magnitude off(x) N

[-] I I I I I I I I I ] x

Fig. 1. Output of conventional gradient operators.

l MGfJ) ~ x

Fig. 2. Result of the multiscale gradient algorithm for ramp edges.

A multiscale gradient algorithm for image segmentation using watersheds 2045

f(x)

( ( [ ~ B i ) - ( f O B i ) ) O B i . 1 when i < d l 2

_ _

m x

MG(D LM 2 LM 3

_ _

LM 4

LM = Local minimum

LM5 LM6 S

[ I ( ( l ~ B i ) - ( [ O B i ) ) O B i . 1 when i_>d/2

MG(¢)

Fig. 3. Interaction between edges.

x

x

x

The multiscale gradient algorithm is very robust to edge interaction. The location of gradient maxima corresponding to one edge is not disturbed by the pre- sence of other edges. Fig. 3 shows two pairs of adjacent step edges. The distance between two adjacent edges is denoted by d. When i < d/2, ( ( f @ Bi) - - (fOBi) )OBi_l correctly produces gradient maxima which coincide with the edges. While i >_d/2, (Oe @Bi) - (fOBi))(~Bi l fills the gap between the two adjacent maxima with the value of the smaller maximum. By averag- ing the values produced by (Or Q B i ) - OeOBi))(~Bi-1 for all i, MG(f) keeps the gradient maxima at the correct locations. For the interaction between ramp edges, the situation is similar. It should be noted that the distance d between edges must be at least three pixels since B1 is a 3 x 3 square. If the distance is smaller than three pixels, the image should be upsampled before computing the gradient, as done in reference (6).

The structuring elements B i in equation (2) could be of any shape satisfying the relation Bo C_ B1 C_ . . . C_ Bn. We use a group of square structuring elements because of its low computational cost. Moreover, one may use several groups of directional structuring elements, e.g. line segments, for specific applications. The gradi- ents of different directions are computed with the corre- sponding groups of structttring elements and the maximum value of these gradients is taken as the final result. According to our experiments, this approach produces more local minima than using square structur- ing elements.

(MG(t))~B s

, , , x

t~(ree)[(MG(t))~Bs+h, MG(t)]

7 [

Fig. 4. Elimination of small local minima.

3. ELIMINATION OF SMALL LOCAL MINIMA

Small local minima is defined as local minima con- sisting of a small number of pixels or having a low contrast with their neighbors. This kind of local minima in gradient images is generally caused by noise or quantization error, and therefore should be eliminated.

Local minima consisting of a small number of pixels are eliminated by dilation with a square structuring element Bs of 2 x 2 pixels, denoted by (MG(f)) q3 Bs. To remove the local minima with a low contrast a constant denoted by h is first added to the dilated gradient image. Then the local minima with a contrast lower than h can be filled using the reconstruction by erosion (6'7) of MG(f) from ( M G ( f ) ) @ B s + h . Hence, the final gradient image can be expressed as 0(rec)[(MG)(f))® Bs + h, MG(f)].

This algorithm is illustrated in Fig. 4, where MG(D has six local minima. Local minima (LM) 3 and 5 consisting of one pixel are removed by the dilation, while local minima 2, 4 and 6 having a contrast lower than h are eliminated by the reconstruction by erosion. Local mini- mum 1 is not removed because it is both wide and deep. The constant h is used to control the number of segmen- tation regions. As h increases, the number of regions produced decreases. The reconstruction by erosion fills all of the local minima where the contrast is lower than h, irrespective of their absolute values. However, thres- holding removes only the minima with low absolute values. For the MG(f) shown in Fig. 4, thresholding cannot remove local minima 2 and 3.

2046 D. WANG

4. USE OF MULTISCALE GRADIENT FOR REGION MERGING

Generally, the above-mentioned algorithms followed by watershed transformation produce meaningful image segmentations. Some applications, however, may require further merging of some regions. For region merging, a number of similarity (between regions) criteria have been proposed, each of which has its specific applica- tions. (1'5'9'14"15) Some criteria are based on edge height along the common contour between two regions. (9'14) If such a criterion is required, the multiscale gradient algorithm is suitable to provide edge height information since it responds effectively to blurred edges as well as step ones.

Considering contour orientation, directional gradients should be used for the purpose of region merging. A directional (horizontal, for example) gradient can be computed with equation (2) using a group of line seg- ments of corresponding orientation (horizontal) as struc- turing elements. If two horizontally adjacent pixels belong to two different regions, i.e. a vertical contour passing between the pixels, the horizontal gradient values corresponding to these pixels are taken as the edge height. Otherwise, i f the pixels are vertically adjacent, the corresponding vertical gradient values are involved.

5. EXPERIMENTAL RESULTS

In the experiments, the images to be segmented were firstly filtered to reduce noise disturbance, as done in references (4-6). Since morphological filters are efficient in removing noise while preserving edges, °6) the open- closing by reconstruction with a structuring element of 3 x 3 pixels (6'7) was used as the filter. The filtered images were then segmented using the new algorithms followed by watershed transformation. This segmentation proce- dure was applied to a number of images. Fig. 5 illustrates one of the images, which has 720 × 480 pixels and 256 gray levels. The image contains a textured background

(wall), sharp edges such as those of the tennis table and low contrast blurred edges formed by the wrinkles in the clothes. The edge of the person's back is also a little blurred. Fig. 6 shows the segmentation results when the constant h, described in Section 3, is set to 8, 12 and 16, respectively. The segmentation results contain, respec- tively, 138, 85 and 61 regions. It can be seen from this figure that there are almost no irrelevant regions in the segmentation results. With a low value of h, the regions delimited by the wrinkles in the clothes can be segmen- ted. As the value of h increases, the contours correspond- ing to low contrast edges disappear.

For comparison purposes, this image was also seg- mented using a mono-scale morphological gradient op- erator followed by thresholding and the first derivative of a Gaussian filter followed by thresholding. The structur- ing element used in the mono-scale morphological op- erator is a square of 3 x 3 pixels. The resulting gradient image is then thresholded in order to eliminating irrele- vant local minima, as done in reference (5,6). The segmentation results contain 2479, 1648 and 1261 re- gions when the threshold is equal to 8, 12 and 16, respectively, as shown in Fig. 7. One can see from this figure that the mono-scale gradient operator followed by thresholding leads to many small and irrelevant regions. The regions delimited by the wrinkles in the clothes are not well segmented. When the threshold is increased to 16, some of the irrelevant regions disappear, but the contour along the back of the person disappears as well.

The first derivative of a Gaussian filter was used in the comparison because it is an efficient approximation of Canny's edge detector °°) and was used in watershed- based image segmentation. (5) The support of the Gaus- sian filter is of 5 x 5 pixels. (aT) Let a(x,y) denote the image filtered by the Gaussian filter. The first derivative of the Ganssian filter is computed by

([a(x,y) - a(x + 1,y)] 2 + [a(x,y) - a (x ,y + 1)]2} 1/2.

Fig. 5. Image to be segmented.

A multiscale gradient algorithm for image segmentation using watersheds 2047

(a)

(b)

~=~ -

(c)

Fig. 6. Segmentation results using the new algorithms with different values of h: (a) h = 8, 09) h = 12, and (c) h = 16.

2048 D. WANG

(a)

(b)

(c)

Fig. 7. Segmentation results using the mono-scale morphological gradient operator followed by thresholding at different values h: (a) h 8, (b) h = 12, and (c) h 16.

A mnltiscale gradient algorithm for image segmentation using watersheds 2049

(a)

(b)

(c)

Fig. 8. Segmentation results using the first derivative of the Gaussian filter followed by thresholding at different values h: (a) h = 2, (b) h = 4, and (c) h = 6.

2050 D. WANG

(a)

(b)

(c)

Fig. 9. Region merging results, (a) with the new algorithms, (b) with the mono-scale morphological gradient operator followed by thresholding, (c) with the first derivative of the Gaussian filter followed by

thresholding.

A multiscale gradient algorithm for image segmentation using watersheds

Table 1. Computational times (s) of segmentation using different gradient algorithms

2051

Algorithm Filtering Gradient Watershed Region merging Total

New algorithms 5.35 2.92 0.27 2.38 10.99 Mono-scale gradient operator 5.43 0.37 0.30 159.80 165.95 First derivative of Gaussian filter 5.36 0.87 0.28 97.02 103.58

Fig. 8 shows the segmentation results when the threshold is set to 2, 4 and 6. These results contain 2087,1205 and 893 regions, respectively. Generally, Canny's edge de- tector and the first derivative of the Gaussian filter are very robust to noise. However, they are not effective for enhancing blurred edges because they are designed according to the step edge model.

Evidently, the segmentation results obtained by the mono-scale morphological gradient operator and the first derivative of the Gaussian filter cannot be directly used for any application. To get meaningful segmentations, a low threshold should be used and a region merging step has to be performed on the results of watershed trans- formation. However, to merge such a large number of regions will require intensive computation.

To compare computational costs, the three segmenta- tion procedures including region merging were tested on a Sun workstation ULTRA 1. In the test, one of the region merging criteria described on page 412 of reference (9) was adopted. The edge height along the common contour between two regions was computed using the method mentioned in Section 4. If the edge height at a contour pixel is lower than a predetermined threshold T, this pixel is considered as a weak edge pixel. The two regions are merged if w / l > 0, where I is the length of the common contour and w is the number of weak edge pixels along the contour. The order in which regions are merged strongly influences the results of region merging. (a4~. To get the best results, the two regions with highest w / l are merged, then the corresponding w / l values are updated accordingly. The process is repeated until no further regions can be merged. After this, every region smaller than 69 pixels (0.02% of the total image area) is merged to the region with which the w / l value is the greatest.

Table 1 reports the computational times for the image in Fig. 5 when h is set to 8 for the new algorithms, the thresholds are set to 8 and 2 for the mono-scale mor- phological gradient operator and the first derivative of the Gaussian filter, respectively. The parameters Tand 0 for region merging are equal to 12 and 1/3, respectively. From this table, it can be seen that watershed transfor- mation is extremely fast. The mono-scale morphological gradient operator and the first derivative of the Gaussian filter require less computational time than the multiscale gradient algorithm. However, region merging for the mono-scale operator has to deal with 2479 regions [shown in Fig. 7(a)], requiring 159,80 s of CPU time and that for the first derivative of the Gaussian filter requires 97.02 s. In contrast, region merging for the new algorithms deals with only 138 regions, requiring only 2.38 s. Hence, the entire segmentation procedure with the

mono-scale operator and the first derivative of the Gaus- sian filter are 10 times more costly than that with the new algorithms. In addition, the results of region merging for the mono-scale gradient operator (containing 137 re- gions) and the first derivative of the Gaussian filter (containing 163 regions) are not as good as that for the new algorithms (containing 70 regions) because of the disturbance of irrelevant regions, as shown in Fig. 9.

6. CONCLUSION

The performance of watershed-based image segmen- tation methods largely depends on the gradient algorithm used in the methods. With a conventional gradient op- erator, watershed transformation produces many irrele- vant regions. A region merging algorithm must be then employed to merge these irrelevant regions. To merge so many regions usually requires a long computational time which greatly reduces the speed of the entire segmenta- tion procedure. In this paper, we have presented a multi- scale gradient algorithm and an algorithm for eliminating small local minima in the resulting gradient images. The multiscale gradient algorithm responds effectively to both step edges and blurred edges. The algorithm for eliminating small local minima can efficiently remove irrelevant minima caused by noise and quantization error in the resulting gradient images. Hence, watershed trans- formation with the proposed algorithms results in mean- ingful segmentation, even without a region merging step. Experimental results show that the proposed algorithms can significantly reduce the computational cost of wa- tershed-based image segmentation methods while effi- ciently improving segmentation accuracy.

Acknowledgements--The author gratefully acknowledges Andr6 Vincent, Limin Wang, James Tam, Peter Haighton and Andr~ Mainguy for their help in this work. Thanks to the anonymous reviewers for their useful suggestions.

REFERENCES

1. R. M. Haralick and L. G. Shapiro, , Comput. Vision Graphics Image Process. 29, 100-132 (1985).

2. N. R. Pal and S. K. Pal, A review on image segmentation techniques, Pattern Recognition 26, 1277-1294 (1993).

3. S. Beucher, Segmentation d'image et morphologie mathe- matique, Ph.D. Dissertation, School of Mines, Paris, France (June 1990).

4. L. Vincent and P. Soille, Watersheds in digital spaces: An efficient algorithm based on immersion simulations, IEEE Trans. Pattern Analysis Mach. lntell. 13, 583-598 (1991).

5. K. Hafts, S. N. Efstratiadis, N. Maglaveras and C. Pappas, Hybrid image segmentation using watersheds, SP1E Proc. Visual Communications and Image Processing'96, 2727, 1140-1151, Orlando, Florida, U.S.A. (1996).

2052 D. WANG

6. R Salembier, Morphological mulfiscale segmentation for image coding, Signal Process. 38, 359-386 (1994).

7. L. Vincent, Morphological grayscale reconstruction in image analysis: Applications and efficient algorithms, IEEE Trans. Image Process. 2, 176-201 (1993).

8. M. W. Hansen and W. E. Higgins, Watershed-driven relaxation labeling for image segmentation, Proc. 1994 IEEE Int. Conf. Image Processing, 3, 460-464, Austin, Texas, U.S.A. (November 1994).

9. A. K. Jain, Fundamentals of Digital Image Processing. Prentice-Hall, New Jersey (1989).

10. J. Canny, A computational approach to edge detection, IEEE Trans. Pattern Anlysis Mach. Intell. 8, 679-698 (1986).

11. J. Shen and S. Castan, An optimal linear operator for step edge detection, CVGIP: Graphical Models Image Process. 54, 112-133 (1992).

12. J. Serra, Image analysis and Mathematical Morphology. Academic Press, London (1982).

13. R. M. Haralick, S. R. Steinberg and X. Zhuang, Image analysis using mathematical morphology, IEEE Trans. Pattern Analysis Mach. lntell. PAMI-9, 532-550 (1987).

14. D. Wang and C. Labit, Morphological spatio-temporal simplification for video image segmentation, Signal Process.: Image Commun. (to appear).

15. D. Cortez, P. Nunes, M. Menezes de Sequeira and F. Pereira, Image segmentation towards new image repre- sentation methods, Signal Process.: Image Commun. 6, 485-498 (1995).

16. D. Wang, V. Haese-Coat, A. Bruno and J. Ronsin, Some statistical properties of mathematical morphology, IEEE Trans. Signal Process. 43, 1955-1965 (1995).

17. P. J. Butt, Fast filter transforms for image processing, Comput. Graphics lmage Process. 16, 20-51 (1981).

About the Au tho r - -DEMIN WANG received the B.S. and M.S. degrees in Electrical Engineering from Shandong Polytechnic University, China, in 1982 and 1985, respectively, and the Ph.D. degree from the Institut National des Sciences Appliquees (INSA) de Rennes, France, in 1992. In 1985, he joined Shandong Polytechnic University where he last served as a Professor of Electrical and Computer Engineering from 1992 to 1993. He was a Visiting Researcher at the University of Sherbrooke, Canada, from 1993 to 1994 and a Visiting Professor at the Institut National de Recherche en Informatique et en Automatique (INRIA) at IRISA Rennes, France, from 1994 to 1995. Since 1996, he has been with the Communications Research Centre, Canada, as an Research Scientist. His research interests include image and video coding, image representation, texture analysis and non- linear filtering.