Ellipse detection and matching with uncertainty

Tim Ellis, Ahmed Abbood and Beatrice Brillault

This paper considers the fitting of ellipses to edge-based pixel data in digital images. It infers the ellipses as projected circles, viewed obliquely, and uses this infor- mation to constrain the position, viewpoint and scale of model objects projected into the image. A central theme of the work is the explicit use of uncertainty, which reflects observation errors in the image data, and which is proagated through the model matching stages, provid- ing a consistent and robust representation of errors.

Keywords: ellipse fitting, Kalman fitter, Mahalanobis distance, perceptual groupings, measurement error, model matching

INTRODUCTION

Model-based vision attempts to matcn image primitives (e.g. straight lines, corners, arcs) to 3D object repre- sentations, typically based on geometric models. One of the major problems in such matching lies in estimating the viewpoint of the image sensor with respect to the scene. When available, initial estimates of viewpoint are used to provide a 30 ==+ 20 trans- formation of the model, followed by verification of remaining image features. Alternatively, the image primitives are used to infer 3D structures in the data, which are directly matched with the model data’,2.

Simple image primitives tend to provide a poor source of data from which to estimate the viewpoint. Much recent work’ has used the detection of percep- tual groups in the image to enrich the data and constrain the range of viewpoints from which to initiate a match. As an example, parallel lines in the scene are broadly insensitive to viewpoint, if we can assume that the range of depths in the scene allow the associated objects to be treated as if under orthographic projec- tion (an assumption used in this work). In addition, this approach has been combined with a more realistic approach to matching, which employs rigorous models of uncertainty applied to the measurement data, and

Machine Vision Group, Centre for Information Engineering, City University. London, EClV OHB. UK

Puper received: IO October 1991; revised papers received: 20 January

1992

propagates these uncertainties through the matching process’,“.‘.

There has been considerable interest in recent years in the detection of ellipses in image dataSmX and the more general problem of fitting conic sections to scattered data”-’ ’ . In computer vision, ellipses are commonly interpreted as oblique projections of circular features in the image’ “. Such features are typically rare in scenes, and hence are excellent cues if they can be detected reliably’. A further advantage of circular features is that they constrain the viewpoint to a greater degree than simpler line features or groups of features (e.g. parallel lines). As such they can provide a more reliable and robust initial estimate of viewpoint.

As with parallel lines, ellipses can also be grouped in order to infer more complex underlying structures. West and RosinI grouped ellipses inferred to belong to planar surfaces (containing multiple circular features), cylindrical/conical objects (axis or revolution) and circular concentric features. Their method used the Hough paradigm to identify ellipses (projected circles) associated with these groupings.

In this paper we consider two aspects of the wider task of matching image to model primitives. Firstly, we extract ellipses from the image data and use them (singly or in groups) to estimate viewpoint and infer structures in the scene. Secondly, we match these structures to model structures. The feature extraction process, based on the Kalman filter, calculates the uncertainty associated with the ellipse fitting. This uncertainty is propagated throughout the model match- ing stage, in order to relate the uncertainty in the feature extraction to the location and recognition of the model object.

Initial results have been applied to suitable demon- stration images which indicate the performance of the ellipse detection algorithm on a variety of projected circles. In addition, examples of ellipse grouping are also presented.

ELLIPSE FITTING

Directly fitting ellipses to pixel edge data is computa- tionally expensive and prone to gross errors in all but the simplest images. A more practical alternative exploits connectivity of edges in the scene and initiates

0262-8856/92/005271-06 0 1992 Butterworth-Heinemann Ltd

VOl 10 no 5 jitne 1992 271

fitting on connected edges. Further efficiencies can be obtained by reducing connected edges to polygonal approximations, utilising only the line end-points for initial fits,

We treat ellipse fitting as a three stage process. Following application of a suitable edge detector (Marr-~ildreth, Canny, Shen’“, S-connected edge seg- ments are extracted from the image and straight line and arc data are generated5. Ellipses are initially fitted to the detected arc segments using a least-mean-square (LMS) error fitting to a general conic function, which maintains an estimate (uncertainty) of the fit through the covariance matrix. Tnitially fitting to the line data results in a more efficient calculation.

These initial fits may be improved by extending the arcs using existing edge connectivity information and more global information of unconnected line segments. To allow updating of the LMS algorithm, we use a recursive form, the Kalman filter. Initializing this‘” algorithm with the parameters from the initial LMS fit (and in particular, the covariance matrix) avoids numerical instabilities in the Kalman filter. In addition, the Kalman filter is updated with the original pixel edge data points, rather than the polygonal end-points, in order to improve the accuracy of fit.

We describe the ellipse by the normal parametric form for a conic section:

f(Xj, yi) = ax2 + bXy + C_V' + dX + ey t-f= 0 (1)

selecting the simple normalization of f= I, which merely precludes representation of ellipses which pass through the image origin (i.e. 0, 0) co-ordinate, see appendix 1). Hence, the ellipse is described by the vector x = (a b c d e 1) and the updating of the parameter vector and its covariance are given by14:

xi=x;_1 - (z; _ hfXi_, ) @- i hJ (r2 + h:S,+, h,

s, -_ ,w_ _ (Si-1 hXS;-, hi)’ r r I

cr2+h:S; ,hi

(2)

(31

where:

z=-(ax2fbxy+cy2+dx+ey+ 1) (4)

h’= (x2xyy2xy 1)’ (5)

cr2 = 4~2((~~+~y+~)~ + (bx+ cy+e)‘) (6)

and n2 is an estimte of Ihe spatial noise. for which we currently use a value of 0.1 (see Porril’).

The Kalman filter is initialized (x,), St,) with values calculated from the LMS method.

In extending the arcs, we use a Mahalanobis distance (MD) measure to select candidate line segments of suitable orientation and location (with respect to the current ellipse fit). The MD test weights the difference between the current fit and a candidate feature (line segment) by the current estimate of the uncertainty and hence provide a metric for selecting features with a given statistical sampling. As shown in Figure 1, we use a mid-point representation for the polygonally approxi- mated line segments, estimating the tangent to the data

Fitted ellipse

Line segment, /

Figure 1. (a) Criteria for mutching global line data to ellipse; fb) determinatioi~ of the x.y-coordinate of the mid-point, Located on the pixei data

from the direction of the line. To minimize errors of fit we select the x-y coordinate of the mid-point corres- ponding to the equivalent position on the edge1 data for that segment. This is determined by the intersection of the projection of the tangent normal (from the line segment midpoint) with the pixel data (see Figure lb). On accepting the segment, the Kalman filter is again updated with the pixel edge data associated with the line segment.

The MD test is performed on ~(~ g) which is equal to M’C&’ M < t (see Appendix 2). and is given by:

f 2(xu Yi) + g2,, (u, v>

c; c; <t

where f(xi, yi) is defined as in equatidn (l), a; is the variance off with respect to the euclidean co-ordinates (x,, yj), and the ellipse parameters, ai is the variance of g with respect to the euclidean co-ordinates, the ellipse parameters and the tangential directions (u, v) and:

gt,,(u, v)=2axu+b(yu+xv)+2cyv+du+ev (8)

We select a value for the threshold t (from a x2 distribution with two degrees of freedom) to provide a suitable trade-off between accepting a high proportion of good features, and rejecting poor ones. In noisier images, the threshold can be reduced to make the test more selective.

A disadvantage of only fitting connected edge data is that connectivity of edges is not assured, due to noise and other interfering effects. Hence, the final stage of our fitting process searches for global ~unconnected) line segments in the image that wiil improve the ellipse fit using the same Mahalanobis distance test given above.

Although the general conic (equation (1)) describes the conic family of ellipses, parabolas and hyperbola, prior to updating the Kalman filter we reject data which result in non-elliptical parameters. This is achieved by ensuring that b2 < 4ac before accepting the updating of the Kalman filter. Otherwise, the data is rejected, and the search for further possible line segments continues.

272

MODEL FITTING

For simplicity, we assume that the projection is orthographic. (Under perspective distortion the image curve from a circular feature in the scene remains elliptical, though there will be an error in the para- meters. In practical terms, for the real scenes we are analysing, perspective does not have a significant effect). Using parameters derived from the equation of the ellipse, we can infer the radius of the projected circle (the major axis), the angle of the oblique viewpoint (cosine (minor/major), .the tilt), and the orientation of the plane (with respect to the viewpoint) in which the corresponding 3D circular feature lies (see Figure 2).

Following the detection of multiple ellipses, grouping of the ellipses can proceed. We avoid the use of the Hough transform for grouping in favour of the Mahalanobis distance/Kalman filter combination.

The model structures in this work are solids of revolution, typified in our case by pipes. The detection of these structues is implemented as a two stage process, grouping appropriate sets of ellipses. Initially the ellipses are grouped on the basis of similar tilt (T) and orientation (0). In 3D, such groupings may be attributed to co-planar or parallel-planar sets. For each ellipse, estimates of the tilt angle and orientation are calculated from the conic parameters. The MD test is performed on P( 7, H), and is equal to PC’;’ P<t (with two degrees of freedom), where C,, is the covariance matrix associated with P. and is given by:

The test selects appropriate ellipses in order to update a Kalman filter, which maintains the current estimate of the plane (ax+ by + cz = 0). The second grouping stage detects the axis of revolution by fitting a straight line (ax + by + c = 0) through the centroids of the parallel planar ellipses, ensuring that ellipses whose centres are co-linear are also co-directional with the current estimte for the axis. (Since the ellipses we have detected are assumed to be from circular features, this implies that the cross-section of the solid of revolution is perpendicular to its axis). For an axis of revolution, R(7, 0. D), the MD test is R’C,’ R < t (with three degrees of freedom), where D is the perpendicular distance between the centroid of the candidate ellipse and the current axis of revolution (as shown in Figure 3), and CR is the covariance of R.

The Mahalanobis distance/Kalman filter combina-

Figure 3. Ellipses are grouped along a plaus- ible axis of revolution. Ellipses distant from this axis will be rejected

tion allows the grouping process to maintian the uncertainty associated with the ellipse estimation, and to propagate this uncertainty to the normal of the plane of the grouped ellipses and the axis of revolution.

The second part of our task is to match the detected (projected) circles with similar features in a 3D model. The inferred circles in the image provide position, viewpoint and scale information which can be directly used to map the model description via a 3 D =+ 20 transformation onto the image feature data. In the case of pipes, this entails locating the parallel (or converg- ing, if perspective distortion is significant) edges of the pipe cylinder. Using the axis of revolution and the diameter of the pipe (estimated as the major axis of the ellipse and corresponding to the radius of the obliquely- viewed circular cross-section) we can derive a further MD test to identify suitable (pairs of) straight lines in the image, to verify their occluding boundary in the model.

Our intention is to find the rotation and translational parameters for the model-to-image transformation using a Kalman filter. The advantage of the iterative operation of the Kalman filter is that is allows the combination of evidence from different types of features (e.g. corners. circles, edges) in a consistent manner. Again, uncertainties associated with the measurement data are propagated through the trans- formation in order to generate projected model features which reflect the uncertainty in the measure- ment. As before, the fitting or matching process uses the Mahalanobis distance for selecting appropriate feature matches and a Kalman filter for refining the parameters of the viewpoint transformation.

RESULTS

Figure 4 shows the detection and fitting of ellipses on a scene rich in projected circles. Edges are detected with a modified version of Shen’s algorithm’3.‘. and are shown in Figure 4b. Considerable noise in the detected image is derived in part from shadows, and from a high sensitivity setting for the detection algorithm. Follow- ing polygonal approximation and arc detection (detected arcs are shown in Figure 4~). ellipses are constructed as detailed above. Figure 4d shows the detected ellipses as estimated from the original line and arc data, and then augmented with locally connected and global line segments. Ellipses which are supported

vol IO no 5 june 1992 273

Figure 4. Grey arcs, detected grouped ellipses

image, edges, ellipses and

b

d i

by only a small length of edge data in the image are ignored, since such ellipses are typically of high uncertainty, and will exhibit a poor fit. Ellipses have been grouped for several axes of revolution, though poorer detection performance on the toy cone has failed to group the resulting ellipses for that object. The axes and associated ellipses are shown overlaid onto the original image in Figure 4e.

CONCLUSION AND FURTHER WORK

It is well known that ellipse fitting is subject to a high uncertainty” and it is essential to take this into account for a 3D interpretation of such features in images. A combination of the Mahalanobis distance and Kalman filter, used throughout the interpretation process, allows this to be achieved in a consistent way.

Figure 5 shows the same processing steps applied to a The accuracy of the ellipse fitting is increased by real scene (Figure 5a), taken from an industrial grouping primitive image features (i.e. line segments) environment. As can be seen, the image is very which may be part of the same ellipse. Multiple ellipses complex but some useful ellipses are still detected, and may then be grouped into parallel planar and co-axial a grouping has been constructed. Figure 5b shows the sets, allowing the inference of 3D structures (e.g. solids edges detected, and Figure 5c shows the ellipses. The of revolution) in the scene. As uncertainty has been final grouping is shown in Figure 5d. propagated to these structures, this may be taken into

274 image and vision computing

I1

c

J

, \

t111 l-~ -I . . . . ~IIIIII~- C ~ . ~ "--~-k~l%,~

.~--. ' ~ : ~ - ~ ~ ~ ~ - ~ - ~ _ ~ u - J ' l /

i_ ~_. - - ~ : ~ ..... "~--'~'~ *.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I/"/5"~-~ . . . . . ~I" I//" ~ j l #

d

Figure 5. Grev level data, edges, and detected and grouped ellipses in a real scene

b

account during subsequent matching processes with a model.

One disadvantage of applying the ellipse detection to solids of revolution lies in the fact that, inevitably, a maximum of 50 per cent of the curve will be visible in the image, due to self-occlusion (unless the object is transparent). [n contrast, locating more complete circular features (e.g. a clock on a wall) would provide more reliable data for the fitting process.

Although it is efficient to make use of arc detection to suggest possible ellipses in the image, this limits the detection of some ellipses, and has a tendancy to miss important elliptical edge data. This is one aspect of the problem to which we are currently giving attention.

The problem in using the MD test is that when the uncertainty is high, in order not to select too many false features the threshold must be reduced, and it is not simple to find an optimal value. Current work is in hand to replace the MD test with a liklihood ratio (LR) test to determine this optimal threshold.

In conclusion, dealing in a consistent way with the measurement error, and propogating uncertainty infor- mation through the low and intermediate processing stages provide robust support for matching against high-level symbolic object descriptions.

REFERENCES

1 Brillault, B A Probabalistic Approach to 3D Interpretation of Monocular Images, PhD Thesis, City University (in preparation)

2 Ayaehe, N and Faugeras, O D 'Building registrar-

ing and fusing noisy visual maps'. Proc. 1st ICCV London, UK (1987) pp 73-82

3 Lowe, D G Perceptual Organisation and Visual Recognition, Kluwer, Netherlands (1985)

4 Grimson, W E L and Huttenioeher, D P 'Percep- tual grouping of circular arcs under projection', Proc. BMVCgO, Oxford, UK (1990) pp 379-382

5 Rosin, P L and West, G A W 'Segmenting curves into elliptic arcs and straight lines'. Proc. 3rd ICCV, Osaka, Japan (1990) pp 75 78

6 Porrill, J "Fitting ellipses and predicting confi- dence envelopes using a bias corrected Kalman filter', Image & Vision Comput., Vol 8 No 1 (1990) pp 37 41

7 Maseiangelo, S '3-D cues from a single view: detection of elliptical arcs and model-based pers- pective backprojection' . Pro:'. BMVCgO, Oxford, U K (1990) pp 223-;!28

8 Safaee-Rad, R, Tehoukanov, I, Benhabib, B and Smith, K C "Accurate parameter estimation of quadratic curves from grey-level images', Comput, Vision, Graph & Image Process., Vol 54 (1 r ) pp 259-274

9 Bookstein, F L 'Fitting conic sections to scattered data' , Comput. Vision, Graph & hnage Process.. Vol 9 (1991) pp 56-71

10 Sampson, P "Fitting conic sections to very scattered data' , Compit. Vision, Graph & hnage Process., Vol 19 (1982) pp 97-1(18

11 Cooper, D B and Yalabik, N "On the computa- tional cost of approximating and rccognising noise- perturbed straight lines and quadratic arcs in thc

vol 10 no 5june 1992 275

12

13

14

15

plane’, IEEE Trans. Cornput., Vol 25 (1976) pp 1020-1032 Rosin, P L and West, G A W ‘Perceptual grouping of circular arcs under projection’, Proc. ~~VC9~, Oxford, UK (1990) pp 379-382 Shen, J and Castan, S ‘An optimal linear operator for edge detection’, Proc. CVPR ‘86 (June 1986) Rosin, P L A note on the least squares fitting of ellipses, Technical Report No. 9, Cognitive Systems Group, Curtin University of Technology, Western Australia KaIman, R E ‘A new approach to linear filtering and prediction problems’, Trans. AS~E J. Basic Eng. (March 1960) pp 35-45

APPENDIX 1 : A NOTE ON NORMALIZATION

The least squares curve fitting process determines the optimum fit of the data to some parametric curve (e.g. equation (1)) by minimizing the sum of the squares of some appropriate metric between the data and the parametric function. An obvious choice for such a metric is the Euclidean distance, though in this case it can be problematic to calculate, and computationalIy simpler metrics, commonly based on the residue” are used. This residue is approximately proportional to the ratio of the square of the distance from the centre of the ellipse to the data point, divided by the square of the distance (along the same direction) from the centre to the conic. Using this residue introduces an error (or bias) into the metric, which results in an overestimate of the eccentricity of the ellipse. A variety of methods have been employed to correct for this”.s which tend to reduce the overall computational efficiency.

Normalization is used to determine unique solutions and to avoid trivial ones, of the parametric function (e.g. for equation (1)) of the form a = b = c= d=e =f=O. Whilst the simplest normalization which preserves the conic function uses f = 1 (and hence eliminating this from the equation merely requires dividing the other coefficients by unity), it has been shown” that as well as excluding as solutions, all ellipses that pass through the origin (0, 0), the fit will not be invariant to Euclidean motions (i.e. rotations, trans- lations and scaling), generting different coefficients for ellipses subject to such motions. However, the use of f= 1 is computationally simpler, and better con- ditioned than a + c = 1.

Using empirical data, RosinI has suggested that by re-sampling the pixel data associated with the arc using a sampling interval inversely proportional to the curva- ture, that this tends to reduce the bias by utilising more samples from the high curvature region of the pixel arc data. (This re-sampling is conveniently provided by a

line segmentation algorithmn developed by Rosin5 and adapted from Lowe, which generates a poIygona1 approximation to the edge1 data and has been used in the present work.

APPENDIX 2: MD EQUATIONS

The following briefly defines the equations associated with the Mahalanobis Distance test for selecting suitable lines for updating the Kalman Filter of the ellipse fitting.

The ellipse equation f(xi, yi) is defined as:

ax2+bxy+cy’+dx-key+f=0. (1)

The tangent equation g,,,(u, v) is found by differen- tiating the ellipse equation with respect to angle, and equating dxld0 = u, dyld0 = v, we find:

2axu f b(yu +xv) + 2cyv + du + ev = 0. (2)

The variance associated with Equation (1) and (2) is determined as follows:

cr;= F’C,F+ G’C,,G

cr;= H’C,,H+Z’C,,Z+ G’C,,,G

where C,, C,, are the covariance of the conic’s parameters, C,, is the covariance of the measurement noise, C,, is the covariance of the directional noise, and:

F’ = (x2 xy y2 x y 1)

G’= (2ax+by+d 2cyi-b.r+e)

N’ = (2XU (yu +xv) 2yv u v)

I’ = (2au+bv 2cv+bu)

The MD test (Equation 7) can also be described in a vector form as:

where:

M’ = (f(xi, yi> gxv y(u, v>

276 image and vision computing