MATCHING CANNY EDGELS TO COMPUTE THE PRINCIPALCOMPONENTS OF OPTIC FLOW

D A Castelow, D W Murray, G L Scott*t and B F Buxton

GEC Research Ltd.,Hirst Research Centre,

East Lane,Wembley,HA9 7PP.

ABSTRACT

A relaxation algorithm for the computation of opticflow at edge elements (edgels) is presented. Flow isestimated only at intensity edges of the image. Edge ele-ments, extracted from an intensity image, are used as thebasis for the algorithm. A matching strength or weightsurface is computed around each edgel and neighbourhoodsupport obtained to enhance the matching strength. Aprincipal moments method is used to determine the flowfrom this weight surface.

The output of the algorithm is, for each edgel, a pairof orthogonal components of the estimate of the flow.Associated with each component is a confidence measure.

Examples of the output of the algorithm are given,and tests of its accuracy are discussed.

I. BACKGROUND

The aim of the ISOR system, described in the paperby Murray et all. is to obtain a three-dimensionalrepresentation of a polyhedral object from an imagesequence and then to recognise the object using 3-d modelmatching.

The algorithm described in this paper was developedto provide estimates of the optic flow near edges in asequence of images. This method forms an essential pre-cursor to the depth from flow (dff) algorithm2 described inMurray et a/1, and the subsequent 3-d edge matching3

described in the paper by Murray and Cook4. One of thefirst stages in this process is to estimate the visual motionat intensity edges in the image.

Edges are extracted from raw images using an imple-mentation5' 6 of the Canny edge detector7' 8. This generatesedge elements (edgels) which contain the following infor-mation;

(i) edge location to sub- pixel accuracy (approximately0.03 pixels on clean step edges),

(ii) edge orientation (accurate to approximately 1.5° on anideal image),

(iii) edge strength: the magnitude of the gradient at thepoint of inflection.

* G L Scott was a GEC Research Associate at the University ofSussex, Falmer, Brighton BN1 9RH, UK at the time this work wascarried out.

t Current address - Dept Engineering Science, University of Ox-ford, Parks Rd, Oxford OX1 3PJThis work was undertaken as part of the Alvey programme DCBS 013,"Spatio- temporal processing and optic flow for computer vision."

The edge maps are refined using the hyster pro-gramme9 which removes edgels of low strength by thres-holding and links edges of high strength by hysteresis usingthe method of Canny7.

These refined edgels are used by the mcs3 pro-

193

gramme10, the algorithm for which is described here.

II. AN OVERVIEW

Edgels are used to compute optic flow for three rea-sons.

(i) For the structure from motion calculations and subse-quent processing we require estimates of flow only atthe edges of polyhedral objects.

(ii) For the images we are considering, edgels are sparserthan pixels, leading to reduced computation times.

(iii) Edgels are more distinctive than pixels, having both astrength and a direction. This information is used inthe computation of the initial matching strength.

Along a straight line segment the match of an indivi-dual edgel is not unique. Only the displacement or motionperpendicular to the edge can be obtained. This is usuallyrefered to as the "aperture problem." The principalmoments decomposition used in the algorithm describedhere is used to capture this information in an unbiasedmanner, as suggested by Scott11. Although individualedgels do not encode any shape (curvature) information,both components of the motion may be obtained reliably ata distinctive shape such as a corner. This is for two rea-sons.

(a) The orientation of individual edgels near a corner maybe sufficiently distinct to give a single very strong ini-tial match.

(b) The support for each match that is sought from neigh-bouring matches in the relaxation stage of the algo-rithm may lead to a single strong match for an edgelnear the distinctive shape.

The algorithm may be described by the followingsteps.

(i) The edges are matched against neighbouring edgels inthe forward and backward frames to produce an initialweight surface. Matching to both the forward and thebackward frames ensures that the solutions sought willbe symmetric with respect to time inversion.

(ii) Support for each match (point on weight surface) isobtained from neighbouring edgels.

(iii) The resulting matching strengths are used to weightthe least squares estimate of the visual motion oroptic flow. AVC 1987 doi:10.5244/C.1.27

The least-squares estimate of the velocity vector isfound by computing the mean displacement of the edgelfrom the weight surface and projecting this onto the princi-pal axes of the scatter matrix. These define the com-ponents of the motion that can be estimated with greatestand least reliability11'12. The eigenvalues of the scattermatrix are used to compute the confidences of the motionestimates. This ensures that we obtain the appropriateinformation along an extended edge segment and at a dis-tinctive shape such as a corner. It also ensures that theestimates are not biased by the choice of image co-ordinateaxes.

These three stages are described in the next three sec-

The initial matching strength of edgel i to an edgel j in theforward frame is

tions.

III. INITIAL WEIGHTS

As mentioned above, three frames of edgels are usedand are referred to as the central, backward and forwardframes. An estimate of the visual motion is obtained foreach edgel in the central frame.

Around each edgel, i, in the central frame is a neigh-bourhood, Nt, which contains all edgels that lie within adistance, Ljv, of the edgel, i. Around the same position inthe forward and backward frames are constructed twolarger locales, F,- and 5,- with radii Lp and Lg respectively.Figure 1 illustrates the neighbourhood and forward localearound an edgel. Each edgel, j , in the forward and back-ward frames has associated with it a neighbourhood set,Nj.

Computing the matching strengths of an edgel to theforward and backward frames uses the same method. Wedescribe the computation for the forward frame.

Initial matches are made between the edgel, i, and itsneighbours in the forward frame, F,-.

Figure 1. Edgels I and j and their neighbourhoodsare shown. Edgel i is in the central frame, edgel jin i's locale, F,- or 5,- in the forward or backwardframe (here forward). Edgel ZieJV;, is a neighbourof j , and keNj is a neighbour of j (in the forwardframe).

where 6<a = a,- - a, modulo 2n and 5J;J- = st - Sj are,respectively,the differences in orientation, a, and strength, s, of thepair of edgels, i and j . The range of edge strengths ororientations which lead to high initial match strengths arespecified by the "widths", a, and aa respectively. Thisheuristic gives a high matching strength between thoseedgels which have similar orientation and edge strength.

This function implies that the best matches are oneswhere there is no rotation of the edgel and no change in itsstrength between successive frames. Typical values of theparameter, fja, are of order 5° to 10°. These values arechosen because of the variability in the directions of edgelsas determined by the edge detection stage and to allowsome rotation of the whole edge feature from frame toframe. For example, if we ignore the variability in the com-puted edge orientation and if our image sequence camefrom successive frames of a video, then specifying a valueof o a = 5° would only have a significant effect for objectsrotating at a speed of a third of a revolution per second orfaster.

IV. NEIGHBOURHOOD SUPPORT

Once the initial matching strengths, p,^°\ have beenfound, support for every match is then sought from theneighbours of i. The matching strength is updated accord-ing to

\c(i,j;h ,

(IV.l)

where Nj is the neighbourhood of j in the forward frameand n(Nj) is the number of edgels in A',. The compatibilityfunction, c, is based on differences in relative displace-ment:

(Ar;A)

(Ar,J2+(Ar,7-Arw)2

where

A r l 7 = Tj- r ,

(IV .2)

(IV.3)

and r, is the position in the image of edgel i. The deriva-tion for the backward frame is similar, with the locale F;

replaced by £,-.

This scheme may be iterated, allowing good matchesto be enhanced relative to bad matches by the support ofneighbours. This has been tested, and is discussed furtherin section VIII.

V. MATCHING SURFACES AND THE LEASTSQUARES CALCULATION

The least squares method used to determine the dis-placement at each pixel (the optic flow) requires weightedsums of displacements.

The algorithm is made symmetric with respect to for-ward and backward frames by associating displacementsAr,-; with p,j1J for forward matches i—*jeFj but displace-ments -Arif with pip for backward matches i->/efi ; .

194

First the mean displacement, < Ar,> , is found:

Ar,-> =

jeF.

(V.I)

Deviations from the mean,

8r0- = Ar,7 - < Ar,-> , (V.2)

are then used to construct the scatter matrix, M :

M = < " ) '

jeF.

The error term is

feB.P<")'/(-8rir)(-8r,7)r(.V.3)

Ez= vTMv (V.4)

to be minimised over unit vectors, v. The minimum is Xj,the smallest eigenvalue of M. The corresponding eigenvec-tor, Vj, defines the direction along which we have highestconfidence in our estimate of the optic flow at the edgel i,< 8r,-> . The estimate is projected into the co-ordinate sys-tem defined by the two eigenvectors, V] and v2. Theconfidence measures in these components of the estimateare the inverses of the corresponding eigenvalues.

VI. A COMPARISON WITH OTHERMATCHING TECHNIQUES

In the computation of the weight surface, an edgel canmatch, with various probabilities, to many edgels in thesubsequent or preceding frames. It may not be possible todecide upon a unique match when the edgel belongs to anextended straight edge segment. For this reason one-to-oneor many-to-one matches cannot be insisted upon. Evenwith the neighbourhood support, it can take very manyiterations of the support algorithm, (IV.1), for the uniqueinformation from corners and other distinctive parts of anedge to diffuse to the middle of a straight portion.

One-to-one matching constraints are applied in thematching algorithms of Ibison and Zapalowski13, Pollard,Mayhew and Frisby (PMF)14, and Lloyd15.

These algorithms, however, are quite different to ours.

The PMF algorithm and that of Lloyd are designed tofind unique matches in a pair of stereo images. Eventhough they match edge elements as here, one-to-onematching can be insisted upon because in binocular stereothe camera geometry is usually known accurately from cali-bration of the system. The epipolar lines are thereforeknown and an edgel can only match to another edgel onthe same epipolar (usually chosen to be a raster orarranged to be so after rectification of the images). Ambi-guous matches along an extended edge segment do nottherefore arise unless the edge is nearly parallel to the epi-polar, when it leads to the well known "horizontal edgeproblem."

The relaxation labelling scheme of Ibison andZapalowski13 matches distinct feature points from oneframe to another. It therefore does not suffer from theaperture problem associated with the ambiguity of match-ing edgels along a line and hence can utilize a constraintinsisting on at most one-to-one matching, and can thusavoid the aperture problem, and hence insist upon one toone matching.

Instead of trying to solve the matching problem, thealgorithm described here uses one to many matches to con-struct a weight surface, the mean and principal axes of

which are identified using the least squares techniquedescribed in section 5.

VII. ACCURACY OF THE ALGORITHM

Figures 2 to 14 are the results of applying the methodto three simple test image sequences.

The first test was on the motion of some simpleplanar image shapes. One image from the sequence used isshown in Figure 2. Each of the shapes was moved onepixel in both the vertical and horizontal directions betweenframes. Edges were extracted using the Canny edge detec-tor5. The edge map obtained is shown in Figure 3. Theestimate of the flow at each edgel is shown in Figures 4 to6. In Figure 4 the estimate of the full flow is plotted. Nei-ther component of the flow has been thresholded. Comparethis with the estimate of the major component of the flow,shown in Figure 5.

In all these diagrams, the estimated flow is not drawnto scale.

Figure 2. The image used in the accuracy experi-ments.

r

y///////y

r ^

J

Figure 3. The edge map of Figure 2, produced usingthe Canny edge detector. Parameters were c = 1.0,thresholds 6 and 2.

195

. \ \ \ \ \ \ WWWV

Figure 4. The estimate of the full flow from thegeometric shapes image. The neighbourhood supportregion was of radius 3.0 pixels, the locale size was 5.0pixels. Matching strength parameters were oa = 10°,og = 10. No thresholding was applied to the com-ponents of the estimate. A single iteration of equa-tion (IV.l) was used to obtain the neighbourhoodsupport.

\ \

Figure 5. The estimate of the major component ofthe flow for the shapes image. Parameters as in Fig-ure 4.

At the corners, the direction of the major axis is notperpendicular to the edge, partly because of the change inthe estimated orientation of the edgels (by the Cannyoperator), but mainly due to the support obtained from ad-jacent vertical edgels. Hence the major component esti-mates show a deviation towards the true motion.

At A in Figure 5, the estimate of the major com-ponent is almost zero. In addition, this estimate has amoderate weight (0.44). The minor component estimatehas a weight of 0.22. The full flow estimate at this point isthe worst one in the figure. It was (-0.75,-0.53)pixels/frame, which must be compared to the veridicalmotion of (-1.0,-1.0).

v \ U i » . . . . » « u ; J J jy

,, I 11 i * \ I I U '

Figure 6. The minor component of the flow vectorfrom the shapes image. To show these clearly theyhave been rotated by 90°. Parameters as in Figure 4.

q

. \ T T T T T T ! T T T T T T M / -V <>

•s.

9

P

^ H T T T T T T T T T T T T M \ ^

Figure 7. A map of the confidences associated withthe minor component of the flow estimate, plottedalong the major component axes. Parameters as inFigure 4.

To explain why this estimate is so poor in comparisonwith those for other edgels, note that this is the only pointwhere the motion is approximately parallel to the edgeldirection. In addition, the edgel below and to its right isoriented similarly. As a result, a strong match will bemade from edgel A not only to the translation of itself butalso to the translation of this neighbour. Because themotion is almost parallel to the edgel, this reduces the esti-mate (if the two matches were of identical strength, theestimate of the flow would be (-0.5,-0.5)).

Along the horizontal line of edges at the top of thesquare, the estimates of the major components are allwithin 2% of the correct value (-1.0 pixels/frame), with themajority of the results within 0.1%. The major componentestimates are best at the centre of the edge, where there isno influence from the corners.

Along straight edges the minor component has a lowconfidence associated with it. This is shown graphically inFigure 7.

196

Figure 8. A synthetic edge map for an idealisedcorner feature.

Figure 10. The estimate of the flow for pure transla-tion of the corner feature for Figure 8. The cornerwas moved diagonally, as described in the text. Otherparameters as for Figure 9.

Figure 9. The major component of the flow estimatefor a pure rotation of the corner feature in of Figure8. The rotation was 2° per frame about the apex ofthe corner. Lw = 5., Lc = i» = 3., a = 5°, a = 5. 1iteration of (IV.1).

Further tests have been carried out using sythetic edgemaps. These edges are not the output of an edge detector,but are drawn to sub-pixel accuracy. The motion of eachextended edge in the images may be specified to sub-pixelaccuracy. This was done to show the quality of the resultswhen edges are moved distances which are not an integernumber of pixels.

These experiments are of particular relevence in test-ing the response of the algorithm near a rotating feature.The edges used are shown in Figure 8, and the two esti-mates of the optic flow shown in Figures 9 and 10.

In Figure 9, the motion of the corner was a pure rota-tion about the point of intersection of the two line seg-ments. Far from the corner the estimates of the velocity

are correct, but near the corner they are systematicallyaltered. All the minor components of the optic flow arebadly estimated, as expected on such a featureless image.

The experiments showed that because of the way thematching surface is used to compute the mean displace-ment, < Ar,-> , the algorithm very badly estimates themotion of points close to a high curvature corner which isalso the centre of rotation. However, the occurrence of acorner which has high curvature at a point which appearsto be a centre of rotation is a rare event and hence thisapparent defect of the algorithm is not thought to besignificant. Any possible problem is further alleviatedbecause we are using edge elements from a Canny edgedetector7' 8 which reduces the curvature of corners.

To demonstrate that this error is indeed the result of afortuitous coincidence of the corner and centre of rotation,Figure 10 shows the estimate of the motion obtained whenthe corner feature of Figure 8 was translated, rather thanbeing rotated. The motion of the corner was (0.5,0.3)pixels/frame. As can be seen, the estimates of the flow aregood. The accuracy may be assessed by noting that themajor component of the velocity estimate for edgelsfurther than 2 pixels away from the corner are within 1%of the veridical motion. Close to the corner the estimatesare reduced, to about 50% of the true velocity, but thedirections of the estimates approach that of the veridicalmotion.

The most important question to be asked of the algo-rithm is whether the accuracy achieved is good enough forthe purpose for which it was devised.

This computation of optic flow from the edge map ofan image was undertaken in order to determine the struc-ture and motion of a polyhedral object, and to recognisethe object by matching the 3-d representation of its shapeso obtained to models stored in the computer memory.Using the algorithm described here to compute the opticflow, a balsa wood chipped block, identical in shape to theblock of Figure 11, was recognised and its motionestimated correctly, as is described in the paper by Murrayet all. in this conference.

197

(a) A single iteration of equation (IV.1) was usedto obtain the neighbourhood support.

Figure 11. A photograph of the block image used inthe experiments concerning the stability of the itera-tion scheme.

(b) Five iterations of equation (IV.1).

Figure 12. A thresholded edge map from the imagein Figure 11. The AIVRU implentation of Canny wasused, with a smoothing Gaussian of width 2.0 pixels.The image has been thresholded with thresholds 2.0and 6.0. See the documentation of canny5 and hy-ster9.

VIII. ITERATIONS

We have looked at the effects of iterating the algo-rithm (IV.1) before computing the optic flow. In this casewe applied the algorithm to the image of the chipped block,Figure 11. This image was synthesised using the construc-tive solid body modeller, WINSOM16.The motion of the block in three dimensions was vertical,with no rotation. Note that this motion is not parallel toany of the edges in the image. The block was moved sothat the displacement of its image was approximately 1% ofthe image size (2.56 pixels) between successive frames.

The edge map is shown in Figure 12. The "Cannyedgels" were produced using the Canny operator5 andrefined using Canny's hysteresis algorithm9.

(c) Fifty iterations of equation (IV.1).

Figure 13. The major component of the flow es-timated from a sequence of images of the block, Fig-ure 11. The neighbourhood support region was ofradius 3.0 pixels, the locale size was 10.0 pixels. Nothresholding was applied to the components of theestimate.

The major component of the estimates of the flow areshown in Figure 13, after 1, 5 and 50 iterations of equation(IV.1). Apart from a few points near vertices, these resultsare in essence the same. The effect of iterating the algo-rithm has not been to alter the major components of theestimate significantly. The full flow estimates are altered asthe neighbourhood support resolves ambiguities in thematches, reflected in the minor components of the flow.This can be seen in Figure 14, where the full flow is plot-ted ater 1, 5 and 50 iterations.

198

(a) One iteration of equation (IV.1).

(b) Five iterations of equation (IV.1).

Fifty iterations of equation (IV.1).

Figure 14. The full flow estimate from a sequence ofimages of the block, Figure 11. Other details as inFigure 13.

The initial effect of iterating equation (IV.1) is tosmooth the full flow estimates. This is particularly evidentalong the top edge of the triangular facet.

After 50 iterations the estimates are much smoother,but there is a definite bias. For example, in some places,the effect of iterating the algorithm (IV.1) is to decreasethe correctness of the full flow estimate. The regionmarked B in Figure 14(c), which is close to a vertex, is anarea of this type. Because there are no good matches to anedgels' left, the least squares solution is biased to the right.

In other places, the full flow estimates approach theveridical motion (vertical). An example of this is seen inthe region marked C in Figure 14(c). The reason for thisregion of good estimates of the full flow is probably due toa distinct edgel, associated with a discontinuity in the origi-

nal image and to a lack of anti-aliasing of the diagonaledge.

IX. POSSIBLE IMPROVEMENTSAND ADDITIONAL TESTS

The algorithm as described here has performedimpressively to date and met the (very) stringent require-ments imposed by the subsequent structure-from-motioncalculations carried out in the ISOR system. However, wenote the following points where improvements might bemade or additional tests carried out.

(i) The matching strategy does not include the disparitygradient limit14, despite the use of pairs of matches.Inclusion of this limit could reduce the weight givento false matches.

(ii) Multiplicative updating of the matching strengths ofthe form used by Ibison and Zapalowski13 could beused in place of the updating scheme (equation IV.1).

(iii) The algorithm in use10 already includes provision tothreshold the major and minor components of theflow, but this aspect of the programme has not beeninvestigated.

(iv) The matching function described gives equal weightfor matches to edge elements of equal strength andorientation, and does not take any account of theirlength. A geometric factor could be added to the ini-tial weight function, equation (III.l), to take accountof this. However, this might reduce the discrimina-tion of the algorithm where the edge element is to bematched to a slightly curved line.

(v) The algorithm has not been extensively tested onhighly textured scenes, where an edgel might matchnot only to many edgels on a line in the subsequent orprevious frames, but also to edgels on nearby edges.However, initial experiments on images of a texturedwooden block suggest that the optic flow may beestimated sufficiently well that the orientation of theplanes may be determined to reasonable accuracyusing a planar facet algorithm17. The motion of theblock was given, and the three visable facets of theblock were hand segmented. Two of the three planeswere at right angles, and their relative orientation wasestimated as 94°.

X. CONCLUSIONS

We have described a method for determining opticflow from a sequence of images, and indicated the qualityof the results obtainable.

The full flow estimates are good at feature points inthe edge image, such as corners, while along extendedstraight edges the major component of the flow is the onlysignificant contributor to the estimate.

We have experimented with iterating the neighbour-hood support function, equation (IV.l). The increase insupport for particular matches seems not to alter the majorcomponent of the flow on a straight edge segment, whichremains perpendicular to the edge. The full flow estimateis altered. In the image studied in detail, the effect isalmost always to move the full flow estimate towards theveridical motion.

199

ACKNOWLEDGEMENTS

The authors thank the AIVRU at Sheffield Universityand IBM UK, GEC Research's Alvey partners on IKBSProject 025 for use of the "Canny" edge detector and theWINSOM CSG body modeller, respectively.

REFERENCES

1. D W Murray, D A Castelow, and B F Buxton, "Froman Image Sequence to a Recognised PolyhedralModel," Submitted to AVC '87, 1987.

2. D W Murray, "Dff - depth from optic flow," inISOR User Documentation, ed. D A Castelow, GECResearch Ltd, Hirst Research Centre, Wembley, 1987.In preparation.

3. D W Murray, "Edgeprune - edge interpretation treepruner," in 75OR User Documentation, ed. D ACastelow, GEC Research Ltd, Hirst Research Centre,Wembley, 1987. In preparation.

4. D W Murray and D B Cook, "Using the orientationof fragmentary 3D edge segments for polyhedralobject recognition," Submitted to Int Journal of Com-puter Vision, 1986.

5. University of Sheffield AIVRU, "Canny - an imple-mentation of the Canny Edge detector," in GDB UserDocumentation release 1.0, vol. AIRVU 014, 1985.

6. B P D Ruff, "Icanny - Integer arithmetic Canny edgedetector," in ISOR User Documentation, ed. D ACastelow, GEC Research Ltd, Hirst Research Centre,Wembley, 1987. In preparation.

7. John F Canny, "Finding Lines and Edges in Images,"MIT Artificial Intelligence Laboratory TechnicalReport AI-TR-720, June 1983.

8. J F Canny, "A computational approach to edge detec-tion," IEEE Transactions on Pattern Analysis andMachine Intelligence, vol. PAMI- 8, no. 6, pp.679- 698, November 1986.

9. University of Sheffield AIVRU, "Hyster - edgethresholding with hysteresis," in GDB User Documen-tation release 1.0, vol. AIRVU 014, 1985.

10. D A Castelow and D W Murray, "Mcs3 - A methodfor computing optic flow from Canny edgels," inISOR User Documentation, ed. D A Castelow, GECResearch Ltd, Hirst Research Centre, Wembley, 1987.In preparation.

11. G L Scott, '"Four- line' method of locally estimatingoptic flow," Image and Vision Computing, vol. 5, no. 2,pp. 67-72, 1987.

12. D H Ballard and C M Brown, Computer Vision,Prentice- Hall Inc, New Jersey, USA, 1982.

13. M C Ibison and L Zapalowski, "On the use of relaxa-tion labelling in the correspondence problem," PatternRecognition Letters, vol. 4, no. 2, pp. 103-109, NorthHolland, 1986.

14. Stephen B Pollard, John E W Mayhew, and John PFrisby, "PMF - A Stereo Correspondence AlgorithmUsing a Disparity Gradient Limit," Perception, vol. 14,no. 4, pp. 449-470, 1985.

15. S A Lloyd, "Binocular stereo algorithm based on thedisparity- gradient limit and using optimisationtheory," Image and Vision Computing, vol. 3, no. 4,pp. 177-182, 1985.

16. P Quarendon, "WINSOM User's Guide," UKScientific Centre report number UKSC 123, IBM,Winchester, Hants, 1984.

17. B F Buxton, Buxton H, Murray D W, and Williams NS, "3D solutions to the aperture problem," inProceedings of ECAI-84: Advances in Artificial Intelli-gence, ed. T O'Shea, pp. 631-640, Elsevier SciencePublishers BV, Amsterdam, 1984.

200