[5] The Role of Disparity Gradient in Stereo Vision

Harit P Trivedi! and Sheelagh A Lloyd-

GEC ResearchLaboratoriesHirst Research Centre, Wembley, HA9 7PP, UK

Reprin ted, with permission of Pion Ltd, from Perception, 1985, 14, 685-690,

Abstract

Burtand Julesz experimentally demonstrated that, in additionto Panum's fusional area, a quantity defined by them andnameddisparity gradient also plays a crucial part in decidingwhether the human visual system would be able to fuse theimages seen by the left and right eyes. The physicalmeaning of this quantity remains obscure despite attempts tointerpret it in terms of depth gradient. Neverthele~s, it hasbeen found to be an effective selector of matches In stereocorrespondence algori~hms. A proof is .pro,:,ided that adisparity gradient limit of less than 2 implies that thematchesbetween the two images preserve the topology of theimages. The result, which is invariant ,unde~ rotations andunder relative as well as overall magnifications, holds forpairs of points separated in any direction, not just alongepipolar lines. This in tum can be shown to preventcorrespondences being established between points whichwould have to be located in three dimensions on a surfaceinvisibleto one eye, assuming opaque surfaces.

Lcft im_lc

f(Po

I}'.

f(PI}

Ripu imalc

1 Introduction

Binocular stereo VISIon entails reconstructing depthinformation from two images of a three-dimensional (3-D)scene taken from slightly different viewpoints. Thisinvolves establishing correspondences between points(solving the 'correspondence problem') and computing thedepthby triangulation. In comp.utervision, pr0'1~ed that thecamera geometry is known, this last step IS trivial, ll?d SOsolving the correspondenceproblem guarantees stereopsis,

In the case of human vision, there is also the concept ofbinocularfusion, which is when a stereoscopically presentedimage appears single, and this is not the same as stereopsis.It is well known that stereopsis can occur with fusion (egdepth perception despite diplopia), and fusion . ~ithoutstereopsis (eg in amblyopes). For computer VISIon, ofcourse, there is no concept of fusion. Burt and Julesz(1980a, 1980b) conducted some interesting experiments toinvestigate the effect of nearby points on binocular fusion.They defined the disparity gradient (figure 1) between twonearbypoints as the difference in their disparities divided,bytheirseparation in visual angle, and demonstrated that fUSIOnof at least one point fails when this gradient exceeds a criticalvalue (approximately 1). Although their experiments werenecessarily concerned with horizontal disparities only (sincethearrangementof human eyes means that vertical disparitiesare usually very small), the definition of disparity gradient is

1 now with BP International Ltd, Sunbury Research Centre,Chertsey Road, Sunbury-on-Thames, Middlesex, TW16 7LN, UK.

2 now with Hewlett Packard Laboratories, Filton Road, StokeGifford, Bristol, BS12 6QZ, UK.

Figure 1. Vector rL in the left image and the correspondingvector rR in the right image. The disparity gradient is definedas IrL - 'RVl l2l' L +'R1.

not restricted to points with only horizontal disparities.There will be situations in computer vision (eg largevergence and/or gaze angles) where points will have bothhorizontal and vertical disparities, so it is useful to keep theconcept of disparity general and not restrict the considerationsto horizontal disparities only.

Despite the fact that this limit applies to fusion, v.:hich, ismeaningless in a computer vision conte~t, the. disparitygradient limit has provided a powerful disambiguator ofcorrespondences in several stereo algorithms (Lloyd 1984;Pollard et al 1985). The success of these algorithms isperhaps surprising: why should a constraint on human fusionbe a good constraint for computer stereopsis? Equally~ w~y

should the disparity gradient limit be a good constraint In

human vision? There are certainly some empirical reasons:most false matches will cause the disparity gradient limit tobe exceeded and so will be rejected by the algorithm, whereasmost pairs of correct matches will satisfy the disparitygradient limit (pollard et al 1985) . E--:en ~n surf~ces

containing apirs of points exceeding the dispanty gradientlimited (consider, for example, a plane inclined away fromthe viewer so that it is close to horizontal), there will still bemany pairs of points with disparity gradient less than thelimit, and so the algorithm may still be able to solve thecorrespondence problem correctly. While not disputing inany way the success of these algorithms. we were, however,still unconvinced of the reasons for this success.

I,'

We therefore propose an explanation of the disparity gradientlimit based on the physics of image formation . Thissuggests a reason why it is sensible, both for human fusionand for computer stereopsis, to impose a disparity gradientlimit. We shall show that imposing a disparity gradientlimit of less than 2 ensures that no matches will be madebetween points that would have to be located in space on asurface invisible to one eye (assuming opaque surfaces).Observe that the converse (that all pairs of points on a non­self-occluding opaque surface satisfy the disparity gradientlimit) does not follow; the example mentioned earlier of aplane inclined away from the viewer would be an obviouscounter example. Imposing a disparity gradient limit is aconservative policy; it prevents false matches, but alsodisallows some correct ones. Bun and Julesz suggest a figureof I, although some of their experimental data might suggesta higher figure. This is still safely below 2 and in perfectaccord with our theory. It is interesting to note that, whenrestricted to pairs of points lying on the same epipolar line, adisparity gradient limit of less than 2 reduces to the familiar'ordering constraint' used in many stereo algorithms, whichdemands that points on an epipolar line lie in the same orderin both images.

We shall, in fact, show that imposing a disparity gradientlimit of less than 2 ensures that the correspondence betweenthe two images preserves their topology. In other words:

(a) each point in the left image corresponds to a unique pointin the right image and vice versa;

(b) ifwe were given one image painted on a rubber sheet, wecould, without tearing the shet or glueing it to itself, deformit so that we obtained the other image.

This may seem very obscure, but if we consider how the twoimages arise , things may become clearer. Let us supposethat we have a continuous non-self-occluding surface as infigure 2. Now, for each eye, the projection which transformsthe surface into the image preserves the topology, and so eachimage must have the same topology as the surface does.Hence they must have the same topology as each other.Conversely, if we have a self-occluding surface as in figure 3,the appropriate matches will not preserve the topology. Thiscan be seen quite easily by considering the point E, where theray from D intersects the surface again. In the left eye theimage of this point will appear in the same place as theimage of D, that is DL' whereas the two will be distinct inthe right image. So DL in the left image corresponds to twopoints (DR and Ea) in the right image, and this implies thatthe topology cannot be preserved since condition (a) abovewould be violated.

2 Proof

Suppose that we have two images IL and IR and acorrespondence f between them . So, for each point P in IL'we know the point or points in IR corresponding to P.Suppose further that f obeys the disparity gradient limit of 2k(where k« I), that is, ifPI and P2 are points in IL' then

In order to prove that the topology is preserved, we need toshow that

48

(i) f is one-to-one (ie each point in IR corresponds to aunique point in IU:(ii) f-l is one -to-one (ie each point in IL corresponds to aunique point in Ia);(iii) f is continuous (roughly, nearby points in IR correspondto nearby points in 10;(iv) f-l is continuous (roughly, nearby points in ILcorrespond to nearby points in IR>.

Since the disparity gradient limit is symmetric, we need onlyprove (i) and (iii), and then (ii) and (iv) will follow bysymmetry.

In order to prove (i), suppose that a single point PR in IRcorresponds to two points in PlL and P2L in IL. Considerthe disparity gradient, ID, between this pair of matches :

But this is not allowed because we are supposing that all thedisparity gradients are less than 2. So PR must correspondto a unique point PL in IL: that is, f is one-to-one.

We now need to prove that f is continuous. Suppose that PIand P2 are two distinct points in IL and let us writefL = PI - P2 and fR = f(Pl) - f(P2)' In order to provecontinuity we need to show that whatever smal positivenumber e we are given, we can always find another positivenumber d so that

IfLI < d implies that IrRI < e .

The proof which follows is straightforward but technical. Wesimply suppose that IfLI < d and apply the fact that f obeysthe disparity gradient limit of 2k with k < 1, and deduce that

2l /2(I + k)2I'RI < d 2

l-k

Turning this round, we see that, given e, we simply choose

d =e_2-_1:-/2_(--:-I-~k2_)(l +k)2

and we are done. It is now clear why we need to stipulatethat k < 1.

We now give the details of the proof.

Let us write fL = (XL.YU and '"R = (XR ,YR). Suppose thatIfLI < d, then txLI, lyLI < d. Now the disparity gradient limitcan be rewritten

(XL - XR)2 + (YL - YR)2 ~

e[(XL + XR)2 + (YL +YR)2Jor

xi (e-l )+2XLXR(e+1)+X~ (e-1) +e(YL +YR)2_ (YL -YR)2 ~ O.

49

surface 5-.....---7,

Figure 2 (Above) Points A, B, and C on a surface S (A, B, C not meant to lie on the intersection of the surface S and a plane) and theirimages. (Below) The two images are topologically equivalent.

A.:......__~,

•Da

Ca Ea-..:.-••~... A

• • a.. .. ,....'Ba

Figure 3 (Above) Surface S of figure 2. In addition, points D and E and a tangent ray passing through the left eye are also shown.(Below) The topologies of the two images are clearly not equivalent,

This is a quadratic in XR, and the coefficient of xi isnegative. The above inequality can only be satisfied whenthere are two real roots and XR lies between them. Thecondition for the quadratic to have real roots is that

4xl (P+ 1)2-

4(k2-1)[xl (k2 - 1) + k2(Y L +YR)2-(YL -YR)2] ~ 0

or

- (I - eYyi +2YLYR (1- e) (1 +e) +4eX~ - (l_e)2Y~ ~ o.

Again, this is a quadratic in YR with leading coefficientnegative. It has two real roots, so the inequality is satisfiedwhen .

2YL (1 - k 4) - 4k (1 - k 2

) Ird2(I-kl)2 ~YR

2YL (1 - k4) + 4k (1 - k2) Ird

YRs 2(1-k'-'j2

or

50

3 Discussion

In their attempt to incorporate the notion of disparity gradientlimit in Panum's fusional area, Burt and Julesz (1980a)mention a "cone-shaped forbidden zone, symmetric about theline of sight" around a point object. This is readilyunderstood in terms of our explanation if the cone is taken tobe defined by the surface k = 1 passing through the point.(The extended lines of sight to each eye lie on this surface,for instance.) Then the matches between the left and the rightimages of the original point (at the tip of the cone) and otherpoints - some inside and some outside the cone - will not, ingeneral, preserve the topology of the images.

Our interpretation of the disparity gradient limit constraint inview of what we have said so far is that the human visualsystem expects to find surfaces, and that it expects thesurfaces to be opaque. If a vision system is to solve thestereo correspondence problem before interpolating surfacesthrough the three-dimensional points so computed, theequivalent topology (or disparity gradient limit) constraintalong with this interpretation would guarantee that no part ofthe surface will be obscured by another: a very powerfulguarantee ensuring no conflict between stereo matching andsurface interpolation at a subsequent stage of processing.

4 Conclusion

So certainly

The tworoots of the quadratic in XR are

(1+k)2IYRI ~ d 2 •

l-k

We have proved that a disparity gradient limit of less than 2implies topological equivalence between left and rightimages. Further, we have shown that this guarantees that agroup of three-dimensional points obeying the disparitygradient limit cannot lie on a surface which would have beenobscured to one eye. Incorporation of the disparity gradientlimit constraint in stereo matching makes it possible toproceed to surface interpolation in a bottom-up fashion.

-2xL(I+k2)±2{4k 2xL2_Ck2_1)[k2CYL+YR)2_CYL-YR)2JP/2 R fe erences2(1-k2)

.':

Now

(since the left-hand side is a quadratic in YR with maximumvaluethe expression on the right-hand side) so

1/2

---"""":""7'''-7""'l':'''"-_.....L- < i 1+ k)2- 1=7i2

Hence

So, given e > 0, take d = e[2-1/2(1-k2)/(I+k)2], and then 11'1.1< d implies that 1"R1 < e. Hence f is continuous. This is theendof the proof.

Note that the result is invariant to rotations and to relative aswell as overall magnifications.

Bun, P. and Julesz, B. (1980a) Modifications of the classicalnotion of Panurn's fusional area. Perception, 9,671-682.

Burt, P. and Julesz, B. (1980b) A disparity gradient limit forbinocular fusion. Science, 208,615-617.

Pollard, S.B., Mayhew, J.E.W., Frisby, J.P. (1985) PMF:A stereo correspondence algorithm using a disparity gradientlimit. Perception, 14, 449-470.

Lloyd, S.A. (1984) A new stereo algorithm. MemoCSRL/157, GEC Research Laboratories, Hirst ResearchCentre, Wembley, UK.