www.oeaw.ac.at

www.ricam.oeaw.ac.at

Simultaneous optical flow andsource estimation: space-time

discretization andpreconditioning

R. Andreev, O. Scherzer, W. Zulehner

RICAM-Report 2014-23

Simultaneous optical flow and source estimation:space-time discretization and preconditioning

Roman Andreev1, Otmar Scherzer1,2, and Walter Zulehner3

1 Johann Radon Institute for Computational and Applied Mathematics (RICAM)Austrian Academy of Sciences

Altenberger Str. 69, 4040 Linz, Austria2 Computational Science Center

University of ViennaOskar-Morgenstern Platz 1, 1090 Wien, Austria

3 Institute of Computational MathematicsJohannes Kepler University

Altenberger Str. 69, 4040 Linz, Austria

Abstract. We consider the simultaneous estimation of an optical flowfield and an illumination source term in a movie sequence. The particularoptical flow equation is obtained by assuming that the image intensity is aconserved quantity up to possible sources and sinks which represent vary-ing illumination. We formulate this problem as an energy minimizationproblem and propose a space-time simultaneous discretization for theoptimality system in saddle-point form. We investigate a precondition-ing strategy that renders the discrete system well-conditioned uniformlyin the discretization resolution. Numerical experiments complement thetheory.

November 27, 2014

1 Introduction

Optical flow is the apparent motion of objects, surfaces, and edges in a visualscene. In Computer Vision the optical flow is calculated as the flow field regis-tering pixels of a movie, which we understand as a sequence of ordered images.The classical optical flow equation is based on the following assumptions:

1. Brightness constancy along each characteristic of the flow.2. A slowly varying image sequence.

The first assumption implies that changes of intensities caused by varying il-lumination (for instance shades) influence and distort the recovered flow. Inthis paper we admit violations of this assumption, and aim at estimating globalchanges of illumination and the effect of them onto the flow field. Indeed, theproblem we consider here consists in simultaneously determining the optical flow,the brightness, and its possible sources and sinks in space and time. While stan-dard optical flow algorithms process two successive frames of a movie sequenceat a time [16,17,18,3,13,6,1], we work with a continuous space-time simultaneous

2 R. Andreev, O. Scherzer and W. Zulehner

formulation similarly to [20]. An brief overview of gradient based optical flowmodels and algorithms may be found in [14].

Related to our work is [21]. Therein, the standard optical flow constraint wasreplaced by the continuity equation ∂tρ+∇ · (ρv) = 0, motivated by mass con-servation in fluid flow. We relax the continuity equation by allowing distributedsources g, and replace the momentum ρv by one unknown j, called the flux ; wethus obtain the constraint ∂tρ+∇ · j = g.

The work [15], see also references therein, suggested incorporating physicalmodels to account for illumination changes, such as the diffusion of the heatobserved by infrared cameras. The parameters governing the physical model arethen estimated in parallel with the optical flow. Our ansatz is more naive in thatwe treat the possible change in illumination as an additional unknown, but aphysical description can in principle be used instead.

As a further difference to the bulk of optical flow literature we treat thedensity ρ as an unknown. We thus estimate (ρ, j, g) in parallel by minimizing afunctional J which consists of a discrepancy term for ρ and regularization termsfor j and g. This principle similar to [7]; however, using the flux j instead of thevelocity v results in linear optimality conditions characterizing the minimizer ofthe functional J . While this is convenient for computation, it leads to somewhatunexpected flow patterns (see Sections 5–6).

It is important to note that these optimality conditions form a set of equationsthat are coupled in space-time. This is typical for optimal control problems withspatio-temporal constraints due to the coupling of the original problem forwardin time and the adjoint problem backward in time. The main contribution ofthis work is to propose and test a preconditioner for the discretized space-timesystem in saddle-point form that is robust in the discretization parameters.

The outline of the paper is as follows. In Section 2 we introduce our opticalflow model, which consists in minimization of a quadratic cost functional over aspace of space-time dependent functions subject to an optical flow constraint. InSection 3 the functional analytic framework is established. In Section 4 we rewritethe optimality conditions in a saddle-point form and discuss its discretization andpreconditioning. In Section 5 we report on our numerical experiments. Section6 concludes the paper.

In the following, 〈·, ·〉 denotes the duality pairing, while 〈·, ·〉X and ‖·‖X de-note the scalar product and the norm of a Hilbert space X. The usual Lebesgueand Sobolev spaces on a domain D are written as Lp(D) and Hs(D). We abbre-viate ‖·‖L2(D) as ‖·‖D, and similarly for other L2 spaces.

2 Optical flow model

We consider a movie sequence of time length T > 0, with each frame defined ona rectangular domain D ⊂ R2. We write J := (0, T ) for the temporal interval.Our optical flow model is based on the scalar transport equation

∂tρ+∇ · j = g in J ×D. (1)

Optical flow with source term 3

Here, ρ denotes the density which is the intensity of the image considered as apiecewise constant function; j denotes the flux which is the optical flow weightedby the density; and g is a function that models varying spatial illumination.The divergence ∇· acts on the spatial variable only. The functions ρ and gare scalar-valued, while j is vector-valued on J × D. Integrating (1) over asubdomain D′ ⊂ D and using the divergence theorem for the j term, one seesthat the density ρ is either transported in or out of D′ over its boundary, or iscreated/annihilated by means of the source term g.

In this paper we consider the problem of identifying the flux j and the sourceg from a finite number of frames of a movie, indexed by T ⊆ J :

ρτ ∈ L2(D), τ ∈ T . (2)

We aim at minimizing the data fidelity functional

FT (ρ) :=1

2

1

T

∑τ∈T‖ρτ − ρ(τ)‖2D (3)

subject to the transport equation (1) and further constraints on the flux j andthe source g as discussed below. Here, the symbol

∑τ∈T denotes the averaged

sum 1#T∑τ∈T .

For comparison purposes we recall the standard optical flow equation, whichreads as follows:

∂tρ+ Φ · ∇ρ = 0 in J ×D. (4)

Equation (4) can be formally derived from (1) by identifying j and ρΦ andneglecting small terms: Indeed, from (1) we get

∂tρ+ Φ · ∇ρ+ ρ∇ · Φ = 0 in J ×D. (5)

Hence, if the term ρ∇·Φ is negligible, and if there are no changes in illumination,the equations are identical.

3 Functional analytic framework

We now introduce function spaces for which the transport equation (1) is well-defined. We shall work with image intensity ρ ∈ H1,0 := H1(J ;L2(D)), fluxj ∈ H0,div := L2(J ;Hdiv(D)), and source g ∈ H0,0 := L2(J ;L2(D)). The super-scripts indicate the Sobolev smoothness in time and space, respectively. Here,Hdiv(D) is the space of vector-valued functions in [L2(D)]2 with distributionaldivergence in L2(D). As a consequence of the Fubini–Tonelli theorem [22, Sec-tion 0.3], the space H0,0 is isometrically isomorphic to L2(J × D). The normson H1(J) and Hdiv(D) are defined by

‖f‖2H1(J) := T−2 ‖f‖2J + ‖f ′‖2J , f ∈ H1(J),

‖v‖2Hdiv(D) := diam(D)−2 ‖v‖2D + ‖∇ · v‖2D , v ∈ Hdiv(D),

4 R. Andreev, O. Scherzer and W. Zulehner

where the scaling is so as to match the units. We introduce the product space

X := H1,0 ×H0,div ×H0,0

and associate it with the norm

‖(ρ, j, g)‖2X := ‖ρ‖2H1,0 + ‖j‖2H0,div + ‖g‖2H0,0 . (6)

The transport equation constraint (1) is reformulated via the linear operator

G : X → H0,0, G(ρ, j, g) := ∂tρ+∇ · j − g. (7)

Since G is continuous, the preimages of closed subsets are closed; in particular,its kernel

Γ := G−1(0) ⊂ X (8)

is a closed linear subspace. It contains precisely the triples (ρ, j, g) ∈ X thatsatisfy the transport equation (1).

Given two regularization parameters αj > 0 and αg > 0, we define thepenalization functional

R(j, g) :=αj2‖j‖2H0,div +

αg2‖g‖2H0,0 , (j, g) ∈ H0,div ×H0,0, (9)

and, recalling FT from (3), the cost functional

J (ρ, j, g) := FT (ρ) +R(j, g), (ρ, j, g) ∈ X. (10)

The parameters αj and αg are dimensionless. The following lemma is straight-forward.

Lemma 1. The functional J is Gateaux differentiable on X. With ρτ from(2), its Gateaux derivative at (ρ, j, g) ∈ X is the continuous linear functionalJ ′(ρ, j, g) = A(ρ, j, g)− `, where A : X → X ′ and ` ∈ X ′ are given by

〈A(ρ, j, g), (ρ, j, g)〉 =1

T

∑τ∈T〈ρ(τ), ρ(τ)〉D + αj〈j, j〉H0,div + αg〈g, g〉H0,0 , (11)

`(ρ, j, g) =1

T

∑τ∈T〈ρτ , ρ(τ)〉D, ∀(ρ, j, g) ∈ X. (12)

Moreover, A is continuous, self-adjoint, and Γ -elliptic,

∃α > 0 : 〈Av, v〉 ≥ α ‖v‖2X ∀v ∈ Γ. (13)

Further, A induces a seminorm |||·||| on X given by

|||(ρ, j, g)|||2 :=1

T

∑τ∈T‖ρ(τ)‖2D + αj ‖j‖2H0,div + αg ‖g‖2H0,0 . (14)

This seminorm is a norm Γ = G−1(0), where it is equivalent to ‖·‖X .

Optical flow with source term 5

With the help of this lemma and standard arguments from variational calculus,one can show strict convexity and coercivity properties of the functional J :

Lemma 2. The functional J defined in (10) is continuous and convex on X.Moreover, it is strictly convex on Γ ,

J (λv+ + (1− λ)v−) < λJ (v+) + (1− λ)J (v−) ∀v± ∈ Γ, λ ∈ (0, 1), (15)

v+ 6= v−, and coercive on Γ ,

∃α > 0, β ≥ 0 : J (v) ≥ α ‖v‖2X − β ∀v ∈ Γ. (16)

Our optical flow problem now reads as follows:

Definition 1. For given regularization parameters α = (αj , αg), the optical flowuα is the unique minimizer of the functional J over Γ .

Existence of the minimizer is due to standard arguments of the calculus of vari-ations [12], because J is non-negative and proper (J 6= ∞) on Γ . Uniquenessfollows from the strict convexity (15) of J on Γ .

The minimizer uα of J is equivalently characterized in terms of first orderoptimality conditions. By [12, Theorems 1.3–1.4 in §3.1.3], we have J ′(uα) = 0,where J ′(u) ∈ Γ ′ is the Gateaux derivative of J at u ∈ Γ . To simplify thenotation we will omit the dependence on α. Using Lemma 1, the requirementJ ′(u) = 0 ∈ Γ ′ is equivalent to the variational problem

〈Au− `, v〉 = 0 ∀v ∈ Γ. (17)

The fact that this variational problem is posed on the implicitly defined subspaceΓ = G−1(0) leads to the saddle-point problem introduced in the next section.

4 Numerical solution

In order to minimize J we solve the equivalent variational problem (17). Theconstraint u ∈ Γ is implemented using a Lagrange multiplier λ to complement(17) to a saddle-point problem: Find (u, λ) ∈ X ×H0,0 such that

〈Au, v〉 + 〈Gu, µ〉 + 〈Gv, λ〉 = 〈`, v〉 ∀(v, µ) ∈ X ×H0,0. (18)

4.1 Discretization

In order to discretize (18) we introduce finite dimensional subspaces Xh ⊂ X andMh ⊂ H0,0, as specified below, and consider the discrete saddle-point system:Find (uh, λh) ∈ Xh ×Mh such that

〈Auh, v〉 + 〈Guh, µ〉 + 〈Gv, λh〉 = 〈`, v〉 ∀(v, µ) ∈ Xh ×Mh. (19)

6 R. Andreev, O. Scherzer and W. Zulehner

With Ah : Xh → X ′h and Gh : Xh → M ′h, defined by Ahv := (Av)|Xhand

Ghv := (Gv)|Mh, v ∈ Xh, as well as `h := `|Xh

, the system (19) is equivalent to

Ah(uhλh

):=

(Ah G

′h

Gh 0

)(uhλh

)=

(`h0

), (20)

where G′h : Mh → X ′h is the H0,0-adjoint.Analogously to the continuous case, we set

Γh := {uh ∈ Xh : 〈Guh, µ〉 = 0 ∀µ ∈Mh}. (21)

The Brezzi equivalence theorem, see [8, Satz III.4.3], states that the left-hand-side of (19) defines an isomorphism Xh ×Mh → X ′h ×M ′h if and only ifthe following two conditions are fulfilled:

1. A is Γh-elliptic,

∃αh > 0 : 〈Av, v〉 ≥ αh ‖v‖2X ∀v ∈ Γh. (22)

2. The discrete inf-sup constant is positive,

βh := infλ∈Mh\{0}

supv∈Xh\{0}

〈Gv, λ〉‖v‖X ‖λ‖H0,0

> 0. (23)

We shall employ a discretization Xh ×Mh ⊂ X × H0,0 for which the tworather restrictive conformity conditions

Γh ⊂ Γ and Mh ⊂ GXh (24)

hold. The first, together with Γ -ellipticity (13) of A, implies Γh-ellipticity (22) ofA, and the second immediately implies the discrete inf-sup condition (23) withβh ≥ 1.

The square domain D is partitioned into non-overlapping open rectanglesand/or triangles, collected in Dh. Geometric compatibility conditions are im-posed following [11, p.51]: each edge of any geometric element K ∈ Dh is eitherpart of the boundary ∂D or is also an edge of some other geometric elementK ′ ∈ Dh. By D∂

h we mean the set of edges E of all geometric elements in Dh

(shared edges occur only once). The temporal interval J = (0, T ) is partitionedinto open subintervals collected in Jh, such that J =

⋃I∈Jh I. Quantities in

H0,0, such as ∂tρ, ∇ · j, g, and λ, are discretized as piecewise constants on eachspace-time geometric element I ×K, (I,K) ∈ Jh ×Dh. The conserved quantityρ ∈ H1,0 is discretized by continuous functions J → L2(D) that are affine oneach I ∈ Jh and have values in the space of piecewise constant functions on Dh.The flux j ∈ H0,div is discretized by assigning a flux density jIE to each pair(I, E) ∈ Jh ×D∂

h, such that jIE = 1|I||E|

∫I

∫Ej(t, x) · n(x)dσ(x)dt, and interpo-

lated linearly into the inner of each geometric element I ×K ∈ Jh×Dh. This istrivially possible on each rectangle, and corresponds to Raviart–Thomas inter-polation on triangles [8, p.141]. It is then clear that the conformity conditions(24), and therefore (22) and (23), are satisfied.

Optical flow with source term 7

4.2 Preconditioning

Several classes of preconditioners have been developed for saddle-point systemsof the form (20), see the survey [4]. Preconditioners based on approximations ofA−1h and of the Schur complement here Sh = GhA

−1h G′h are most widely used.

However, our Ah is not, in general, invertible on the whole finite-dimensionalspace Xh. Indeed, recall from (11) that A has the block structure

〈Ahv, v〉 =1

T

∑τ∈T‖ρ(τ)‖2D + αj ‖j‖2H0,div + αg ‖g‖2H0,0 , (25)

where v = (ρ, j, g), so that the first block is not positive definite on H1,0.In [5] a similar saddle-point system for a related problem from image regis-

tration was obtained, for which block-triangular preconditioners were proposed.We replace the triangular preconditioner by the symmetric and indefinite pre-conditioner Ph : Xh ×Mh → X ′h ×M ′h, given by

Ph :=

(Ah 0Gh I

)(A−1h 0

0 −Sh

)(Ah G

′h

0 I

), (26)

see [2]. It was shown in [19] that this allows to transform the indefinite saddle-point system into a symmetric positive definite one for a new inner product(see (27) below), so that the conjugate gradient (CG) method can be applied.This is not possible for the preconditioned system of [5] due to nonreal spectrum.Another well-known method to transform the indefinite saddle-point system intoa symmetric positive definite one was proposed in [10]. There it is also requiredthat Ah be positive definite on Xh, which is not true in our case.

It was shown in [19, Theorem 2.1] that if

Nh :=

(Ah −Ah 0

0 GhA−1h G′h − Sh

)(27)

is positive definite on Xh ×Mh then P−1h Ah is symmetric positive definite withrespect to the scalar product defined by Nh. To that end, assume:

1. Ah > Ah and GhA−1h G′h > Sh to assert that Nh > 0, as well as

2. Ah ≤M0Ah on kerGh, for some real constant M0 ≥ 1,3. GhA

−1h G′h ≤M1Sh for some real constant M1 ≥ 1.

Under those assumptions, the spectrum of P−1h Ah is positive and is containedin an interval determined by the constants M0 and M1 [19, Theorem 2.2]. Im-

portantly, Ah ≤ M0Ah in the second condition is only required to hold onkerGh = Γh, where Ah is positive definite by (22), and not on all of Xh.

We now specify our choice of Ah and Sh. For Ah we take the block operator(25) with the first block replaced by

2Cρ

((#T )−2 ‖∂tρ‖2H0,0 + T−2 ‖ρ‖2H0,0

), (28)

8 R. Andreev, O. Scherzer and W. Zulehner

while the second and the third nonzero blocks are multiplied by two. Here,Cρ > 0 is computed to satisfy

T−1∑τ∈T|f(τ)|2 ≤ Cρ

((#T )−2 ‖f ′‖2J + T−2 ‖f‖2J

)(29)

for all f ∈ H1(J). This entails Ah ≥ 2Ah, so that Ah > Ah is fulfilled in the firstassumption. The second assumption follows from the norm equivalence statedin Lemma 1. The constant Cρ can be chosen independently of T and T , as longas the subintervals defined by T are of comparable length.

For Sh we take the Cλ-fold of the Riesz isomorphism on H0,0, with, forinstance, 0 < Cλ < β2

h‖Ah‖−1, where βh > 0 is the discrete inf-sup constant (23).

Then GhA−1h G′h > Sh is fulfilled in the first assumption. The third assumption

follows by continuity of GhA−1h G′h on H0,0. In the numerical experiments below

we set Cλ = 12 .

It can be seen from the factorization (26) the application of the inverse of

the preconditioner only requires the application of A−1h and S−1h (as well as Ghand G′h). The application of S−1h is trivial since we work with piecewise constant

basis functions. In order to apply the block-diagonal operator A−1h , it is useful

to observe that each block of A has the Kronecker product form T ⊗ X. Theapplication of T⊗X and (T⊗X)−1 = T−1⊗X−1 is done by means of the matrixidentity (T ⊗X)±1Vec(u) = Vec(X±1uT±T), where Vec stacks the columns ofa matrix one after another into one long vector. In our Matlab implementation,we use this identity and apply the direct solver to compute X−1u column-wise,uT−T row-wise, etc. This technology is an important feature of space-time tensorproduct discretizations.

5 Numerical examples

5.1 Robustness of preconditioner

We investigate the robustness of the discretization and the preconditioner withrespect to the spatial discretization. The image domain is D := (0, 3) × (0, 4)and the temporal interval is J := (0, 2). A sequence of partitions D2−k , k =0, 1, 2, . . . , 7, of D is obtained by subdividing D into (3 × 2k) × (4 × 2k) equalrectangles. In each case, the partition Jh of J consists of 100 subintervals ofequal length (this choice is arbitrary). The input data (2) consists of two framesρ0 and ρ2, hence T = {0, 2}, as shown in Figure 1. The parameter choices in thepenalization functional (9) are αj = 10−4 and αg = 1, which promotes the fluxand puts a heavy penalty on the source term. We performed 20 CG iterationswith the preconditioner (26) and the energy norm (27) as described in Section4.2. The residual in the energy norm is reduced by the iteration by a factor≈ 106, for each choice of the spatial resolution level k. The results are depictedin Figure 1. In Figure 2 (left) we report on the value of the cost functional in thecourse of the CG iteration, for different spatial resolutions levels k. We observe

Optical flow with source term 9

that the iteration is robust in k. In Figure 2 (right), the convergence of the time-averaged flux over the horizontal midline (for the final CG iterate) as a functionof the spatial resolution is shown. First order convergence in terms of the totalnumber of degrees of freedom is observed.

x

y

0 0.5 1 1.5 2 2.5 3

0

0.5

1

1.5

2

2.5

3

3.5

4 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

x

y

0 0.5 1 1.5 2 2.5 3

0

0.5

1

1.5

2

2.5

3

3.5

4 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

x

y

0 0.5 1 1.5 2 2.5 3

0

0.5

1

1.5

2

2.5

3

3.5

4

−1.5

−1

−0.5

0

0.5

1

1.5

x

y

0 0.5 1 1.5 2 2.5 3

0

0.5

1

1.5

2

2.5

3

3.5

4

−1.5

−1

−0.5

0

0.5

1

1.5

x

y

0 0.5 1 1.5 2 2.5 3

0

0.5

1

1.5

2

2.5

3

3.5

4

−1.5

−1

−0.5

0

0.5

1

1.5

x

y

0 0.5 1 1.5 2 2.5 3

0

0.5

1

1.5

2

2.5

3

3.5

4

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

x 10−3

x

y

0 0.5 1 1.5 2 2.5 3

0

0.5

1

1.5

2

2.5

3

3.5

4

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

x 10−3

x

y

0 0.5 1 1.5 2 2.5 3

0

0.5

1

1.5

2

2.5

3

3.5

4

−1

−0.8

−0.6

−0.4

−0.2

0

0.2

0.4

0.6

0.8

1

x 10−3

Fig. 1. Top: Input data ρτ for τ = 0 (left) and τ = 2 (right). Middle, left to right : Thedivergence of the computed flux, ∇ · j, at t ≈ 0, t ≈ 1 and t ≈ 2. For the purpose ofvisualization, the flux is interpolated from the edges to the midpoint of each geometricelement. Bottom: The computed source term g at t = 0, 1, 2. Note the small scale of10−3 in accordance with the choice of αg/αj = 104.

5.2 Vienna Prater movie

In this example we apply the method to a movie showing the Riesenrad at theVienna Prater park turning clockwise. A vertical dark strip moving from left toright across the image has been superimposed artificially simulating shading, seeFigure 3. The movie consists of 200 frames of 256× 216 pixels each. This data ismapped to the domain D = (0, 4)× (0, 3) and the temporal interval J = (0, 2).The parameter choices in (9) is again αj = 10−4 and αg = 1. We look at frames91 and 92 and zoom in on two regions highlighted in Figure 3. The computed

10 R. Andreev, O. Scherzer and W. Zulehner

0 5 10 15 20

10−2

100

# iterations

J(ite

rate

)

104

106

10−8

10−7

10−6

Total #dofs

Fin

al a

ve

rag

e m

idflu

x e

rro

r

Fig. 2. Left : The value of the cost functional J in the course of the conjugate gradientiteration. The curves for different spatial resolutions overlap. Right : First order con-vergence of the time-averaged flux over the horizontal midline as the spatial resolutionis increased. See Section 5.1.

discrete flux between these two frames in those two regions is shown in Figure 3.It is interesting to note that the model generates a strong, almost uniform fluxj across the superimposed dark strip in the opposite direction of the movementof the strip in order to transport the bright intensity from its bow to its stern.One observes finer flow pattern within the strip away from its boundary causedby the rotation of the Riesenrad, as the graycolor displaying ∇ · j in Figure 3suggests. On the other hand, the moving bright spokes of the Riesenrad causethe intensities to flow across them in the same direction (as one might expectfrom the optical flow model).

6 Conclusions

We have considered a version of the in optical flow equations in which the imagebrightness evolves as a conserved quantity up to possible sources (or sinks). Wehave formulated the problem of estimating the optical flow and the source asan energy minimization problem. We have investigated a space-time discretiza-tion and preconditioning strategy for the resulting saddle-point equations. Thediscretization was shown to be stable in the Galerkin sense, and the precondi-tioner to be robust in the discretization resolution. In particular, applying theconjugate gradient method (with a suitable scalar product for which the systemmatrix is symmetric and positive definite) allows to solve the complete space-time problem within a few iterations. We have found that the transport equationmodel with the chosen penalization functional produces rather global flows; wedo not expect this to happen if the “kinetic energy” ‖j‖2/ρ is penalized insteadof the divergence of the flux j and wish to investigate this type of penalizationnext. We note, however, that our model is typical for control problems with aquadratic cost functional subject to a linear time-dependent PDE constraint,and we therefore expect the results to be applicable elsewhere. It would furtherbe of interest to investigate robustness of the proposed preconditioner in thepenalization parameters.

Optical flow with source term 11

x

y

0 0.5 1 1.5 2 2.5 3 3.5 4

0

0.5

1

1.5

2

2.5

3 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

x

y

0 0.5 1 1.5 2 2.5 3 3.5 4

0

0.5

1

1.5

2

2.5

3 0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

x

y

0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

−3

−2

−1

0

1

2

3

4

x

y

1 1.05 1.1 1.15 1.2 1.25 1.3 1.35 1.4 1.45 1.5

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

−3

−2

−1

0

1

2

3

4

Fig. 3. Top: Two consecutive frames ρτ for τ ≈ 0.91 in the example in Section 5.2.Highlighted in white are the regions of [33, 64]× [1, 36] pixels (left) and [65, 96]× [1, 36]pixels (right). The Riesenrad rotates clockwise and the superimposed dark vertical stripmoves eastwards (→). Bottom: Computed discrete flux for the example in Section 5.2.Left: The region of [33, 64]× [1, 36] pixels. Right: The region of [65, 96]× [1, 36] pixels.In both cases, there is a strong flux westward (←) across the superimposed dark strip,which moves eastwards (→). There is a less distinct flux directed north-east (↗) acrossthe bright spokes of the Riesenrad. The background color shows the divergence ∇ · j.

Acknowledgements

OS and WZ were supported by the Austrian Science Fund (FWF) within theNFN Geometry and Simulation, projects S11702 and S11704. OS was also sup-ported by the project Modeling Visual Attention as a Key Factor in VisualRecognition and Quality of Experience funded by the Wiener Wissenschafts undTechnologie Funds – WWTF. The Vienna Prater movie was provided by Ch. Val-uch, Cognitive Science Research Platform, Cognitive Psychology, University ofVienna, and preprocessed by A. Patrone, University of Vienna.

References

1. Gilles Aubert, Rachid Deriche, and Pierre Kornprobst. Computing optical flow viavariational techniques. SIAM J. Appl. Math., 60(1):156–182, 2000.

12 R. Andreev, O. Scherzer and W. Zulehner

2. Randolph E. Bank, Bruno D. Welfert, and Harry Yserentant. A class of iterativemethods for solving saddle point problems. Numer. Math., 56(7):645–666, 1990.

3. Steven S. Beauchemin and John L. Barron. The computation of optical flow. ACMComput. Surv., 27(3):433–466, 1995.

4. Michele Benzi, Gene H. Golub, and Jorg Liesen. Numerical solution of saddle pointproblems. Acta Numer., 14:1–137, 2005.

5. Michele Benzi, Eldad Haber, and Lauren Taralli. A preconditioning technique fora class of PDE-constrained optimization problems. Adv. Comput. Math., 35(2-4):149–173, 2011.

6. Michael J. Black and P. Anandan. The robust estimation of multiple motions:Parametric and piecewise-smooth flow fields. Comput. Vis. Image Und., 63:75–104, 1996.

7. Alfio Borzi, Kazufumi Ito, and Karl Kunisch. Optimal control formulation fordetermining optical flow. SIAM J. Sci. Comput., 24(3):818–847, 2002.

8. Dietrich Braess. Finite Elemente. Springer, 4th edition, 2007.9. James H. Bramble and Joseph E. Pasciak. Corrigenda: “A preconditioning tech-

nique for indefinite systems resulting from mixed approximations of elliptic prob-lems”. Math. Comp., 51(183):387–388, 1988.

10. James H. Bramble and Joseph E. Pasciak. A preconditioning technique for in-definite systems resulting from mixed approximations of elliptic problems. Math.Comp., 50(181):1–17, 1988.

11. Philippe G. Ciarlet. The finite element method for elliptic problems. North-Holland,1978.

12. Bernard Dacorogna. Direct methods in the calculus of variations. Springer, 1989.13. Alberto Del Bimbo, Paolo Nesi, and Jorge L. C. Sanz. Analysis of optical flow

constraints. IEEE T. Image Process., 4(4):460–469, 1995.14. David J. Fleet and Yair Weiss. Optical flow estimation. In Nikos Paragios, Yun-

mei Chen, and Olivier D. Faugeras, editors, Handbook of Mathematical Models inComputer Vision, chapter 15. Springer, 2006.

15. Horst W. Haussecker and David J. Fleet. Computing optical flow with physicalmodels of brightness variation. IEEE T. Pattern Anal., 23(6):661–673, 2001.

16. Berthold K. P. Horn and Brian G. Schunck. Determining optical flow. Artif. Intell.,17:185–203, 1981.

17. Bruce D. Lucas and Takeo Kanade. An iterative image registration techniquewith an application to stereo vision. In Proceedings of the 7th international jointconference on Artificial intelligence, volume 2, pages 674–679. Morgan KaufmannPublishers Inc. San Francisco, CA, USA, 1981.

18. Nans-Helmut Nagel and Wilfried Enkelmann. An investigation of smoothness con-straints for the estimation of displacement vector fields from image sequences.IEEE T. Pattern Anal., 8:565–593, 1986.

19. Joachim Schoberl and Walter Zulehner. Symmetric indefinite preconditioners forsaddle point problems with applications to PDE-constrained optimization prob-lems. SIAM J. Matrix Anal. Appl., 29(3):752–773 (electronic), 2007.

20. Joachim Weickert and Christoph Schnorr. Variational optic flow computation witha spatio-temporal smoothness constraint. J. Math. Imaging Vis., 14(3):245–255,2001.

21. Richard P. Wildes, Michael J. Amabile, Ann-Marie Lanzillotto, and Tzong-ShyngLeu. Recovering estimates of fluid flow from image sequence data. Comput. Vis.Image Und., 80(2):246–266, 2000.

22. Kosaku Yosida. Functional analysis. Springer, 1980.