Mechanical Regularization of Optical Flow:General Framework Using Finite-Elements

Petr JordanDivision of Engineering and Applied Sciences

Harvard University, Cambridge, MApjordan@deas.harvard.edu

Todd E. ZicklerDivision of Engineering and Applied Sciences

Harvard University, Cambridge, MAzickler@eecs.harvard.edu

Simona SocrateDepartment of Mechanical Engineering

MIT, Cambridge, MAssocrate@mit.edu

Robert D. HoweDivision of Engineering and Applied Sciences

Harvard University, Cambridge, MAhowe@deas.harvard.edu

Abstract

We present a general framework for regularization ofoptical flow by mechanical finite-element models derivedfrom the theory of continuum mechanics. We enforceimage-based local motion estimates as lumped body forcesapplied at mesh nodes of an underlying mechanicalmodel. The lumped body forces are formulated as virtualsprings displaced by local motion estimates. The choiceof each virtual spring stiffness reflects local texturalquality and associated local motion confidence. In thisformulation, the choice of image similarity measure, localsearch algorithm, image-mechanics confidence coupling,and material mechanics is application and user specific.Complex nonlinear viscoelastic materials can be usedfor regularization, as the modularity of the frameworkfacilitates the use of commercially available finite-elementsolvers. Results from synthetic uniaxial deformation ofliver parenchyma are provided and compared to traditionalimage-based regularizers. We demonstrate that knowledgeof underlying material mechanics significantly improvesmotion estimates, even in situations where mechanicalboundary conditions are not known.

1. IntroductionThe estimation of visual motion from image sequences

is one of the classical problems in Computer Vision.Due to its inherent ill-posedness, the solutions generallyrequire regularization and fall into the categories of localand global optimization problems. In local techniques,such as Lucas-Kanade [15], the regularization is achievedimplicitly by imposing local motion models (constant

motion, affine transformation, etc.) in order to obtainan additional constraint necessary for well-posedness. Inglobal techniques, such as Horn & Schunck [13], anexplicit Tikhonov-type regularization is used to enforceglobal smoothness of the resulting motion field. Detailedperformance and accuracy evaluation of the commonly usedoptical flow techniques can be found in [2, 3].

Applications in medical imaging often deal withestimation of motion from noise-contaminated volumetricdata with sparse textural information. The traditionaloptical flow techniques, relying on image-basedregularization, generally do not have an appropriatephysical motivation. It has been suggested that physics-based models can provide more accurate motion priors fromelastic [1, 9, 20, 16, 10] and viscoelastic [5, 6] solid bodymechanics, as well as relax image constancy assumptionsby coupling volume density to image irradiance [8]. Thesetechniques draw upon the assumption that knowledge ofmaterial mechanics and boundary conditions observed inan image sequence can contribute to a better estimate ofunderlying motion.

This work presents a modular framework for the solutionof motion tracking problems under constraints derivedfrom continuum mechanics. We enforce image-basedlocal motion estimates as lumped body forces applied atmesh nodes of an underlying mechanical model. Thelumped body forces are formulated as virtual springsdisplaced by local motion estimates. The choice of eachvirtual spring stiffness reflects local textural quality andassociated local motion confidence. Due to the image-mechanics coupling through lumped body forces, solutionof the mechanics problem does not require completeknowledge of boundary conditions. The key benefit

1

of this approach is the freedom of choice of individualcomponents, including the image similarity measure, localmatching algorithm, motion confidence metric, materialconstitutive law and parameters, and boundary conditions.Most importantly, this technique can be implemented with acommercially available finite-element package (ABAQUS,HKS, Providence, RI, USA) and state-of-the-art materialconstitutive models used in the solid mechanics, materialscience, and biomechanics communities. This techniqueis suitable for large-deformation tracking not only becauseof the finite-deformation formulation of the finite-elementmethod, but also due to the fact that the stress state ispropagated from frame to frame by the mechanical model.This approach not only helps to eliminate multi-frameaccumulation error, but also provides proper mechanicsover time, unlike memoryless mechanical regularizers [16],for time/rate dependent viscoelastic materials. Sincethe choice of local motion tracking algorithm is userand application specific, accurate large deformation localtracking can be achieved through statistical block matchingmethods [4] that do not suffer from the small motionassumptions of the optical flow constraint equation.

2. Background And Related WorkIn the following section we will discuss classical

formulation and regularization of the optical flow problemand relate mechanical regularization to some commonlyused image-based regularizers. The traditional differentialoptical flow techniques, such as Horn and Schunck [13] andLucas-Kanade [15] rely on two fundamental assumptions:intensity constancy and intensity smoothness. Under theintensity constancy constraint, one can express the motionof a point from time t to time t+ δt as

I (x+ uxδt, y + uyδt, z + uzδt, t+ δt) = I (x, y, z, t) ,(1)

where I (x, y, z, t) is the voxel intensity and (ux, uy, uz)are vector components of the corresponding voxel motion.If local intensity smoothness is assumed, Eq. 1 may beapproximated with a first-order Taylor series, yielding theoptical flow constraint equation

∂I

∂xux +

∂I

∂yuy +

∂I

∂zuz +

∂I

∂t= 0. (2)

2.1. Optical Flow Regularization

Since enforcement of the optical flow constraint (Eq.2) at each image voxel results in an under-constrainedsystem of linear equations, further regularization isrequired for well-posedness. The available regularizationtechniques range from homogeneous first-order smoothing

to formulations reflecting true material mechanics. Athorough summary of image-based regularization operatorsis provided by Bruhn and Weickert [19]. In thefollowing sections, we will demonstrate that mechanicalregularization provides a general, physically-realistic,regularization framework and, under certain conditions,can be related to the common image-based regularizationoperators.

2.1.1 Image-Based Regularizers

The traditional two-dimensional optical flow techniqueformulated by Horn and Schunck regularizes the solution byenforcing a first-order smoothness of resulting displacementfield. The functional being minimized is written as

Ψ(ux,uy) =∫ ∫

(Ef + α2Es)dxdy. (3)

While the Ef term is the deviation from the optical flowconstraint (Eq. 2), the authors propose two differentdefinitions of the smoothness constraint Es. The morewidely used (and the one carried through in the originalpaper) is

Es =(∂ux

∂x

)2

+(∂ux

∂y

)2

+(∂uy

∂x

)2

+(∂uy

∂y

)2

. (4)

The second formulation of the smoothness constraint,

Es =(∇2ux

)2+

(∇2uy

)2, (5)

is more interesting because it enforces Laplaciansmoothness and can be closely related to continuummechanics. As ux and uy are vector components ofdisplacement field u, minimization of Ef + α2

(∇2u)2

is analogous to the solution of Poisson’s equation(∇2ψ(x, y)− F (x, y) = 0) over the image domain.

When considering local optical flow techniques, such asLucas-Kanade [15] or Singh [17], the smoothness constraintcannot be explicitly related to mechanical regularization.Since these methods rely on local assumptions ofsmoothness (either constant or affine transformation withinlocal neighborhood), the only parameter that determinesthe deformation field smoothness is the size of thelocal neighborhood. In implementations where the localneighborhood is sampled by Gaussian weighting functions,the standard deviation of the sampling kernel may beconsidered as a measure of smoothness.

2.1.2 Linear Elasticity

In general, static problems in three-dimensional linearelasticity require the solution of 15 scalar fields that satisfy15 field equations. These field equations consist of 6strain-displacement equations (Eq. 6), 3 force equilibriumequations (Eq. 7), and 6 constitutive law equations (Eq. 8),

ε =12(∇u + (∇u)T ), (6)

∇ · σT + f = 0, (7)

σ = 2µε+ λ(tr ε)I, (8)

where ε is the strain tensor

ε =(∂u∂x

,∂u∂y,∂u∂z,∂u∂x

+∂u∂y,∂u∂y

+∂u∂z,∂u∂x

+∂u∂z

)T

,

σ is the stress vector

σ = (σx, σy, σz, τxy, τyz, τxz)T = Dε,

D is the elasticity matrix (see [18], pg. 40), µ and λ arethe Lame material constants, u is the displacement vector,and f is the body force vector. The field equations oflinearized elasticity can be combined in various ways toeliminate unknowns and thus arrive at reduced forms of thefield equations, involving a reduced number of equationsand unknowns. The problem of finding solutions to theequations of elasticity can be restated in terms of eitherfinding a displacement field u that satisfies the Lame-Navier equations or finding a stress field σ that satisfies theequations of equilibrium and Michell’s equations [18]. TheLame-Navier equations may be written as

(λ+ µ)∇(∇ · u) + µ∇2u + f = 0. (9)

The Lame constants may be related to a material’s Young’smodulus E and Poisson’s ratio ν as λ = Eν

(1+ν)(1−2ν) andµ = E

2(1+ν) . Rewriting the Lame-Navier equations in termsof E and ν results in

E

(1 + ν)(1− 2ν)∇(∇·u)+

E

2(1 + ν)∇2u+ f = 0. (10)

It is important to note that boundary conditions may onlybe stated in terms of displacements in this formulationand, therefore, the deformation field u is governed by itsboundary values (if specified) and the imposed body force

f . In the following sections we will relate body forcef to image-based motion content and discuss its role inmechanical regularization.

Under the assumption of an incompressible material(ν = 1

2 and ∇ · u = 0), the Lame-Navier equations reduceto the Poisson’s equation

E

3∇2u + f = 0. (11)

Therefore, the Laplacian-smooth regularization suggestedby Horn and Schunck is loosely equivalent to mechanicalregularization with an incompressible linear isotropicmaterial. Consequently, the two methods are not equivalentfor more complex nonlinear materials. This observation isimportant because it implies that Laplacian smooth image-based regularizers bias towards incompressible linearisotropic body mechanics.

3. Mechanical RegularizationThis paper presents a general regularization framework

that links local image motion to a mechanical model toprovide a global and mechanically accurate dense motionfield (Fig. 1). As discussed earlier, problems in three-dimensional elasticity are governed by Eqns. 6, 7, and 8.We propose to deform a mechanical model (Fig. 1, blockB) by a body force f lumped at nodal locations. Threevirtual springs attached to each node of the model act aslumped body forces in the 3 orthogonal directions (Fig. 2).The virtual springs displacements uOF

x,y,z are obtained froma local motion tracking algorithm (Fig. 1, block A) andtheir stiffnesses are adjusted to reflect local textural qualityand associated motion estimate confidence cx,y,z. The localimage motion can be obtained from an arbitrary motiontracking source or algorithm (modified Lucas-Kanade inour case). The choice of mapping between local motionconfidence and spring stiffness is discussed in detail in thelater sections of this paper. The mechanically regularizedoptical flow uFEM

x,y,z (Fig. 1, block C) is obtained fromthe solution of the mechanically deformed finite-elementmodel. Finally, dense motion fields (Fig. 1, block D)providing per-voxel displacements ux,y,z are computedby interpolation of the nodal displacements uFEM

x,y,z withelemental shape functions.

The proposed technique assumes that the mechanicalproperties of the imaged medium are known or canbe reasonably well estimated, however, the boundaryconditions do not have to be known, but can be used toprovide further motion constraints.

3.1. Local Optical Flow Estimation

We choose to estimate local optical flow uOFx,y,z by a

modified Lukas-Kanade algorithm. As shown earlier, the

Figure 1. Under the general mechanical regularization framework, local motion estimates are coupled to a mechanical model as lumpedbody forces. Interpolation of the resulting nodal displacements by element shape functions provides dense motion field estimates.

motion of each voxel can be estimated with the optical flowconstraint equation (Eq. 2). We sample the neighborhoodof each mesh node and assemble a system of equationsweighted by the corresponding elemental shape functionNi (x, y, z) (see [21] pg. 136-138 for the linear tetrahedralshape function definition).

Ni∂Ii∂x

uOFx +Ni

∂Ii∂y

uOFy +Ni

∂Ii∂z

uOFz = −Ni

∂Ii∂t

(12)

This local system of equations can be rewritten as

AuOFx,y,z = b,

where

A =

Ni∂Ii

∂x Ni∂Ii

∂y Ni∂Ii

∂z

· · ·· · ·

Nn∂In

∂x Nn∂In

∂y Nn∂In

∂z

, (13)

uOFx,y,z =

[uOF

x , uOFy , uOF

z

]T, (14)

b =[−Ni

∂Ii∂t, · · ·,−Nn

∂In∂t

]T

, (15)

and the nodal displacement can be estimated using thepseudo-inverse

uOFx,y,z =

(AAT

)−1AT b. (16)

Local Lucas-Kanade motion is computed at each meshnode, providing a globally unconstrained set of localmotion estimates. Each nodal motion estimate has anassociated confidence cOF

x,y,z, which is often computed fromthree eigenvectors (direction of confidence) and eigenvalues(level of confidence) of the square ATA matrix. In ourcase, we choose to compute the confidence by summingthe absolute values of image gradients contained in eachelement, such that

cx =n∑

i=1

Ni

∣∣∣∣∂Ii∂x

∣∣∣∣ ,

cy =n∑

i=1

Ni

∣∣∣∣∂Ii∂y

∣∣∣∣ , (17)

cz =n∑

i=1

Ni

∣∣∣∣∂Ii∂z

∣∣∣∣ .

Figure 2. Nodal image-mechanics coupling at a single lineartetrahedral element with local optical flow estimates coupled via 3orthogonal linear springs at each nodal position.

3.2. Mechanics-Based Deformation

Once the local motion estimates along with theassociated confidence levels are computed, the mechanicalmodel is deformed with virtual springs corresponding tolocal image motion. This relationship can be captured byan energy balance between the potential energy containedin all virtual springs (computed from local spring extensionxi

x,y,z and stiffness ki) and the internal elastic strain energyover the body domain Ω.

N∑

i=1

(12ki

x,y,z

(xi

x,y,z

)2)

=∫

Ω

σT εdV (18)

In order to relate the image-based confidence valuescOFx,y,z to physically relevant stiffnesses ki

x,y,z of theaforementioned virtual springs (as shown in Fig. 2), it isimportant to note that the stiffness of each spring is not onlya function of local confidence, but also of the local nodalstiffness of the mechanical model. Therefore, the stiffnessof each spring is computed as

kix = βKi

xcx,

kiy = βKi

ycy, (19)

kiz = βKi

zcz,

where β is the regularization parameter and K is thenodal stiffness of the mechanical model. Nodal stiffnessvalues are contained on the diagonal of the global stiffnessmatrix, which is assembled from contributions of individualelemental stiffness matrices (see [21] pg. 27 for details).It is important to note that the choice of the confidencemeasure is application specific and other candidates includecondition numbers of the linear system or curvatures ofthe auto-correlation and cross-correlation function of imageintensity, as shown by Singh [17].

Since high-frequency components of the motion field aresmoothed out during regularization, the elastic strain energyof the regularized problem is always less than or equal to thestrain energy of local optical flow field. The parameter βdetermines the amount of regularization by controlling theamount of potential energy stored in virtual springs. Whenconsidering the limiting cases, as β → 0 the correspondingregularized motion consists of low-frequency (rigid bodymotion) content only, while as β → ∞ the entire spectrumof the local motion estimates is preserved and therefore noregularization if performed. Similar to the regularizationparameter α in the Horn & Schunck method, β providesa way to control confidence between the image informationand the mechanical model and therefore should reflect noisecharacteristics of the image data. It should also be noted thatβ is independent of the structural properties (mechanics andgeometry) of the mechanical model.

The solution of the mechanical finite-element modelconsists of regularized motion estimates uFEM

x,y,z evaluatednodal positions of the underlying mesh. In order toobtain regularized dense motion field ux,y,z , the nodaldisplacements uFEM

x,y,z are interpolated over the uniformvolumetric grid corresponding to voxel locations using thefinite-element model’s shape functions.

4. Evaluation

4.1. Synthetic Motion Recovery

In order to obtain ground truth motion field forperformance evaluation, a synthetic deformation scene(Fig. 3) was generated by an unconstrained compression(εz = 0.03) of a cube (5 × 5 × 5 cm) ofvolumetric texture obtained by imaging liver parenchymawith three-dimensional ultrasound (SONOS 7500, PhilipsMedical Systems, Andover, MA, USA). A registeredfinite-element model with corresponding geometry andrealistic material properties (E = 10 kPa, ν =0.25) was used to compute the resulting mechanicaldeformation field. The deformation field was subsequentlyused to warp the volumetric texture, resulting in asynthetic volumetric deformation sequence. This motionsequence is used in subsequent performance evaluationsas it provides a simple ground-truth environment forbasic evaluation of the algorithm’s properties. Themotion tracking accuracy is evaluated in terms of meanmagnitude error (MME) and mean angular error (MAE)for the mechanics-based regularization, gradient-smoothimage-based regularization, Laplacian-smooth image-basedregularization [14], rigid body motion, and unregularizedlocal optical flow.

The choice of parameter β, as in other globalregularization schemes, is key to optimal motion estimation.In our synthetic deformation sequence, the optimal

Figure 3. Ground truth deformation field of a linear elasticmaterial. The deformation vector field is depicted with orientedcones with size and color proportional to motion magnitude.

regularization point (Fig. 4) is achieved at β =5.72, while lower values produce smoother motionfields and higher values preserve high-frequency content(noise). The simulation results demonstrate that forour chosen geometry, mechanical properties, and imagingcharacteristics, the mechanics-based regularizer is superiorto Horn & Schunck and Lucas-Kanade (Table 1) in therange of β = 〈1.179, 187.4〉 (Fig. 4). It should benoted that the choice of an optimal regularization parameterα in the first-order smooth and Laplacian smooth Horn& Schunck implementation is made such that the meanmagnitude error (MME) is minimized.

Table 1. Mean magnitude error (MME) and mean angular error(MAE) of common optical flow techniques compared to theproposed mechanically constrained approach.

Method MME MAE[voxels] []

Mechanics (β = 5.72, ν = 0.25) 0.1041 3.6941∇u Horn & Schunck (α = 16) 0.1329 4.7162∇2u Horn & Schunck (α = 6) 0.1391 9.0781Local (nodal) 0.1358 4.8191Rigid Body Motion 0.4332 15.436

4.2. Material Parameter Sensitivity

In order to demonstrate the sensitivity of this techniqueto the choice of underlying material parameters, the MMEwas computed for various values of Poisson’s ratio ν.This parameter determines the level of coupling between

strains in each axis, such that εx = −νεy = −νεz fora homogeneous isotropic material, and its plausible rangeis −1/2 < ν < 1/2. The results (Fig. 4, right)suggest that optimal regularization is obtained when ν =0.25, corresponding to the underlying material used in thesynthetic deformation sequence. More importantly, highaccuracy is achieved in the range of ν = 〈0.05, 0.45〉which spans the majority of engineering and biologicalmaterials. As ν → 0.5, the motion fields approachesthe incompressible body regularization obtained from theLaplacian smooth Horn & Schunck algorithm.

4.3. Noise Analysis

In order to obtain an upper bound on the accuracy ofthe algorithm, noise analysis was performed on the linearelastic deformation sequence by injecting varying levels ofmultiplicative Gaussian noise into local motion estimates(Fig. 11). The level of noise varies from noise-free localmotion (corresponding to the ground-truth motion field)to Gaussian distributed with standard deviation σN =0.8. These simulations demonstrate that as the level ofnoise increases, mechanical regularization (β near optimalregularization point) provides increasing benefit over localmethods (β →∞).

10−5

100

105

0

0.1

0.2

0.3

0.4

0.5

MM

E [v

oxel

s]

β

σN

= 0

σN

= 0.2

σN

= 0.4

σN

= 0.6

σN

= 0.8

Rigid Body Motion Local Optical Flow

Figure 5. Mean magnitude error (MME) and mean angularerror (MAE) with varying levels of local optical flow error(multiplicative, Gaussian-distributed noise with variance σN

injected into ground-truth local optical flow).

4.4. Nonlinear Mechanics and Boundary ConditionEffects

While linear elasticity is an adequate approximation ofmaterial mechanics in the small-deformation regime, mostmaterials (especially biological) possess strongly nonlinearstress-strain relationship. We demonstrate these effects byselecting two materials laws (linear elastic and 2nd-orderreduced polynomial hyperelastic) and parameters, whosestress-strain characteristics coincide at low strains (lessthan 1%) and diverge at higher strains. The goal of this

10−5

100

105

0

0.1

0.3

0.5

MM

E [v

oxel

s]

β

Rigid Body Motion

Local Optical Flow

Optimal Regularization Point

100

101

102

103

0.1

0.12

0.14

0.16

0.18

Modified Lucas−Kanade

Gradient−SmoothHorn & Schunck

MM

E [v

oxel

s]

β0 0.25 0.49

0.1

0.11

0.12

0.13

ν

MM

E [

voxe

ls]

Figure 4. Linear elastic material deformation: the effects of the regularization parameter β on the mechanically regularized optical flow(left), the range of β in which mechanical regularizer performs better than gradient-smooth and Laplacian-smooth image-based regularizer(middle), and the sensitivity of the mean magnitude error to ν, the Poisson’s ratio of the material (right).

simulation is to demonstrate that a nonlinear regularizer canbe easily implemented in this framework and can providefurther improvements in deformation tracking of materialswith nonlinear mechanics.

In this simulation we generate a nonlinear torsionaldisplacement field by constraining the bottom surface of thepreviously described cube and applying torsional force of0.01N, resulting in the deformation field shown in Fig. 6.In this case, however, the material mechanics are nonlinearby selecting a 2nd-order reduced polynomial hyperelasticformulation [12] (C10 = 2.0 × 104, C20 = 5.0 × 105,D10 = 1.5 × 10−5, D20 = 0). A linear elastic materialfrom the previous example (E = 10 kPa, ν = 0.25)may be rewritten as a 1st-order reduced polynomial withthe coefficients C10 = 2.0 × 105 and D10 = 1.5 × 10−5

and may serve as a small-deformation approximation of the2nd-order material.

In experimental scenarios where boundary forces maybe directly measured or controlled, the knowledge of theseconstraints further improves the motion prior accuracy andthe subsequent motion estimate. The improvement gainedby the knowledge of boundary conditions is demonstratedin Fig. 7 as a dependence of MME on the amount oftorsional force applied to the material surface. Theseresults suggest that while a linear elastic regularizer withno knowledge of the boundary force (FN = 0) canprovide better accuracy than image-based regularizers,the knowledge of nonlinear mechanics and boundaryconditions (FN = 1) can further improve the deformationestimates. It is important to note that this example is meantto demonstrate a trend of improvement in a single frame-to-frame deformation and will result in an incrementalbenefit in long-time, large-deformation, multiple-frametracking scenarios. In these situations, nonlinear mechanicsbecome even more important and the knowledge/control ofboundary conditions provides a constraint on accumulationerrors commonly seen in mutliple-frame optical flow

tracking.

Figure 6. Ground truth torsional deformation field of a nonlinearhyperelastic material. The deformation vector field is depictedwith oriented cones with size and color proportional to motionmagnitude.

5. DiscussionWe have presented a general framework for

regularization of optical flow by a mechanical finite-element model deformed by virtual springs correspondingto local motion estimates. The key advantage of thisapproach is the modular nature of its formulation, underwhich the choice of image similarity measure, localsearch algorithm, image-mechanics confidence coupling,and most importantly, the mechanical model’s materiallaw, is completely user and application specific. Theimplementation of this method is not significantly morecomplex than standard optical flow algorithms, as itcan rely on using commercially available finite-element

0 0.5 1 1.5 2 2.50.04

0.06

0.08

0.1

0.12

0.14

MM

E [

voxe

ls]

Normalized Boundary Force

Mechanical Regularization(Reduced Polynomial)

Gradient−SmoothHorn & Schunck

Mechanical Regularization(Linear Elastic)

Figure 7. Deformation tracking MME for image-based regularizer,linear elastic regularizer, and nonlinear hyperelastic regularizer asa function of underlying torsional boundary force.

solvers. This is of key importance, as it enables the useof nonlinear, viscoelastic material models of arbitrarycomplexity without the need for a reliable custom-madeFEM solver. While a 2nd-order reduced polynomialhyperelastic material was used to demonstrate thisability, any hyperelastic formulation, such as Mooney-Rivlin or Arruda-Boyce, may be easily implementedand even combined with rate-dependent viscous andporous materials. An additional advantage is the abilityto propagate the internal stress state of the material fromframe-to-frame, allowing for mechanically accurate long-time solutions and limiting multi-frame accumulation error.Finally, the method accounts for spatial variation in texturequality and associated motion tracking confidence.

It should also be noted that other local motion modelsmay be attractive for specific implementations. Forinstance, a maximum likelihood maximization block-matching algorithm [7, 4] was shown to be a goodchoice for local motion and associated confidence inultrasound applications. Furthermore, the local image-based motion can be complemented with alternativemotion estimates (electromagnetic trackers, diffusion tensorimaging, Doppler ultrasound, or even manual fiducialtracking) by introducing additional virtual springs withconfidence levels corresponding to the noise characteristicsof the measurement source.

While mechanics have been used by others forregularization of optical motion in the past, theimplementations are generally limited to linear elasticmaterials [11], do not allow for spatial variability ofconfidence [11], require complete knowledge of boundaryconditions [16], or require custom FEM solvers [11, 16].The use of time-dependent mechanical laws is oftenimpossible, as it requires propagation of the model stressstate from frame to frame. All of the above mentionedlimitation can be overcome by the framework proposed.In comparison to the traditional image-based regularizers

[13, 19], the finite-element formulation is attractive becauseof its formulation as an integration problem. Therefore,it does not require computation of often noisy high-order spatial derivatives of image data. As shown in theAppendix, an implementation of Laplacian smooth Hornand Schunck algorithm through a finite-difference iterativescheme requires computation of fourth-order spatial imagederivatives.

Our synthetic deformation scene has shown that accuratemotion results can be obtained even in the cases whereboundary conditions are completely unknown. Whilethe knowledge of boundary conditions of the mechanicalproblem will clearly improve the motion estimates, the userhas the freedom of specifying them fully, partially, or not atall. It is often the case in Computer Vision problems thatthe boundary conditions are not known and therefore ourmethod is a good candidate for motion tracking of mediawith known mechanical properties but unknown boundaryconditions.

6. AcknowledgementsThis project was funded by US Army grant, contract

number DAMD 17-01-1-0677 and the Harvard-MITDivision of Health Sciences and Technology.

References[1] R. Bajcsy and S. Kovacic. Multi-resolution elastic matching.

Computer Vision, Graphics, and Image Processing, 46:1–21,1989. 1

[2] J. Barron, D. Fleet, S. Beauchemin, and T. Burkitt.Performance Of Optical Flow Techniques. CVPR, 92:236–242, 1994. 1

[3] S. S. Beauchemin and J. L. Barron. The Computation ofOptical Flow. ACM Computing Surveys, 27(3):433–467,1995. 1

[4] D. Boukerroui, J. A. Noble, and M. Brady. Velocityestimation in ultrasound images: a block matching approach.In Information Processing in Medical Imaging, 2003. 2, 8

[5] M. Bro-Nielsen and C. Gramkow. Fast fluid registrationof medical images. In Proceedings of Visualization inBiomedical Computing, volume 1131 of Springer LectureNotes in Computer Science, pages 267–276, 1996. 1

[6] G. Christensen, S. Joshi, and M. Miller. Volumetrictransformation of brain anatomy. IEEE Trans. Med. Imag.,16(6):864–877, 1997. 1

[7] B. Cohen and I. Dinstein. New maximum likelihood motionestimation schemes for noisy ultrasound images. PatternRecognition, 35:455–463, 2002. 8

[8] T. Corpetti, E. Memin, A. Santa-Cruz, D. Heitz, andG. Arroyo. Optical flow estimation in experimental fluidmechanics. In Proceedings of ISSPA, 2003. 1

[9] C. Davatzikos. Spatial transformation and registration ofbrain images using elastically deformable models. ComputVis Image Underst, 66(2):207–222, May 1997. 1

[10] M. Ferrant. Physics-based Deformable Modeling of Volumesand Surfaces for Medical Image Registration. PhD thesis,Universite’ Catholique de Louvain, 2001. 1

[11] M. Ferrant, S. K. Warfield, C. R. G. Guttmann, R. V.Mulkern, F. A. Jolesz, and R. Kikinis. 3D Image MatchingUsing a Finite Element Based Elastic Deformation Model.In MICCAI, pages 202–209, 1999. 8

[12] Hibbitt, Karlsson & Sorensen, Inc., Providence, RI.ABAQUS Manual, version 5.4 edition, 2005. 7

[13] B. K. P. Horn and B. G. Schunck. Determining Optical Flow.Artificial Intelligence, 16(1-3):186–203, August 1981. 1, 2,8

[14] P. Jordan. Implementation of Laplacian-smooth image-basedregularization in Horn & Schunck optical flow. Technicalreport, Harvard University, 2006. 5

[15] B. D. Lucas and T. Kanade. An Iterative Image RegistrationTechnique with an Application to Stereo Vision (IJCAI).In Proceedings of the 7th International Joint Conferenceon Artificial Intelligence (IJCAI ’81), pages 674–679, April1981. 1, 2

[16] X. Papademtris, A. Sinusas, D. Dione, and J. Duncan.3D Cardiac Deformation from Ultrasound Images. InProceedings of MICCAI 99, pages 420–429, 1999. 1, 2, 8

[17] A. Singh. Image-flow computation: Estimation-theoreticframework, unification and integration. PhD thesis,Columbia University, 1990. 2, 5

[18] J. Slaughter. Linearized Theory of Elasticity. Elsewier, 1999.3

[19] J. Weickert and C. Schnrr. A theoretical framework forconvex regularizers in pde-based computation of imagemotion. International Journal of Computer Vision,45(3):245–264, 2001. 2, 8

[20] F. Yeung, S. F. Levinson, D. Fu, and K. J. Parker. Feature-adaptive motion tracking of ultrasound image sequencesusing a deformable mesh. IEEE Trans Med Imaging,17(6):945–956, Dec 1998. 1

[21] O. C. Zienkiewicz. The Finite Element Method. McGraw-Hill, 3 edition, 1977. 4, 5

Dropped Material7. Dropped Figures

Figure 8. The undeformed (left) and deformed (right) finite-element model used to generate the synthetic deformationsequence.

8. Appendix8.1. The Finite-Element Method

The fundamentals of small-deformation finite-elementtheory are summarized below. It is yet to be determinedwhether it is appropriate to include this appendix. Also,large-deformation finite-element formulation should bediscussed and perhaps formulated as well.

The energy-balance formulation of a solid bodymechanics can be stated as sum of internal strain energyof the body and external energy imposed on the boundaries.

E =12

∫

Ω

σTεdV +∫

Ω

FudV (20)

where ε is the strain vector

ε =(∂u∂x

,∂u∂y,∂u∂z,∂u∂x

+∂u∂y,∂u∂y

+∂u∂z,∂u∂x

+∂u∂z

)T

= Lu (21)

and σ is the stress vector

σ = (σx, σy, σz, τxy, τyz, τxz)T = Dε (22)

u =Nnodes∑

i=1

Neli uel

i (23)

The field equations are interpolated over the elements bylinear shape functions:

Neli =

16V

(ai + bix+ ciy + diz) (24)

Figure 9. Ground truth deformation field (top) and deformationfield recovered by mechanically constrained optical flow (bottom).The deformation vector field is depicted with oriented cones withsize and color scaled by motion magnitude. (colorbar needs to beadded)

where

6V = det

1 xi yi zi

1 xj yj zj

1 xm ym zm

1 xp yp zp

(25)

ai = det

xj yj zj

xm ym zm

xp yp zp

0 20 40 60 80 100

0.1

0.3

0.5

α

MM

E [v

oxel

s]

Laplace−SmoothHorn & Schunck

First−Order−SmoothHorn & Schunck

0 0.25 0.490.1

0.11

0.12

0.13

ν

MM

E [

voxe

ls]

Figure 10. The effects of α on MME in the first-order smooth andLaplacian smooth Horn & Schunck algorithm (left). The MME asa function of material property (Poisson’s ratio, ν) is shown on theright.

10−5

100

105

0

0.1

0.2

0.3

0.4

0.5

MM

E [v

oxel

s]

β

σN

= 0

σN

= 0.2

σN

= 0.4

σN

= 0.6

σN

= 0.8

Rigid Body Motion Local Optical Flow

10−5

100

105

0

2

4

6

8

10

12

β

MA

E [d

egre

es]

σN

= 0

σN

= 0.2

σN

= 0.4

σN

= 0.6

σN

= 0.8

Rigid Body Motion Local Optical Flow

Figure 11. Mean magnitude error (MME) and mean angularerror (MAE) with varying levels of local optical flow error(multiplicative, Gaussian-distributed noise with variance σN

injected into ground-truth local optical flow).

0 2 4 6 8 10 120

2000

4000

6000

8000

10000

12000

14000

16000

18000

strain (%)

stre

ss [

Pa]

linear

reduced polynomial

0 0.5 1 1.5 2 2.50.04

0.06

0.08

0.1

0.12

0.14

MM

E [

voxe

ls]

Normalized Boundary Force

Mechanical Regularization(Reduced Polynomial)

Gradient−SmoothHorn & Schunck

Mechanical Regularization(Linear Elastic)

Figure 12. Stress-strain relationship (left) for the linear andnonlinear material. Mechanics-based regularization vs. Horn &Schunck (right).

bi = det

1 yj zj

1 ym zm

1 yp zp

ci = det

xj 1 zj

xm 1 zm

xp 1 zp

di = det

xj yj 1xm ym 1xp yp 1

The energy balance in Eq. 20, can be discretized andrewritten in matrix form as

E(uel

1 , ...,uelNnodes

)=

12

∫

Ω

Nnodes∑

i=1

Nnodes∑

j=1

(uel

i

)T (Bel

i

)TDBel

j uelj dΩ

+∫

Ω

Nnodes∑

i=1

FNeli uel

j dΩ (26)

where

Bi =

∂Ni

∂x 0 00 ∂Ni

∂y 00 0 ∂Ni

∂z∂Ni

∂y∂Ni

∂x 00 ∂Ni

∂z∂Ni

∂y∂Ni

∂z 0 ∂Ni

∂x

(27)

D =E(1− ν)

(1 + ν)(1− 2ν)(28)

×

1 ν1−ν

ν1−ν 0 0 0

ν1−ν 1 ν

1−ν 0 0 0ν

1−νν

1−ν 1 0 0 00 0 0 1−2ν

2(1−ν) 0 00 0 0 0 1−2ν

2(1−ν) 00 0 0 0 0 1−2ν

2(1−ν)

Solution of Eq. 20 can be obtained by minimization ofE with respect to the nodal displacements uel

i

∂E(uel

1 , ...,uelNnodes

)

∂ueli

= 0

where i = 1, ..., Nnodes, resulting in the system ofequations

∫

Ω

Nnodes∑

j=1

(Bel

i

)TDBel

j uelj dΩ =

−∫

Ω

FNeli dΩ

where j = 1, ..., Nnodes, which can be rewritten as

Keluel = −Fel (29)

as the global stiffness matrix Kelij is assembled by

integration of all elements

Kelij =

∫

Ω

(Bel

i

)TDBel

j dΩ=(Bel

i

)TDBel

j Ve (30)

and the nodal forces are computed as

Feli =

∫

Ω

FNeli dΩ

Consequently, the finite-element formulation results insparse linear system

Ku = −F (31)

which is generally solved by Cholesky factorization orGauss-Jordan elimination.

8.2. Laplacian Motion Field Smoothness (Horn &Schunck extension)

Since there is no clear reference for development of theLaplacian-smooth implementation of the Horn & Schunckalgorithm it may be appropriate to include it here.

Extending the Horn & Schunck algorithm to Laplacian-smooth regularization requires reformulation of thefunctional Φ(u, v, w). While the overall form remains as

Φ(u, v, w) =∫

Ω

((∂I

∂xu+

∂I

∂yv +

∂I

∂zw +

∂I

∂t

)2

+ αEs

)dxdydz

(32)

the Es is in this case defined as,

Es =(∇2u

)2+

(∇2v)2

+(∇2w

)2

where the Laplacian operator ∇2 is defined as

∇2 =∂2

∂x2+

∂2

∂y2+

∂2

∂z2

The minimization of the functional Φ(u, v, w) inEq. ?? can be achieved through calculus of variations.The Euler-Lagrange equations are the essential tool invariational problems and are analogous to zero-slopeestimation (setting partial derivatives to zero) in calculus.Minimization of Φ(u, v, w) is again performed by theEuler-Lagrange equations, which in this case yield thefollowing system of equations

∇4u =1α

(∂I

∂xu+

∂I

∂yv +

∂I

∂zw +

∂I

∂t

)∂I

∂x

∇4v =1α

(∂I

∂xu+

∂I

∂yv +

∂I

∂zw +

∂I

∂t

)∂I

∂y(33)

∇4w =1α

(∂I

∂xu+

∂I

∂yv +

∂I

∂zw +

∂I

∂t

)∂I

∂z

where ∇4 is the biharmonic operator defined as

∇4 = ∇2∇2 =(∂2

∂x2+

∂2

∂y2+

∂2

∂z2

)2

=∂4

∂x4+

∂4

∂y4+

∂4

∂z4

+2∂4

∂x2y2+ 2

∂4

∂x2z2+ 2

∂4

∂y2z2

Similarly, the biharmonic operator in 3D can be obtainedby 3D self-convolution of a 7-point Laplacian 3D kernel,yielding the following finite-difference approximation

∇4ψi,j,k = −42ψi,j,k

+ 12(ψi+1,j,k + ψi−1,j,k + ψi,j+1,k + ψi,j−1,k + ψi,j,k+1

+ ψi,j,k−1)− 2(ψi+1,j+1,k + ψi+1,j−1,k + ψi−1,j+1,k

+ ψi−1,j−1,k)− 2(ψi+1,j,k+1 + ψi−1,j,k+1 + ψi,j+1,k+1

+ ψi,j−1,k+1)− 2(ψi+1,j,k−1 + ψi−1,j,k−1 + ψi,j+1,k−1

+ψi,j−1,k−1)− (ψi+2,j,k +ψi−2,j,k +ψi,j+2,k +ψi,j−2,k

+ ψi,j,k+2 + ψi,j,k−2)

We can then proceed by expressing the biharmonic termas

∇4ψi,j,k = ¯ψi,j,k − ψi,j,k

where ¯ψi,j,k is the neighborhood mask defined as

¯ψi,j,k =1242

(ψi+1,j,k + ψi−1,j,k + ψi,j+1,k + ψi,j−1,k + ψi,j,k+1

+ ψi,j,k−1)− 242

(ψi+1,j+1,k + ψi+1,j−1,k + ψi−1,j+1,k

+ψi−1,j−1,k)− 242

(ψi+1,j,k+1 +ψi−1,j,k+1 +ψi,j+1,k+1

+ψi,j−1,k+1)− 242

(ψi+1,j,k−1 +ψi−1,j,k−1 +ψi,j+1,k−1

+ψi,j−1,k−1)− 142

(ψi+2,j,k+ψi−2,j,k+ψi,j+2,k+ψi,j−2,k

+ ψi,j,k+2 + ψi,j,k−2)

Under this definition of ¯ψi,j,k the iterative Gauss-Seidelequations have a form identical to the standard Horn &Schunck formulation

uk+1 = ¯uk −

(∂I∂x

¯uk + ∂I∂y

¯vk + ∂I∂z

¯wk + ∂I∂t

)∂I∂x

α+(

∂I∂x

)2+

(∂I∂y

)2

+(

∂I∂z

)2

vk+1 = ¯vk −

(∂I∂x

¯uk + ∂I∂y

¯vk + ∂I∂z

¯wk + ∂I∂t

)∂I∂y

α+(

∂I∂x

)2+

(∂I∂y

)2

+(

∂I∂z

)2

wk+1 = ¯wk −

(∂I∂x

¯uk + ∂I∂y

¯vk + ∂I∂z

¯wk + ∂I∂t

)∂I∂z

α+(

∂I∂x

)2+

(∂I∂y

)2

+(

∂I∂z

)2