Go With The FlowA New Manifold Modeling and Learning Framework for Image Ensembles

Richard G. BaraniukRice University

Aswin Sankaranarayanan Chinmay Hegde Sriram Nagaraj

Go With The FlowA New Manifold Modeling and Learning Framework for Image Ensembles

Richard G. BaraniukRice University

Sensor Data Deluge

Sensor Data Deluge

Sensor Data Deluge

Concise Models

• Efficient processing / compression requires concise representation

• Sparsity of an individual image

pixels largewaveletcoefficients

(blue = 0)

Concise Models

• Efficient processing / compression requires concise representation

• Our interest in this talk: Collections of images

Concise Models

• Our interest in this talk:Collections of images parameterized by q 2 Q

– translations of an object q: x-offset and y-offset

– rotations of a 3D object: q pitch, roll, yaw

– wedgeletsq: orientation and offset

Concise Models

• Our interest in this talk:Collections of images parameterized by q 2 Q

– translations of an object q: x-offset and y-offset

– rotations of a 3D object: q pitch, roll, yaw

– wedgeletsq: orientation and offset

• Image articulation manifold

Image Articulation Manifold• In practice: N-pixel images:

• In theory:

• K-dimensional articulation space

• Thenis a K-dimensional manifoldin the ambient space

• Very concise model

articulation parameter space

Smooth IAMs• In practice: N-pixel images:

• In theory:

• If images are smooththen so is

• Local isometry: image distance parameter space distance

• Linear tangent spacesare close approximationlocally

articulation parameter space

Smooth IAMs• In practice: N-pixel images:

• In theory:

• If images are smooththen so is

• Local isometry: image distance parameter space distance

• Linear tangent spacesare close approximationlocally

articulation parameter space

Ex: Manifold Learning

LLEISOMAPHE …

• K=1rotation

Ex: Manifold Learning

• K=2rotation and scale

Theory/Practice Disconnect: Smoothness

• Practical image manifolds are not smooth!

• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho, Grimes]

Local isometryLocal tangent approximations

Theory/Practice Disconnect: Smoothness

• Practical image manifolds are not smooth!

• If images have sharp edges, then manifold is everywhere non-differentiable [Donoho, Grimes]

Local isometryLocal tangent approximations

Failure of Local Isometry

• Ex: translation manifold

• Local isometry

all blue imagesare equidistantfrom the red image

0 20 40 60 80 1000

0.5

1

1.5

2

2.5

3

3.5

Translation in [px]

Euc

lidea

n di

stan

ce

Failure of Tangent Plane Approx.

• Ex: cross-fading when synthesizing / interpolating images that should lie on manifold

Input Image Input Image

Geodesic Linear path

Image Articulation Manifold

• Linear tangent space at is K-dimensional

– provides a mechanism to transport along manifold

– problem: since manifold is non-differentiable, tangent approximation is poor

• Our goal: replace tangent spacewith new transport operator that respects the nonlinearity of the imaging process

Tangent space at 0I

Articulations

0

1 2

2I1

I0I IAM

Optical Flow Transport• Brightness constancy: Given two images I1 and I2,

we seek a displacement vector field f(x, y) = [u(x, y), v(x, y)] such that

Ex:

• Linearized brightness constancy (LBC)

)),(),,((),( 12 yxvyyxuxIyxI

),()(),()(),(),( 1112 yxvIyxuIyxIyxI YX

History of Optical Flow

• Dark ages (<1985): special cases solved– LBC an under-determined set of linear equations

• Horn and Schunk (1985): – Regularization term: smoothness prior on the flow

• Brox et al (2005)– shows that linearization of brightness constancy is

a bad assumption– develops optimization framework to handle BC directly

• Brox et al (2010), Black et al (2010), Liu et al (2010)– practical systems with reliable code

Optical Flow Manifold

0

1 2

IAM

OFM at 0I

2I1

I0I

Articulations

• Consider a reference imageand a K-dimensional articulation

• Collect optical flows fromto all images reachable by aK-dimensional articulation

• Provides a mechanism to transport along manifold

Optical Flow Manifold

0

1 2

IAM

OFM at 0I

2I1

I0I

Articulations

• Consider a reference imageand a K-dimensional articulation

• Collect optical flows fromto all images reachable by aK-dimensional articulation

• Provides a mechanism to transport along manifold

• Theorem: Collection of OFs is a smooth, K-dimensional manifold(even if IAM is not smooth)[N,S,H,B,2010]

Flow Metric• Replace intensity-based image distance

– root cause of IAM non-differentiability

… with distance along the OF field from I1 to I2

• Enables differential geometric tools– ex: vector fields, curvature, parallel transport, …

curve c

IAMFlow Field Vt

OFM is Smooth (Translation)

Euclidean distance between image intensities

Flow metric between images(globally linear)

OFM is Smooth (Rotation)

00 30 60 30 60

-80 -60 -40 -20 0 20 40 60 800

50

100

150

200

Articulation in [ ]

Inte

nsity

-80 -60 -40 -20 0 20 40 60 80-20

-10

0

10

20

Articulation in [ ]

Opt

ical

flow

vx i

n pi

xels

Articulation θ in [⁰]

Inte

nsi

ty I(θ

)O

p. flow

v(θ

)Euclidean distance between image intensities

Flow metric between images(nearly linear)

The Story So Far…

0

1 2

IAM

OFM at 0I

2I1

I0I

Articulations

flow metric

Euclidean metric

Tangent space at 0I

Articulations

0

1 2

2I1

I0I IAM

Input Image Input Image

Geodesic Linear pathIAM

OFM

OFM Synthesis

ISOMAP embedding error for OFM and IAM

2D rotations

Reference image

Manifold Learning

Manifold Learning

-6 -4 -2 0 2 4 6-5

-4

-3

-2

-1

0

1

2

3

4

5Two-dimensional Isomap embedding (with neighborhood graph).

Embedding of OFM

2D rotations

Reference image

Data196 images of two bears moving linearlyand independently

IAM OFM

TaskFind low-dimensional embedding

OFM Manifold Learning

IAM OFM

Data196 images of a cup moving on a plane

Task 1Find low-dimensional embedding

Task 2Parameter estimation for new images(tracing an “R”)

OFM ML + Parameter Estimation

• Point on the manifold such that the sum of geodesic distances to every other point is minimized

• Important concept in nonlinear data modeling, compression, shape analysis [Srivastava et al]

Karcher Mean

10 images from an IAM

ground truth KM OFM KM linear KM

• Goal: build a generative model for an entire IAM/OFM based on a small number of base images

• El CheapoTM algorithm:– choose a reference image randomly– find all images that can be generated from this image by OF– compute Karcher (geodesic) mean of these images– compute OF from Karcher mean image to other images– repeat on the remaining images until no images remain

• Exact representation when no occlusions

Manifold Charting

Manifold Charting

IAM

• Goal: build a generative model for an entire IAM/OFM based on a small number of base images

• Ex: cube rotating about axis

• All cube images can be representing using 4 reference images + OFMs

• Many applications– selection of target

templates for classification– “next-view” selection for

adaptive sensing applications

Summary

• IAMs a useful concise model for many image processing problems involving image collections and multiple sensors/viewpoints

• But practical IAMs are non-differentiable– IAM-based algorithms have not lived up to their promise

• Optical flow manifolds (OFMs)– smooth even when IAM is not– OFM ~ nonlinear tangent space– support accurate image synthesis, learning, charting, …

• Barely discussed here: OF enables the safe extension of differential geometry concepts– Log/Exp maps, Karcher mean, parallel transport, …

Related Work

• Analytic transport operators– transport operator has group structure [Xiao and Rao 07]

[Culpepper and Olshausen 09] [Miller and Younes 01] [Tuzel et al 08]

– non-linear analytics [Dollar et al 06]

– spatio-temporal manifolds [Li and Chellappa 10]

– shape manifolds [Klassen et al 04]

• Analytic approach limited to a small class of standard image transformations (ex: affine transformations, Lie groups)

• In contrast, OFM approach works reliably with real-world image samples (point clouds) and broader class of transformations

Open Questions

• Our treatment is specific to image manifolds

• What are the natural transport operators for other data manifolds?

dsp.rice.edu

Open Questions

• Theorem: random measurements stably embed aK-dim manifoldwhp [B, Wakin, FOCM ’08]

• Q: Is there an analogousresult for OFMs?