what energy functions can be minimized using graph cuts? shai bagon advanced topics in computer...

What Energy Functions Can be Minimized Using

Graph Cuts?

Shai Bagon

Advanced Topics in Computer Vision

June 2010

What is an Energy Function?

E

a number

suggested solution

For a given problem:Image Segmentation:

237-20

Useful Energy function:

1. Good solution Low energy

2. Tractable Can be minimized

Families of Functions or Outline

• F2 submodular

• Non submodular

• F3

• Beyond F3

Foreground SelectionLet

yi – color of ith pixel

xi ϵ {0,1} BG/FG labels (variables)

Given BG/FG scribbles:Pr(xi|yi)=How likely each pixel to be FG/BG

Pr(xm|xn)=Adjacent pixels should have same label

F2 energy:

E(x)=∑iEi(xi)+∑ijEij(xi,xj)

xm xn

xi

yi

Submodular

Known concept from set-functions:

E(x) = ∑i Ei(xi) + ∑ij Eij (xi, xj), xi ϵ {0,1}

Syxyxfyxfyfxf ,

20,01,11,00,1 Sffff

1 C D

0 A B

xj

xi0 1

Eij(xi,xj):

What does it mean?

B+C-A-D ≥ 0

How toMinimize?

E(x) = ∑i Ei(xi) + ∑ij Eij (xi, xj), xi ϵ {0,1}

Local “beliefs”:

Data termPrior knowledge:

Smoothness term

F2 submodular

Graph Partitioning

A weighted graph G=( V E w )

Special Nodes: s t

s-t cut:

Cost of a cut:

Nice property: 1:1 mapping

s-t cut ↔ {0,1}|V|-2

wV E

wij

s

t

TjSi

wijTSCut,

(,)

VTSTS

TtSsVTVS

,

,,,

s

t

Graph Partitioning - Energy

E(x) = ∑i Ei(xi) + ∑ij Eij (xi, xj)

Graph Partitioning

i j

Ej(1)

D-C

B+C-A-D

Ei(0)

1 C D

0 A B

xj

xi

0 1

Eij(xi,xj)

C-AC-A

00

D-C0

D-C0

00

B+C-A-D0= A + + +

C-A

s

t

Graph Partitioning - Energy

E(x) = ∑i Ei(xi) + ∑ij Eij (xi, xj)

Graph Partitioning

i j

Ej(1)

B+C-A-D

Ei(0)

C-A

Tv

Svxv 1

0

xE

DACBCDEEA

wATScut

ij

TjSiij

01

,,

D-C

st cut binary assignment

cut cost energy of assignment

min cut Energy min.

B=Eij(0,1)

Recap

F2 submodular:

E(x) = ∑i Ei(xi) + ∑ij Eij (xi, xj)

Eij(1,0)+Eij(0,1)≥Eij(0,0)+Eij(1,1)

Mapping from energy to graph partition

Min Energy = computing min-cut

Global optimum in poly timefor submodular functions!

Next…

Multi-label F2

E(x)=∑i Ei(xi) + ∑ij Eij(xi,xj) s.t. xi ϵ {1,…,L}

– Fusion moves: solving binary sub-problems– Applications to stereo, stitching, segmentation…

●

Currentlabeling

suggestedlabeling

“Alpha expansion”

=

Fusion

Solve Binary problem: xi=0 xi=1

Stereo matching see http://vision.middlebury.edu/stereo/

Ground truthPairwise MRF[Boykov et al. ‘01]

slide by Carsten Rother, ICCV’09

Input:

Panoramic stitching

slide by Carsten Rother, ICCV’09

Panoramic stitching

slide by Pushmeet Kohli, ICCV’09

AutoCollage

http://research.microsoft.com/en-us/um/cambridge/projects/autocollage/ [Rother et. al. Siggraph ‘05 ]

Next…

Multi-label F2

E(x)=∑i Ei(xi) + ∑ij Eij(xi,xj) s.t. xi ϵ {1,…,L}

– Fusion moves: solving binary sub-problems– Applications to stereo, stitching, segmentation…

Non-submodular

Beyond pair-wise interactions: F3

Merging Regionsinput image regions (Ncuts) “edge” prob.

pi

0:1:

1Prii xi

ixi

i ppx

1: 0:

1loglogi ixi xi

ii ppx

“weak” edge

“strong” edge

pi – prob. of boundary being edgeGOAL: Find labeling xiϵ{0,1} that max:

i

j

min:

Taking -log

Merging Regions

ii i

i

xiii

ii

xii

xii

xii

xii

xp

pC

ppp

pppp

i

iiii

11

log

log1loglog

loglog1loglog

0:

0:0:0:1:

x

Adding and subtracting the same number

1: 0:

1loglogi ixi xi

ii ppx merged be likey to210

edgean be likely to210

1log :

def

ii

ii

i

ii

pw

pw

p

pw

i

ii xwC

Merging Regions

Solving for edges:

Consistency constraints:No “dangling” edge

i iix xwCminarg

J

x1 x2 x3 EJ

0 0 0 0

1 1 1 0

0 1 1 0

0 0 1 λ

wi

xi

No longer pair-wise:

F3

31321

321

11

11

xxxxx

xxxEJ

Minimization trick

21min1 3211,0

321

xxxzxxxz

Freedman D., Turek MW, Graph cuts with many pixel interactions: theory and applications to shape modeling. Image Vision Computing 2010

1min

11,01

KxzxK

i iz

K

ii

Merging Regions

The resulting energy:

+ Pair-wise

- Non submodular!

Jnml nlnmllmn

n nn

xxxxxz

xwE

,,11

min

zx

Quadratic Pseudo-Boolean Optimization

s

i j

ti j

Kolmogorov V., Carsten R., Minimizing non-submodular functions with graph cuts – a review. PAMI’07

+ All edges with positive capacities

- No constraint

Labeling rule:

partial labeling

s

i j

t

i j

ii 1

otherwise

, if1

, if0

SiTi

TiSi

yi

Quadratic Pseudo-Boolean Optimization

Properties of partial labeling y:

1. Let z=FUSE(y,x) E(z)≤E(x)

2. y is subset of optimal y*

y is complete:

1. E submodular

2. Exists flipping

(inference in trees)

s

i j

t

i j

Quadratic Pseudo-Boolean Optimization

0?????

rp q s t

000?? 0010?

rp q s t

rp q s tQPBO:

Probe Node p:0 1

What can we say about variables?

•r -> is always 0•s -> is always equal to q•t -> is 0 when q = 1 slide by Pushmeet Kohli, ICCV’09

QBPO - Probing

• Probe nodes in an order until energy unchanged

• Simplified energy preserves global optimality and (sometimes) gives the global minimum

slide by Pushmeet Kohli, ICCV’09

QBPO - Probing

Merging Regions

Result using QPBO-P:

Resultregions (Ncuts)input image

Recap

• F3 and more– Minimization trick

• Non submodular– QPBO approx. – partial labeling

Beyond F3…

[Kohli et. al. CVPR ‘07, ‘08, PAMI ’08, IJCV ‘09]

Image Segmentation

E(X) = ∑ ci xi + ∑ dij |xi-xj|i i,j

E: {0,1}n → R

0 →fg, 1→bg

n = number of pixels

[Boykov and Jolly ‘ 01] [Blake et al. ‘04] [Rother et al.`04]

Image Unary Cost Segmentation

Pn Potts Potentials

Patch Dictionary

(Tree)

Cmax 0

{0 if xi = 0, i ϵ p Cmax otherwise

h(Xp) =

p

[slide credits: Kohli]

Pn Potts Potentials

E(X) = ∑ ci xi + ∑ dij |xi-xj| + ∑ hp (Xp) i i,j p

p

{0 if xi = 0, i ϵ p Cmax otherwise

h(Xp) =

E: {0,1}n → R

0 →fg, 1→bg

n = number of pixels

[slide credits: Kohli]

Image Segmentation

E(X) = ∑ ci xi + ∑ dij |xi-xj| + ∑ hp (Xp) i i,j

Image Pairwise Segmentation

Final Segmentation

p

E: {0,1}n → R

0 →fg, 1→bg

n = number of pixels

[slide credits: Kohli]

Application: Recognition and Segmentation

from [Kohli et al. ‘08]

Image

Unaries onlyTextonBoost

[Shotton et al. ‘06]

Pairwise CRF only[Shotton et al. ‘06]

Pn Potts

One super-pixelization

another super-pixelization

Robust(soft) Pn Potts model

{0 if xi = 0, i ϵ p f(∑xp) otherwise

h(xp) =p

p

from [Kohli et al. ‘08]

Robust Pn PottsPn Potts

Application: Recognition and Segmentation

From [Kohli et al. ‘08]

Image

Unaries onlyTextonBoost

[Shotton et al. ‘06]

Pairwise CRF only[Shotton et al. ‘06]

Pn Potts robust Pn Potts robust Pn Potts(different f)

One super-pixelization

another super-pixelization

Same idea for surface-based stereo]Bleyer ‘10[

One input image

Ground truth depth

Stereo with hard-segmentation

Stereo with robust Pn Potts

This approach gets best result on Middlebury Teddy image-pair:

How is it done…

H (X) = F ( ∑ xi )

Most general binary function:

H (X)

∑ xi

concave

0

The transformation is to a submodular pair-wise MRF, hence optimization globally optimal

[slide credits: Kohli]

Higher order to Quadratic

• Start with Pn Potts model:

{0 if all xi = 0C1 otherwise

f(x) = x ϵ {0,1}n

min f(x) min C1a + C1 (1-a) ∑xix =x,a ϵ {0,1}

Higher Order Function

Quadratic Submodular Function

∑xi = 0 a=0f(x) = 0

∑xi > 0 a=1f(x) = C1

[slide credits: Kohli]

Higher order to Quadratic

min f(x) min C1a + C1 (1-a) ∑xix=

x,a ϵ {0,1}

Higher Order Function

Quadratic Submodular Function

∑xi

1 2 3

C1

C1∑xi

[slide credits: Kohli]

Higher order to Quadratic

min f(x) min C1a + C1 (1-a) ∑xix=

x,a ϵ {0,1}

Higher Order Submodular

Function

Quadratic Submodular Function

∑xi

1 2 3

C1

C1∑xi

a=1a=0Lower

envelope of concave

functions is concave

[slide credits: Kohli]

Summary• Submodular F2

• F3 and beyond: minimization trick

• Non submodular– QPBO(P)

• Beyond F3 – Robust HOP

s

i j

t

i j

∑xi

a=1a=0

f2(x)

f1(x)