dr. ulas bagci bagci@ucfbagci/teaching/computervision16/lec13.pdfdr. ulas bagci [email protected]...

CAP5415-ComputerVisionLecture13-AdvancedImageSegmentation

(OptimizationProblem)

[email protected]

Lecture13:AdvancedImageSegmentation

1

Reminders• Oct18GuestLecture(Lecture17)Dr.Borji

–BasicsofHumanVisionSystem• Oct20GuestLecture(Lecture18)Dr.Gong

–DeepLearning(CNN,RNN,LSTM,GRU)

• Selectyourprojectby15th ofOctober(UseWebCourse!)

2


Labeling &Segmentation• Labeling isacommonwayformodelingvariouscomputervisionproblems(e.g.opticalflow,imagesegmentation,stereomatching,etc)

3



• Thesetoflabelscanbediscrete(asinimagesegmentation)

4


L = {l1, . . . , lm} with L = m


• Thesetoflabelscanbediscrete(asinimagesegmentation)

• Orcontinuous(asinopticalflow)

5


L = {l1, . . . , lm} with L = m

L ⇢ Rnfor n � 1

Labelingisafunction• Labelsareassignedtosites (pixellocations)

6


Labelingisafunction• Labelsareassignedtosites (pixellocations)• Foragivenimage,wehave

7


|⌦| = Ncols

.Nrows

Labelingisafunction• Labelsareassignedtosites (pixellocations)• Foragivenimage,wehave• Identifyingalabelingfunction(withsegmentation)is

8


|⌦| = Ncols

.Nrows

f : ⌦ ! L


Weaimatcalculatingalabelingfunctionthatminimizesagiven(total)errororenergy

9


|⌦| = Ncols

.Nrows

f : ⌦ ! L


Weaimatcalculatingalabelingfunctionthatminimizesagiven(total)errororenergy

*Aisanadjacencyrelationbetweenpixellocations10


|⌦| = Ncols

.Nrows

f : ⌦ ! L

E(f) =X

p2⌦

[Edata

(p, fp

) +X

q2A(p)

Esmooth

(fp

, fq

)]

EnergyFunctionTheroleofanenergyfunctioninminimization-basedvisionistwofold:

1. asthequantitativemeasureoftheglobalqualityofthesolutionand

2. asaguidetothesearchforaminimalsolution.

Correctsolutionisembeddedastheminimum.11


SegmentationasanEnergyMinimizationProblem

• Edata assignsnon-negativepenaltiestoapixellocation pwhenassigningalabeltothislocation.

12




• Esmooth assignsnon-negativepenaltiesbycomparingtheassignedlabelsfp andfq atadjacentpositionsp andq.

13





14





Thisoptimizationmodelischaracterizedbylocalinteractionsalongedgesbetweenadjacentpixels,andoftencalledMRF(MarkovRandomField)model.

15


EnergyFunction-Details

16


E(f) =X

p2⌦

[Edata

(p, fp

) +X

q2A(p)

Esmooth

(fp

, fq

)]


• SampleDataTerm:

17


E(f) =X

p2⌦

[Edata

(p, fp

) +X

q2A(p)

Esmooth

(fp

, fq

)]

Edata(p, fp) = (p)

(p = 0) = �logP (p 2 BG)

(p = 1) = �logP (p 2 FG)


• SampleDataTerm:

• SampleSmoothnessTerm:

18


E(f) =X

p2⌦

[Edata

(p, fp

) +X

q2A(p)

Esmooth

(fp

, fq

)]

Edata(p, fp) = (p)

(p = 0) = �logP (p 2 BG)

(p = 1) = �logP (p 2 FG)

(p, q) = Kpq�(p 6= q) where

Kpq =exp(��(Ip � Iq)2/(2�2))

||p, q||


19


E(p) =X

p2⌦

p(p) +X

p2⌦

X

q2A(p)

pq(p, q)

p⇤ = argminp2L

E(p)


• Tosolvethisproblem,transformtheenergyfunctionalintomin-cut/max-flowproblemandsolveit!

20


E(p) =X

p2⌦

p(p) +X

p2⌦

X

q2A(p)

pq(p, q)

p⇤ = argminp2L

E(p)

EnergyFunction

21


UnaryCost(dataterm)

DiscontinuityCost

p q

Recap:ImageasaGraph

22


Node/vertexEdge

pixel

p q

GraphCutsforOptimalBoundaryDetection(BoykovICCV2001)

23


F

B

F

B

F

F F

F B

B

B

• Binarylabel:foregroundvs.background• Userlabelssomepixels• Exploit

– StatisticsofknownFg &Bg– Smoothnessoflabel

• Turnintodiscretegraphoptimization– Graphcut(mincut/maxflow)

Graph-Cut

24


EachpixelisconnectedtoitsneighborsinanundirectedgraphGoal:splitnodesintotwosetsAandBbasedonpixelvalues,andtrytoclassifyNeighborsinthesamewaytoo!

Graph-Cut

25


EachpixelisconnectedtoitsneighborsinanundirectedgraphGoal:splitnodesintotwosetsAandBbasedonpixelvalues,andtrytoclassifyNeighborsinthesamewaytoo!

Graph-Cut

26


• Eachpixel=node• AddtwonodesF&B• Labeling:linkeachpixeltoeitherForB

F

B

F

B

F

F F

F B

B

B

Desiredresult

CostFunction:Dataterm

27


• PutoneedgebetweeneachpixelandbothF&G• Weightofedge=minusdataterm

B

F

CostFunction:Smoothnessterm

28


• Addanedgebetweeneachneighborpair• Weight=smoothnessterm

B

F

Min-Cut

29


• Energyoptimizationequivalenttographmincut• Cut:removeedgestodisconnectFfromB• Minimum:minimizesumofcutedgeweight

B

Fcut

Min-Cut

30


Source

Sink

v1 v2

2

5

9

42

1Graph (V, E, C)

Vertices V = {v1, v2 ... vn}Edges E = {(v1, v2) ....}Costs C = {c(1, 2) ....}

Min-Cut

31


Source

Sink

v1 v2

2

5

9

42

1

Whatisast-cut?

Anst-cut(S,T)divides thenodesbetweensourceandsink.

Whatisthecostofast-cut?

SumofcostofalledgesgoingfromStoT

5 + 2 + 9 = 16

Min-Cut

32


Whatisast-cut?

Anst-cut(S,T)divides thenodesbetweensourceandsink.

Whatisthecostofast-cut?

SumofcostofalledgesgoingfromStoT

Source

Sink

v1 v2

2

5

9

42

1

2 + 1 + 4 = 7

Whatisthest-mincut?

st-cutwiththeminimumcost

Howtocomputemin-cut?

33


Source

Sink

v1 v2

2

5

9

42

1

Solve the dual maximum flow problem

In every network, the maximum flow equals the cost of the st-mincut

Min-cut\Max-flow Theorem

Compute the maximum flow between Source and Sink

Constraints

Edges: Flow < Capacity

Nodes: Flow in & Flow out

Max-FlowAlgorithms

34


Augmenting Path Based Algorithms

1. Find path from source to sink with positive capacity

2. Push maximum possible flow through this path

3. Repeat until no path can be found

Source

Sink

v1 v2

2

5

9

42

1

Algorithms assume non-negative capacity

Flow=0

Max-FlowAlgorithms

35







Source

Sink

v1 v2

2

5

9

42

1

Flow=0

Max-FlowAlgorithms

36







Source

Sink

v1 v2

2-2

5-2

9

42

1

Flow=0+2

Max-FlowAlgorithms

37







Source

Sink

v1 v2

0

3

9

42

1

Flow=2

Max-FlowAlgorithms

38







Source

Sink

v1 v2

0

3

9

42

1

Flow=2

Max-FlowAlgorithms

39







Source

Sink

v1 v2

0

3

9

42

1

Flow=2

Max-FlowAlgorithms

40







Source

Sink

v1 v2

0

3

5

02

1

Flow=2+4

Max-FlowAlgorithms

41







Source

Sink

v1 v2

0

3

5

02

1

Flow=6

Max-FlowAlgorithms

42







Source

Sink

v1 v2

0

3

5

02

1

Flow=6

Max-FlowAlgorithms

43







Source

Sink

v1 v2

0

2

4

02+1

1-1

Flow=6+1

Max-FlowAlgorithms

44







Source

Sink

v1 v2

0

2

4

03

0

Flow=7

Max-FlowAlgorithms

45







Source

Sink

v1 v2

0

2

4

03

0

Flow=7

AnotherExample-MaxFlow

46


source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

Ford & Fulkerson algorithm (1956)

Find the path from source to sink

While (path exists)

flow += maximum capacity in the path

Build the residual graph (“subtract” the flow)

Find the path in the residual graph

End


47


source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3



While (path exists)




End


48


source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3



While (path exists)




End

flow = 3


49

Lecture13:AdvanceImageSegmentation

source

sink

9

5

6-3

8-3

42+3

2

2

5-3

3-3

5

5

311

6

2

3



While (path exists)




End

flow = 3

+3


50


source

sink

9

5

3

5

45

2

2

2

5

5

311

6

2

3



While (path exists)




End

flow = 3

3


51


source

sink

9

5

3

5

45

2

2

2

5

5

311

6

2

3



While (path exists)




End

flow = 3

3


52


source

sink

9

5

3

5

45

2

2

2

5

5

311

6

2

3



While (path exists)




End

flow = 6

3


53


source

sink

9-3

5

3

5-3

45

2

2

2

5

5

3-311

6

2

3-3



While (path exists)




End

flow = 6

3

+3

+3


54


source

sink

6

5

3

2

45

2

2

2

5

5

11

6

2



While (path exists)




End

flow = 6

3

3

3


55


source

sink

6

5

3

2

45

2

2

2

5

5

11

6

2



While (path exists)




End

flow = 6

3

3

3


56


source

sink

6

5

3

2

45

2

2

2

5

5

11

6

2



While (path exists)




End

flow = 11

3

3

3


57


source

sink

6-5

5-5

3

2

45

2

2

2

5-5

5-5

1+51

6

2+5



While (path exists)




End

flow = 11

3

3

3


58


source

sink

1

3

2

45

2

2

2

61

6

7



While (path exists)




End

flow = 11

3

33


59


source

sink

1

3

2

45

2

2

2

61

6

7



While (path exists)




End

flow = 11

3

33


60


source

sink

1

3

2

45

2

2

2

61

6

7



While (path exists)




End

flow = 13

3

33


61


source

sink

1

3

2-2

45

2-2

2-2

2-2

61

6

7



While (path exists)




End

flow = 13

3

33

+2

+2


62


source

sink

1

3

45

61

6

7



While (path exists)




End

flow = 13

3

33

2

2


63


source

sink

1

3

45

61

6

7



While (path exists)




End

flow = 13

3

33

2

2


64


source

sink

1

3

45

61

6

7



While (path exists)




End

flow = 15

3

33

2

2


65


source

sink

1

3-2

4-25

61

6-2

7



While (path exists)




End

flow = 15

3

33+2

2

2-2

+2


66


source

sink

1

1

5

61

4

7



While (path exists)




End

flow = 15

3

35

2

2

2


67


source

sink

1

1

5

61

4

7



While (path exists)




End

flow = 15

3

35

2

2

2


68


source

sink

1

1

5

61

4

7



While (path exists)




End

flow = 15

3

35

2

2

2


69


source

sink

1

1

5

61

4

7


Why is the solution globally optimal ?

flow = 15

3

35

2

2

2


70


source

sink

1

1

5

61

4

7



1. Let S be the set of reachable nodes in the

residual graph

flow = 15

3

35

2

2

2

S


71




1. Let S be the set of reachable nodes in the

residual graph

2. The flow from S to V - S equals to the sum

of capacities from S to V – S

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

S


72


flow = 15



1. Let S be the set of reachable nodes in the residual graph

2. The flow from S to V - S equals to the sum of capacities from S to V – S

3. The flow from any A to V - A is upper bounded by the sum of capacities from A to V – A

4. The solution is globally optimal

source

sink

8/9

5/5

5/6

8/8

2/40/2

2/2

0/2

5/5

3/3

5/5

5/5

3/30/10/1

2/6

0/2

3/3

Individual flows obtained by summing up all paths


73


source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

S

T

cost = 18

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

S

T


74


source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

S

T

cost = 23


75


C(x) = 5x1 + 9x2 + 4x3 + 3x3(1-x1) + 2x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 6x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)

+ 8(1-x5) + 5(1-x6)

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1 x2

x3

x4

x5

x6


76


C(x) = 2x1 + 9x2 + 4x3 + 2x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)

+ 5(1-x5) + 5(1-x6)

+ 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5)

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1 x2

x3

x4

x5

x6


77


C(x) = 2x1 + 9x2 + 4x3 + 2x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 3x6(1-x2) + 2x4(1-x5) + 3x6(1-x5) + 6(1-x4)

+ 5(1-x5) + 5(1-x6)

+ 3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5)

3x1 + 3x3(1-x1) + 3x5(1-x3) + 3(1-x5)

=

3 + 3x1(1-x3) + 3x3(1-x5)

source

sink

9

5

6

8

42

2

2

5

3

5

5

311

6

2

3

x1 x2

x3

x4

x5

x6


78


C(x) = 3 + 2x1 + 6x2 + 4x3 + 5x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 2x5(1-x4) + 6(1-x4)

+ 2(1-x5) + 5(1-x6) + 3x3(1-x5)

+ 3x2 + 3x6(1-x2) + 3x5(1-x6) + 3(1-x5)

3x2 + 3x6(1-x2) + 3x5(1-x6) + 3(1-x5)

=

3 + 3x2(1-x6) + 3x6(1-x5)

source

sink

9

5

3

5

45

2

2

2

5

5

311

6

2

33

x1 x2

x3

x4

x5

x6


79


C(x) = 6 + 2x1 + 6x2 + 4x3 + 5x1(1-x3)

+ 3x3(1-x1) + 2x2(1-x3) + 5x3(1-x2) + 2x4(1-x1)

+ 1x5(1-x1) + 3x5(1-x3) + 5x6(1-x3) + 1x3(1-x6)

+ 2x5(1-x4) + 6(1-x4) + 3x2(1-x6) + 3x6(1-x5)

+ 2(1-x5) + 5(1-x6) + 3x3(1-x5)

source

sink

6

5

3

2

45

2

2

2

5

5

11

6

2

3

3

3

x1 x2

x3

x4

x5

x6


80


C(x) = 15 + 1x2 + 4x3 + 5x1(1-x3)

+ 3x3(1-x1) + 7x2(1-x3) + 2x1(1-x4)

+ 1x5(1-x1) + 6x3(1-x6) + 6x3(1-x5)

+ 2x5(1-x4) + 4(1-x4) + 3x2(1-x6) + 3x6(1-x5)

source

sink

1

1

5

61

4

7

3

35

2

2

2x1 x2

x3

x4

x5

x6 cost = 0

min cut

S

T

cost = 0

• All coefficients positive

• Must be global minimum

S – set of reachable nodes from s

HistoryofMax-FlowAlgorithms

81


Augmenting Path and Push-Relabel

n:#nodesm:#edgesU:maximumedgeweight

Algorithms assume non-negative edge

weights

ExampleSegmentationinaVideo

82


Image Flow GlobalOptimum

pqw

n-links st-cuts

t

= 0

= 1

E(x) x*

SegmentationExample

83


SegmentationExample

84


SegmentationExample

85


Optimality?

86


Seeds GrabCut(iterativeGCWithboxprior)

VascularSurfaceSegmentationviaGC10/1/15

87


SlideCreditsandReferences• Fredo DurandofMIT• M.Tappen ofAmazon• R.Szelisky,Univ.ofWashington/Seatle• http://www.csd.uwo.ca/faculty/yuri/Abstracts/eccv06-tutorial.html• J.Malcolm,GraphCutinTensorScale• InteractiveGraphCutsforOptimalBoundary&RegionSegmentationofObjectsin

N-Dimages.YuriBoykov andMarie-Pierre Jolly.InInternationalConference onComputerVision,(ICCV),vol.I,2001.http://www.csd.uwo.ca/~yuri/Abstracts/iccv01-abs.html

• http://www.cse.yorku.ca/~aaw/Wang/MaxFlowStart.htm• http://research.microsoft.com/vision/cambridge/i3l/segmentation/GrabCut.htm• http://www.cc.gatech.edu/cpl/projects/graphcuttextures/• AComparativeStudyofEnergyMinimizationMethodsforMarkovRandomFields.

RickSzeliski,Ramin Zabih,DanielScharstein,OlgaVeksler,VladimirKolmogorov,Aseem Agarwala,MarshallTappen,Carsten Rother.ECCV2006www.cs.cornell.edu/~rdz/Papers/SZSVKATR.pdf

• P.Kumar,OxfordUniversity.

88


dr. ulas bagci bagci@ucfbagci/teaching/computervision16/lec13.pdfdr. ulas bagci [email protected]...

Documents