Contour Correspondence via Ant Colony Optimization

Oliver van Kaick Ghassan Hamarneh Hao Zhang Paul WightonGrUVi Lab MIAL GrUVi Lab MCL

School of Computing ScienceSimon Fraser University

Outline

1 Introduction

2 Related work on correspondence

3 Review of the ACO framework

4 ACO for shape correspondence

5 Experimental results

6 Conclusions

Contour Correspondence via Ant Colony Optimization 2 / 39

Outline

1 Introduction

2 Related work on correspondence

3 Review of the ACO framework

4 ACO for shape correspondence

5 Experimental results

6 Conclusions

Contour Correspondence via Ant Colony Optimization 3 / 39

Introduction

Shape correspondence

Finding a meaningful matching between two shapes

Focus: 2D contours

Contour Correspondence via Ant Colony Optimization 4 / 39

Introduction

Shape correspondence

Finding a meaningful matching between two shapes

Focus: 2D contours

Contour Correspondence via Ant Colony Optimization 4 / 39

Introduction

Applications in

Computer graphics (shape analysis, morphing, and animation)Computer vision (object tracking, recognition, and retrieval)Medical computing (statistical shape modeling and analysis ofanatomical structures)

3D shape matching and retrieval (Chen et al., 2003)

Contour Correspondence via Ant Colony Optimization 5 / 39

Solving contour correspondence

Common approach

1 Select feature points

2 Compute shape descriptors3 Extract a matching

Greedy best matchingBipartite matchingIterative closest point (ICP) schemeDynamic programming under point ordering

Contour Correspondence via Ant Colony Optimization 6 / 39

Solving contour correspondence

Common approach

1 Select feature points

2 Compute shape descriptors3 Extract a matching

Greedy best matchingBipartite matchingIterative closest point (ICP) schemeDynamic programming under point ordering

Contour Correspondence via Ant Colony Optimization 6 / 39

Incorporating proximity information

Most algorithms do not consider proximity informationProximity: if two points are close on one shape, their correspondingpoints in the second shape should also be close

Better handling of missing parts or a lack of salient features

Contour Correspondence via Ant Colony Optimization 7 / 39

Incorporating proximity information

Most algorithms do not consider proximity informationProximity: if two points are close on one shape, their correspondingpoints in the second shape should also be close

Better handling of missing parts or a lack of salient features

Contour Correspondence via Ant Colony Optimization 7 / 39

Incorporating proximity information

Most algorithms do not consider proximity informationProximity: if two points are close on one shape, their correspondingpoints in the second shape should also be close

Better handling of missing parts or a lack of salient features

Contour Correspondence via Ant Colony Optimization 7 / 39

Incorporating proximity information

Most algorithms do not consider proximity informationProximity: if two points are close on one shape, their correspondingpoints in the second shape should also be close

Better handling of missing parts or a lack of salient features

Contour Correspondence via Ant Colony Optimization 7 / 39

Incorporating proximity information

Most algorithms do not consider proximity informationProximity: if two points are close on one shape, their correspondingpoints in the second shape should also be close

Better handling of missing parts or a lack of salient features

Contour Correspondence via Ant Colony Optimization 7 / 39

Incorporating proximity information

Take advantage of the vertex ordering (contours)

Enforcing order preservation 6= Enforcing proximity

Contour Correspondence via Ant Colony Optimization 8 / 39

QAP

When proximity is incorporated, we can formulate pointcorrespondence via the Quadratic Assignment Problem (QAP)

QAP is one of the most difficult optimization problems

Ant Colony Optimization (ACO) has had great success in solvingthis problem

Contour Correspondence via Ant Colony Optimization 9 / 39

Contributions

We formulate the general point correspondence problem in terms ofQAP incorporating proximity information

We propose the first ACO algorithm to compute the matching

Applicable to contours and unorganized 2D point sets

We extend the framework to enforce order preservation (contours)

Contour Correspondence via Ant Colony Optimization 10 / 39

Outline

1 Introduction

2 Related work on correspondence

3 Review of the ACO framework

4 ACO for shape correspondence

5 Experimental results

6 Conclusions

Contour Correspondence via Ant Colony Optimization 11 / 39

Related work on correspondence

Matching shape descriptors2D shape matching

Bipartite matching solved using the Hungarian algorithmInteger constrained minimization (Maciel and Costeira, 2003)Soft assign algorithm (Gold et al., 1998)Preservation of binary neighborhood information (Zheng andDoermann, 2006)QAP formulation (Berg et al., 2005)

Contour correspondence

Order preservation and dynamic programming (Liu et al., 2004, Scottand Nowak, 2006)

Contour Correspondence via Ant Colony Optimization 12 / 39

Related work on correspondence

Matching shape descriptors2D shape matching

Bipartite matching solved using the Hungarian algorithmInteger constrained minimization (Maciel and Costeira, 2003)Soft assign algorithm (Gold et al., 1998)Preservation of binary neighborhood information (Zheng andDoermann, 2006)QAP formulation (Berg et al., 2005)

Contour correspondence

Order preservation and dynamic programming (Liu et al., 2004, Scottand Nowak, 2006)

Contour Correspondence via Ant Colony Optimization 12 / 39

Related work on correspondence

Matching shape descriptors2D shape matching

Bipartite matching solved using the Hungarian algorithmInteger constrained minimization (Maciel and Costeira, 2003)Soft assign algorithm (Gold et al., 1998)Preservation of binary neighborhood information (Zheng andDoermann, 2006)QAP formulation (Berg et al., 2005)

Contour correspondence

Order preservation and dynamic programming (Liu et al., 2004, Scottand Nowak, 2006)

Contour Correspondence via Ant Colony Optimization 12 / 39

Related work on correspondence

Without using local descriptors:

Physically-based approach (Sederberg and Greenwood, 1992)Pattern matching in the Gaussian map (Tal and Elber, 1999)Deformation-based edit distance (Sebastian et al., 2003)Skeletal and shock graphs (Sundar et al., 2003, Siddiqi et al., 1999)

Transform-based techniques (Shapiro and Brady, 1992, Sclaroff andPentland, 1995, Bronstein et al., 2006, Jain et al., 2007)

Group correspondence

Minimum Description Length (MDL) principle (Davies et al., 2002)

Contour Correspondence via Ant Colony Optimization 13 / 39

Outline

1 Introduction

2 Related work on correspondence

3 Review of the ACO framework

4 ACO for shape correspondence

5 Experimental results

6 Conclusions

Contour Correspondence via Ant Colony Optimization 14 / 39

ACO facts

Finds good solutions to NP-hard problems (Dorigo et al., 1996)Routing, assignment, and schedulingInspiration from nature

Individual ants have a simple behaviorThe ant colony can solve difficult problems

Ant characteristicsForaging behavior (search)Pheromone trail (communication)

Contour Correspondence via Ant Colony Optimization 15 / 39

ACO facts

Finds good solutions to NP-hard problems (Dorigo et al., 1996)Routing, assignment, and schedulingInspiration from nature

Individual ants have a simple behaviorThe ant colony can solve difficult problems

Ant characteristicsForaging behavior (search)Pheromone trail (communication)

Contour Correspondence via Ant Colony Optimization 15 / 39

ACO facts

Finds good solutions to NP-hard problems (Dorigo et al., 1996)Routing, assignment, and schedulingInspiration from nature

Individual ants have a simple behaviorThe ant colony can solve difficult problems

Ant characteristicsForaging behavior (search)Pheromone trail (communication)

Contour Correspondence via Ant Colony Optimization 15 / 39

ACO facts

Finds good solutions to NP-hard problems (Dorigo et al., 1996)Routing, assignment, and schedulingInspiration from nature

Individual ants have a simple behaviorThe ant colony can solve difficult problems

Ant characteristicsForaging behavior (search)Pheromone trail (communication)

Contour Correspondence via Ant Colony Optimization 15 / 39

ACO facts

Finds good solutions to NP-hard problems (Dorigo et al., 1996)Routing, assignment, and schedulingInspiration from nature

Individual ants have a simple behaviorThe ant colony can solve difficult problems

Ant characteristicsForaging behavior (search)Pheromone trail (communication)

Contour Correspondence via Ant Colony Optimization 15 / 39

ACO facts

Finds good solutions to NP-hard problems (Dorigo et al., 1996)Routing, assignment, and schedulingInspiration from nature

Individual ants have a simple behaviorThe ant colony can solve difficult problems

Ant characteristicsForaging behavior (search)Pheromone trail (communication)

Contour Correspondence via Ant Colony Optimization 15 / 39

ACO facts

Finds good solutions to NP-hard problems (Dorigo et al., 1996)Routing, assignment, and schedulingInspiration from nature

Individual ants have a simple behaviorThe ant colony can solve difficult problems

Ant characteristicsForaging behavior (search)Pheromone trail (communication)

Contour Correspondence via Ant Colony Optimization 15 / 39

ACO facts

Finds good solutions to NP-hard problems (Dorigo et al., 1996)Routing, assignment, and schedulingInspiration from nature

Individual ants have a simple behaviorThe ant colony can solve difficult problems

Ant characteristicsForaging behavior (search)Pheromone trail (communication)

Contour Correspondence via Ant Colony Optimization 15 / 39

ACO facts

Finds good solutions to NP-hard problems (Dorigo et al., 1996)Routing, assignment, and schedulingInspiration from nature

Individual ants have a simple behaviorThe ant colony can solve difficult problems

Ant characteristicsForaging behavior (search)Pheromone trail (communication)

Contour Correspondence via Ant Colony Optimization 15 / 39

ACO facts

Finds good solutions to NP-hard problems (Dorigo et al., 1996)Routing, assignment, and schedulingInspiration from nature

Individual ants have a simple behaviorThe ant colony can solve difficult problems

Ant characteristicsForaging behavior (search)Pheromone trail (communication)

Contour Correspondence via Ant Colony Optimization 15 / 39

Review of the ACO framework

Problem modeled with a graph

The solution search involves ants traversing this graph

ACO metaheuristic:

For each iteration:1 Traverse the graph (construct solution)

Heuristic information and pheromones

2 Evaluate solution

Objective function

3 Deposit pheromones on the edges of the graph

Quality of the solution

Contour Correspondence via Ant Colony Optimization 16 / 39

Review of the ACO framework

Problem modeled with a graph

The solution search involves ants traversing this graph

ACO metaheuristic:

For each iteration:1 Traverse the graph (construct solution)

Heuristic information and pheromones

2 Evaluate solution

Objective function

3 Deposit pheromones on the edges of the graph

Quality of the solution

Contour Correspondence via Ant Colony Optimization 16 / 39

Advantages of ACO

Probabilistic approach

Heuristic information

Escape from bad local minima

Parallelizable

Contour Correspondence via Ant Colony Optimization 17 / 39

Outline

1 Introduction

2 Related work on correspondence

3 Review of the ACO framework

4 ACO for shape correspondence

5 Experimental results

6 Conclusions

Contour Correspondence via Ant Colony Optimization 18 / 39

ACO for shape correspondence

i i

i

1

2

3

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

i i

i

1

2

3 jj

j j

14

23

Shapes to be matched

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

i i

i

1

2

3 jj

j j

14

23

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

jj

j j

14

23

i

i

i

2

1

3

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

ACO graph

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

Iteration start

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

π(i2) = j1

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

π(i3) = j3

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

π(i1) = j2

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

Iteration end

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

Correspondence obtained

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

1

2

3

1

2

3

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

Cost = X

Cost is computed

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

j

j

j

j

1

2

3

4

i1

i2

i3

Cost = X

Pheromone is deposited

Contour Correspondence via Ant Colony Optimization 19 / 39

ACO for shape correspondence

For a fixed number of iterations:1 Traverse the graph

Heuristic information and pheromones

2 Evaluate solution

Objective function

3 Deposit pheromones on the edges of the graph

Quality of the solution

4 Retain best solution

Contour Correspondence via Ant Colony Optimization 20 / 39

ACO for shape correspondence

For a fixed number of iterations:1 Traverse the graph

Heuristic information and pheromones

2 Evaluate solution

Objective function

3 Deposit pheromones on the edges of the graph

Quality of the solution

4 Retain best solution

Contour Correspondence via Ant Colony Optimization 20 / 39

Cost function: QAP formulation

QAP =

(1−ν)

Descriptor distance +

ν

Proximity

Descriptor distance =

1− 1

|I |·

∑i∈I

exp(−)

Proximity =

Contour Correspondence via Ant Colony Optimization 21 / 39

Cost function: QAP formulation

QAP = (1−ν)Descriptor distance + ν Proximity

Descriptor distance =

1− 1

|I |·

∑i∈I

exp(−)

Proximity =

Contour Correspondence via Ant Colony Optimization 21 / 39

Cost function: QAP formulation

QAP = (1−ν)Descriptor distance + ν Proximity

Descriptor distance =

1− 1

|I |·

∑i∈I

exp

(−

dist(Ri ,Rπ(i))

2

σ2R

)Proximity =

Contour Correspondence via Ant Colony Optimization 21 / 39

Cost function: QAP formulation

QAP = (1−ν)Descriptor distance + ν Proximity

Descriptor distance =

1− 1

|I |·

∑i∈I

exp

(− dist(Ri ,Rπ(i))

2

σ2R

)

Proximity =

Contour Correspondence via Ant Colony Optimization 21 / 39

Cost function: QAP formulation

QAP = (1−ν)Descriptor distance + ν Proximity

Descriptor distance = 1− 1

|I |·∑i∈I

exp

(− dist(Ri ,Rπ(i))

2

σ2R

)

Proximity =

Contour Correspondence via Ant Colony Optimization 21 / 39

Cost function: QAP formulation

QAP = (1−ν)Descriptor distance + ν Proximity

Descriptor distance = 1− 1

|I |·∑i∈I

exp

(− dist(Ri ,Rπ(i))

2

σ2R

)

Proximity =∑i∈I

∑i ′ 6=i∈I

exp

(−dist(i , i ′)2

σ2I

)

∣∣dist(i , i ′)−dist(π(i),π(i ′))∣∣

|I |(|I |−1)/2

Contour Correspondence via Ant Colony Optimization 21 / 39

Cost function: QAP formulation

QAP = (1−ν)Descriptor distance + ν Proximity

Descriptor distance = 1− 1

|I |·∑i∈I

exp

(− dist(Ri ,Rπ(i))

2

σ2R

)

Proximity =∑i∈I

∑i ′ 6=i∈I

exp

(−dist(i , i ′)2

σ2I

)∣∣dist(i , i ′)−dist(π(i),π(i ′))∣∣

|I |(|I |−1)/2

Contour Correspondence via Ant Colony Optimization 21 / 39

Cost function: QAP formulation

QAP = (1−ν)Descriptor distance + ν Proximity

Descriptor distance = 1− 1

|I |·∑i∈I

exp

(− dist(Ri ,Rπ(i))

2

σ2R

)

Proximity =∑i∈I

∑i ′ 6=i∈I

exp

(−dist(i , i ′)2

σ2I

)∣∣dist(i , i ′)−dist(π(i),π(i ′))∣∣

|I |(|I |−1)/2

Contour Correspondence via Ant Colony Optimization 21 / 39

Edge traversal

j

j

j

j

1

2

3

4

i1

i2

i3

Edge probability:

pij = α Pheromones + (1−α)Heuristic

Heuristic information:

Contour Correspondence via Ant Colony Optimization 22 / 39

Edge traversal

j

j

j

j

1

2

3

4

i1

i2

i3

Edge probability:

pij = α Pheromones + (1−α)Heuristic

Heuristic information:

Contour Correspondence via Ant Colony Optimization 22 / 39

Edge traversal

j

j

j

j

1

2

3

4

i1

i2

i3

Edge probability:

pij = α Pheromones + (1−α)Heuristic

Heuristic information:

Heuristic−1 =dist(Ri ,Rj)×

Descriptor similarity

|dist(i , i ′)−dist(j ,π(i ′))| ×Proximity to last vertex

|dist(i , i ′′)−dist(j ,π(i ′′))| ×Proximity to second last vertex

Contour Correspondence via Ant Colony Optimization 22 / 39

Edge traversal

j

j

j

j

1

2

3

4

i1

i2

i3

Edge probability:

pij = α Pheromones + (1−α)Heuristic

Heuristic information:

Heuristic =

(e

−dist(Ri ,Rj )2

σ2R

)×(

1− e−dist(i ,i ′)2

σ2I |dist(i , i ′)−dist(j ,π(i ′))|

)×(

1− e−dist(i ,i ′′)2

σ2I |dist(i , i ′′)−dist(j ,π(i ′′))|

)

Contour Correspondence via Ant Colony Optimization 22 / 39

Order preservation

j

j

j

j

1

2

3

4

i1

i2

i3

Hard constraints: by removingedges that should not be traversed

Order preservation

Soft constraints: by modifying theprobabilities

Contour Correspondence via Ant Colony Optimization 23 / 39

Order preservation

j

j

j

j

1

2

3

4

i1

i2

i3

Hard constraints: by removingedges that should not be traversed

Order preservation

Soft constraints: by modifying theprobabilities

Contour Correspondence via Ant Colony Optimization 23 / 39

Order preservation

j

j

j

j

1

2

3

4

i1

i2

i3

Hard constraints: by removingedges that should not be traversed

Order preservation

Soft constraints: by modifying theprobabilities

Contour Correspondence via Ant Colony Optimization 23 / 39

List of ACO parameters and values

ACO Parameters Symbol ValueNumber of ants m 1Number of iterations T 1000Influence of pheromones α 0.3Pheromone evaporation rate ρ 0.1Pheromone deposition constant δ 0.01Initial pheromone levels τ0 1Minimum pheromone levels τmin 0.1 · 1

|I |Influence of proximity ν 0.7Gaussian width in X σI 0.1 · Imax

Gaussian width in S σR 0.1 ·Rmax

Parameters used in our ACO algorithm and their chosen values

The values are fixed in all the experiments

Contour Correspondence via Ant Colony Optimization 24 / 39

Outline

1 Introduction

2 Related work on correspondence

3 Review of the ACO framework

4 ACO for shape correspondence

5 Experimental results

6 Conclusions

Contour Correspondence via Ant Colony Optimization 25 / 39

Experimental results

Contours extracted from the Brown dataset (Sharvit et al., 1998)

Shape context (rotation-variant) descriptor (Belongie et al., 2002)

Contour Correspondence via Ant Colony Optimization 26 / 39

ACO pheromone deposition

Contour Correspondence via Ant Colony Optimization 27 / 39

ACO solution computation

Contour Correspondence via Ant Colony Optimization 28 / 39

ACO solution computation

Contour Correspondence via Ant Colony Optimization 29 / 39

ACO solution computation

Contour Correspondence via Ant Colony Optimization 30 / 39

ACO solution computation

Contour Correspondence via Ant Colony Optimization 31 / 39

Order preservation (OP) vs. proximity

(a) Contours to match (b) OP

(c) ACO (Proximity only) (d) ACO (Proximity + OP)

Contour Correspondence via Ant Colony Optimization 32 / 39

Handling of occlusion or missing parts

(a) (b)

(c) (d)

Matchings computed by ACO for contourswith occlusion or structure change

Contour Correspondence via Ant Colony Optimization 33 / 39

Handling of open contours

(a) Frame (b) Matching

Matching computed by ACO for an open contour of a left ventricle

Contour Correspondence via Ant Colony Optimization 34 / 39

Evaluation against ground-truth correspondence

Ground truth provided by a human user

Error is the sum of geodesic distances between corresponded verticesand ground truth (Karlsson and Ericsson, 2006)

Compared to Hungarian and COPAP (Scott and Nowak, 2006)

Shape class Hungarian COPAP ACOAirplanes 223.16 32.55 13.02Fish 56.85 21.67 22.80Four-legged 235.57 32.58 25.48Hands 375.94 94.86 121.95Humans 482.27 53.75 20.95Rabbits 190.01 80.01 53.44Stingrays 30.55 5.88 5.16Tools 204.36 35.29 22.48

Deviation from ground truth

Contour Correspondence via Ant Colony Optimization 35 / 39

Timing

0 50 100 150 200 250 300 350 400 4500

500

1000

1500

2000

2500

3000

Number of vertices in contour

Exe

cutio

n tim

e (s

)

COPAPHungarianACO

Execution time comparison between Hungarian, COPAP, and ACO

Contour Correspondence via Ant Colony Optimization 36 / 39

Outline

1 Introduction

2 Related work on correspondence

3 Review of the ACO framework

4 ACO for shape correspondence

5 Experimental results

6 Conclusions

Contour Correspondence via Ant Colony Optimization 37 / 39

Conclusions and future work

Proximity is incorporated

The QAP is solved using the proposed ACO algorithm

Advantages include

Proximity improves the resultsResults are generally betterResource requirements scale moderatelyApplicable to contours and unorganized 2D point setsHard and soft constraints can be easily incorporated

Future work

Extension to 2D manifoldsParallelization

Contour Correspondence via Ant Colony Optimization 38 / 39

Conclusions and future work

Proximity is incorporated

The QAP is solved using the proposed ACO algorithm

Advantages include

Proximity improves the resultsResults are generally betterResource requirements scale moderatelyApplicable to contours and unorganized 2D point setsHard and soft constraints can be easily incorporated

Future work

Extension to 2D manifoldsParallelization

Contour Correspondence via Ant Colony Optimization 38 / 39

Conclusions and future work

Proximity is incorporated

The QAP is solved using the proposed ACO algorithm

Advantages include

Proximity improves the resultsResults are generally betterResource requirements scale moderatelyApplicable to contours and unorganized 2D point setsHard and soft constraints can be easily incorporated

Future work

Extension to 2D manifoldsParallelization

Contour Correspondence via Ant Colony Optimization 38 / 39

Acknowledgments

Thank you!

Funding for this project was provided by

Faculty of Applied SciencesSimon Fraser University

Contour Correspondence via Ant Colony Optimization 39 / 39