robust solvers for square jigsaw puzzles

Robust Solvers for Square Jigsaw Puzzles

Debajyoti MondalYang Wang Stephane Durocher

Department of Computer ScienceUniversity of Manitoba

CRV 2013 231/05/2013

What are Jigsaw Puzzles?

CRV 2013 331/05/2013

Square Jigsaw Puzzles

24×18 = 432 puzzle pieces

CRV 2013 431/05/2013

State-of-Art Solvers

Pomeranz, Shemesh and Ben-Shahar

CVPR 2011

Cho, Avidan and Freeman

CVPR 2010

CVPR 2012

Andrew Gallagher

Solved puzzles with 432 pieces

Average 10% accuracy on432 piece puzzles

Solved puzzles with 3300 pieces

Average 94% accuracy on432 piece puzzles

Solved puzzles with 9600 pieces

Average 95% accuracy on432 piece puzzles

http://www.cs.bgu.ac.il/faculty/person/dolevp.html http://www.cs.bgu.ac.il/faculty/person/shemeshm.html http://www.cs.bgu.ac.il/~ben-shahar/

http://www.eng.tau.ac.il/~avidan/ http://people.csail.mit.edu/taegsang/ http://people.csail.mit.edu/billf/

http://chenlab.ece.cornell.edu/people/Andy/

CRV 2013 531/05/2013

Why Solving Jigsaw Puzzles ?

Restore Torn Apart Documents

http://www.bouldercitysocial.com/wp-content/uploads/2011/04/paperShredding.jpg

Fossil Reconstructionhttp://www.aim.uzh.ch/morpho/wiki//CAP/3-2

Ancient art/document reassemblyhttp://www.edgarlowen.com/n1/b7300.jpg

CRV 2013 631/05/2013

Our Robust Jigsaw Solver (Noise and Missing Boundary)

CRV 2013 731/05/2013

Our Robust Jigsaw Solver (Noise and Missing Boundary)

CRV 2013 831/05/2013

How to Solve a Puzzle?

Xi Xj

Xi Xk

Xi Xj

Xi Xk

Xi Xj

Xi Xk

Xi Xj

Xi Xk

Xi Xj

Xi Xk

Xi Xj

Xi Xk

CRV 2013 931/05/2013

Successful Strategies

Pomeranz et. al. [CVPR 2011]Sum of Squared Distance (SSD)

Gallagher [CVPR 2012]Mahalanobis Gradient Compatibility (MGC)

SSD ( xi , xj ) = DLR ( xi , xj ) MGC ( xi , xj ) = f (μi , Gij)

CRV 2013 1031/05/2013

Our Approach: M+S

(M+S) Compatibility Score = MGC( xi , xj ) SSD( xi , xj )1/q.

MGC

SSD

M+S,4

M+S,5

M+S,6

M,S,7MGC

SSD

M+S

20 images, each with 432 Puzzle Pieces of size 28×28×3

CRV 2013 1131/05/2013

Further Refinements

MGCScoring matrix

(M+S) Compatibility Score = MGC( xi , xj ) SSD( xi , xj ) 1/q.

5 3 9 1 6 7 2 4 8

| MGC(3,1) - MGC(3,2) | < σ

Row 3

CRV 2013 1231/05/2013

How to Refine this further?

MGCScoring matrix

(M+S) Compatibility Score = MGC( xi , xj ) SSD( xi , xj ) 1/q.

5 3 9 1 6 7 2 4 8

| MGC(3,1) - MGC(3,2) | < σ

Row 3

Greedy choice!No global Agreement!

CRV 2013 1331/05/2013

Selectively Weighted MGC (wMGC)

MGCScoring matrix

3

2 2

3

CRV 2013 1431/05/2013

Selectively Weighted MGC (wMGC)

MGCScoring matrix

A bijection with optimum weight

CRV 2013 1531/05/2013

Selectively Weighted MGC (wMGC)

5

2

MGCScoring matrix

5 3 9 1 6 7 2 4 8

Row 2

wMGC (xi , xj) =

Column 4

(M+S) Score, if ‘Conflict’MGC Score, otherwise.

CRV 2013 1631/05/2013

Selectively Weighted MGC (wMGC)

5

2

MGCScoring matrix

5 3 9 1 6 7 2 4 8

Row 2

wMGC (xi , xj) =

Column 4

(M+S) Score, if ‘Conflict’MGC Score, otherwise.

CRV 2013 1731/05/2013

Experimental ResultsMGC

SSD

M+S

wMGC,4

(M+S) Compatibility Score = MGC( xi , xj ) SSD( xi , xj )1/q.

wMGC (xi , xj)

=(M+S) Score, if ‘Conflict’MGC Score, otherwise.

20 images, each with 432 Puzzle Pieces of size 28×28×3

MGC

SSD

M+S

wMGC

CRV 2013 1831/05/2013

Gallagher’s Reassembly [CVPR 2012]

Scoring Matrix

Construct Spanning Tree Trimming Filling

CRV 2013 1931/05/2013

Results

Perfect Noisy Cropped 0

50

100

150

200

250

300

350

Forest

SSDMGCOurs

Perfect Noisy Cropped 0

50

100

150

200

250

300

350

City

SSDMGCOurs

MIT scene database, 328 images of forest, 308 images of city81 pieces per puzzle, each piece of size 28×28×3

CRV 2013 2031/05/2013

Future Research

Image Filtering?

How much does it help if we know the image category?

Robust functions for compatibility scoring.

Thank You