region filling and object removal by exemplar based image inpainting

Region Filling and Object Removal by Exemplar-Based Image Inpainting

Criminisi, A., Perez, P., and Toyama, K. (2004)

Lee, WoongheeM.S. student at the Big Data Mining Lab.

Department of computer science and engineering at the Hanyang UniversityOctober 4th, 2015

Contents Prerequisite Key Observations Region Filling Algorithm Results and Comparisons

Prerequisite Texture synthesis Inpainting

PrerequisiteTo fill large image repetitively 2-D texture synthesis

inpainting

Exmaples from Ashikimin [1]

PrerequisiteProperties

• Cheap cost to generate image• Effectively generating image• Difficult to fill holes in photos• A complex product of mutual

influences between different boundaries

texture synthesis

inpainting

PrerequisiteTo fill holes in images by propagating linear structures (called isophote)

texture synthesis

inpainting

PrerequisiteTo fill holes in images by propagating linear structures (called isophote)

Depends on Gestalt Law of Continuation

texture synthesis

inpainting

PrerequisiteGestalt Law of Continuation texture synthesis

inpainting

PrerequisiteGestalt Law of Continuation texture synthesis

inpainting

PrerequisiteGestalt Law of Continuation

Human perceives a dotted line as a full line by implicit continuation.

texture synthesis

inpainting

PrerequisitePropagation direction texture synthesis

inpainting

propagate along isophotes

PrerequisiteProperties

• Effective to fill speckles, scratches, and overlaid text

• Causes noticeable blur to fill large regions

• Extremely slow (83’-158’ on a 384 X 256 image)

texture synthesis

inpainting

Main Idea

To combine the advantages of “texture synthesis” and “inpainting”

Key ObservationsA. Exemplar-Based Synthesis SufficesAlgorithm Core: Isophote-driven image-sampling process

Key ObservationsA. Exemplar-Based Synthesis Suffices

Key ObservationsB. Filling Order is Critical

artefacts

Key ObservationsB. Filling Order is Critical

Onion peel(concentric-layer odering) causes “over shooting” → To achieve balancing between the structured regions and texture regions.

Region Filling Algorithm1) Computing Patch Priorities𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝

𝐶 𝑝 =Σ𝑞∈Ψ𝑝∩ 𝐼−Ω 𝐶(𝑞)

|Ψ𝑝|

𝐷 𝑝 =|∇𝐼𝑝

⊥∙𝑛𝑝|

𝛼

Initialization: 𝐶 𝑝 = 0, ∀𝑝∈ Ω and 𝐶 𝑝 = 1, ∀𝑝∈ 𝐼 − Ω

Region Filling Algorithm1) Computing Patch Priorities𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝

𝐶 𝑝 =Σ𝑞∈Ψ𝑝∩ 𝐼−Ω 𝐶(𝑞)

|Ψ𝑝|

𝐷 𝑝 =|∇𝐼𝑝

⊥∙𝑛𝑝|

𝛼

Initialization: 𝐶 𝑝 = 0, ∀𝑝∈ Ω and 𝐶 𝑝 = 1, ∀𝑝∈ 𝐼 − Ω

higher priority

lower priority

Region Filling Algorithm1) Computing Patch Priorities𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝

𝐶 𝑝 =Σ𝑞∈Ψ𝑝∩ 𝐼−Ω 𝐶(𝑞)

|Ψ𝑝|

𝐷 𝑝 =|∇𝐼𝑝

⊥∙𝑛𝑝|

𝛼

Initialization: 𝐶 𝑝 = 0, ∀𝑝∈ Ω and 𝐶 𝑝 = 1, ∀𝑝∈ 𝐼 − Ω

similar priority

Region Filling Algorithm2) Propagating Texture and Structure InformationAfter computing priorities, setting the highest priority Ψ 𝑝

To avoid diffusion, propagating image texture from the source region

Ψ 𝑞 = arg𝑚𝑖𝑛Ψ𝑞∈Φ𝑑(Ψ 𝑝, Ψ𝑞)

Region Filling Algorithm3) Updating Confidence ValuesAfter filling the patch Ψ 𝑝, the confidence term is updated

𝐶 𝑝 = 𝐶 𝑝 , ∀𝑝∈ Ψ 𝑝 ∩ Ω

It does not require additional parameter to specify image.

Region Filling AlgorithmThe 𝑡 indicates the current iteration.

Region Filling AlgorithmProperties of the region filling algorithm

Recall 𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝

The priority equation achieves balance of effects and an organic synthesis

Region Filling AlgorithmProperties of the region filling algorithm

𝑃 𝑝 = 𝐶 𝑝 𝐷 𝑝

• avoids an arbitrary fill order.• eliminates the risk of “broken-structure”

artefacts.• propagates strong edges.• reduces blocky and misalignment artefacts

without additional step.

Region Filling AlgorithmImplementation Details

The target 𝛿Ω is manually selected.The normal direction 𝑛𝑝 is computed as1) Contour’s “control” points are filtered via

2D Gaussian kernel2) estimated as the orthogonal unit vector of

𝛿Ω

Region Filling AlgorithmImplementation Details

The gradient ∇𝐼𝑝is computed as the MAX value in Ψ𝑝 ∩ 𝐼

Pixels are classified as belonging to• The target region Ω• The source region• The remainder

Results and Comparisons

Experimental environment was a 2.5-GHz Pentium IV with 1GB of RAM.

To compare with the results of earlier work.

Results and ComparisonsKanizsa Triangle and the Connectivity Principle

Results and ComparisonsComparing Different Filling Orders

original image target region

raster-scan concentric

Harrison’s2 m 45 s

Ours5 s

Results and ComparisonsComparing Different Filling Orders

original image target region

Results and ComparisonsComparing Different Filling Orders

raster-scan concentric

Results and ComparisonsComparing Different Filling Orders

Harrison’s45 m

Ours2 s

Results and ComparisonsComparing Different Filling Orders

Using only data term leads the “over shoot”

Results and ComparisonsComparisons With Diffusion-Based Inpainting

original image target region

Results and ComparisonsComparisons With Diffusion-Based Inpainting

onion peel ours

Results and ComparisonsComparisons With Diffusion-Based Inpainting

onion peel ours

Results and ComparisonsComparisons With Diffusion-Based Inpainting

onion peel ours

Results and ComparisonsComparisons With Diffusion-Based Inpainting

onion peel ours

Results and ComparisonsComparisons With Diffusion-Based Inpainting

priority function for before image

Priority function is 0 for inside and 1 for outside Final priorities made the continuation of the pole

Results and ComparisonsComparisons With Diffusion-Based Inpainting

original image target region

Results and ComparisonsComparisons With Diffusion-Based Inpainting

Isophotes hits the thin boundary

Results and ComparisonsComparisons With Diffusion-Based Inpainting

ours traditional image inpainting (blurry)

Results and ComparisonsComparisons With Diffusion-Based Inpainting

target region

texture and structure inpainting (blurry) ours

Comparison With Drori et al.

Results and Comparisons

Drori et al. (blurry)

Examples on PhotographsResults and Comparisons

Drori et al. (blurry)

Examples on PhotographsResults and Comparisons

Drori et al. (blurry)

onion peel ours

Examples on PhotographsResults and Comparisons

ours

“bow-tie” effect

Thank you for listening