real-time exemplar-based face sketch synthesis

Real-Time Exemplar-Based Face Sketch Synthesis

Pipeline illustration

Note: containing animations

Yibing Song1 Linchao Bao1 Qingxiong Yang1 Ming-Hsuan Yang2

1City University of Hong Kong2University of California at Merced

Our assumption: a database containing photo-sketch pairs

1. photo database 2. sketch database

Aligned

Coarse Sketch GenerationStep 1: KNN search

p

Test photo patch Test photo

Training photo dataset

𝑻 𝒑𝑻 𝒑

𝑻 𝒑

Matched photo patch

Relative position

Similarly

Matched photo patch

Relative position

∆𝒑𝑲[ ]∆𝒑 =

Test photo patch

Matched photo patch

Matched photo patch

Matched photo patch

𝒙𝒑𝟏 ∙ +𝒙𝒑

𝟐 ∙ +𝒙𝒑𝑲 ∙ ¿

2. Compute linear mapping function defined by

Coarse Sketch GenerationStep 2: Linear Estimation from Photos

Matched sketch pixel

p

Matched sketch pixel

Test photo

𝑺𝑷 +∆𝒑𝟏

❑

𝑺𝑷 +∆𝒑𝟐

❑

Matched sketch pixel𝑺𝑷 +∆𝒑𝑲

❑

𝒙𝒑𝟏 ∙ +𝒙𝒑

𝟐 ∙ +𝒙𝒑𝑲 ∙ ¿

Estimation on pixel p

Repeat for every pixel

Coarse sketch

Coarse Sketch GenerationStep 3: Apply Linear Mapping to Sketches

𝑬𝒑

Because: coarse sketch image is not natural. is not a good similarity measurement between p and r.

Denoising: State-of-the-art Image Denoising Algorithms

Coarse sketch

Nonlocal Means (NLM)

p

r

𝑆𝑝𝑁𝐿𝑀=¿ 𝐸𝑟

𝑤(𝑝 ,𝑟 )+⋯

For all pixels in the neighbor of p:

Little improvement

After NLM

q

𝐸𝑞𝑤(𝑝 ,𝑞)+¿

[NLM] A. Buades, B. Coll and J.-M. Morel, A non-local algorithm for image denoising, CVPR 2005.

Motivation – BM3D

BM3D groups correlated patches in the noisy image to create multiple estimations.

Our idea for sketch denoising: group highly similar sketch estimations.

How BM3D works

[BM3D] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3D transform-domain collaborative filtering,” IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080-2095, August 2007.

𝑤(𝑝 ,𝑞) ∙

Proposed Spatial Sketch Denoising Algorithm (SSD)

Test photo

q

𝑺𝒒+∆𝒒𝟏

❑

p

Matched sketch

𝑺𝒑+∆𝒒𝟏

❑

Similarly ,

𝑺𝒑+∆𝒒𝟐

❑

𝑺𝒒+∆𝒒𝟐

❑

,

𝑺𝒒+∆𝒒𝑲

❑

𝑺𝒑+∆𝒒𝑲

❑𝒙𝒒𝟏 ∙ +𝒙𝒒

𝟐 ∙ +𝒙𝒒𝑲 ∙ ¿

𝑬𝒑𝒒

p

Estimations from pixels in local region

r 𝑬𝒑𝒓

Averaging estimations to generate output sketch value.

Nonlocal Means (NLM):

𝑆𝑝𝑁𝐿𝑀=¿ 𝐸𝑞 +⋯𝐸𝑟𝑤(𝑝 ,𝑟 )∙+¿

Proposed SSD:

𝑆𝑝𝑆𝑆𝐷=¿ +⋯+¿1 ∙𝑬𝒑

𝒒 1 ∙𝑬𝒑𝒓

p

Proposed SSD is robust to

Input 5x5 local region

11x11 local region

17x17 local region

23x23 local region

Note: When is sufficient large (i.e., >100), the proposed SSD can effectivelysuppress noise while preserving facial details like the tiny eye reflections (see close-ups).

Robustness to the region size - the only parameter involved