real-time exemplar-based face sketch synthesis
DESCRIPTION
Real-Time Exemplar-Based Face Sketch Synthesis. Pipeline illustration. Qingxiong Yang 1. Ming-Hsuan Yang 2. Yibing Song 1. Linchao Bao 1. 1 City University of Hong Kong. 2 University of California at Merced. Note: containing animations. - PowerPoint PPT PresentationTRANSCRIPT
Real-Time Exemplar-Based Face Sketch Synthesis
Pipeline illustration
Note: containing animations
Yibing Song1 Linchao Bao1 Qingxiong Yang1 Ming-Hsuan Yang2
1City University of Hong Kong2University of California at Merced
Our assumption: a database containing photo-sketch pairs
1. photo database 2. sketch database
Aligned
Coarse Sketch GenerationStep 1: KNN search
p
Test photo patch Test photo
Training photo dataset
๐ป ๐๐ป ๐
๐ป ๐
Matched photo patch
Relative position
Similarly
Matched photo patch
Relative position
โ๐๐ฒ[ ]โ๐ =
Test photo patch
Matched photo patch
Matched photo patch
Matched photo patch
๐๐๐ โ +๐๐
๐ โ +๐๐๐ฒ โ ยฟ
2. Compute linear mapping function defined by
Coarse Sketch GenerationStep 2: Linear Estimation from Photos
Matched sketch pixel
p
Matched sketch pixel
Test photo
๐บ๐ท +โ๐๐
โ
๐บ๐ท +โ๐๐
โ
Matched sketch pixel๐บ๐ท +โ๐๐ฒ
โ
๐๐๐ โ +๐๐
๐ โ +๐๐๐ฒ โ ยฟ
Estimation on pixel p
Repeat for every pixel
Coarse sketch
Coarse Sketch GenerationStep 3: Apply Linear Mapping to Sketches
๐ฌ๐
Because: coarse sketch image is not natural. is not a good similarity measurement between p and r.
Denoising: State-of-the-art Image Denoising Algorithms
Coarse sketch
Nonlocal Means (NLM)
p
r
๐๐๐๐ฟ๐=ยฟ ๐ธ๐
๐ค(๐ ,๐ )+โฏ
For all pixels in the neighbor of p:
Little improvement
After NLM
q
๐ธ๐๐ค(๐ ,๐)+ยฟ
[NLM] A. Buades, B. Coll and J.-M. Morel, A non-local algorithm for image denoising, CVPR 2005.
Motivation โ BM3D
BM3D groups correlated patches in the noisy image to create multiple estimations.
Our idea for sketch denoising: group highly similar sketch estimations.
How BM3D works
[BM3D] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, โImage denoising by sparse 3D transform-domain collaborative filtering,โ IEEE Trans. Image Process., vol. 16, no. 8, pp. 2080-2095, August 2007.
๐ค(๐ ,๐) โ
Proposed Spatial Sketch Denoising Algorithm (SSD)
Test photo
q
๐บ๐+โ๐๐
โ
p
Matched sketch
๐บ๐+โ๐๐
โ
Similarly ,
๐บ๐+โ๐๐
โ
๐บ๐+โ๐๐
โ
,
๐บ๐+โ๐๐ฒ
โ
๐บ๐+โ๐๐ฒ
โ๐๐๐ โ +๐๐
๐ โ +๐๐๐ฒ โ ยฟ
๐ฌ๐๐
p
Estimations from pixels in local region
r ๐ฌ๐๐
Averaging estimations to generate output sketch value.
Nonlocal Means (NLM):
๐๐๐๐ฟ๐=ยฟ ๐ธ๐ +โฏ๐ธ๐๐ค(๐ ,๐ )โ+ยฟ
Proposed SSD:
๐๐๐๐๐ท=ยฟ +โฏ+ยฟ1 โ๐ฌ๐
๐ 1 โ๐ฌ๐๐
p
Proposed SSD is robust to
Input 5x5 local region
11x11 local region
17x17 local region
23x23 local region
Note: When is sufficient large (i.e., >100), the proposed SSD can effectivelysuppress noise while preserving facial details like the tiny eye reflections (see close-ups).
Robustness to the region size - the only parameter involved