Face Sketch Synthesis

using Stagewise GAN

Introduction

• Photo and sketch

• The same content

• Different in geometry and in texture

• Different domain

Introduction

• Photo to sketch

• Enhance the artistry of pictures

• Criminal investigation

• can only search using sketch

• Solved the problem of large work-load of

manual creation sketch

Related work

Sketch Synthesis

• Face Photo-Sketch Synthesis and Recognition(2009)

• state-of-the-art approach using a multi-scale Markov random field (MRF)

• Markov Weight Fields for Face Sketch Synthesis(2012)

• the target sketch patch is represented by a linear combination of some candidate sketch

patches(MWF)

• Real-Time Exemplar-Based Face Sketch Synthesis(2014)

• casted the face sketch synthesis problem into a spatial sketch denoising (SSD) problem,local

fusion the patches after divided.

Related work

Synthesis Model

• Deep Belief Net-work(DBN)

• Generative Adversarial Networks(GANs)

• Variational Autoencoder(VAE )

Image to Image

• GANs

• cGANs

• cycleGANs

• Pix2pix

Our Model

• Use GAN model as main model

• Using Encoder-Decoder as Generator

• Using patch-Decoder

• 5 stage training

• End-to-end net

• Input data :photo and 5 different stage sketches

photos

Generator stage1

Generated stage1

Groundtruth stage1

Generator stage2

Discriminator stage1

Generated stage1

concat

patch 0,1

concat

Generator stage3

concat

Generator stage4

concat

Generator stage5

concat

Generated stage2

Groundtruth stage2

Discriminator stage1

concat

patch 0,1

Discriminator stage1

concat

patch 0,1

Discriminator stage1

concat

patch 0,1

Discriminator stage1

concat

patch 0,1

Generated stage2

Generated stage3

Generated stage4

Groundtruth stage3

Groundtruth stage4

Groundtruth stage5

Generated stage3

Generated stage4

Generated stage5

（final）

Full model

Conv1

Conv2

Conv3Conv8

Deconv7

Deconv6

Conv5

Conv4

Conv9

Pooling2

Poling3

Photo

or

Photo +last

250x200

16

32

64

128

128

outputoutput

128

64

32

1

Sigmoid

Euclidean

Loss

Generator

d_Conv1

d_Conv2

d_Conv3

dropout1

d_Pooling1

dropout2

dropout3

d_Pooling3

d_fc4

dropout4

d_fc4

SigmoidLoss

ConcatConv9

SketchPatch

output

64

128

256

128

1

Discriminator

16 Patches in every

discriminator

Experiments

• Use faces in CUFS data sketches are painted by our artists

• Use 62 faces for training and 32 for testing

• The size of every image is 200x250

Database

• CUHK Face Sketch database (CUFS) is for research on face sketch synthesis and face

sketch recognition.

• 188 faces from the Chinese University of Hong Kong (CUHK) student database(done)

• 123 faces from the AR database . (painting)

Database

因为数据没有画完，现在用的训练数据由62张，测试数据31张。

Data

Synthesis

SSD MWF MRF RSLCR GAN

Experiments

Experiments

(a） (b） (c） (d） (e） (f） (g） (h） (i）

(a) face photo;(b) sketch by artist from CUFS; (c) SSD; (d) MEF ;(e) MRF ;(f) RSLCR ;(g) GAN ; (h) our model ; (i)

sketch by our artist;