Neural  Art    by  Mark  Chang  

A  Neural  Algorithm  of  Ar6s6c  Style  

•  Author：  – Leon  A.  Gatys.    – Alexander  S.  Ecker.    – Ma@hias  Bethge    

•  Organiza6on：  – Werner  Reichardt  Centre  for  Integra6ve  Neuroscience  and  Ins6tute  of  Theore6cal  Physics,  University  of  Tubingen,  Germany.      

– Bernstein  Center  for  Computa6onal  Neuroscience,  Tubingen,  Germany.  

The  Mechanism  of  Pain6ng  

Brain  Ar6st  

Scene   Style   ArtWork  

Computer   Neural  Networks  

Overview  

•  Visual  Percep6on  •  Computer  Vision  •  Neural  Art  •  Demo  

Visual  Percep6on  

•  Neuron  •  Visual  Pathway  •  Misconcep6on  

Neuron  

•  Neuron   •  Ac6on  Poten6al  

Dendrite  

Axon  

Cell  Body  

Time  

Voltage  

Threshold  

Visual  Pathway  

Re6na  

Visual  Area  V1  

Visual  Area  V4  Inferior  Temporal  Gyrus  (IT)  

Visual  Pathway  

Visual  Area  V1  

Inferior  Temporal  Gyrus  (IT)  

Recep6ve  Fields  

Visual  Area  V4  

Misconcep6on  

Computer  Vision  

•  Neural  Networks    •  Convolu6onal  Neural  Networks  •  VGG  19  

Neural  Networks    

n W1

W2

x1

x2

b Wb

y

nin = w1x1 + w2x2 + wb

nout

=1

1 + e�nin

Sigmoid  

Rec6fied  Linear  

nout

=

⇢nin

if nin

> 0

0 otherwise

Neural  Networks    

x

y

n11

n12

n21

n22

b b

z1

z2

Input    Layer

Hidden  Layer

Output  Layer

W12,y

W12,x

W11,y

W11,b W12,b

W11,x W21,11

W22,12

W21,12

W22,11

W21,b W22,b

Convolu6onal  Neural  Networks  

•  Convolu6onal  Layer

depth  

width  width    depth  

weights  weights  

height  

shared  weight  

Convolu6onal  Neural  Networks  

•  Stride •  Padding

Stride  =  1  

Stride  =  2  

Padding  =  0  

Padding  =  1  

Convolu6onal  Neural  Networks  

•  Pooling  Layer  1 3 2 4

5 7 6 8

0 0 4 4

6 6 0 0

4 5

3 2 no  overlap  

no  padding   no  weights  

depth  =  1  

7 8

6 4

Maximum  Pooling  

Average  Pooling  

Convolu6onal  Neural  Networks  

Convolu6onal  Layer  

Convolu6onal  Layer     Pooling  

Layer    

Pooling  Layer    

Recep6ve  Fields  Recep6ve  Fields  

Input  Layer  

Convolu6onal  Neural  Networks  

Input  Layer  

Convolu6onal  Layer  with  

Recep6ve  Fields:  

Max-­‐pooling  Layer  with  

Width  =3,  Height  =  3  

Filter  Responses  

Filter  Responses  

Input  Image  

VGG  19  

•  Karen  Simonyan  &  Andrew  Zisserman.  Very  Deep  Convolu6onal  Networks  for  Large-­‐scale  Image  Recogni6on.  

•  ImageNet  Challenge  2014  •  19  (+5)  layers  – 16  Convolu6onal  layers  (width=3,  height=3)  – 5  Max-­‐pooling  layers  (width=2,  height=2)  – 3  Fully-­‐connected  layers  

VGG  19  

depth=64  conv1_1  conv1_2  

maxpool

depth=128  conv2_1  conv2_2

maxpool

depth=256  conv3_1  conv3_2  conv3_3  conv3_4

depth=512  conv4_1  conv4_2  conv4_3  conv4_4

depth=512  conv5_1  conv5_2  conv5_3  conv5_4

maxpool maxpool maxpool

size=4096  FC1  FC2  

size=1000  sogmax

Neural  Art  

•  Content  Genera6on  •  Style  Genera6on  •  Artwork  Genera6on  

Content  Genera6on  

Brain  Ar6st  Content  

Canvas  Minimize  

the  difference  

Neural  S6mula6on  

Draw  

Content  Genera6on  Filter    

Responses  VGG19  

Update  the  color  of    the  pixels  

Content  

Canvas  

Result  

Width*Height  Depth  

Minimize  the  

difference  

Content  Genera6on  

Layer  l’s  Filter  l  Responses:  

Lcontent

(P l, Cl) =1

2

X

i,j

(Cl

i,j

� P l

i,j

)2

Layer  l’s  Filter    Responses:  Lcontent

(P l, Cl) =1

2

X

i,j

(Cl

i,j

� P l

i,j

)2Input  Photo:  L

content

(p, c, l) =1

2

X

i,j

(Cl

i,j

� P l

i,j

)2

Lcontent

(p,x, l) =1

2

X

i,j

(X l

i,j

� P l

i,j

)2

@Lcontent

(p,x, l)

@X l

i,j

= X l

i,j

� P l

i,j

X l

X li,j

Input  Canvas:  x

Width*Height  (j)  

Depth  (i)  

Width*Height  (j)  

Depth  (i)  

Content  Genera6on  

•  Backward  Propaga6on  Layer  l’s  Filter  l  Responses:  X l

Input  Canvas:  

x

VGG19  

@Lcontent

@x=

@Lcontent

@X l

@X l

x

x x� ⌘@L

content

@xUpdate  Canvas  

Learning  Rate  

Content  Genera6on  

Content  Genera6on  

VGG19  

conv1_2   conv2_2   conv3_4   conv4_4   conv5_2  conv5_1  

Style  Genera6on  

•  ”Style”  is  posi6on-­‐independent  

style  extrac6on  

Style  Genera6on  

VGG19  Artwork  

G  

G  

Filter  Responses   Gram  Matrix  

Width*Height  

Depth  

Depth  

Depth  

Posi6on-­‐  dependent  

Posi6on-­‐  independent  

Style  Genera6on  

1. .5

.5

.5

1.

1.   .5 .25 1.

.5 .25 .5

.25 .25

1. .5 1.

Width*Height  

Depth  

k1   k2  

k1  

k2  

Depth  

Depth  

Layer  l’s  Filter  Responses  Gram  Matrix  

F l1

F l2

F l3

F l4

F l1

F l2

F l3

F l4

G        

Gli,j = F l

i · F lj

Gl4,1 = F l

4 · F l1

= 1⇥ 1 + 0⇥ 0.5 + 0⇥ 0 + ...

= 1

Style  Genera6on  

VGG19  Filter  

 Responses  Gram    Matrix  

Minimize  the  

difference  

G  

G  

Style  

Canvas  

Update  the  color  of    the  pixels  Result  

Style  Genera6on  

Lstyle(a,x, l) =1

2

X

i,j

(X li,j �Al

i,j)2

@Lstyle(a,x, l)

@F li,j

= ((F l)T (X l �Al))j,i

Layer  l’s    Filter  Responses  

Layer  l’s    Gram  Matrix  

Layer  l’s    Gram  Matrix  

F li,j

Ali,j X l

i,j

Input  Artwork:  

Input  Canvas:  a

x

Style  Genera6on  

Style  Genera6on  

VGG19  

Conv1_1   Conv1_1  Conv2_1  

Conv1_1  Conv2_1  Conv3_1  

 

Conv1_1  Conv2_1  Conv3_1  Conv4_1  

Conv1_1  Conv2_1  Conv3_1  Conv4_1  Conv5_1  

Artwork  Genera6on  

Ltotal

= ↵Lcontent

+ �Lstyle

a

Lcontent

(p, c, l) =1

2

X

i,j

(Cl

i,j

� P l

i,j

)2

x

x x� ⌘@L

total

@xx

Filter  Responses  VGG19  

Lcontent

(p,x)

Lstyle(a,x)

Gram  Matrix  

Artwork  Genera6on  

VGG19   VGG19  

Lcontent

(p,x) Lstyle(a,x)

Conv1_1  Conv2_1  Conv3_1  Conv4_1  Conv5_1  

Conv4_2  

Ltotal

= ↵Lcontent

+ �Lstyle

Artwork  Genera6on  

Demo  

•  Content  v.s.  Style  •  Different  Ini6al  State  •  Different  VGG  Layers  •  Sketch  &  Watercolor  •  Pain6ng  &  Poem  

Content  v.s.  Style  

0.15   0.05  

0.02   0.007  

↵

�

Different  Ini6al  State  noise   0.9  *noise  +  0.1*photo   photo  

Different  VGG  Layers  

Conv1_1  Conv2_1  

Conv1_1  Conv2_1  Conv3_1  

Conv1_1  Conv2_1  Conv3_1  Conv4_1  

Conv1_1  Conv2_1  Conv3_1  Conv4_1  Conv5_1  

↵

�= 0.002

Sketch  &  Watercolor  

Pain6ng  &  Poem  

Further  Reading  •  A  Neural  Algorithm  of  Ar6s6c  Style  –  h@p://arxiv.org/abs/1508.06576  

•  Texture  Synthesis  Using  Convolu6onal  Neural  Networks  –  h@p://arxiv.org/abs/1505.07376  

•  Convolu6onal  Neural  Network  –  h@p://cs231n.github.io/convolu6onal-­‐networks/  

•  Neural  Network  Back  Propaga6on  –  h@p://cpmarkchang.logdown.com/posts/277349-­‐neural-­‐network-­‐backward-­‐propaga6on  

•  Computa6onal  Poetry：  –  h@p://www.slideshare.net/ckmarkohchang/computa6onal-­‐poetry  

Code

•  Python  Tensorflow  – h@ps://github.com/ckmarkoh/neuralart_tensorflow  

•  Python  Theano  – h@ps://github.com/woonketwong/ar6fy  

•  Python  Theano  (ipython  notebook)  – h@ps://github.com/Lasagne/Recipes/blob/master/examples/styletransfer/Art%20Style%20Transfer.ipynb  

•  Python  deeppy  – h@ps://github.com/andersbll/neural_ar6s6c_style  

Image  URL  

•  h@p://www.taipei-­‐101.com.tw/upload/news/201502/2015021711505431705145.JPG      

•  h@ps://github.com/andersbll/neural_ar6s6c_style/blob/master/images/starry_night.jpg?raw=true  

Acknowledgement  

•  NTU  imlab  

About  the  Speaker  

•  Email:  ckmarkoh  at  gmail  dot  com  •  Blog:  h@p://cpmarkchang.logdown.com  •  Github:  h@ps://github.com/ckmarkoh  

Mark  Chang  

•  Facebook:  h@ps://www.facebook.com/ckmarkoh.chang  •  Slideshare:  h@p://www.slideshare.net/ckmarkohchang  •  Linkedin:  h@ps://www.linkedin.com/pub/mark-­‐chang/85/25b/847