Learning to Shade Hand-drawn Sketches

CVPR 2020 Oral Presenter: Yu Han

2020/04/05

1

Outline • Authors • Background • Network • Experiment • Conclusion

2

Qingyuan Zheng* · University of Maryland

3

Zhuoru Li* · Project HAT

4

Adam Bargteil · Assistant professor at University of Maryland

· Phd at UCB · Postdoc at CMU Graphics Lab

Research Interest: · Computer graphics and animation, especially using physics-based animation

5

Outline • Authors • Background • Network • Experiment • Conclusion

6

Background• Pix2pix

7

Image-to-Image Translation with Conditional Adversarial Networks. In CVPR, 2017.

Background• DeepNormal

8

Deep Normal Estimation for Automatic Shading of Hand-Drawn Characters. In ECCV, 2018.

Background• Sketch2Normal

9

Interactive Sketch-Based Normal Map Generation with Deep Neural Networks. In SIGGRAPH, 2018.

Background• Squeeze and Excitation(SE) Block

10

SE Net：Squeeze-and-Excitation Networks. In CVPR, 2018.

Outline • Authors • Background • Network • Experiment • Conclusion

11

Network

12

Data

13

· 1160 paired hand-drawn line drawings and shadows · Tagged with a lighting direction · 8x3+2=26 directions

Network Architecture 1. Generative Network 2. Discriminator Network 3. Loss Function

14

Generative Network

15

Two parts: • Shape net: encodes the underlying 3D structure from 2D sketches • Render net: renders artistic shadows based on the encoded structure

Generative Network

16

Render net: • Self-attention: enhance the visual reasoning 3D structure from 2D sketches. • SE blocks: filter out unnecessary features • Extract two supervision side outputs, s1 and s2

Network Architecture 1. Generative Network 2. Discriminator Network 3. Loss Function

17

Discriminator Network

18

• Self-attention: make discriminator sensitive to the distant features

Network Architecture 1. Generative Network 2. Discriminator Network 3. Loss Function

19

Loss Function

20

Outline • Authors • Background • Network • Experiment • Conclusion

21

Experiment

22

• Shift, zoom in/out, and rotate to augment the dataset • Input: 320x320 • 80 000 iterations

Experiment

23

Experiment

24

Experiment

25

Experiment

26

User Study

27

All people.

People with drawing experience.

Ablation Study

28

Ablation Study

29

Calculate the first two orders of the distance between distributions in Gaussian space.

Outline • Authors • Background • Network • Experiment • Conclusion

30

Conclusion • New dataset

• Propose a network that “understands” the structure and 3D spatial

relationships implied by line drawings and produces highly-detailed

and accurate shadows

31

Thank you

32