![Page 1: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/1.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
DeepFont: Font Recognition and Similarity Based on Deep LearningHailin Jin | Principal Scientist | Adobe Research
![Page 2: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/2.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
![Page 3: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/3.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Problem definition
Font recognition Font similarity Why does it matter? Use fonts effectively
Design inspiration
Text/image editing
3
![Page 4: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/4.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Challenges
Extremely large scale recognition problem Over 100,000 known fonts
Beyond object recognition Recognize subtle design styles
Extremely difficult to collect real-world training data Have to use synthetic training data
Mismatch between training and test data
4
![Page 5: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/5.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Our approach – DeepFont
Deep Learning – convolutional neural networks Effective at large-scale
Effective at fine-grained recognition
No need to do OCR End-to-end training and data-driven
5
![Page 6: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/6.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
DeepFont system overview
Localization Network
Recognition Network
6
![Page 7: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/7.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
GPUCPU
DeepFont training
Fonts Font maps
Offline rendering Online
rendering
Words
Images
Data augmentation
CNN training
![Page 8: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/8.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Data augmentation
Noise Blur Perspective deformation Shading Variable character spacing Variable aspect ratio
9
![Page 9: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/9.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Effects of data augmentation
10
![Page 10: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/10.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Beyond data augmentation
Cannot enumerate all possible degradation Background, font decoration
May introduce degradation bias in training Leverage abundant unlabeled real-world images
11
![Page 11: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/11.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Network decomposition for domain adaptation
Decompose the network into two parts Train the two parts separately
12
Unsupervised cross-domainvariable layers
Supervised domain-specific
![Page 12: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/12.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Network decomposition for domain adaptation
Stacked convolutional auto-encoder Synthetic data and unlabeled real data
Unsupervised training Fix the subnetwork Train the rest in a supervised way Labeled synthetic data
13
![Page 13: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/13.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Qualitative evaluation
4383 real-world images collected from font forums
Vary the number of unsupervisedly trained layers
14
Model Augmentation Decomposition Top-1 accuracy Top-5 accuracy
LFE (CVPR’04) Y N/A 42.6% 60.3%
DeepFont N N 42.5% 49.2%
DeepFont Y N 66.7% 79.2%
DeepFont Y Y 71.4% 81.8%
0 1 2 3 4 5
Train 91.5% 90.1% 88.8% 87.5% 84.8% 82.1%
Test 79.3% 79.7% 81.8% 81.0% 77.5% 74.0%
![Page 14: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/14.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Text localization
Localization model
![Page 15: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/15.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Text localization results
![Page 16: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/16.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
DeepFont for font similarity
Similarity Network
LushScript.ttf
Query Sample
Similar fonts
![Page 17: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/17.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Font similarity
18
![Page 18: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/18.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Demo in Photoshop
19
![Page 19: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/19.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential. 20
Mobile
![Page 20: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/20.jpg)
© 2016 Adobe Systems Incorporated. All Rights Reserved. Adobe Confidential.
Summary
DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization
Photoshop Mobile Dataset available at http://www.atlaswang.com/deepfont.html References Chen, Yang, Jin, Brandt, Shechtman, Agarwala, and Han. Large-scale Visual Font Recognition. CVPR 2014. Wang, Yang, Jin, Brandt, Shechtman, Agarwala, and Huang. DeepFont: Identify Your Font from An Image. ACM
Multimedia 2015.
21
![Page 21: New DeepFont: Font Recognition and Similarity Based on Deep … · 2016. 4. 10. · DeepFont Font recognition Font similarity Data augmentation Model decomposition Text localization](https://reader035.vdocuments.us/reader035/viewer/2022071218/604e9813b6275633e75fbc4a/html5/thumbnails/21.jpg)