![Page 1: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/1.jpg)
Learning Representations for Automatic Colorization
Experiment Presentation - 09/21/16Tushar Nagarajan
![Page 2: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/2.jpg)
Introduction
Colorization
Levin et al. (2004) Wesch et al. (2002)
Larsson et al. (2016)
Previous attempts: Transfer, Scribble
![Page 3: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/3.jpg)
Idea
Predict the color histogram for each pixel
Wiki User:SharkDThinkstock
Why HSL?
![Page 4: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/4.jpg)
Model
Representing a pixel - Image hypercolumninterpolate
conv1
conv2
![Page 5: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/5.jpg)
Model
Larsson et al. (2016)
![Page 6: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/6.jpg)
Model
Image hypercolumn features : pre-trained VGG
Larsson et al. (2016)
A vector represents a histogram
![Page 7: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/7.jpg)
Model
Image hypercolumn features : pre-trained VGG
Larsson et al. (2016)
![Page 8: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/8.jpg)
Why just two predictions?
Lightness information already present
![Page 9: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/9.jpg)
Results
Larsson et al. (2016)Demo: http://colorize.ttic.edu/
![Page 10: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/10.jpg)
Results
Larsson et al. (2016)
Why is this important?
![Page 11: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/11.jpg)
Experiment
![Page 12: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/12.jpg)
Experiment - Foreground Consistency
Photo credit: Peter Zelewski
![Page 13: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/13.jpg)
Not the best colorization we’ve seen...
![Page 14: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/14.jpg)
![Page 15: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/15.jpg)
Source of inconsistency?
![Page 16: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/16.jpg)
![Page 17: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/17.jpg)
![Page 18: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/18.jpg)
- Averaged over 15 models- Errors for 64 backgrounds
![Page 19: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/19.jpg)
Background class 1
![Page 20: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/20.jpg)
Background class 2
![Page 21: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/21.jpg)
Qualitative Analysis
![Page 22: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/22.jpg)
Qualitative Analysis
![Page 23: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/23.jpg)
Qualitative Analysis
![Page 24: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/24.jpg)
Do colorization errors in the background trickle down to the foreground?
Ans: Not too much, sorry.
R = 0.414
![Page 25: Learning Representations for Automatic Colorizationvision.cs.utexas.edu/381V-fall2016/slides/nagarajan-expt.pdf · Larsson et al. (2016) Previous attempts: Transfer, Scribble. Idea](https://reader033.vdocuments.us/reader033/viewer/2022060214/5f05a1a47e708231d413ec39/html5/thumbnails/25.jpg)
Summary
- Background coloring influences foreground coloring to some extent
- Hypercolumn features = extra background information
- Low L scenes contribute less to the top of the hypercolumn than the foreground?