![Page 1: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/1.jpg)
Deep Learning for the High Dynamic
Range Imaging PipelineDemetris Marnerides
Warwick Centre of Predictive Modelling (WCPM)
The University of Warwick
Project Supervisors: Dr Kurt Debattista (Primary – WMG), Dr Igor Khovanov (Secondary – WCPM)
![Page 2: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/2.jpg)
Contents
Introduction to HDR
Introduction to Deep Learning
Relevant Work
Motivation
ExpandNet
Results
Future Work
2WCPM Seminar Series, December 2017
![Page 3: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/3.jpg)
Introduction to HDR
Low/Standard Dynamic Range (LDR)
▪ Limited Luminance range
▪ Limited Colour gamut
▪ 8 bit quantization [0-255]
High Dynamic Range (HDR)
▪ Real-World Lighting
▪ 32-bit floats
3WCPM Seminar Series, December 2017
![Page 4: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/4.jpg)
Introduction to HDR (2)
Most content is LDR
HDR LDR straightforward (Tone Mapping)
Inverse is hard (LDR HDR)
▪ Expert knowledge / heuristics
▪ Quantization, clipping
▪ Non-linear local luminance shifts
Proposed data-driven solution
▪ Learn relevant information from data
4WCPM Seminar Series, December 2017
?
![Page 5: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/5.jpg)
Artificial Neural Networks
Single hidden layer
𝒚 = 𝐶𝒇 𝐴𝒙+ 𝒃 + 𝒅
Activations: Sigmoid, Tanh, Rectifiers (ReLU, PReLU, ELU, SELU) ...
Find parameters that minimize some ‘loss’ between model and data
▪ Euclidean distance (least squares regression)
𝑖
ഥ𝒚𝑖 − 𝒚𝑖2
Stochastic Gradient Descent with Backpropagation
5WCPM Seminar Series, December 2017
![Page 6: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/6.jpg)
Going Deeper
𝒚 = 𝐴𝑵+𝟏𝒇𝑵 𝐴𝑵 …𝒇𝟐 𝐴𝟐𝒇𝟏 𝐴𝟏𝒙+ 𝒃𝟏 + 𝒃𝟐 …+ 𝒃𝑵 + 𝒃𝑵+𝟏
Depth
▪ Exponentially more expressive with less parameters
▪ Computationally more efficient
▪ Aids generalization over memorization
6WCPM Seminar Series, December 2017
![Page 7: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/7.jpg)
Convolutions
7WCPM Seminar Series, December 2017
𝑦1 = 𝑥1 𝑓1+ 𝑥2 𝑓2+ 𝑥3 𝑓3
![Page 8: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/8.jpg)
Convolutions
8WCPM Seminar Series, December 2017
𝑦2 = 𝑥2 𝑓1+ 𝑥3 𝑓2+ 𝑥4 𝑓3
![Page 9: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/9.jpg)
Convolutions
9WCPM Seminar Series, December 2017
𝑦3 = 𝑥3 𝑓1+ 𝑥4 𝑓2+ 𝑥5 𝑓3
![Page 10: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/10.jpg)
Convolutions
10WCPM Seminar Series, December 2017
𝑦4 = 𝑥4 𝑓1+ 𝑥5 𝑓2+ 𝑥6 𝑓3
![Page 11: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/11.jpg)
Convolutions
11WCPM Seminar Series, December 2017
Likewise for 2D vectors (matrices, images)
𝑓𝑖𝑙𝑡𝑒𝑟 =1 0 10 1 01 0 1
![Page 12: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/12.jpg)
Classification
Modular improvements:
▪ e.g. residual connections
Auto-encoders
Deep Convolutional Neural Networks
12WCPM Seminar Series, December 2017
LeNet-5
![Page 13: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/13.jpg)
Inverse problems in Imaging
13WCPM Seminar Series, December 2017
Colorful Image
ColorizationZhang et al., 2016
Globally and Locally
Consistent
Image CompletionIizuka et al., 2017
![Page 14: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/14.jpg)
Motivation for a new architecture
14WCPM Seminar Series, December 2017
UNet-like architectures:
▪ Abstract representations
▪ Multiscale context
▪ However prone to artefacts
• E.g. from the pix2pix
Semantic Segmentation results
https://phillipi.github.io/pix2pix/images/cityscapes_cGAN_AtoB/latest_net_G_val/index.html
![Page 15: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/15.jpg)
ExpandNet
15WCPM Seminar Series, December 2017
![Page 16: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/16.jpg)
Workflow (training)
16WCPM Seminar Series, December 2017
![Page 17: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/17.jpg)
Workflow (testing)
17WCPM Seminar Series, December 2017
![Page 18: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/18.jpg)
1000 nits (cd/m2)
▪ PU-PSNR
▪ PU-SSIM
▪ PU-MSSSIM
▪ HDR-VDP
Significance
▪ p < 0.01
Results
18WCPM Seminar Series, December 2017
![Page 19: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/19.jpg)
Results (2)
19WCPM Seminar Series, December 2017
HDR-VDP-2 – Detection Probability Maps
![Page 20: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/20.jpg)
Results (3)
20WCPM Seminar Series, December 2017
Image comparisons with other CNNs
![Page 21: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/21.jpg)
Branches
21WCPM Seminar Series, December 2017
All branches
Local (masked D + G) Dilated (masked L + G)
![Page 22: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/22.jpg)
Branches (2)
22WCPM Seminar Series, December 2017
Training combinations of branches
![Page 23: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/23.jpg)
Future Work
23WCPM Seminar Series, December 2017
Reducing compression artefacts
Hallucinate under/over exposed regions
▪ Generative Adversarial Networks (GANs)
HDR Super-resolution
LDR to HDR Video
▪ Recurrent Neural Networks
![Page 24: Deep Learning for the High Dynamic Range Imaging Pipeline · Deep Learning for the High Dynamic Range Imaging Pipeline Demetris Marnerides Warwick Centre of Predictive Modelling (WCPM)](https://reader033.vdocuments.us/reader033/viewer/2022042219/5ec5aaedbd278d405c141f0f/html5/thumbnails/24.jpg)
Thank you!
24WCPM Seminar Series, December 2017
PhD is funded by the EPSRC