deep learning for the high dynamic range imaging pipeline · deep learning for the high dynamic...

Deep Learning for the High Dynamic

Range Imaging PipelineDemetris Marnerides

Warwick Centre of Predictive Modelling (WCPM)

The University of Warwick

D.Marnerides@warwick.ac.uk

Project Supervisors: Dr Kurt Debattista (Primary – WMG), Dr Igor Khovanov (Secondary – WCPM)

Contents

Introduction to HDR

Introduction to Deep Learning

Relevant Work

Motivation

ExpandNet

Results

Future Work

2WCPM Seminar Series, December 2017

Introduction to HDR

Low/Standard Dynamic Range (LDR)

▪ Limited Luminance range

▪ Limited Colour gamut

▪ 8 bit quantization [0-255]

High Dynamic Range (HDR)

▪ Real-World Lighting

▪ 32-bit floats

3WCPM Seminar Series, December 2017

Introduction to HDR (2)

Most content is LDR

HDR LDR straightforward (Tone Mapping)

Inverse is hard (LDR HDR)

▪ Expert knowledge / heuristics

▪ Quantization, clipping

▪ Non-linear local luminance shifts

Proposed data-driven solution

▪ Learn relevant information from data

4WCPM Seminar Series, December 2017

?

Artificial Neural Networks

Single hidden layer

𝒚 = 𝐶𝒇 𝐴𝒙+ 𝒃 + 𝒅

Activations: Sigmoid, Tanh, Rectifiers (ReLU, PReLU, ELU, SELU) ...

Find parameters that minimize some ‘loss’ between model and data

▪ Euclidean distance (least squares regression)

𝑖

ഥ𝒚𝑖 − 𝒚𝑖2

Stochastic Gradient Descent with Backpropagation

5WCPM Seminar Series, December 2017

Going Deeper

𝒚 = 𝐴𝑵+𝟏𝒇𝑵 𝐴𝑵 …𝒇𝟐 𝐴𝟐𝒇𝟏 𝐴𝟏𝒙+ 𝒃𝟏 + 𝒃𝟐 …+ 𝒃𝑵 + 𝒃𝑵+𝟏

Depth

▪ Exponentially more expressive with less parameters

▪ Computationally more efficient

▪ Aids generalization over memorization

6WCPM Seminar Series, December 2017

Convolutions

7WCPM Seminar Series, December 2017

𝑦1 = 𝑥1 𝑓1+ 𝑥2 𝑓2+ 𝑥3 𝑓3

Convolutions

8WCPM Seminar Series, December 2017

𝑦2 = 𝑥2 𝑓1+ 𝑥3 𝑓2+ 𝑥4 𝑓3

Convolutions

9WCPM Seminar Series, December 2017

𝑦3 = 𝑥3 𝑓1+ 𝑥4 𝑓2+ 𝑥5 𝑓3

Convolutions

10WCPM Seminar Series, December 2017

𝑦4 = 𝑥4 𝑓1+ 𝑥5 𝑓2+ 𝑥6 𝑓3

Convolutions

11WCPM Seminar Series, December 2017

Likewise for 2D vectors (matrices, images)

𝑓𝑖𝑙𝑡𝑒𝑟 =1 0 10 1 01 0 1

Classification

Modular improvements:

▪ e.g. residual connections

Auto-encoders

Deep Convolutional Neural Networks

12WCPM Seminar Series, December 2017

LeNet-5

Inverse problems in Imaging

13WCPM Seminar Series, December 2017

Colorful Image

ColorizationZhang et al., 2016

Globally and Locally

Consistent

Image CompletionIizuka et al., 2017

Motivation for a new architecture

14WCPM Seminar Series, December 2017

UNet-like architectures:

▪ Abstract representations

▪ Multiscale context

▪ However prone to artefacts

• E.g. from the pix2pix

Semantic Segmentation results

https://phillipi.github.io/pix2pix/images/cityscapes_cGAN_AtoB/latest_net_G_val/index.html

ExpandNet

15WCPM Seminar Series, December 2017

Workflow (training)

16WCPM Seminar Series, December 2017

Workflow (testing)

17WCPM Seminar Series, December 2017

1000 nits (cd/m2)

▪ PU-PSNR

▪ PU-SSIM

▪ PU-MSSSIM

▪ HDR-VDP

Significance

▪ p < 0.01

Results

18WCPM Seminar Series, December 2017

Results (2)

19WCPM Seminar Series, December 2017

HDR-VDP-2 – Detection Probability Maps

Results (3)

20WCPM Seminar Series, December 2017

Image comparisons with other CNNs

Branches

21WCPM Seminar Series, December 2017

All branches

Local (masked D + G) Dilated (masked L + G)

Branches (2)

22WCPM Seminar Series, December 2017

Training combinations of branches

Future Work

23WCPM Seminar Series, December 2017

Reducing compression artefacts

Hallucinate under/over exposed regions

▪ Generative Adversarial Networks (GANs)

HDR Super-resolution

LDR to HDR Video

▪ Recurrent Neural Networks

Thank you!

24WCPM Seminar Series, December 2017

PhD is funded by the EPSRC