image restoration with neural networks · several types of noise involved in the image formation:...

Work with Hang Zhao, Iuri Frosio, Jan Kautz

IMAGE RESTORATION WITH NEURAL NETWORKSOrazio Gallo

MOTIVATIONThe long path of images…

Demosaic Denoise

Bad Pixel

Correction

Image

Enhancing

Tone

Mapping

Lens

Correction

Black

Level

Metering

AF/AE

Image Signal Processor (ISP)

DEMOSAICINGcolors by interpolation

Image c

redit

: W

ikip

edia

Image c

redit

: M

arc

Levoy

DENOISINGSeveral types of noise involved in the image formation:

• Photon shot noise

• Dark current (AKA thermal noise)

• Photo-response non-uniformity

• Vignetting

• Readout noise:

• Reset noise (charge-to-voltage transfer)

• White noise (during voltage amplification amplification)

• Quantization noise (ADC)

DENOISING

MOTIVATION

Demosaic Denoise

Bad Pixel

Correction

Image

Enhancing

Tone

Mapping

Lens

Correction

Black

Level

Metering

AF/AE

Demosaicing before denoising changes the

statistics of the noise. And the best de-noising

algorithms require to know what the noise looks like.

MOTIVATION

Denoise Demosaic

Bad Pixel

Correction

Image

Enhancing

Tone

Mapping

Lens

Correction

Black

Level

Metering

AF/AE

Denoising first can change the color reproduction

accuracy as the three channels may be denoised

differently.

PSF CFA Noise

[1] Heide et al., ACM SIGGRAPH Asia 2012 (ToG)

FLEXISP1

A Flexible Camera Image Processing Framework

CAN WE DO IT WITH A NEURAL NETWORK?

Can we do it with a neural network, which moves

the heavy lifting to the training stage and inference

is very quick?

JOINT DEMOSAICING AND DENOISING

JOINT DEMOSAICING AND DENOISINGNetwork architecture

convolu

tion

convolu

tion

convolu

tion

bilin

ear

inte

rpola

tion

MEASURING IMAGE QUALITY

Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/

Original

Higher sensitivity to errors in texture-less regions!

MEASURING IMAGE QUALITY

Wang, et al. "Image quality assessment: from error visibility to structural similarity." IEEE TIP (2004)

MEASURING IMAGE QUALITY

Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/

Original

0.988 0.662

MEASURING IMAGE QUALITYHigher sensitivity to errors in texture-less regions!

JOINT DEMOSAICING AND DENOISINGNetwork architecture

convolu

tion

convolu

tion

convolu

tion

bilin

ear

inte

rpola

tion

JOINT DEMOSAICING AND DENOISINGNetwork training

Training data 31 x 31 patches from 700, 999x666 RGB

images (MIT-Adobe FiveK dataset)

Input - noisy image (realistic noise model)

- bilinear interpolation

Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

Ground truth

Noisy

RESULTSVisual comparison (+ unsharp masking)

Noisy BM3D (state of the art) Ground truth

Noisy

RESULTSVisual comparison (+ unsharp masking)

Noisy BM3D (state of the art) Ground truth

JOINT DEMOSAICING AND DENOISING: RESULTS

Average image quality metrics on the testing dataset

DOES IT GENERALIZE?

JPEG ARTIFACT REMOVAL&

SUPER-RESOLUTION

JPEG ARTIFACT REMOVALNetwork training

Training data 31 x 31 patches from 700, 999x666 RGB

images (MIT-Adobe FiveK dataset)

Input JPEG compressed image, 25% quality

Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

JPEG ARTIFACT REMOVAL: RESULTSVisual comparison (+ unsharp masking)

Ground truthL1 + MS-SSIML2JPEG

JPEG ARTIFACT REMOVAL: RESULTSNumerical comparison

Average image quality metrics on the testing dataset

SUPER-RESOLUTIONNetwork training

Training data 31 x 31 patches from 700, 999x666 RGB

images (MIT-Adobe FiveK dataset)

Input 2x downsampled image + upsampled

with bilinear interpolation

Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM

SUPER-RESOLUTION: RESULTSVisual comparison (+ unsharp masking)

L1 + MS-SSIML2Low rez

SUPERRESOLUTION: RESULTSNumerical comparison and literature

LEARNINGS?

LEARNINGSA closer look at the different losses

LEARNINGSand

LEARNINGSand

0.3939

0.3896

• seems to have more convergence issues.

• converges faster and speeds up the convergence or other losses, too.

LEARNINGSA closer look at the different losses

LEARNINGSSSIM and MS-SSIM

“Higher sensitivity to errors in texture-less regions!”

• Multi-scale is helpful when dealing with transition regions.

LEARNINGSA closer look at the different losses

RESULTSWhy mixing MS-SSIM and ?

CONCLUSIONS

• Even a shallow network can produce state-of-the-art results…

• …if you train it carefully.

• Perceptually-motivated loss functions can help!

• But you have to be aware of their limitations!

What have we learnt?

Thanks!

Zhao, Gallo, Frosio, and Kautz,“Loss Functions for Image Restoration with Neural Networks”,

IEEE Trans. on Comp. Imaging, 2017