image restoration with neural networks · several types of noise involved in the image formation:...
TRANSCRIPT
Work with Hang Zhao, Iuri Frosio, Jan Kautz
IMAGE RESTORATION WITH NEURAL NETWORKSOrazio Gallo
MOTIVATIONThe long path of images…
Demosaic Denoise
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black
Level
Metering
AF/AE
Image Signal Processor (ISP)
DEMOSAICINGcolors by interpolation
Image c
redit
: W
ikip
edia
Image c
redit
: M
arc
Levoy
DENOISINGSeveral types of noise involved in the image formation:
• Photon shot noise
• Dark current (AKA thermal noise)
• Photo-response non-uniformity
• Vignetting
• Readout noise:
• Reset noise (charge-to-voltage transfer)
• White noise (during voltage amplification amplification)
• Quantization noise (ADC)
DENOISING
MOTIVATION
Demosaic Denoise
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black
Level
Metering
AF/AE
Demosaicing before denoising changes the
statistics of the noise. And the best de-noising
algorithms require to know what the noise looks like.
MOTIVATION
Denoise Demosaic
Bad Pixel
Correction
Image
Enhancing
Tone
Mapping
Lens
Correction
Black
Level
Metering
AF/AE
Denoising first can change the color reproduction
accuracy as the three channels may be denoised
differently.
PSF CFA Noise
[1] Heide et al., ACM SIGGRAPH Asia 2012 (ToG)
FLEXISP1
A Flexible Camera Image Processing Framework
CAN WE DO IT WITH A NEURAL NETWORK?
Can we do it with a neural network, which moves
the heavy lifting to the training stage and inference
is very quick?
JOINT DEMOSAICING AND DENOISING
JOINT DEMOSAICING AND DENOISINGNetwork architecture
convolu
tion
convolu
tion
convolu
tion
bilin
ear
inte
rpola
tion
MEASURING IMAGE QUALITY
Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/
Original
Higher sensitivity to errors in texture-less regions!
MEASURING IMAGE QUALITY
Wang, et al. "Image quality assessment: from error visibility to structural similarity." IEEE TIP (2004)
MEASURING IMAGE QUALITY
Image adapted from https://ece.uwaterloo.ca/~z70wang/research/ssim/
Original
0.988 0.662
MEASURING IMAGE QUALITYHigher sensitivity to errors in texture-less regions!
JOINT DEMOSAICING AND DENOISINGNetwork architecture
convolu
tion
convolu
tion
convolu
tion
bilin
ear
inte
rpola
tion
JOINT DEMOSAICING AND DENOISINGNetwork training
Training data 31 x 31 patches from 700, 999x666 RGB
images (MIT-Adobe FiveK dataset)
Input - noisy image (realistic noise model)
- bilinear interpolation
Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM
Ground truth
Noisy
RESULTSVisual comparison (+ unsharp masking)
Noisy BM3D (state of the art) Ground truth
Noisy
RESULTSVisual comparison (+ unsharp masking)
Noisy BM3D (state of the art) Ground truth
JOINT DEMOSAICING AND DENOISING: RESULTS
Average image quality metrics on the testing dataset
DOES IT GENERALIZE?
JPEG ARTIFACT REMOVAL&
SUPER-RESOLUTION
JPEG ARTIFACT REMOVALNetwork training
Training data 31 x 31 patches from 700, 999x666 RGB
images (MIT-Adobe FiveK dataset)
Input JPEG compressed image, 25% quality
Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM
JPEG ARTIFACT REMOVAL: RESULTSVisual comparison (+ unsharp masking)
Ground truthL1 + MS-SSIML2JPEG
JPEG ARTIFACT REMOVAL: RESULTSNumerical comparison
Average image quality metrics on the testing dataset
SUPER-RESOLUTIONNetwork training
Training data 31 x 31 patches from 700, 999x666 RGB
images (MIT-Adobe FiveK dataset)
Input 2x downsampled image + upsampled
with bilinear interpolation
Training cost function L2 / L1 / SSIM / MS-SSIM / L1 + MS-SSIM
SUPER-RESOLUTION: RESULTSVisual comparison (+ unsharp masking)
L1 + MS-SSIML2Low rez
SUPERRESOLUTION: RESULTSNumerical comparison and literature
LEARNINGS?
LEARNINGSA closer look at the different losses
LEARNINGSand
LEARNINGSand
0.3939
0.3896
• seems to have more convergence issues.
• converges faster and speeds up the convergence or other losses, too.
LEARNINGSA closer look at the different losses
LEARNINGSSSIM and MS-SSIM
“Higher sensitivity to errors in texture-less regions!”
• Multi-scale is helpful when dealing with transition regions.
LEARNINGSA closer look at the different losses
RESULTSWhy mixing MS-SSIM and ?
CONCLUSIONS
• Even a shallow network can produce state-of-the-art results…
• …if you train it carefully.
• Perceptually-motivated loss functions can help!
• But you have to be aware of their limitations!
What have we learnt?
Thanks!
Zhao, Gallo, Frosio, and Kautz,“Loss Functions for Image Restoration with Neural Networks”,
IEEE Trans. on Comp. Imaging, 2017