resolution enhancement of low-quality videos using a high-resolution frame

Resolution enhancement of

low-quality videos using a

high-resolution frame

Tuan Pham 1

Lucas van Vliet 1

Klamer Schutte 2

1. Quantitative Imaging Group, Delft University of Technology, The Netherlands

2. TNO Physics and Electronics Laboratory, The Netherlands

Visual Communications and Image Processing 2006

SPIE paper 6077-8

© Tuan Pham ([email protected]) 2

Contents

1. Influence of compression on Super-Resolution

2. Example-based SR in the DCT domain

3. Application in video compression


Super-Resolution reconstruction

•Problems with compressed inputs:

– Registration: less accurate motion vectors due to compression error

– Fusion: outlier intensities due to ringing and blocking artifacts

– Deblur: distorted and truncated frequency spectrum due to

quantization


•Each 8x8 image block is projected onto 64 orthogonal bases

→ 64 DCT coefficients per block

Discrete Cosine Transform

Input image 64 DCT basis functions DCT coefficients

•The coefficients are then quantized (lossy) & compressed (lossless)


Quantization noise and blur

Quantization noiseJPEG quality 80

24

68

24

684

5

6

7

8

xy

MS

E o

f all

8x8

bloc

ks

Average error variance of 8x8 blocks

Average DCT attenuation factor


SR by texture synthesis

Hi-Res texture source

Interpolation

Multi-frame fusion

or

Texture transfer

Hi-Res blurred intermediate

video

Super-Resolved output video

Lo-Res compressed input video

•Example-based SR1 aims to fill-in the missing frequencies:

1. Freeman et al., “Example-based Super-Resolution”, IEEE Comp.Graph. & App., 2001

•We propose a DCT-domain SR synthesis algorithm which

can be applied directly on the compressed video stream.


Resize versus SR in DCT domain

•Super-resolution in DCT domain:

– fill-in plausible high-frequency coefficients instead of zero-ing

1. Mukherjee and Mitra, “Image resizing in the compressed domain using subband DCT”, CSVT, 2002

•Image resize1 in DCT domain:

– zero filling of high-frequency content followed by transcoding

transcodeenlarge

0 fill-in

8x8DCT

LR DCT

16x16 DCT 2x HR DCT

0

0 0

8x8DCT

8x8DCT

8x8DCT

8x8DCT

8x8DCT


Example-based SR in DCT domain

MeanAbs + ε

8x8 quantized DCT

15 LR DCTs 33 overlapping pixels

DC

Best Match

Training data

SR decoded image

+

AC

÷

×α

×

•Block-wise SR synthesis in a raster-scan order:

– Correspondence constraint: low-freq. coefficients of LR - HR blocks should match

– Spatial continuity constraint: boundary pixels of adjacent HR blocks should match


DCT-domain SR

Texture source: CIF frame 0

DCT-domain SR synthesis

+

Input QCIF frame 20 (JPEG quality 80)

Example-based SR (Freeman)

vs spatial domain SR


DCT-domain SR vs spatial domain SR

Texture source: CIF frame 0

DCT-domain SR

+

Input QCIF frame 50 (JPEG quality 80)

Local Linear Embedding SR (CVPR’04)


Importance of a good texture source

input (JPEG quality 50) source from 30-frame away

wrong source

QCIF input good but not striking poor SR result

+


•LR compressed video + a regularly updated HR source

SR using updated HR frames

QCIF input at JPEG quality 80 CIF output using 1 HR frame / sec

•Application: video coding for devices with limited resources


Conclusions

•SR reconstruction of compressed image is difficult because of:

– truncated frequency spectrum due to heavy quantization

– space-variant quantization noise

•DCT-domain SR is better than spatial domain SR because it:

– produces better results

– is more efficient

•Application of SR synthesis using a HR frame:

– video compression for systems with limited resources (phones, cameras)


Tuan Pham

[email protected]

Thank you !