A Fast and Efficient Multi-View Depth Image Coding Method Based on Temporal and Inter-View Correlations of Texture

Images

Jin Yong Lee

Ho Chen Wey

Du Sik Park

IEEE Transactions on CSVT 2011

Outline

IntroductionProposed Method

Temporal CorrelationTexture and Depth View SynthesisInter-View Correlation

EvaluationsCoding PerformanceEncoding Complexity AnalysisSubjective Quality Assessment

Introduction

Encode video information of each view individually with the H.264/AVC

A multi-view video coding structure with hierarchical B pictures

Multi-view video plus depth

Outline

IntroductionProposed Method

Temporal CorrelationTexture and Depth View SynthesisInter-View Correlation

EvaluationsCoding PerformanceEncoding Complexity AnalysisSubjective Quality Assessment

Temporal Correlation

Stationary regions have similar pixel values for successive frames

Use sum of the squared difference(SSD) to measure the temporal correlation

If SSD => Strongly Correlated Texture

images

Depth images

I P

Temporal Correlation

Texture images

Depth images

P IB

Temporal Correlation

Outline

IntroductionProposed Method

Temporal CorrelationTexture and Depth View SynthesisInter-View Correlation

EvaluationsCoding PerformanceEncoding Complexity AnalysisSubjective Quality Assessment

Texture and Depth View Synthesis

Synthesis the next depth view by 3D image warping

Translate gray values into real depth

Project the original points into 3D world

Reproject these 3D points into the image plane of

neighboring view

Hole-filling

Texture and Depth View Synthesis

The pixel position(x,y) at the reference frame can be projected into a 3D point (u,v,w)

The corresponding pixel location of the virtual image

[ 𝑢𝑣𝑤]=𝑅𝑟×𝐴𝑟− 1×[𝑥𝑦1 ]×𝑍𝑟 (𝑥 , 𝑦 )+𝑇 𝑟

()

[𝑥′

𝑦 ′

𝑧 ′ ]=𝐴𝑡×𝑅𝑡− 1×{[ 𝑢𝑣𝑤]−𝑇 𝑡}

Translate gray values into real depth

Project the original points into 3D world

Reproject these 3D points into the image plane of

neighboring view

Hole-filling

Texture and Depth View Synthesis

Some pixels in the synthesized image are missing or undefinedOcclusionPixel position quantization

Only consider the neighboring pixelsaround the holeIf a view is warped to the right view

position, the hole is filled with its left pixel

Translate gray values into real depth

Project the original points into 3D world

Reproject these 3D points into the image plane of

neighboring view

Hole-filling

Texture and Depth View Synthesis

Outline

IntroductionProposed Method

Temporal CorrelationTexture and Depth View SynthesisInter-View Correlation

EvaluationsCoding PerformanceEncoding Complexity AnalysisSubjective Quality Assessment

Inter-View Correlation

1. A depth image of I-view(), a texture image of I-view() and P-view() are encoded

2. Synthesized a virtual texture image from the reconstructed and

3. If SSD => skip the block

Texture images

Depth images

I-view P-view

Synthesized Image

texturedepth

Inter-View Correlation

Evaluations

Coding performanceEncoding complexity analysisSubjective Quality Assessment

Coding performance

Simulation conditions

Test sequences

Coding performance

Balloons

Kendo

Book Arrival

Newspaper

Champagnetower

Pantomime

BDBR : Average bit-rate difference in % over the whole range of PSNRBDPR : Average PSNR difference in dB over the whole range of bit-rates

Coding Performance

Balloons

Kendo Book Arrival

Newspaper Champane tower Pantomime

Encoding Complexity Analysis

ETR and RM indicate the total encoding time reduction and the reduced number of the modes performing the RD optimization in percentage

SB and VS represent the proportion of skipped macro blocks in depth image and the time required for the view synthesis process

Encoding Complexity Analysis

Subjective Quality Assessment

15 professional subjectsTwo stereoscopic view pairs were rendered using

the depth images decoded by the original method and the proposed method respectively

Subjective Quality Assessment

Y-axis indicates a differential score that subtract a score of the original method from the proposed method