an efficient multiview video compression scheme

1

An Efficient Multiview Video Compression An Efficient Multiview Video Compression SchemeScheme

2006. 10. 262006. 10. 26Lee,ByunghwaLee,Byunghwa

([email protected])([email protected])

2Computer EngineeringKyungpook National University

ContentsContents IntroductionIntroduction Neighbor-Based Multiview Video CompressionNeighbor-Based Multiview Video Compression Experimental ResultsExperimental Results ConclusionsConclusions


IntroductionIntroduction Multiview video compressionMultiview video compression

Critical part to the success of a real-time 3D video application over Critical part to the success of a real-time 3D video application over networksnetworks

Traditional videoTraditional video Essentially a two-dimensional medium and only provides a passive Essentially a two-dimensional medium and only provides a passive

way for viewers to observe the eventway for viewers to observe the event 3D video3D video

Allows viewers to actively move their own eyes to interested targets Allows viewers to actively move their own eyes to interested targets that are not the focus of the cameraman. 3D video normally needs that are not the focus of the cameraman. 3D video normally needs to capture multiple synchronized video streams, which is usually cato capture multiple synchronized video streams, which is usually called multiview video, and new viewpoints are created by advanced glled multiview video, and new viewpoints are created by advanced graphics rendering algorithms such as Image-Based Rendering (IBR)raphics rendering algorithms such as Image-Based Rendering (IBR)


IntroductionIntroduction 3D video can be created by two general approaches3D video can be created by two general approaches

model-based approachmodel-based approach Multiview video from sparsely-arranged cameras and extracts 3D modeMultiview video from sparsely-arranged cameras and extracts 3D mode

ls from the images and then renders the new viewpoints with the help ls from the images and then renders the new viewpoints with the help of a 3D scene modelof a 3D scene model

Advantage - reducing the acquisition cost by using few cameras. Advantage - reducing the acquisition cost by using few cameras. Disadvantage - increases the algorithmic complexity and very difficult pDisadvantage - increases the algorithmic complexity and very difficult p

roblem to extract scene models from general real-world scenes in real roblem to extract scene models from general real-world scenes in real timetime


IntroductionIntroduction pure image-based approachpure image-based approach

uses densely arranged cameras to acquire high-resolution light fields auses densely arranged cameras to acquire high-resolution light fields and then uses image-based rendering to generate images at the new vind then uses image-based rendering to generate images at the new viewpointsewpoints

Advantage - reconstructing new views without the scene model.Advantage - reconstructing new views without the scene model. Disadvantage - demands more storage and transmission bandwidth for Disadvantage - demands more storage and transmission bandwidth for

multiview video data. multiview video data.

It makes multiview video compression indispensable fIt makes multiview video compression indispensable for the practical use of 3D videoor the practical use of 3D video


IntroductionIntroduction Multiview video compressionMultiview video compression

Simultaneously reduce temporal and spatial redundancy among muSimultaneously reduce temporal and spatial redundancy among multiple synchronized video streamsltiple synchronized video streams

MVP(MPEG-2 Multiview Profile) proposes a block-based steMVP(MPEG-2 Multiview Profile) proposes a block-based stereoscopic coding(BBSC) to encode the stereo videoreoscopic coding(BBSC) to encode the stereo video MCP(Motion-compensated prediction)MCP(Motion-compensated prediction)

Reduce the temporal redundanciesReduce the temporal redundancies DCP(Disparity-compensated prediction)DCP(Disparity-compensated prediction)

Reduce the spatial redundanciesReduce the spatial redundancies


IntroductionIntroduction Main StreamMain Stream

Central stream among all the streamsCentral stream among all the streams Encoded using only a MCP-based MPEG2-like Encoded using only a MCP-based MPEG2-like

algorithmalgorithm Secondary StreamSecondary Stream

Compressed with MCP and DCP based on the main Compressed with MCP and DCP based on the main streamstream

Center Approach


Neighbor-based Multiview Video CompressionNeighbor-based Multiview Video Compression

Main StreamMain Stream Encoded using only a MCP-based MPEG2-like Encoded using only a MCP-based MPEG2-like

algorithmalgorithm Secondary StreamSecondary Stream

Compressed using DCP from multiple nearest Compressed using DCP from multiple nearest neighbors and MCPneighbors and MCP

Neighbor Approach

Better spatial redundancy reduction than center approach and thus improve the video quality under same bit rate



Basic IdeaBasic Idea Use Multiple nearest neighbor frames for DCPUse Multiple nearest neighbor frames for DCP

However, the synchronized frames in multiview video are coded in However, the synchronized frames in multiview video are coded in a specific order and the early-coded frames may not find optimal na specific order and the early-coded frames may not find optimal nearest neighbor frames as reference framesearest neighbor frames as reference frames

Encoding Order

S2S3S1S4S5

Two nearest stream is S1 and S5.

But, S4 have frames in stream in S1 and S2 as reference frames.



The key challenge of the neighbor approach is to decide thThe key challenge of the neighbor approach is to decide the stream encoding order so that the number of streams whe stream encoding order so that the number of streams which have the optimal neighbor streams as reference frames ich have the optimal neighbor streams as reference frames can be maximized and thus the maximal decorrelation of scan be maximized and thus the maximal decorrelation of spatial coherence can be achieved.patial coherence can be achieved.


Multiview Video


Experimental ResultsExperimental Results Neighbor approach has a better video quality Neighbor approach has a better video quality

than both the center approach and MPEG2than both the center approach and MPEG2 Neighbor approach uses the nearest neighbor Neighbor approach uses the nearest neighbor

frames as reference frames and can better exploit frames as reference frames and can better exploit the spatial coherence than center approach and the spatial coherence than center approach and MPEG2 that uses only temporal prediction and no MPEG2 that uses only temporal prediction and no spatial predictionspatial prediction

The number of the neighbor frames The number of the neighbor frames increases, the reconstructed video quality increases, the reconstructed video quality also increasesalso increases Since more neighbor frames lead to more spatial Since more neighbor frames lead to more spatial

redundancy reductionredundancy reduction


Experimental ResultsExperimental Results PSNR improvement is limited as the number of neighbor frPSNR improvement is limited as the number of neighbor fr

ames increases.ames increases. Since only a limited number of macroblocks will be predicted by neSince only a limited number of macroblocks will be predicted by ne

w added neighbor frames while most other macroblocks already haw added neighbor frames while most other macroblocks already have a small prediction error and it is hard to futher diminish the predve a small prediction error and it is hard to futher diminish the prediction error with the added neighbor framesiction error with the added neighbor frames


Experimental ResultsExperimental Results


Experimental ResultsExperimental Results Each macroblock in the proposed compression scheme caEach macroblock in the proposed compression scheme ca

n be predicted using one of 6 modes: Forward (F), Backwarn be predicted using one of 6 modes: Forward (F), Backward (B), Disparity (D), Forward and Backward interpolation (Fd (B), Disparity (D), Forward and Backward interpolation (FB), Forward and Disparity Interpolation (FD), Backward and B), Forward and Disparity Interpolation (FD), Backward and Disparity Interpolation (BD)Disparity Interpolation (BD)


Experimental ResultsExperimental Results

The average percentage of macroblocks used for disparity prediction related types in both the center and the neighbor approach at the bit rate 1Mbps for ea

ch stream in the case of 14 streams.

disparity-related compensation accommodates most prediction types and plays a significant role to reduce prediction error and improve the image quality. The results also show that the neighbor approach has a higher percentage macroblocks predicted by disparity than the center approach. Thus the better image quality in the neighbor approach should obviously be attributed to the spatial redundancy reduction. From the data, it is also evident that the benefits from increasing the number of neighbor frames is limited and the gain becomes smaller and smaller as the number of neighbor frames continually increase

s.


ConclusionsConclusions Proposed Algorithm is an efficient to find a Proposed Algorithm is an efficient to find a

nearly optimal stream encoding order which nearly optimal stream encoding order which maximizes spatial redundancy reduction.maximizes spatial redundancy reduction.

Experimental results show that the proposed Experimental results show that the proposed compression scheme can achieve better compression scheme can achieve better video quality over the center approach and video quality over the center approach and MPEG2 under same bit rate.MPEG2 under same bit rate.

an efficient multiview video compression scheme

Documents

multiview video data

video quality

d video application

mcpbased mpeg2

multiview profile

d models

synchronized frames

d scene modeladvantage