Download - 3D Realistic Broadcast System
-
8/2/2019 3D Realistic Broadcast System
1/38
Investigate of 3D realistic
broadcast system
-
8/2/2019 3D Realistic Broadcast System
2/38
Outline
Introduction
Video processing
Transmission Rendering
What are the most technical challenge?
Conclusion References
-
8/2/2019 3D Realistic Broadcast System
3/38
I. Introduction
Trends on broadcasting technology
-
8/2/2019 3D Realistic Broadcast System
4/38
Realistic broadcast systems
-
8/2/2019 3D Realistic Broadcast System
5/38
Requirements for realistic
-
8/2/2019 3D Realistic Broadcast System
6/38
Requirements for realistic
Multi-modal information
- 3D video with depth information at multiple
viewpionts.
- Multi-channel audio.
- CG model.
- Haptic Data. High quality data.
User friendly interaction.
-
8/2/2019 3D Realistic Broadcast System
7/38
Main components
-
8/2/2019 3D Realistic Broadcast System
8/38
II. Video processing
Multi-view video
-
8/2/2019 3D Realistic Broadcast System
9/38
Multi-view camera system
-
8/2/2019 3D Realistic Broadcast System
10/38
Multi-view codec system
-
8/2/2019 3D Realistic Broadcast System
11/38
Coding of multi-view video
-
8/2/2019 3D Realistic Broadcast System
12/38
Multi video coding
-
8/2/2019 3D Realistic Broadcast System
13/38
Cu trc don
Dongia cc min khng gian
-
8/2/2019 3D Realistic Broadcast System
14/38
Php donda trn H264/VAC
-
8/2/2019 3D Realistic Broadcast System
15/38
3D video system
-
8/2/2019 3D Realistic Broadcast System
16/38
Depth information
-
8/2/2019 3D Realistic Broadcast System
17/38
Depth camera system
-
8/2/2019 3D Realistic Broadcast System
18/38
Hybird camera system
-
8/2/2019 3D Realistic Broadcast System
19/38
Image of hybird camera
-
8/2/2019 3D Realistic Broadcast System
20/38
Problems of depth map
-
8/2/2019 3D Realistic Broadcast System
21/38
High quality depth map generation
-
8/2/2019 3D Realistic Broadcast System
22/38
III. Transmission
-
8/2/2019 3D Realistic Broadcast System
23/38
Cu trc cab m ha H264/AVC
-
8/2/2019 3D Realistic Broadcast System
24/38
NAL
Gip d dng nh xdliu H264/AVC titngvnchuynnh: RTP/IP chobtkdchv internet thi gian thc.
nhdng file thng dnglutr, truyndn.
H.23x cho dchvmthoi.
Hthng MPEG-2 cho dchvqung b.
Dliu video c m ha cspxp voNAL units.1 NAL unit gms nguyn lnbyte.
-
8/2/2019 3D Realistic Broadcast System
25/38
1 hthngdchvxut
-
8/2/2019 3D Realistic Broadcast System
26/38
3D contents generation
-
8/2/2019 3D Realistic Broadcast System
27/38
IV. Multiview Video Rendering
Belongs to the broad research field of image-
based rendering.
Multi-stereoscopic video with depth maps:
A number of video sequences captured from
diferent view points.
A single depth map is available to facilitate the
virtual view rendering.
-
8/2/2019 3D Realistic Broadcast System
28/38
Multi-stereoscopic video with depth
map
An example multiview data set.
-
8/2/2019 3D Realistic Broadcast System
29/38
Rendering and weight map generation
The rendering process from multiview video
-
8/2/2019 3D Realistic Broadcast System
30/38
Rendering Process
Split the to be rendered view into light rays.
For each light ray, trace the light ray to the surfaceof the depth map, obtain the intersection, andreprojected the intersection into nearby cameras(Cam 3 & Cam4).
The intensity of the light ray is thus the weightedaverage of the projected light rays in Cam 3 andCam 4.
The weight is the angular difference between thelight ray to be rendered and the light ray beingprojected.
-
8/2/2019 3D Realistic Broadcast System
31/38
Rendering Process (Cont)
When the virtual view point moves away from Cam4(where the depth map is given), the will be occlusionsand holes when computing the light ray/geometryintersection.
-
8/2/2019 3D Realistic Broadcast System
32/38
Rendering Process (Cont)
Convert the given depth map into 3D mesh surface, whereeach vertex corresponds to one pixel in the depth map.
The mesh surface is then projected to the capturing camerasto compute any potential occlusions in the captured images.
Finally, the mesh is projected to the virtual rendering pointwith multi-texture blending.
-
8/2/2019 3D Realistic Broadcast System
33/38
Rendering Process (Cont)
For each vertex being rendered, it is projected
to the nearby captured images to locate the
corresponding texture coordinate.
This process takes into consideration the
occlusion computed earlier: if a vertex is
occluded in a nearby view, its weight for that
camera will be set to zero.
-
8/2/2019 3D Realistic Broadcast System
34/38
The weight maps
The weight maps generated by the redering process
-
8/2/2019 3D Realistic Broadcast System
35/38
The weight maps (Cont)
The brighter pixels are the ones with largerweights.
During compression of the multiview video,
the pixels with high weights shall be encodedwith high quality, while the rest pixels can beencoded with low quality QuantizationParameter map: portion of the image that havelow weights (and thus high QP) are much morecoarsely quantized
-
8/2/2019 3D Realistic Broadcast System
36/38
Quantization Parameter Maps
QP values for weight maps
-
8/2/2019 3D Realistic Broadcast System
37/38
Rendering Process (Cont)
A segment of enceded images from cameras 2, 3 and 4
-
8/2/2019 3D Realistic Broadcast System
38/38
V. Technically ChallengingDepth/Disparity Estimation
Sub-pixel accuracy Temporal enhancement to reduce flickering effects
Depth map refinement for distorted depth map
Coding of Multiview Video + Depth Map Coding structure
Depth map coding scheme
Bit allocation for depth map coding
Intermediate View Synthesis
View synthesis method for depth map distortion
Filtering along object boundaries