3d video coding - xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/ppt/karstenmuller.pdf · 3d...
TRANSCRIPT
![Page 1: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/1.jpg)
Fraunhofer Institute for TelecommunicationsHeinrich-Hertz-Institut Berlin Einsteinufer 3710587 BerlinGermany
3D Video Formats and Coding Standards
+49 30 310 02 – [email protected] http://www.hhi.fraunhofer.de
Karsten Müller
![Page 2: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/2.jpg)
02.11.2009© Fraunhofer HHI
Folie 2
Kars
ten
Mül
ler
Overview 3D Video
• Introduction, scenarios, requirements
• 3D video formats
• 3D coding standards
• Towards a new 3D video coding standard
![Page 3: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/3.jpg)
02.11.2009© Fraunhofer HHI
Folie 3
Kars
ten
Mül
ler
3D Video Introduction
• Provision of left and right eye view• Added depth sensation • Resolved 2D viewing
ambiguity• Additional features
(e.g. free viewpoint, depth-controlled object insertion)
3D Video is more than adding 1 dimension to 2D! Source: Philips
![Page 4: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/4.jpg)
02.11.2009© Fraunhofer HHI
3D video scenarios3D cinema• Increasing number of 3D productions• First studios start to release all productions also in 3D• Technology: stereoscopic 3D, glasses based
3D home entertainment• Different types of displays available: stereoscopic, auto-stereoscopic
with 2 … N views• Various technologies, input formats and display sizes• Glasses based systems may not be acceptable
3D mobile• auto-stereoscopic 2 view display with fixed viewing position• Good 3D viewing in spite of small display sizes
Slide 4
Philip
p M
erkl
e
![Page 5: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/5.jpg)
02.11.2009© Fraunhofer HHI
Folie 5
Kars
ten
Mül
ler
Requirements: 3D processing chain
• Stereo / multiview capturing, depth provision
• Production format specification (e.g. SMPTE)
• Rectification, color correction, format conversion
• 3D format specification
• 3D video coding (e.g. MPEG)
• High quality intermediate view synthesis(e.g. 2 view + 2 depth N view)
• High-resolution stereo/multiview display
Capture
Post-processing
Coding
View Synthesis
ProductionFormat
TransportFormat
Transport Format
Display
Display Format
![Page 6: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/6.jpg)
02.11.2009© Fraunhofer HHI
Folie 6
Kars
ten
Mül
ler
The plethora of 3D Video Formats
• Conventional stereo video (CSV)• Mixed resolution stereo (MRS)• Video plus depth (V+D)
• Multiview video (MVV)• Multiview video + depth (MVD)• Layered depth video (LDV)
• Depth-enhanced stereo (DES)
![Page 7: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/7.jpg)
02.11.2009© Fraunhofer HHI
Slide 7
Philip
p M
erkl
e
assuming multiple views with color & depth…
Image Source: Interactive Visual Media Group of Microsoft Research
![Page 8: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/8.jpg)
02.11.2009© Fraunhofer HHI
Slide 8
Philip
p M
erkl
e
Stereo video (CSV)
CSV
![Page 9: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/9.jpg)
02.11.2009© Fraunhofer HHI
Slide 9
Philip
p M
erkl
e
Mixed resolution stereo (MRS)
MRS
![Page 10: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/10.jpg)
02.11.2009© Fraunhofer HHI
Slide 10
Philip
p M
erkl
e
Video plus depth (V+D)
V+D
![Page 11: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/11.jpg)
MVV
02.11.2009© Fraunhofer HHI
Slide 11
Philip
p M
erkl
e
Multiview video (MVV)
![Page 12: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/12.jpg)
MVD
02.11.2009© Fraunhofer HHI
Slide 12
Philip
p M
erkl
e
Multiview video plus depth (MVD)
![Page 13: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/13.jpg)
02.11.2009© Fraunhofer HHI
LDV
Slide 13
Philip
p M
erkl
e
Layered depth video (LDV)
![Page 14: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/14.jpg)
02.11.2009© Fraunhofer HHI
DES
Slide 14
Philip
p M
erkl
e
Depth-enhance Stereo (DES)
![Page 15: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/15.jpg)
02.11.2009© Fraunhofer HHI
Folie 15
Kars
ten
Mül
ler
3D Video Coding Standard(s)
• Advanced Video Coding (MPEG-4 AVC / H.264)– Simulcast– Stereo SEI– Auxilliary Picture Synthax
• MPEG-C part 3 (container format for V+D, e.g. with AVC coding)
• Multi-view Video Coding (MPEG-4 MVC / H.264)
All coding approaches are based on MPEG-4 AVC / H.264 standards family
![Page 16: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/16.jpg)
02.11.2009© Fraunhofer HHI
Folie 16
Kars
ten
Mül
ler
Towards a new 3D Video Coding Standard
• MPEG develops a new 3D video coding standard
• Motivation for new standard:– Decouple production from coding format– MPEG-4 AVC/H.264 only optimized for 2D color video, but not for
depth information
![Page 17: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/17.jpg)
3D components under consideration:– Transport format– 3D video coding methods– Intermediate view synthesis
02.11.2009© Fraunhofer HHI
Folie 17
Kars
ten
Mül
ler
Coverage of 3D Video Coding
Capture
Post-processing
Coding
View Synthesis
ProductionFormat
Transport Format
Transport Format
Display
Display Format
![Page 18: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/18.jpg)
02.11.2009© Fraunhofer HHI
Folie 18
Kars
ten
Mül
ler
Constraints for 3DVC
• Consider capturing technology, i.e. maximal 2-3 recorded views
• Break linear dependency of coding bitrate from number of target views (e.g. MVC)
• Provide scene geometry data in general form, i.e. pixel-wise depth data
• Consider statistical properties of depth (and supplementary) data
• Consider new quality evaluation methods for intermediate views
• Provide high-quality view synthesis for continuous viewing range
• Decrease depth and coding errors
![Page 19: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/19.jpg)
02.11.2009© Fraunhofer HHI
Folie 19
Kars
ten
Mül
ler
Summary
• 3D Video technology is maturing due to world wide development from capturing to display
• Although a variety of 3D/stereo formats exist,…• …only one underlying coding method with variations is
used: MPEG-4/AVC•
• For 3D video entertainment, new 3D Video coding beyond AVC/MVC is required and being developed
![Page 20: 3D Video Coding - Xidiansee.xidian.edu.cn/conference/mpegjpeg/workshop/PPT/karstenmuller.pdf · 3D video scenarios. 3D cinema ... • First studios start to release all productions](https://reader030.vdocuments.us/reader030/viewer/2022011802/5b1f40157f8b9ae6418c858b/html5/thumbnails/20.jpg)
02.11.2009© Fraunhofer HHI
Folie 20
Kars
ten
Mül
ler
Acknowledements
• Special Thanks to:
– Philipp Merkle