© 2018 InterDigital, Inc. All Rights Reserved.© 2019 InterDigital, Inc. All Rights Reserved.
Volumetric Video: The “MPEG Metadata for Immersive Video”Distribution Format
© 2018 InterDigital, Inc. All Rights Reserved.© 2019 InterDigital, Inc. All Rights Reserved.
Volumetric Video:Key Use Cases
• Volumetric content is the next generation of video
• Users can experience the sensations of depth and parallax
• Volumetric video enables increased immersion into a content
Volumetric Video Creates an Immersive Experience
© 2019 InterDigital, Inc. All Rights Reserved.3
Volumetric Video is mandatory in VR
Users use volumetric video for:
• Up to 360° video with parallax
• Content that is not flat
• To enhance the overall video experience
• More immersive
• More natural
• Less discomfort
Experience of Parallax in VR
© 2019 InterDigital, Inc. All Rights Reserved.4
On television, users can experience the sensation of depth and parallax using volumetric video
A key example:
• Dynamic Window Experience
Experience of Parallax on TV
© 2019 InterDigital, Inc. All Rights Reserved.5
On smartphones, volumetric video can create new experiences on any 2D screen
Experience of Parallax on Smartphones
© 2019 InterDigital, Inc. All Rights Reserved.6
© 2018 InterDigital, Inc. All Rights Reserved.© 2019 InterDigital, Inc. All Rights Reserved.
CreatingVolumetric Videofrom Multi-ViewsContent
To create a volumetric experience, there is a strong correlation between the degree of freedom of the experience and video rig design
Adapt Capture Setup to the Degree of Freedom of the
Experience
© 2019 InterDigital, Inc. All Rights Reserved.8
Viewing
Capture
Dynamic Window VR
• Utilize light field camera arrays
• Leverage the relationship between degree of freedom and rig design
• Consider the challenge of capturing a large environment
Capture Real Video
© 2019 InterDigital, Inc. All Rights Reserved.9
• Creating an up to 360 °video with a virtual rig on a computer-generated (CG) scene
Develop Computer-Generated Content
© 2019 InterDigital, Inc. All Rights Reserved.10
• Develop a scene composition for non full light field capture
• Provides large field of view content
• Use VFX compositing tools
Create a Composite
© 2019 InterDigital, Inc. All Rights Reserved.11
© 2018 InterDigital, Inc. All Rights Reserved.© 2019 InterDigital, Inc. All Rights Reserved.
How to Create Specific Volumetric Streams
© 2019 InterDigital, Inc. All Rights Reserved.13
The Encoder Pre-Processing StepsTransforming Multi-Views and Depth Information into Video Streams
MULTI-VIEWS + DEPTH
Pruner Step: Remove redundancies between multi-views
Create patches from non-redundant information for each view
Packing Step: Create texture and depth atlas information for each frame
TEXTURE + DEPTH + METADATA
streams to encode
Finding RedundanciesBetween Multi-Views
© 2019 InterDigital, Inc. All Rights Reserved.14
• The Pruner Step
• The Packing Step
© 2019 InterDigital, Inc. All Rights Reserved.15
© 2019 InterDigital, Inc. All Rights Reserved.16
© 2018 InterDigital, Inc. All Rights Reserved.© 2019 InterDigital, Inc. All Rights Reserved.
Understandingthe MPEG Metadata for Immersive Video(MIV) Distribution Format
MPEG Metadatafor Immersive Video Format
• The MPEG MIV format has three streams:
✓ Texture
✓ Depth
✓ Metadata
• Content is encoded usingstandard compression codec (HEVC)
• Metadata includescamera parameters and patch list
ROADMAP
• Reference Software V1 available
• MPEG Committee draftbegins in 2020 (MPEG-I Part 12)
© 2019 InterDigital, Inc. All Rights Reserved.18
DECODER
Depth HEVCdecoding
MetadataTexture HEVC
decoding
POST-PROCESSING
PRE-PROCESSING
Depth HEVCcoding
MetadataTexture HEVC
coding
ENCODER
MULTI-VIEWS + DEPTH
DEVICE DEPENDANT VIEWPORT
STANDARDSTREAMS
MPEG Metadata for Immersive Video Format: Full Workflow
MPEG MIV encoder block diagram MPEG MIV decoder block diagram
• From Multi-views + depth
• To input streams for compression
• From decoded streams
• To device specific viewport
Basic Views (T+D)
Basic Camera Parameters List
Additional Views (T+D)
Additional Camera Parameters List
View Optimizer
Source Camera Parameters
Source Views(T+D)Source Views
(T+D)Source Views
(T+D)
CameraParameters List
Atlas Constructor
Pruner
Aggregator
Path Packer
AtlasParameters List
MetadataComposer
Video Texture
Encoder
Video Depth
Encoder
Decoder
TextureVideo
Decoder
Depth Video
Decoder
MetadataParser
ReferenceRenderer
Atlas Patch Occupancy Map
Generator
Bitstream
Viewport
Viewing positionViewing orientationEncoder
Texture component
Depthcomponent
Atlas
A B C
A
B
CA
B
C
D
E
© 2019 InterDigital, Inc. All Rights Reserved.19
Encoder Decoder
© 2018 InterDigital, Inc. All Rights Reserved.© 2019 InterDigital, Inc. All Rights Reserved.
How to RenderVolumetric Video
• View interpolation takes sparse information to create a smooth rendering of immersive content
• View interpolation is an important quality assessment point for the user
View Interpolation
© 2019 InterDigital, Inc. All Rights Reserved.21
© 2018 InterDigital, Inc. All Rights Reserved.© 2019 InterDigital, Inc. All Rights Reserved.
MIV Distribution Format: The Results
© 2019 InterDigital, Inc. All Rights Reserved.23
© 2019 InterDigital, Inc. All Rights Reserved.24
© 2019 InterDigital, Inc. All Rights Reserved.25
Adam Content,
Unity
© 2018 InterDigital, Inc. All Rights Reserved.© 2019 InterDigital, Inc. All Rights Reserved.
What’s Next for MPEG MIV Format and Volumetric Video?
© 2019 InterDigital, Inc. All Rights Reserved.27
Unlocking the Potential of Volumetric Video
To achieve the benefits and opportunities of volumetric video we must:
• Identify new content types where volumetric essence is core
• Develop editing tools to ingest true volumetric content
• Create deep-learning solutions to scale content creation & unlock rendering technologies
• Implement encoding and rendering tools for the next generation of capture and display devices
• Develop video formats & distribution solutions that are adapted to diverse use cases and market timelines and show a path to extended immersive experiences