Advanced Computer Technology IIFTV and 3DV
KyungHee Univ.Master Course
2009311004Kim Kyung Yong
04/19/23
INDEX
• Introduction• Definition of FTV• Overview of FTV Technologies• FTV Applications and Services• Standardization Items for FTV• Requirements of FTV video• 3DV Introduction, Motivation• Problem statement• Proposed Technical Solution• Reference
04/19/23 2
Introduction
• Evolution of image systems
04/19/23 3
Digital camera
Digital cinema
Digital TV
FTV
Photography
Cinema
TV
Pixel-basedsyste
m
Ray-basedsyste
m
past
present
future
Digitalize
More views
More pixels
FTV : Free viewpoint TV
Definition of FTV
• FTV (Free viewpoint TV)– A new and very broad type of audio-visual system that
allows each user to view the real 3D space from different user viewpoints.
Overview of FTV Technologies
• FTV System and FTV data format
DVB : Digital Video Broadcasting
Overview of FTV Technologies
• Configuration of FTV System
04/19/23 6
Acquisition Correction Encoding
Transmission / Storage
DecodingView
GenerationDisplay
Overview of FTV Technologies
• 1. Acquisition
Linear arrangement
1D-arc arrangement
Circular arrangement
2D-array arrangement
Overview of FTV Technologies
• 2. Correction
Before
After
1) Correction of Misalignment 2) Color Correction
Camera A Camera B
- Before
- AfterCamera A Camera B
Overview of FTV Technologies
• 3. Encoding and Decoding– The standardization is progressing with MVC in JVT.– This is work to do us.
Overview of FTV Technologies
• 4. View Generation– For the linear camera arrangement
– For the circular camera arrangement
Overview of FTV Technologies
• 4. View Generation [Cont.]– Example of generated FTV images at various times and
viewpoints
Overview of FTV Technologies
• 5. Display– 2D display with head tracking
Overview of FTV Technologies
• 5. Display [Cont.]– 3D display with and without head tracking
Overview of FTV Technologies
• 5. Display [Cont.]– 3D displays with various numbers of views
Overview of FTV Technologies
• 5. Display [Cont.]– 360 degree ray-reproducing 3D display (SeeLINDER)
FTV Applications and Services
• Applications of FTV
04/19/23 16
Field Examples
communication/broadcast very realistic, user-centered
Entertainment game, sports, event, exhibition, Expo
Advertisement new type of presentation
nature observation bird watching, safari, undersea park
sightseeing barrier-free
museum new type of exhibition
art/content creation of new media art and digital content
video production capturing and production can be separatedunrealistic camera motions can be achieved
archive spatio-temporal archive, living national treasures, traditional entertainment
education distant education, educational tool
medicine examination, operation, education
security, surveillance store, factory, building, street, public facilities
transportation traffic monitoring at intersection, ITS
FTV Applications and Services
• System configurations of FTV Services
04/19/23 17
Standardization Items for FTV
• Position of depth search and interpolation
04/19/23 18
Standardization Items for FTV
• FTV reference model with items considered for standardization– FTV Data Format
• Multi-view video, camera parameters, depth data
– Decoder– Interpolation
04/19/23 19
CaptureCorrection
/ConversionEncoder Decoder Display
Transmission/Storage
Rendering
FTVData Format
CaptureCorrection
/ConversionEncoder Decoder Display
Transmission/Storage
Interpolation
FTVData Format
CaptureCorrection
/ConversionEncoder Decoder Display
Transmission/Storage
Rendering
FTVData Format
CaptureCorrection
/ConversionEncoder Decoder Display
Transmission/Storage
Interpolation
FTVData Format
Requirements of FTV video
• Requirements for Data Format on Visual information– Video data
• Video and audio data shall be supported in the data format. Video data may be corrected from captured images by real multiple cameras, so that misalignment of camera geometry and colors can be removed.
– Depth maps• Depth maps shall be supported in the data format. Depth maps may be
generated by conversion from captured images by real multiple cameras or through special depth cameras or by other means. Types of depth information should be defined that are able to derive the original object distance.
– Camera parameter• Camera parameters, both extrinsic and intrinsic parameters, shall be supported in
the data format.
– Additional data• Additional data could be necessary to make the system more efficient. E.g. data
to represent transparency or specular reflection.
04/19/23 20
Requirements of FTV video
– Low complexity for editing• The data format should allow for editing with low complexity.
– Applicability• The data format should be applicable for both natural and synthetic scenes.
04/19/23 21
Requirements of FTV video
• Requirements for Compression on Depth– Compression efficiency
• Depth maps should be coded efficiently.
– Synthesis accuracy• The impact on the visual distortion on synthesized views should be minimized.
04/19/23 22
Requirements of FTV video
• Requirements for Interpolation on Visual information– Control of rendering images
• The range of free viewpoint should be controlled.
– Specification of rendering parameters• Rendering control by specifying parameters in bit-stream should be supported.
– Low complexity• Conversion from FTV data to rendered data should be low complexity.
– Display types• Various capabilities of displays, e.g. display types, number of views should be
supported.
– Scalability• Scalable rendering should be supported for real-time processing.
04/19/23 23
Introduction, Motivation
• 3D Video– 3D video and free viewpoint video are advanced types of
visual media that expand user sensation.– The functionality of 3D video shall denote depth
perception of the observed scenery, also known as stereo.
– Free viewpoint video provides the user the functionality selecting an arbitrary viewpoint onto the observed scenery within practical limits.
04/19/23 24
Problem statement
• It illustrates 3DV and FVV systems and applications that are not efficiently supported yet by available standards
• Conventional stereo displays only provide an unnatural 3D impression, since head motion parallax viewing can not be supported with 2 fixed views.
04/19/23 25
Problem statement
• Auto-stereoscopic display– This is only possible, if the available input data support
rendering a continuum of output views.
04/19/23 26
Problem statement
• Holografika’s HV RC640 wide angle 3D display– The more views are used the wider the effect.
04/19/23 27
Proposed Technical Solution
• Multi-view video plus depth (MVD) format[Cont.]– System 1.
• Very limited free viewpoint functionality is supported as view selection.
04/19/23 28
Proposed Technical Solution
• Multi-view video plus depth (MVD) format[Cont.]– System 2.
• However, quality of virtual views decreases with distance from the available camera, due to dis-occlusion and other problems.
• Therefore single video plus depth is not suitable to support wide range free viewpoint navigation and wide angle 3D displays that require more than 2 views displayed at the same time.
04/19/23 29
Proposed Technical Solution
• Multi-view video plus depth (MVD) format[Cont.]– Proposed System
• Multi-view video plus depth is a combination of the 2 formats illustrated in Fig. 3 and Fig. 4.
04/19/23 30
Proposed Technical Solution
• Generation of MVD typically involves a number of sophisticated image processing algorithms. (M > N)
04/19/23 31
N original images
Rectification
Disparity Matching
N-1 rectified image pairs
N-1 rectified pairs video plus disparity
Depth Map Creation
Analysis
De-Rectification
N video-plus-depth representations
• • • •
• •
• • • •
• •
• • N-1 rectified pairs video plus depth
Proposed Technical Solution
• Multi-view video plus depth (MVD) format[Cont.]
04/19/23 32
Reference
• MPEG N9466– Applications and Requirements on FTV
• MPEG M14417– Proposal on Requirements for FTV
• JVT-W100– Multi-view video plus depth (MVD) format for advanced
3D video systems
04/19/23 33