advanced computer technology ii ftv and 3dv kyunghee univ. master course 2009311004 kim kyung yong...

Advanced Computer Technology IIFTV and 3DV

KyungHee Univ.Master Course

2009311004Kim Kyung Yong

04/19/23

INDEX

• Introduction• Definition of FTV• Overview of FTV Technologies• FTV Applications and Services• Standardization Items for FTV• Requirements of FTV video• 3DV Introduction, Motivation• Problem statement• Proposed Technical Solution• Reference

04/19/23 2

Introduction

• Evolution of image systems

04/19/23 3

Digital camera

Digital cinema

Digital TV

FTV

Photography

Cinema

TV

Pixel-basedsyste

m

Ray-basedsyste

m

past

present

future

Digitalize

More views

More pixels

FTV : Free viewpoint TV

Definition of FTV

• FTV (Free viewpoint TV)– A new and very broad type of audio-visual system that

allows each user to view the real 3D space from different user viewpoints.

Overview of FTV Technologies

• FTV System and FTV data format

DVB : Digital Video Broadcasting

Overview of FTV Technologies

• Configuration of FTV System

04/19/23 6

Acquisition Correction Encoding

Transmission / Storage

DecodingView

GenerationDisplay

Overview of FTV Technologies

• 1. Acquisition

Linear arrangement

1D-arc arrangement

Circular arrangement

2D-array arrangement

Overview of FTV Technologies

• 2. Correction

Before

After

1) Correction of Misalignment 2) Color Correction

Camera A Camera B

- Before

- AfterCamera A Camera B

Overview of FTV Technologies

• 3. Encoding and Decoding– The standardization is progressing with MVC in JVT.– This is work to do us.

Overview of FTV Technologies

• 4. View Generation– For the linear camera arrangement

– For the circular camera arrangement

Overview of FTV Technologies

• 4. View Generation [Cont.]– Example of generated FTV images at various times and

viewpoints

Overview of FTV Technologies

• 5. Display– 2D display with head tracking

Overview of FTV Technologies

• 5. Display [Cont.]– 3D display with and without head tracking

Overview of FTV Technologies

• 5. Display [Cont.]– 3D displays with various numbers of views

Overview of FTV Technologies

• 5. Display [Cont.]– 360 degree ray-reproducing 3D display (SeeLINDER)

FTV Applications and Services

• Applications of FTV

04/19/23 16

Field Examples

communication/broadcast very realistic, user-centered

Entertainment game, sports, event, exhibition, Expo

Advertisement new type of presentation

nature observation bird watching, safari, undersea park

sightseeing barrier-free

museum new type of exhibition

art/content creation of new media art and digital content

video production capturing and production can be separatedunrealistic camera motions can be achieved

archive spatio-temporal archive, living national treasures, traditional entertainment

education distant education, educational tool

medicine examination, operation, education

security, surveillance store, factory, building, street, public facilities

transportation traffic monitoring at intersection, ITS

FTV Applications and Services

• System configurations of FTV Services

04/19/23 17

Standardization Items for FTV

• Position of depth search and interpolation

04/19/23 18

Standardization Items for FTV

• FTV reference model with items considered for standardization– FTV Data Format

• Multi-view video, camera parameters, depth data

– Decoder– Interpolation

04/19/23 19

CaptureCorrection

/ConversionEncoder Decoder Display

Transmission/Storage

Rendering

FTVData Format

CaptureCorrection

/ConversionEncoder Decoder Display

Transmission/Storage

Interpolation

FTVData Format

CaptureCorrection

/ConversionEncoder Decoder Display

Transmission/Storage

Rendering

FTVData Format

CaptureCorrection

/ConversionEncoder Decoder Display

Transmission/Storage

Interpolation

FTVData Format

Requirements of FTV video

• Requirements for Data Format on Visual information– Video data

• Video and audio data shall be supported in the data format. Video data may be corrected from captured images by real multiple cameras, so that misalignment of camera geometry and colors can be removed.

– Depth maps• Depth maps shall be supported in the data format. Depth maps may be

generated by conversion from captured images by real multiple cameras or through special depth cameras or by other means. Types of depth information should be defined that are able to derive the original object distance.

– Camera parameter• Camera parameters, both extrinsic and intrinsic parameters, shall be supported in

the data format.

– Additional data• Additional data could be necessary to make the system more efficient. E.g. data

to represent transparency or specular reflection.

04/19/23 20

Requirements of FTV video

– Low complexity for editing• The data format should allow for editing with low complexity.

– Applicability• The data format should be applicable for both natural and synthetic scenes.

04/19/23 21

Requirements of FTV video

• Requirements for Compression on Depth– Compression efficiency

• Depth maps should be coded efficiently.

– Synthesis accuracy• The impact on the visual distortion on synthesized views should be minimized.

04/19/23 22

Requirements of FTV video

• Requirements for Interpolation on Visual information– Control of rendering images

• The range of free viewpoint should be controlled.

– Specification of rendering parameters• Rendering control by specifying parameters in bit-stream should be supported.

– Low complexity• Conversion from FTV data to rendered data should be low complexity.

– Display types• Various capabilities of displays, e.g. display types, number of views should be

supported.

– Scalability• Scalable rendering should be supported for real-time processing.

04/19/23 23

Introduction, Motivation

• 3D Video– 3D video and free viewpoint video are advanced types of

visual media that expand user sensation.– The functionality of 3D video shall denote depth

perception of the observed scenery, also known as stereo.

– Free viewpoint video provides the user the functionality selecting an arbitrary viewpoint onto the observed scenery within practical limits.

04/19/23 24

Problem statement

• It illustrates 3DV and FVV systems and applications that are not efficiently supported yet by available standards

• Conventional stereo displays only provide an unnatural 3D impression, since head motion parallax viewing can not be supported with 2 fixed views.

04/19/23 25

Problem statement

• Auto-stereoscopic display– This is only possible, if the available input data support

rendering a continuum of output views.

04/19/23 26

Problem statement

• Holografika’s HV RC640 wide angle 3D display– The more views are used the wider the effect.

04/19/23 27

Proposed Technical Solution

• Multi-view video plus depth (MVD) format[Cont.]– System 1.

• Very limited free viewpoint functionality is supported as view selection.

04/19/23 28

Proposed Technical Solution

• Multi-view video plus depth (MVD) format[Cont.]– System 2.

• However, quality of virtual views decreases with distance from the available camera, due to dis-occlusion and other problems.

• Therefore single video plus depth is not suitable to support wide range free viewpoint navigation and wide angle 3D displays that require more than 2 views displayed at the same time.

04/19/23 29

Proposed Technical Solution

• Multi-view video plus depth (MVD) format[Cont.]– Proposed System

• Multi-view video plus depth is a combination of the 2 formats illustrated in Fig. 3 and Fig. 4.

04/19/23 30

Proposed Technical Solution

• Generation of MVD typically involves a number of sophisticated image processing algorithms. (M > N)

04/19/23 31

N original images

Rectification

Disparity Matching

N-1 rectified image pairs

N-1 rectified pairs video plus disparity

Depth Map Creation

Analysis

De-Rectification

N video-plus-depth representations

• • • •

• •

• • • •

• •

• • N-1 rectified pairs video plus depth

Proposed Technical Solution

• Multi-view video plus depth (MVD) format[Cont.]

04/19/23 32

Reference

• MPEG N9466– Applications and Requirements on FTV

• MPEG M14417– Proposal on Requirements for FTV

• JVT-W100– Multi-view video plus depth (MVD) format for advanced

3D video systems

04/19/23 33