overview of 360-degree video standards and efficient

1 © 2017 Gachon University. All rights reserved.

Overview of 360-degree Video Standards

and Efficient Streaming Technologies

Aug. 29. 2017

제1회 VR/AR 비디오 표준 기술 및 통신 네트워크 융합 강좌@가천대학교 가천관 B1 대강당

Eun-Seok Ryu (류은석; [email protected])

Multimedia Communications and Systems Lab(MCSL)

http://mcsl.gachon.ac.kr

Department of Computer Engineering

Gachon University


Contents

• 이론• HEVC and SHVC Standards

• 360 Video Processing for Virtual Reality (VR)

• 실습포함• HM Reference SW and Common Test Condition (CTC)

• JVET 360Lib Tools for 360-degree Video


1. 이론:

HEVC and SHVC Standards


Organization

• ISO/IEC JTC1 SC29 MPEG

• ISO (International Organization for Standardization)

• IEC (International Electrotechnical Commission)

• JTC 1/SC 29 (Subcommittee of the Joint Technical Committee); Topic: Coding of

audio, picture, multimedia and hypermedia information

• MPEG (Moving Picture Experts Group)

• ITU-T VCEG

• ITU (International Telecommunication Union)

• ITU-T (ITU Telecommunication Standardization Sector)

• VCEG (Video Coding Experts Group)

• Joint Collaboration Effort (MPEG with VCEG)

• JVT (Joint Video Team): H.264/AVC, MPEG-4 Part 10

• JCT-VC (Joint Collaborative Team on Video Coding)• MPEG-H Part 2, H.265/HEVC

• JVET (Joint Video Exploration Team)• FVC (Future Video Coding)


JCT-VC Document Management System

• URL: http://phenix.int-evry.fr/jct/

http://phenix.int-evry.fr/jct/


Service Menus

• All Meetings• 4 times / year (normally, Jan.-Feb., Mar.-Apr., Jun.-Jul., Oct.-Nov.)

• Next Meeting Proposal template download


Document Register System Details

• JCTVC-(Meeting Letter) (Document Number)

• e.g. JCTVC-X0001

• Also provides MPEG number: m (document number)

• e.g. m28763

• AHG(Ad-hoc group) and CE (Core Experiment) Reports: Normally with

documents number 0001 ~ 002X

• Overall Meeting Report and Standard Draft by Editors: Normally with

documents number 1000 ~ 102X

• well…Strategy? (could ruin the fair competition system)

• Registration with vague proposal title first.

• Update the proposal title and upload documents on meeting.


Document Types and Example

• Technical Proposal

• For future patent licensing / own product

protecting

• CTC (Common Test Condition) proposal

• Cross Check Document

• To verify the proposal from other

company/institute

• Experiment multiple test sets / check

proposal documents consistency

Zip compressed file- Proposal Document (.docx)- Presentation Slides (.ppt)- Excel spread sheets with

experimental results (.xls)

Presentation Slides

Proposal Documents with Patent rights declaration


Common Test Condition (CTC) - HEVC

• Huge test points

• 24 Test sequences (Class A ~ F) x 3 coding

structure (AI, RA, LD-B, LD-P (option)) x 4 QPs x

2 (Results of proposed and control groups)

• Early days, there were High / Low profiles in

HEVC (later, only Main profile): x2 times

• Currently, main profile and High efficiency (10 bit)

profile• Intra, main

• Intra, high efficiency, 10 bit

• Random access, main

• Random access, high efficiency, 10 bit

• Low delay, main

• Low delay, high efficiency, 10 bit

• Low delay, main, P slices only (optional)

• Low delay, high efficiency, P slices only, 10 bit (optional)

• QP values: 22, 27, 32, and 37

Class Sequence nameResolution

Frame count

Frame rate

Bit depth

Intra Random access

Low-delay

A Traffic2560x1600

150 30fps 8 Main/ Main10

Main/ Main10

A PeopleOnStreet2560x1600


Main/ Main10

A Nebuta2560x1600


Main/ Main10

A SteamLocomotive2560x1600


Main/ Main10

B Kimono1920x1080


Main/ Main10

Main/ Main10

B ParkScene1920x1080


Main/ Main10

Main/ Main10

B Cactus1920x1080


Main/ Main10

Main/ Main10

B BQTerrace1920x1080


Main/ Main10

Main/ Main10

B BasketballDrive1920x1080


Main/ Main10

Main/ Main10

C RaceHorses832x480


Main/ Main10

Main/ Main10

C BQMall832x480


Main/ Main10

Main/ Main10

C PartyScene832x480


Main/ Main10

Main/ Main10

C BasketballDrill832x480


Main/ Main10

Main/ Main10

D RaceHorses416x240


Main/ Main10

Main/ Main10

D BQSquare416x240


Main/ Main10

Main/ Main10

D BlowingBubbles416x240


Main/ Main10

Main/ Main10

D BasketballPass416x240


Main/ Main10

Main/ Main10

E FourPeople1280x720


Main/ Main10

E Johnny1280x720


Main/ Main10

E KristenAndSara1280x720


Main/ Main10

F BaskeballDrillText832x480


Main/ Main10

Main/ Main10

F ChinaSpeed1024x768


Main/ Main10

Main/ Main10

F SlideEditing1280x720


Main/ Main10

Main/ Main10

F SlideShow1280x720


Main/ Main10

Main/ Main10

*Document#: JCTVC-L1100


Common Test Condition (CTC) - SHVC

• Scalable HEVC• Spatial scalabilities (2x, 1.5x), and SNR scalabilities (using QPs) for both HEVC and AVC layers

• QP values:

Mandatory test

conditions

Intra, spatial 2xIntra, spatial 1.5xRandom access, spatial 2xRandom access, spatial 1.5xRandom access, SNRLow delay, B slices, spatial 2xLow delay, B slices, spatial 1.5xLow delay, B slices, SNR

Optional test conditions

Low delay, P slices only, spatial 2xLow delay, P slices only, spatial 1.5xLow delay, P slices only, SNR

Scalability ratio BL QP EL delta QP

Spatial 2x and 1.5x 22, 26, 30, 34 0, 2

SNR 26, 30, 34, 38 -6, -4

* Document#: JCTVC-Q1009

Class Sequence nameFrame count

Frame rate, Hz

BL resolution

EL resolution

IntraRandom access

Low-delay P or B

A Traffic 150 301280x800

2560x1600

2560x1600

2560x1600

Spatial 2x Spatial 2x

SNR

Spatial 2x

SNR

A PeopleOnStreet 150 301280x800

2560x1600

2560x1600

2560x1600

Spatial 2x Spatial 2x

SNR

Spatial 2x

SNR

B Kimono 240 24

960x540

1280x720

1920x1080

1920x1080

1920x1080

1920x1080

Spatial 2x

Spatial 1.5x

SNR

Spatial 2x

Spatial 1.5x

SNR

Spatial 2x

Spatial 1.5x

SNR

B ParkScene 240 24

960x540

1280x720

1920x1080

1920x1080

1920x1080

1920x1080

Spatial 2x

Spatial 1.5x

SNR

Spatial 2x

Spatial 1.5x

SNR

Spatial 2x

Spatial 1.5x

SNR

B Cactus 500 50

960x540

1280x720

1920x1080

1920x1080

1920x1080

1920x1080

Spatial 2x

Spatial 1.5x

SNR

Spatial 2x

Spatial 1.5x

SNR

Spatial 2x

Spatial 1.5x

SNR

B BasketballDrive 500 50

960x540

1280x720

1920x1080

1920x1080

1920x1080

1920x1080

Spatial 2x

Spatial 1.5x

SNR

Spatial 2x

Spatial 1.5x

SNR

Spatial 2x

Spatial 1.5x

SNR

B BQTerrace 600 60

960x540

1280x720

1920x1080

1920x1080

1920x1080

1920x1080

Spatial 2x

Spatial 1.5x

SNR

Spatial 2x

Spatial 1.5x

SNR

Spatial 2x

Spatial 1.5x

SNR


Experimental Results under CTC

• Example: JCT-VC J0034 for Scalable HEVC by InterDigital• Excel Spread Sheets under CTC (note: provides summary table and plots automatically)

• Decoding Time Results


Example of Meeting Notes

• JCTVC-W1000-v2

• Title: Meeting report of the 23rd meeting of the Joint Collaborative Team on Video

Coding (JCT-VC), San Diego, US, 19–26 Feb. 2016

• Author: Gary Sullivan (Microsoft / VCEG side) and Jens-Rainer Ohm (RWTH Aachen

University / MPEG side)

Proposal

Decision by chairs


HEVC Links (arranged by Fraunhofer HHI)

• HEVC webpage by HHI Fraunhofer (http://hevc.info/) – to be updated after meetings

• JCT-VC documents: http://phenix.int-evry.fr/jct/

• ITU-T Recommendation H.265: High Efficiency Video Coding: http://www.itu.int/rec/T-

REC-H.265

• Common Test Conditions and Software Reference Configurations for HM (HEVC

Reference Tool): http://phenix.it-

sudparis.eu/jct/doc_end_user/current_document.php?id=7281

• HEVC Reference SW (HM): https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware

• HM software manual:

https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/trunk/doc/software-manual.pdf

• HEVC Software Development Guidelines (JCTVC-H1001): http://phenix.int-

evry.fr/jct/doc_end_user/documents/8_San%20Jose/wg11/JCTVC-H1001-v1.zip

Extra:

• JCT-VC Bitstream directory: ftp://ftp.kw.bbc.co.uk/hevc/

• Test Sequences (ID/password required): ftp://ftp.tnt.uni-hannover.de/testsequences

• (Extra) RTP Payload Format for HEVC by IETF: https://tools.ietf.org/html/rfc7798

• Github OpenHEVC Decoder SW: https://github.com/OpenHEVC/openHEVC

http://hevc.info/


http://www.itu.int/rec/T-REC-H.265

http://phenix.it-sudparis.eu/jct/doc_end_user/current_document.php?id=7281

https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware

https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/trunk/doc/software-manual.pdf

http://phenix.int-evry.fr/jct/doc_end_user/documents/8_San Jose/wg11/JCTVC-H1001-v1.zip

ftp://ftp.kw.bbc.co.uk/hevc/

ftp://ftp.tnt.uni-hannover.de/testsequences

https://tools.ietf.org/html/rfc7798

https://github.com/OpenHEVC/openHEVC


SHVC Links

• Scalable HEVC (= HEVC Scalable Extension) – finalized in Oct. 2014.

• Same Repositories with HEVC.

• Login information is required for ftp sites.

• Common SHM test conditions and software reference configurations (JCTVC-

Q1009): http://phenix.it-

sudparis.eu/jct/doc_end_user/current_document.php?id=9106

• SHM (SHVC reference SW) repository:

https://hevc.hhi.fraunhofer.de/svn/svn_SHVCSoftware/

• Test sequences: ftp://[email protected]/testsequences/ and

ftp://[email protected]/scalable/sequences/

• AVC base layer bitstreams: ftp://[email protected]

hannover.de/scalable/bitstreams/AVC_CTC

• Latest SHM SW version 12.0

• Simple usage:

TAppEncoder -c cfg/encoder_randomaccess_scalable.cfg -c cfg/per-sequence-svc/BasketballDrive-2x.cfg -c cfg/layers.cfg -q0 22 -q1 22 -b str/BasketballDrive.bin -o0 rec/BasketballDrive_l0_rec.yuv -o1 rec/BasketballDrive_l1_rec.yuv TAppDecoder -b str/BasketballDrive.bin -ls 2 -o0 rec/BasketballDrive_l0_drec.yuv -o1 rec/BasketballDrive_l1_drec.yuv For AVC base layer tests the following should be used: cfg/layers_avcbase.cfg configuration file -ibl <BLrecon.yuv> option to specify the reconstructed base layer input

http://phenix.it-sudparis.eu/jct/doc_end_user/current_document.php?id=9106

https://hevc.hhi.fraunhofer.de/svn/svn_SHVCSoftware/

ftp://[email protected]/testsequences/

ftp://[email protected]/scalable/sequences/

ftp://[email protected]/scalable/bitstreams/AVC_CTC


The History of Video Coding Standards

201420121988 1990 1992 1994 1996 1998 2000 2002 2004 2006 2008 2010 2016

H.261

MPEG-1

H.263

MPEG-4Part 2

MPEG-2/H.262

JOINT

H.264/MPEG-4 Part 10 (AVC)

HEVCH.264/MPEG-4 AVC

SVC & MVC Ext.

HEVC SVC & MVC Ext.

MPEG-4 Version 2

H.263 + H.263 ++VideoConference

PMP

UHD,Mobile FHD

VC-2VC-1

(from WMV) (from Dirac)

AdobeFlash Video

(8K)

*김찬열, HEVC Overview, 삼성전자 DMC 연구소, Dec. 2014.


High Efficiency Video Coding (HEVC)

• ISO/IEC and ITU-T: The Joint Collaborative Team on Video Coding (JCT-VC)

• JCT-VC에서 2013년에 새롭게 제정한 국제 비디오 표준

• 기존 H.264/AVC 대비 약 50% 압축 효율 향상

• 4K/8K Ultra High Definition (UHD) 비디오 지원

• 새로운 coding tree 구조 및 병렬 tools

Divide intocoding units

(CU) withquad tree-based

partitioning

Divide intotransform units

(TU) withquad tree-based

partitioning

Divide intoprediction units

(PU) withquadrant and

asymmetric motion partitioningIn transform phase In prediction phase


Typical HEVC Video Encoder

Decoded Picture Buffer

Inter-Picture Prediction

Intra-Picture Prediction

DeblockingFilter

Sample Adaptive

Offset(SAO)

Inverse Transform

De-quantization

Transform Quantization Entropy Coding+

+

CodedBitstream

Input Video Signal (Splited into CTUs)


Block Partitioning for Prediction and Transform Coding

CTU

TU

CU

PU

HEVC가지원하는 4종류의블록

CTU: 슬라이스를 구성하는 한 개 이상의 블록

CU(코딩의 기본 단위): CTU를 구성하는 한 개이상의 블록

PU(예측의 기본 단위): CU를 분할하여 생성한한 개 이상의 블록

TU(변환의 기본 단위): CU를 분할하여 생성한한 개 이상의 블록

• HEVC는 Block-based hybrid video coding 구조로 설계됨

• 기존 H.264/AVC는 16x16이라는 고정적인 크기의 Macroblock을 기반으로Encoding 및 Decoding

• HEVC는 64x64, 32x32, 16x16의 가변적인 크기를 갖는 CTU를 사용

• CU = Luma CB(Coding Block), Cb CB, Cr CB로 구성


HEVC Parallelization Tools

• Existing Parallelization Techniques• Picture-Level Parallelization

• Slice-Level Parallelization

• Block-Level Parallelization

• New Parallelization Tools in HEVC• Tiles

• Wavefront Parallel Processing (WPP)

Video Sequence

Pic1 Pic2 Pic4Pic3

. . .

CPUCore

1Core

3

Picture-Level Parallelization의예시

Core2

Core4

Block-level Parallelization의예시(Macroblock-level Pipeline)

Core1 Core3Core2

dedicated forprediction

dedicated forin-loop filter

dedicated forentropy coding


Tile & WPP Parallelization

• Tile Parallelization

• Picture를 가변적인 크기의 사각형으로 분할

• Tile간에 의존성을 갖지 않으며, 헤더를추가로 사용하지 않음

• Wavefront Parallel Processing (WPP)

Parallelization

• CTU Row 단위의 병렬 처리 기법

• CTU간의 의존성을 고려한 병렬처리 기법

Thread1

Tile Parallelization

Thread2 Thread3 Thread4


Intra-Picture Prediction

Mode 0: Intra_Planar

Mode 1: Intra_DC+

휘도성분의 35가지예측모드 Directional prediction의예시

Current PU

Boundary samplesfrom decoded PUs

• Picture를 구성하는 각 픽셀들 간의 Spatial redundancy를 제거

• HEVC는 35가지의 Intra-picture prediction 모드를 지원

• 임의 접근 및 에러 강인성을 위해 영상 중간 중간에 삽입


Inter-Picture Prediction

Pic1 Pic2 Pic4Pic3

. . .

Pic1 Pic2 Pic4Pic3

. . .

Uni-prediction의예시 Bi-prediction의예시

• Video sequence를 구성하는 Picture들 간의 Temporal redundancy를 제거

• Reference Picture List에서 현재 블록과 유사한 예측 블록을 선택하여 현재PU와의 Residual을 부호화

• P-Slice는 단방향 예측(Uni-prediction) 수행, B-Slice는 양방향 예측(Bi-prediction) 수행


Transform and Quantization

• HEVC Transform

• 비디오 압축에서의 변환 부호화는 공간 영역(Spatial domain)의 신호를주파수 영역(Frequency domain)의 신호로 변경하여 표현하는 것

• HEVC는 4x4, 8x8, 16x16, 32x32 크기의 TU에 DCT(Discrete Cosine

Transform) 적용

• 4x4 크기의 TU는 예외적으로 DST(Discrete Sine Transform) 적용 가능

• HEVC Quantization•Quantization은 비디오 압축 기술에서 유일하게 정보의 손실이 발생되는단계

•Transform의 결과로 구해진 변환 계수를 몇 개의 대표 값으로 근사화 하는과정


In-loop Filters

• Encoding/Decoding loop에 포함된 화질 개선용 필터

• HEVC에는 Deblocking filter와 SAO(Sample Adaptive Offset)라는 2가지필터가 적용

• Deblocking filter는 블록 간의 불연속성(Block artifact)의 제거를 목표로 함

• SAO는 Ringing artifact의 제거를 목표로 함

Loop Filter 적용예


Scalable Video Coding (SVC)

• Supports various device capabilities with single

bitstream

• Reduces total BW

SVC < AVC (around 20%, for 3 Spatial layers)

• No transcoding, No duplicated contents for

every target bitrates

Temporal

Decomposition

Transform/

Entropy coding

Motion

coding

2D spatial

upsampling

Intra prediction

for intra block

Temporal

Decomposition

Transform/

Entropy coding

Motion

coding

Intra prediction

for intra block

2D spatial

Down-

sampling

QP1

QP2 Mu

ltiplex

motion

info.

Video

input

Enhancement Layer Encoder

Base Layer Encoder

SVC

Bitstream

Base Layer

Enhancement Layer n

Enhancement Layer n’

HDTV

Laptop

Mobile

High quality,

Resolution, and

temporal video

Medium quality,

Resolution, and

temporal video

Low quality,

Resolution, and

temporal video

Enhancement Layer Encoder

2D spatial

Down-

sampling


Scalable HEVC (SHVC)

• Higher-layer decoding picture buffer (DPB) uses lower-layer’s

reconstructed picture

• Simple / easy to deploy

Base layer

Enhancement layer

Inter-layer referencepicture

Typical SHVC Encoder Inter-layer Prediction

*US20140010294 A1 (InterDigital Patent Application /2014)


2. 이론:

360 Video Processing for Virtual Reality (VR)


Emerging 360-Degree Video

• Digital Camera 기반의 360 Degrees Video Systems는 최근 크게 주목

• 최근 360 Degree Video Viewer로서 다양한 HMD들이 출시

다양한 HMD의등장(Gear VR, Oculus Rift, Cardboard VR Kit, PlayStation VR)

360 Degree Video의예시


HMD Revenues by Device Type

• World Market, Forecast: 2014 to 2020 (Source: ABI Research)

Device Type Revenues 2014 2015 2016 2017 2018 2019 2020

VR Standalone (US$ Thousands) 935 3,980 19,504 94,592 269,586 501,431 902,575

VR Mobile (US$ Thousands) 55,250 164,093 321,621 561,551 853,557 1,190,712 1,607,462

VR Tether (US$ Thousands) 55,250 111,605 876,546 1,938,042 3,038,851 4,691,985 6,906,602

Enterprise Smart

Glasses

(US$ Thousands) 576,790 1,051,088 2,046,876 3,849,328 6,715,196 9,522,589 11,693,589

Consumer Smart

Glasses

(US$ Thousands) 148,317 27,280 526,339 989,827 1,726,765 2,448,666 3,006,923

Total (US$ Thousands) 836,543 1,601,046 3,790,886 7,433,340 12,603,955 18,355,383 24,117,151


Mobile VR? (source: ABI Research + MCSL조사 2016년 기준)

DEVICE HMD TypeDisplay Type

DisplayResolution

RefreshRate

Field of View

AudioPosition Tracking

Head Tracking InputRelease Date

Price

Oculus Rift CV1

VR tethered

- 2160X1200 90 Hz -Built-in

Headphones

IR Camera 360°

GyroscopeAccelerometer Magnetometer

OculusTouch

Standard controller

Q1 2016US$400(est.)

HTC ViveVR

tethered- 2160X1200 90 Hz 110°

Headphone jack

Lighthouse basestation

360°


Lighthouse controllers Standard controller

Nov 2015

US$500(est.)

Sony’s Project

Morpheus

VR tethered

OLED 1920X1080 120 Hz 100°Headphone

jackIR Camera

360°Gyroscope

Accelerometer

Standard controllerPS Move

1Q 2016US$400(est.)

FOVEVR

tetheredWQHD 2560X1440 90 Hz 100°

Headphone jack

Lighthouse basestation

360°


Eye tracking Standard controller Lighthouse controllers

May 2016

US$399

OSVRVR

tetheredQLED 1920X1080 60 Hz 100° None

IR Camera 210°

GyroscopeAccelerometer

Compass

No standard input

Available now

US$200to

US$300

Google Cardboard

VR mobileDependenton phone

Dependenton phone

Dependenton phone

Dependenton phone

Headphone jack

NoneDependent on

phoneSingle button

Available now

US$5 to US$20

LG 360 VR VR mobile IPS LCD920X720per Eye

- 80°Headphone

jackNone

Gyroscope Accelerometer

2 Buttons(Back, Select)

Available now

US$199(plus

phone)

Samsung Gear VR

VR mobileSuper

AMOLED2560X1440 60 Hz 96°

Headphone jack

NoneGyroscope

Accelerometer Magnetometer

Standard controller Touchpad

Available now

US$199 (plus

phone)

Mattel View-Master

VR mobileDependent on phone

Dependenton phone

Dependenton phone

Dependenton phone

Headphone jack

NoneDependent on

phoneSingle button

Available now

US$30

ZEISS VR ONE

VR mobileDependent on phone

Dependenton phone

Dependenton phone

Dependenton phone

Headphone jack

NoneDependent on

phoneNone

Available now

US$99

StarVRVR

tetheredLCD 5120X1440 - 210°

Headphone jack

IR camera Fiducial markers

360°


No standard input


1도당 들어가는 픽셀 수

4K 수직 해상도의 4배

90fps는 저지연을 통해 메스꺼움을 방지할 수 있음

요구사항 부가설명

360도 서라운드 사운드와 객체지향 오디오 및*고차원 앰비소닉스 지원 요구

사용자의 상호작용과 VR 이미지와 오디오 간의 시간차

픽셀/도

해상도

프레임률

3D 오디오

영상과 음성의지연속도

거리감물체와의 충분한 거리가 있어야사용자가 메스꺼움을 느끼지 않음

시차 사용자 시점에 따른 물체의 가시성 확인 필요

40픽셀/도

11520x6480

90fps

환경기반 또는장면기반오디오

최대 20ms

물체와의충분한 거리

시차와의상호작용

고품질 VR을 위한 요구사항Technicolor, Oct. 2016 (m39532, MPEG 116th Meeting)


1단계 2단계 3단계

Yaw

Roll

Pitch

Yaw

Roll

Pitch

Right

Left

Forward

Backward

Up

Down

Up

YawForward

Roll

Right

PitchDown

Backward

Left

• 2017년까지 3DoF 표준완성

• 앉은상태에서머리를회전하며영상확인

• 기본적으로 360도스트리밍전송

• 가능할경우타일링스트리밍전송

• 2020년까지VR 상업서비스활성화

• 앉은상태에서영상에서움직이면서

머리를회전하며영상확인

• 유저간대화및프로젝션최적화

• 2022년까지 6DoF 지원

• 유저가걷는움직임이영상에반영되고

머리를회전하며영상확인

• 가상환경과상호작용지원

MPEG-I ( Immersive media) 단계별 목표Jan. 2017 (N16718, MPEG 117th Meeting)

• MPEG-I는 MPEG에서 VR관련 표준화를 담당하며 3단계의 목표를 명시

• MPEG-I의 의장은 퀄컴, Technicolor, 삼성으로 구성

• 2020년까지 VR 상업서비스 활성화 목표

• 2017년까지 3DoF 표준 완성 후 2022년까지 6DoF 지원 목표


전통적인방식의DASH 스트리밍

사용자시각에맞는 DASH

스트리밍

VR기기의사양에맞는

적응형스트리밍

웹브라우저기반VR스트리밍

VR컨텐츠를위한다각도스트리밍

Use Cases In

DASH-VR

• MPEG DASH-VR은 VR과 관련된 DASH 표준화를 담당

• DASH-VR과 관련하여 현재까지 엘지, 노키아, 화웨이, Mediatek등 다양한기업에서 34개의 기고문 제안

• DASH-VR은 호환성 및 효율적인 스트리밍을 고려하여 5개의 use case를 구성

MPEG DASH-VR Use caseJan. 2017 (m40057, MPEG 117th Meeting)

1080p

720p

360p

MPEG-DASH 기술


360 Video Processing (1/2)

Image stitching,Equirectangular

mapping

Video rendering on a sphere

Video decoding

Video encoding

Stitched video is coded as regular 2D video by using H.264/AVC and H.265/HEVC

Input from seven cameras

Corresponding stitched image

Stitched image texture mapped onto a sphere


360 Video Processing (2/2)


Region-wise Packing / Unpacking

• Rectangular region-based (e.g., ERP, CMP) 영역 기반 region-wise packing

기법 구현

• Packing시, 상대적으로 덜 중요한 region을 Downscaling하여 Encoding

efficiency 증가

• Decoding 된 프레임을 unpacking하여 Rendering가능한 projection으로 투영

• Unpacking시, packing과정에서 Downscaling했던 region을 Upscaling하여 복구


Tile-Based Encoding

• 병렬처리를 지원하는 Tile기법을 적용하여 encoding

• Viewport기반 타일 선택을 위한 MCTS(motion-constrained tile set)를 적용

• Stream시 원하는 Tile만 선택하여 대역폭 효율 증가


Tile-Based Decoding

• Viewport기반 부분 타일 decoding

• 전체가 아닌 viewport에 해당하는 Tile만 decoding하여 storage 효율 증가

• 화질 별 인코딩을 통하여 Viewport에 해당하는 Tile은 고화질, 아닌 Tile은저화질로 디코딩


x

SHM encoder ver. 12.3

(based on HM 16.10)HM encoder ver. 16.16

(Latest version of HM, Aug. 2017)

PicEL tPicEL t-1

Upsampled PicBL t

When referring to the

same tile, TIP is

performed according to

the existing method


other tiles, ILP is

performed using

Upsampled BL

• EL: Enhancement Layer

• BL: Base Layer

• TIP: Temporal Inter Prediction

• ILP: Inter Layer Prediction

x

PicEL tPicEL t-1


same tile, TIP is

performed according to

the existing method


other tiles, Intra

Prediction is performed

Motion-Constrained Tile Set (MCTS) by Gachon MCSL


Modified The Reference Range by Considering Interpolation

Original TileModified Reference RangeSubtract 3 pixels from top

Subtract 3 pixels from left

Subtract 4 pixels from bottom

Subtract 4 pixels from right

An error occurs when

referring to the orange

region due to luma

interpolation

Tile Boundary Consideration of Current MCTS

*Considering Luma component’s 8 Tab filter


MCTS Performance Comparison

Test Sequence(8192x4096)

Original bitrate(9 Tiles)

Proposed bitrate(4 Tiles)

Proposed bitrate(1 Tiles)

KiteFlite 60 mbps 28 mbps(-53%) 7 mbps(-88%)

Harbor 22 mbps 10 mbps(-53%) 2 mbps(-88%)

Trolley 32 mbps 16 mbps(-50%) 4 mbps(-87%)

GasLamp 10 mbps 5 mbps(-47%) 1 mbps(-86%)

• SHM 10.0 Reference Software / EL QP 22 / BL QP 22 / 300 frame

• Proposed Method : MCTS implemented by Gachon MCSL


…

• 병렬처리와 NAL 단위 추출을 위해 타일과 슬라이스를 병합하여 인코딩

• 독립적 타일 전송이 가능하도록 Interpolation filter를 고려하여 인코더에서 모션벡터 범위수정

• 슬라이스 단위의 NAL을 분석하여 원하는 타일 추출 (SEI Message도움 필요// HHI, LG)

• Viewport 기반 ( 원하는 타일) 디코딩

Tile Extractor for MCTS

Tile & slice 1: 1 merge encoding

For parallel processing and extraction

𝑡0 𝑡1 𝑡2

Motion constrained tile sets

HM Encoder

Tile/

Slice

Tile/

Slice

Tile/

Slice

Tile/

Slice

Tile/

Slice

Tile/

Slice

Tile/

Slice

Tile/

Slice

Tile/

Slice

Modify motion

vector range

For independent

tile transmission

Tile Extractor

slice header Tile1

…

slice header Tile2

slice header Tile3

slice header Tile9

Analyze

slice NAL

unit and

extract

desired tile

HM Decoder

Viewpoint-based

decoding


Tile Partitioning-based Parallel Video Processing (by MCSL)

▪ Motivation: 작업량이 동일한 HEVC Tile영상 처리를 비대칭 모바일 CPU 코어들에 할당 시 효율저하

▪ Proposed Method: 각 코어의 처리능력에 비례한 크기로 HEVC 타일을 분할하여 디코딩 효율 최적화

▪ Achieved decoding time gain: up to 17.4% on Testboard, 29.65% Exynos 8890 Phone (구현/실측)

Conventional tile partitioning method Proposed tile partitioning method

Big1

Big2

Little1

Little2

Little3

Little4

Asymmetric multicore system

Resolution A Resolution B

Complexity B'(=decoding time)

Big/Little cores

Performance Table

Complexity A'(=decoding time)

Big

core

Little

core

(1) Get A'/ B' ratio by using the Big/Little performance ratio (Little/Big)

(2) Get A/B ratio by using the complexity model(e.g. regression model)

(3) Segment a picture with A/B ratio

Segment B

Segment A

(4) Allocate segmented images to Big/Little cores


Tile Prediction Complexity-based Parallel Video Processing (by MCSL)

▪ Motivation: 미리 고정된 크기로 인코딩 된 HEVC Tile 영상의 경우 앞의 방법을 쓸 수 없음.

▪ Proposed Method: 대칭 분할된 HEVC Tile의 복잡도를 PU Partitioning정도로 예측하여 비대칭코어 할당

▪ Achieved decoding speed-up: up to 12.3% on Testboard (구현/실측)

영역별컨텐츠복잡도에따라다르게분할된 PU

Y e l l o w C i r c l e :단 순 한 부 분Green Circle: 복잡한부 분

PU와 Decoding Time(Complexity)의 Scatter Plot


Video Decoding Speed-up for Mobile VR using Tiled-SHVC

and Asymmetric Multicores (by MCSL)

▪ Motivation: Mobile VR의 장점에도 불구하고 Tethered VR 대비 낮은 성능으로 고품질 컨텐츠 제공 어려움

▪ Proposed Method: MCSL연구실의 SHVC기반 Tile처리 최적화 기술과 mmWave기반의 PC offload를통해 성능 극대화 (Base Layer: 360VR, Enhancement Layer: Tiled VR with MCSL-MVP + PC offload)


3. 실습:

HM Reference SW and

Common Test Condition (CTC)


HM (HEVC Reference SW)

• Open SW Tool: Tortoise SVN (download from Sourceforge site)

• Select ‘Choose Items’ -> Tags -> HM-XX (version; example: HM-15.0)

• SVN: https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/

https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/


Open Visual Studio Solution File

• Build -> select visual studio solution file (e.g. HM_vc10.sln)

• Decoder: TAppDecoder -> TAppDecTop.cpp -> TAppDecTop::decode()

• Build: Release mode (x64)


HM Test Sequences

• ftp://hevc:[email protected]/testsequences

• ID: hevc, Passwd: DployV1

• Resolution: UHD / FHD(1080p) / HD(720p), etc.

• Frame rate: 24/30/60

• Format: YUV (10 Seconds’ raw video sequence)

• Explain YUV format: (in class)• Raw file structure (e.g. 4:2:0)

Y

U

V

ftp://hevc:[email protected]/testsequences


JCT-VC and MPEG Web

• JCT-VC (Joint collaborative team on video

coding): http://phenix.int-evry.fr/jct/

• Freely access (recommend)

• MPEG: ISO/IEC SC29, WG11:

http://wg11.sc29.org/

• Needs login information (password) to download

standard documents such as proposal and

information.


http://wg11.sc29.org/


JCT-VC Common Test Conditions (CTC)

• For experiments in a well-defined environment and the comparison of the

outcome of experiments.

• Quantization parameter values: 22, 27, 32 and 37

• Coding structure: Random Access (RA), All Intra (AI), Low-delay B/P (LDB/P)

Name Resolution Frame length Frame rate

PeopleOnStreet 3840×2160 150 30 fps

Traffic 3840×2048 300 30 fps

BasketballDrive 1920×1080 500 50 fps

BQTerrace 1920×1080 600 60 fps

Cactus 1920×1080 500 50 fps

Kimono 1920×1080 240 24 fps

ParkScene 1920×1080 240 24 fps

Official Test Sequences

PeopleOnStreet

Traffic


• Common Coding Options (1)

Encoding with HM 15.0 (1)

#: 특정 라인 주석처리

Coding Tree Unit (CTU) size: 64x64 – 16x16




Fast Motion Search Mode On/Off

Quantization Parameter (QP) Value (편의상 커맨드라인 Parameter로입력하기 위하여 주석처리 하였음)

❖ CTC QP Values: 22, 27, 32 and 37




Deblocking Filter On/Off

SAO : Sample Adaptive Offset Filter On/OffAMP: PU Asymmetric Motion Partitions On/Off




Slice 분할 모드0: 슬라이스 분할 X1: CTU 개수를 기준으로 슬라이스 분할2: Bytes 수를 기준으로 슬라이스 분할3: Tiles 개수를 기준으로 슬라이스 분할

SliceMode 값에 따른 슬라이스의 크기(Slices per Picture 개수 정의)




Tile 분할 모드0: Tile Non-Uniform Partitioning1: Tile Uniform Partitioning

Picture 내의 Tile Columns 개수 - 1

각 Tile의 Width 정의를 위한 CTU 개수(정의되지 않은 가장 오른쪽 Tile의 Width는 나머지 CTU 개수로 자동정의 됨)

각 Tile Boundary를 가로질러 In-loop Filter 적용 On/Off

15 20 Rest

10

Rest

Tile Partitioning Options가위와같을때, Tile Partitioning 예시

Tile 1 Tile 2 Tile 3

Tile 4 Tile 5 Tile 6


• RA Coding Options (encoder_randomaccess_main.cfg)


IntraPeriod: I-Picture 삽입 주기GOPSize : Group of picture의 크기

… …

GOP (8 pictures)

I-Picture

Period of I-Picture (32 pictures)

IntraPeriod= 32, GOPSize= 8일경우의 Coding Structure 예시

I-Picture


• AI Coding Options (encoder_intra_main.cfg)


IntraPeriod: I-Picture 삽입 주기GOPSize : Group of picture의 크기

…

GOP, Period of I-Picture (1 picture)

I-Picture

IntraPeriod= 1, GOPSize= 1일경우의 Coding Structure 예시


• Low-delay B Coding Options (encoder_lowdelay_main.cfg)


IntraPeriod: I-Picture 삽입 주기 (-1 = Only First)GOPSize : Group of picture의 크기

Period of I-Picture (Only first picture)

I-Picture

IntraPeriod= -1, GOPSize= 4일경우의 Coding Structure 예시

… …

GOP(4 Pictures)


• Test Sequence Options (BQTerrace.cfg)


Input YUV Raw Video 파일 경로

Input YUV Raw Video 프레임율

Input YUV Raw Video 해상도


• Example of Encoding with HM 15.0 (for BQSquare Test Sequence)


TAppEncoder.exe-c .\cfg\encoder_randomaccess_main.cfg-c .\cfg\BQSquare.cfg-q 37 -b .\bitstreams\BQSquare_416x240_60_RA_qp37.bin > .\logs\BQSquare_416x240_60_RA_qp37_encoded.txt

Input Options-c (cfg옵션) 사용하는 cfg 파일명과 위치-c (cfg옵션) 사용하는 cfg 파일명과 위치-q (QP옵션) QP값 직접입력-b (bitstream옵션) 저장될 비트스트림 파일명과 위치> (로그출력) 로그내용 저장할 로그 파일명과 위치


• Example of Encoder Logs



• Example of Decoding with HM 15.0 (for BQSquare Test Sequence)

Decoding with HM 15.0 (1)

TAppDecoder.exe -b .\bitstreams\BQSquare_416x240_60_RA_qp37.bin -o rec_BQSquare_416x240_60_RA_qp37.yuv > .\logs\BQSquare_416x240_60_RA_qp37_decoded.txt

Input Options-b (bitstream옵션) 사용할 비트스트림 파일명과 위치-o (비디오파일 생성옵션) 출력될 yuv파일명과 위치> (로그출력) 로그내용 저장할 로그 파일명과 위치


• Example of Decoder Logs



• 인코딩 전후 화질 비교


Differences from Lossy Compression

Original Picture Reconstructed Picture


• Excel Spread Sheet Example: HM15.0 v.s. HM 14.0

HM 15.0 Software Technical Evaluation (1)


• Excel Spread Sheet Example: HM15.0 vs. HM 14.0

HM 15.0 Software Technical Evaluation (2)


HEVC NAL Unit Analyzer (Free SW)

• Analyze HEVC(h265) bitstreams

• Display syntax elements of hevc elementary streams

• 다운로드 : https://github.com/virinext/hevcesbrowser/releases

Syntax Element in NAL Unit

Bit Value of Bitstream

Offset, Length, Type, Info of NAL Unit


4. 실습:

JVET 360Lib Tools for 360-degree Video


• For the input ERP video(e.g. 8K), encoding/decoding is conducted by using

one of 11 projection formats

• The output video has low fidelity(e.g. 4K), otherwise the conversion would

require lots of time

360 Video Testing ProcedureJuly 2017, Torino, Italy, JVET-G1030, 7th JVET meetings

Jill Boyce, Elena Alshina, Adeel Abbas, Yan Ye


Projection Formats Supported in 360Lib

Index Projection format

0 Equirectangular (ERP)

1 Cubemap (CMP)

2 Adjusted Equal-Area (AEP)

3 Octahedron (OHP)

4 Icosahedron (ISP)

5 Truncated Square Pyramid (TSP)

6 Segmented Sphere Projection (SSP)

7 Adjusted Cubemap Projection (ACP)

8 Rotated Sphere Projection (RSP)

9 Equatorial Cylindrical Projection (ECP)

10 Equi-Angular Cubemap (EAC)

CMP 4x3 and 3x2

ERP

OHP


360Lib Installation

• SW package (SVN)

- Check out HM-16.15 https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-

16.15

- Check out 360Lib-3.0

https://jvet.hhi.fraunhofer.de/svn/svn_360Lib/tags/360Lib-3.0

• After install, copy several files from 360Lib to HM


360Lib SW Features (1/4)

• 360Lib Introduction• Based on Projection format conversion tool for 360 video provided

by InterDigital.

• Tools are added as a separated library to the HM/JEM SW.

• Separated application is also provided.

• Bit-depth: 8 bit and more.

• Interpolation filters are also supported.• Luma components: Lanczos3 (default)• Chroma components: Lanczos2 (default)

• Main features with examples• Projection formats: ERP -> CMP

• InputGeormetryType: 0• CodingGeometryType: 1

• Sphere Roation• Yaw, Pitch, Roll: counter-colockwise rotation in degree along Y, -Z, X axis, respectively


360Lib SW Features (2/4) - example

• 3, 4: face row와 face column의미. 즉, CMP 4x3

• Red numbers: face Index

• Number after face index: rotation degrees (counter clockwise) before putting it into the output image

• 따라서, 2 90 6 0 7 0 8 0은 face index 2를 90도 반시계방향으로 돌리고, 6, 7, 8은 virtual face index (empty/null).

1 2 3 4

3


360Lib SW Features (3/4) - example

• Viewport Generation Parameter

• ViewPortSettings: 75.0 75.0 270.0 0.0• Horizontal, vertical field of view (FOV): both 75.0 degrees

• Center of the viewport is set at 270.0 degrees in longitude and 0.0 degree in altitude.

• ViewPortFile: viewport.txt• 프레임 별 viewport settings

• 0~3: 75 75 15 5 // 4~7: 75 75 45 5 …

• DynamicViewPortFile: dynamic_viewport.txtex) 75.0 75.0 0 -45 -15 299 45 15

• FOV: Hor/ver 각각 75.0도, 0과 299: 프레임 번호(POC)의 시작과 끝, 다른 두 숫자는the center of viewport의 yaw, pitch.

• 따라서, POC 0부터 POC 299까지 서서히 변하는 viewport 값을 conversion SW가viewport position을 automatically 계산하여 각 프레임간에 linear interpolation.

• Viewport의 width/height: CodingFaceWidth/CodingFaceHeight 값으로 세팅

• SphereVideo: 값 0: 아무런 projection conversion 없이 비디오 인코딩, 값 1:

projection conversion 수행

0 75 75 15 54 75 75 45 58 75 75 60 5

Frame number


360Lib SW Features (4/4) – encoder_360_CMP.cfg


HM-16.16-360Lib-4.0 Software Preparation

1. check out HM-16.16 (https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware/tags/HM-16.16);

2. check out 360Lib-4.0(https://jvet.hhi.fraunhofer.de/svn/svn_360Lib/tags/360Lib-4.0);

3. copy following directories (./360Lib-4.0) to ./HM-16.16;• copy source files:• copy ./360Lib-4.0/source/Lib/TLib360 to ./HM-16.16/source/Lib/• copy ./360Lib-4.0/source/Lib/TAppEncHelper360 to ./HM-16.16/source/Lib/• copy ./360Lib-4.0/source/App/utils/TApp360Convert to ./HM-

16.16/source/App/utils/

• copy configure files:• copy ./360Lib-4.0/cfg-360Lib to ./HM-16.16/

• copy build files:• copy ./360Lib-4.0/HM-360Lib-4.0-build to ./HM-16.16/

4. build HM-16.l6-360Lib-4.0 software: • use ./HM-360Lib-4.0-build/linux/makefile for Linux platform; the

usage of make command is the same as that for HM-16.16; • use HM_vc2010.sln in ./HM-360Lib-4.0-build/HM_vc2010.sln for

Windows platform;


360Lib Execution (e.g. ERP to CMP)

• Command to converge the ERP video to CMP video and encode

TAppEncoder.exe -c ./encoder_randomaccess_main.cfg

-c ./encoder_360_CMP.cfg -c ./360test_Trolley.cfg

-c ./360test_Trolley_DynamicViewports.cfg --SphFile=./sphere_655362.txt

-i ./Trolley_8192x4096_30fps_8bit_420_erp.yuv

--CodingFaceWidth=1184 --CodingFaceHeight=1184 --IntraPeriod=32

-b test_cmp.bin --SEIDecodedPictureHash=1 -o rec_cmp.yuv -q 27


360Lib Encoding Result (Trolley)

• Input file : 8K(ERP), Output file : 4K(ERP), QP : 27


360Lib Encoding Result (Trolley)

• Input file : 8K(ERP), Output file : 4K(CMP), QP : 27


Thank You !

http://mcsl.gachon.ac.kr

Questions > [email protected]