100x Evolution of Video Codec Chips

Jinjia Zhou1, Dajiang Zhou2, Satoshi Goto2

1Hosei University, Tokyo, Japan2Waseda University, Kitakyushu, Japan

Tribute to Prof. Goto

Prof. Goto has been my supervisor from 2008 to 2015. (M.S -> Ph.D -> PDF)

2

Prof. S. Goto’s Video Coding Research Group

►One of the first Full-HD H.264 encoders, first to use SiS DRAM (VLSI’07 and JSSC’09)

►First 4kx2k@60fps H.264 decoder (VLSI’10)

►First 8kx4k H.264 decoder (ISSCC’12)

►First 8Kx4K H.264 (intra-frame) encoder (VLSI’12)

►First 8Kx4K H.264 ME encoder (VLSI’13 and JSSC’14)

►First 8Kx4K HEVC decoder (ISSCC’16 and JSSC’16)

3

4

Mass media

Video codec: encoder and decoder

Source video data

Encoder

Receiver

Compressed video stream

Decoder

Compression

Decompression

Video camera

Display device

Restored video data stream

Transmit.

Channel/Storage

100%

100%

~1%

~1%

~1%

5

Applications of video codec chips

Enc./Dec. Chip

TV conference Surveillance

Automotive

Mobile/Portable

Small frame delayUltra-low powerHigh compressionHigh video qualityFree-point view….

……

Source of the images: http://www.artesanosdecastillalamancha.org/wp-content/uploads/2015/06/28.pnghttp://www.caradvice.com.au/67890/2011-brakes-camera-action-pedestrian-detection-automated-platooning/photos/

Home entertainment

6

8K UHDTV and free-viewpoint TV

25~30fps

≥120fps

7680 pixels

7

Video coding standards

1990 1995 2000 2005 2010

MPEG-1 MPEG-4

H.261 H.263 (+/++)

H.262

MPEG-2

H.264

MPEG-4 AVC

H.265

HEVC

ITU-T standards

MPEG standards

Joint ITU-T &

MPEG standards

Compression ratio ~50:1 ~100:1 ~200:1

2013

8

High compression at high complexity

►Recent powerful codecs address the huge video throughput in the communication channel

►Their high compression ratio, however, is at the expense of high complexity

9

H.264 (2003)

~480Mbps ~240Mbps

RAW HEVC (2013)

48000Mbps

Complexity of video codecs (norm.)

1

20

307.2

0.1

1

10

100

1000

1080p/MPEG-2 4K/H.264 8K/HEVC

Complexity/pixel Throughput Overall complexity10

Real-time 8K UHDTV codec systems

NHK 8K codec

(2007)

NHK 8K encoder

(2013)

Our target:

Single chip/chipset11

Memory bandwidth issue

►Performance bottleneck >50GBps BW required for decoding 8K UHDTV

>100GBps BW required for encoding 8K UHDTV

►Power consumption Majority of power

consumed by DRAMtraffic

►Fabrication cost BW determines chip pin count

12

Codec

DRAM

Memorytraffic

Data dependency issue

►Video codecs exploit all kinds of data dependencies to strengthen compression Inter-frame prediction

Intra-frame prediction

Context-adaptive entropy coding (CABAC)

►Data dependencies restricts the degree of efficient parallelism/pipelining Power and area issues

Performance issue

13

Challenges summarized

14

Transform & Quant.

Inv. Trans. & Inv. Quant.

Deblocking Filter

Reference Frames

Frame Output

Motion Compensation

Motion Estimation

Intra Prediction

-

Entropy Coding

Source Frm.Decoder

Memory bandwidth

requirements

Computational

complexity

Data

dependencies

Our efforts to address the challenges

15

System

Algorithm

Architecture

Circuits

Device

Evaluation

Bus/interface optimization, 3DLSI

Processing order optimization

Embedded compression

2-D cache

Reduce complexity

Trade-off b/w time & quality

Hardware friendliness, …

Alleviate data dependencies

Processing order optimization

Predictive execution, …

Reduce memory access

/ Increase memory bandwidth

System integration

►FIFO vs RAM

FIFO: simple interface and flexibility

RAM: random accessibility for data reordering

►Proposed BIBO (Block-in-block-out) queues:

combines benefits of the two

<16>

►Words in a block can be written in a random order►A block can be pushed after all words are written►Blocks follows first-in-first-out►Blocks can be in a variable size

Word write & block push

BIBO queueWordBlock

<17>

Word read & block pull

►Words in a block can be read in a random order►A block can be pulled after all words are read►Blocks can be pulled in different sizes as pushed

BIBO queue

Merged

<18>

Merge and split

►Block merging/splitting can beautomated by BIBO given bothword addressing and blockscan follow a Z-scan order

BIB

O q

ueu

e

<19>

Merge and splitB

IBO

qu

eu

e

Push

<20>

Merge and splitB

IBO

qu

eu

e

Pull

<21>

Merge and splitB

IBO

qu

eu

e

<22>

Implemented video codec chips from Goto’s Lab.

Source: http://www.f.waseda.jp/goto/html/chip.html23

Video decoder demo: 4K@FPGA, 8K@chip

24

Performance of codec VLSI chips

25

27.6

1990

249

3981

0

1000

2000

3000

4000

MIT

ASSCC'08

Ours

ISSCC'12

Ours

VLSIC'12

NTU

VLSIC'13

MIT

ISSCC'13

NTT

VLSIC'15

Ours

ISSCC'16

Mpixel/s

H.264 decoder HEVC decoder H.264 encoder HEVC encoder

144x

Shen Li

Thanks to all members who have contributed in the video codec chip design

26

Thank you!

jinjia.zhou.35@hosei.ac.jp

27