system level modeling of a h.264 video encoder

Center for Embedded Computer SystemsUniversity of California, Irvine

System Level Modeling of a H.264 Video Encoder

Che-Wei Chang, Rainer Domer

Technical Report CECS-11-04June 2, 2011

Center for Embedded Computer SystemsUniversity of California, IrvineIrvine, CA 92697-2625, USA

(949) 824-8919

cheweic,[email protected]://www.cecs.uci.edu

cheweic, [email protected]

http://www.cecs.uci.edu


Che-Wei Chang, Rainer Domer

Technical Report CECS-11-04June 2, 2011

Center for Embedded Computer SystemsUniversity of California, IrvineIrvine, CA 92697-2625, USA

(949) 824-8919

cheweic,[email protected]://www.cecs.uci.edu

Abstract

In this report, we demonstrate the results of our system modeling project involving the encoder of the H.264advanced video coding(AVC) standard. The goal is to create aH.264/AVC video encoder model with SpecCSystem Level Design Language (SLDL). First, we briefly introduce the essential features of the H.264/AVC videoencoder algorithm and the properties of the JM reference C source codes. We then describe the modificationswe have performed to make the JM reference C implementation compliant for SpecC SLDL. Our model hasbeen separated into three major behavior blocks, which are Stimulus, Design, and Monitor, to reflect a propersystem design testbench. The design-under-test(DUT) component then has been structured into a hierarchy ofcommunicating behaviors. In this work, we have identified several opportunities for parallel execution in theH.264 encoding which we have explicitly specified in the system model. The report concludes with experimentalresults that validate the correct functionality of our H.264 encoder model by success full simulation.



Contents

1 Introduction 1

2 H.264/AVC Video Encoder Algorithm 22.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . 22.2 Input Video Data Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . 22.3 Coding Structure of H.264/AVC Video Encoder . . . . . . . . . . . . . . . . .. . . . . . . . . 2

3 Reference C implementation of H.264/AVC Video Encoder 43.1 Properties of H.264/AVC Video Encoder JM 13.0 Reference Software. . . . . . . . . . . . . . 43.2 SpecC compliant reference implementation . . . . . . . . . . . . . . . . . . . . . .. . . . . . 5

4 Modeling of H.264/AVC Video Encoder Platform 64.1 Major Behaviors : Stimulus, Design, Monitor . . . . . . . . . . . . . . . . . . . .. . . . . . . 74.2 Communication between top-level behaviors . . . . . . . . . . . . . . . . . . . .. . . . . . . . 11

5 Exploiting parallelism in H.264 Video Encoder 125.1 Luminance/Chrominance residual coding and reconstruction . . . . . . .. . . . . . . . . . . . 125.2 Motion vector searching for multiple reference frames . . . . . . . . . . . .. . . . . . . . . . 13

6 Experiments and Results 14

7 Conclusion and Future Work 15

References 16

ii

List of Figures

1 Basic encoding structure for H.264/AVC for a macroblock . . . . . . . . . .. . . . . . . . . . 32 Hierarchy of Video Stream . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . 44 Example of Coding Order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . 53 Code Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . 65 Top level of H.264 Encoder Model . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . 96 Design under Test Behavior . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . 107 Communication between Stimulus and Design . . . . . . . . . . . . . . . . . . . . . . . .. . . 178 Communication between Design and Monitor . . . . . . . . . . . . . . . . . . . . . . .. . . . 189 Simulation report . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . 19

iii

List of Tables

iv

List of Listings

1 Modes of Intra-Frame Prediction . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . . 32 Properties of the H.264 JM Reference Encoder . . . . . . . . . . . . . . . .. . . . . . . . . . 43 Variables Initialization Modification . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . 74 List of function pointers in JM reference encoder . . . . . . . . . . . . . .. . . . . . . . . . . 75 Before Function Pointer Elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . 86 After Function Pointer Elimination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . 87 Concurrent Execution with par statement . . . . . . . . . . . . . . . . . . . . . .. . . . . . . . 128 Residual Coding Parallelization . . . . . . . . . . . . . . . . . . . . . . . . . . . . .. . . . . 139 Motion Vector Searching Parallelization . . . . . . . . . . . . . . . . . . . . . . . .. . . . . . 14

v


CW. Chang, R DomerCenter for Embedded Computer Systems

University of California, IrvineIrvine, CA 92697-2625, USAcheweic,[email protected]://www.cecs.uci.edu

Abstract

In this report, we demonstrate the results of our sys-tem modeling project involving the encoder of theH.264 advanced video coding(AVC) standard. Thegoal is to create a H.264/AVC video encoder modelwith SpecC System Level Design Language (SLDL).First, we briefly introduce the essential features of theH.264/AVC video encoder algorithm and the proper-ties of the JM reference C source codes. We then de-scribe the modifications we have performed to makethe JM reference C implementation compliant forSpecC SLDL. Our model has been separated intothree major behavior blocks, which are Stimulus, De-sign, and Monitor, to reflect a proper system designtestbench. The design-under-test(DUT) componentthen has been structured into a hierarchy of communi-cating behaviors. In this work, we have identified sev-eral opportunities for parallel execution in the H.264encoding which we have explicitly specified in the sys-tem model. The report concludes with experimentalresults that validate the correct functionality of ourH.264 encoder model by success full simulation.

1 Introduction

High level languages like C language provide us afast path to evaluate our algorithm. However, exceptfor fully sequential behavior, it cannot model systemarchitectures such as parallelism and pipeline struc-ture. Hardware description languages like VHDL orVerilog can perfectly describe those structures, but tomodel the whole system of complex application andrefine it with HDL will be very time consuming. To

simplify the system design flow and evaluate the soft-ware and hardware as a whole at an early design stage,languages such as System Verilog, SystemC or SpecCare developed to fill this gap between high level lan-guage and HDL. SpecC [2] is one of the system leveldescription languages (SLDL) which models the al-gorithm at higher levels of abstraction (than RTL). Inthis project, we use SpecC to create the system modelof a H.264/AVC video encoder.

The most important feature of a SpecC model isthe separation of computation and communication.To accelerate the design of a complex System-on-Chip application, reusability of existing design suchas hard/soft Intellectual Properties (IPs) is a key fea-ture to solve this problem. SpecC clearly separatesthe design model of system-on-chip into two indepen-dent parts: computation and communication, whichare encapsulated into modules/behaviors and chan-nels respectively. Since computation and communi-cation are dealt with independently, the computationbehavior blocks can still be reused without or withonly minor modification even if the communication inan existing design does not fit in the requirements ofthe new project. The communication protocol can beeasily exchanged by using another channel with com-patible interface. In the same manner, we can alsomodify the computation behavior blocks to fit newrequirements without modifying the communicationchannels.

Another important feature of SpecC is its capabil-ity of modeling parallel structures. In a system con-structed with multiple processing elements, pipelin-ing or parallel structure are widely used to gain higher

1



resource utility and better performance. High levelprogramming languages like C language generally areexecuted sequentially and do not support parallel orpipelining structure. In SpecC, programmer can use’par’ statement in the program to model the parallelstructure in the implementation. In this project, thesetwo features are also used to create our system-levelmodel of a H.264/AVC video encoder.

This report is organized as follows: In Section IIwe will briefly introduce the H.264/AVC video cod-ing standard and some features of the JM reference Cimplementation. Section III shows the modificationwe performed on the JM reference C implementationso that it can be compiled by SpecC. In Section IV wewill demonstrate our SpecC model of a H.264/AVCvideo encoder platform. The parallelism we have ex-ploited in our H.264 encoder model will be describedin Section V. The simulation result is shown in Sec-tion VI.

2 H.264/AVC Video Encoder Algo-rithm

Before we demonstrate our SpecC model ofH.264/Video encoder, in this section we first describethe algorithm [5] and the reference JM C implemen-tation [3].

2.1 Background

H.264/AVC is the video coding standard of the ITU-T Video Coding Experts Group and the ISO/IECMoving Picture Experts Group. The main goals ofH.264/AVC standard is to enhance the video compres-sion performance and provide a ’network-friendly’video representation for both ’conversational’ appli-cations (video telephony) and ’non-conversational’(storage, broadcast, or streaming). Compared withexisting standards H.264/AVC has achieved a greatimprovement in rate-distortion efficiency, and it alsoprovides 17 profiles which support different featuresets for various applications [4] .

2.2 Input Video Data Format

The input of a H.264 video encoder and output of aH.264 video decoder are both in YUV format. In aYUV-format file, the information of brightness, calledluma, and the information of color, calledchroma,are stored separately. In a YUV-format file, for ev-ery frame in the video stream, there will be one ar-ray for storing luma information and two arrays forchroma information. Because the human visual sys-tem is more sensitive to luma than chroma, samplingformats in which the chroma component has only onehalf or one fourth of the number of samples than lumacomponent are proposed. The three major samplingformats are listed below:

*4-4-4 sampling: The number of chroma samplesis the same as the number of luma samples in each ofthe chroma arrays.

*4-2-2 sampling: The number of chroma samplesis the same as the number of luma samples in verticaldimension, but half in the horizontal dimension.

*4-2-0 sampling: The number of chroma samplesis half of the number of luma samples in both verticaland horizontal dimensions

In these three sampling formats, the precisions foreach sample are all 8-bits.

2.3 Coding Structure of H.264/AVC VideoEncoder

The basic encoding structure of H.264/AVC is shownin Figure 1. In the following sections, we will brieflydescribe the functions of these blocks.

In H.264/AVC video encoder, each video slice isencoded using intra-frame or inter-frame prediction toremove the spatial and temporal similarity in pictures.

* Intra-Frame Prediction: The spatial prediction inH.264 is called Intra-Frame prediction. It makes useof the characteristic that in certain areas in a picture,the pixels values are all the same or vary with littledifference. In this case, H.264 encoder predicts thepixel values in the current coded macroblock from thepixel values in its neighbor macroblocks to compressthe video data. The reference macroblocks are usu-ally the macroblock above or to the left of the cur-rent coded macroblock. In H.264, encoder can alter-natively select the block-size of 4x4 or 16x16 in the

2

Figure 1: Basic encoding structure for H.264/AVC fora macroblock

inter-frame prediction. For 16x16 and 4x4 block spa-tial predictions, there are four and nine different waysto predict the pixel value respectively. These predic-tion modes are listed in Listing 1.

In H.264, the slice which are coded with intra-frame prediction, are called I-slice.

* Inter-Frame Prediction: The temporal predictionin H.264/AVC video encoder is called Inter-FramePrediction which makes use of the temporal similar-ity between current coded pictures and the referencepictures. A continuous video is formed by a sequenceof pictures, and a picture is often composed of backscene and the object scene. In a continuous video, theback scene generally is still or varies in very limit dif-ference, and the object scene moves in the continuouspictures in certain regular way (for example in cer-tain speed and direction). Encoder temporal predictsa picture by comparing the difference between currentcoded picture and reference pictures in the buffer andobtains a motion vector for motion compensation indecoder. In prior standards, the reference pictures arelimited to the previous pictures in display order. InH.264/AVC video encoder, this limitation is removedand the reference pictures can be previous or futurepictures in display order. The inter-frame predictionperforms the block-based motion estimation on everymacroblock in the current coded frame, and a distinctmotion vector is sent for each macroblock for motioncompensation in H.264 decoder.

1 Mode 0 V e r t i c a l Mode2 Mode 1 H o r i z o n t a l Mode3 Mode 2 DC Mode4 Mode 3 Diagona l Down L e f t Mode5 Mode 4 Diagona l Down Righ t Mode6 Mode 5 V e r t i c a l R igh t Mode7 Mode 6 H o r i z o n t a l Down Mode8 Mode 7 V e r t i c a l L e f t Mode9 Mode 8 H o r i z o n t a l Up Mode

(a) Prediction Mode of Intra 4x4

1 Mode 0 V e r t i c a l Mode2 Mode 1 H o r i z o n t a l Mode3 Mode 2 DC Mode4 Mode 3 P lane Mode

(b) Prediction Mode of Intra 16x16

Listing 1: Modes of Intra-Frame Prediction

In H.264, two types of slices could be encoded withinter-frame prediction, which are P-slice and B-slice.In addition to the intra-frame prediction described inthe previous section, macroblocks in P-slice and B-slice can be encoded using inter-frame prediction. Ininter-frame prediction, both P-slice and B-slice arepredicted by motion estimation toward the referencepictures. The major difference between P-slice andB-slice is that unlike P-slice, in which only one mo-tion vector is obtained for motion compensation, in Bslice the motion vector is the weighted average of twomotion vectors to two different reference pictures.

In H.264/AVC video coding standard, the conceptof B slices is generalized. The pixel values are bi-predicted from the weighted combination of two dif-ferent motion compensated reference frames. Justlike the reference pictures in inter-frame prediction,the reference pictures in bi-prediction can be previousor future pictures in display order.

After spatial or temporal prediction, the differencesbetween input pictures and predicted pictures aretransformed to frequency domain by the block trans-formation function in H.264 coding algorithm. Thereare several unique features about the block transformselected by H.264:

* integer transform design* specified to 8-bit input video data* a 4x4 transform size is supported, rather than just

8x8 transform

3

While the macroblock size is still 16x16, pixelsare divided into 4x4 or 8x8 blocks for transforma-tion. The output coefficients are then quantized andscanned (in zig-zag fashion for frame coding) fromthe lowest frequency coefficient toward the highest.In this way the highest-variance coefficients are or-dered first, which maximizes the number of consec-utive zero-valued coefficients to gain better perfor-mance in entropy coding. In addition to the quantizedcoefficients, information required to decode the codeddata such as encoding parameters, reference frame in-dexes, and motion vectors for motion compensationare coded with entropy coding and form a compressedvideo bitstream.

3 Reference C implementation ofH.264/AVC Video Encoder

SpecC SLDL is a superset of the C language andit is compliant with ANSI-C. It means that everyANSI-C program can be compiled well by SpecCcompiler. To make the H.264 codec modeling eas-ier, rather than developing a H.264/AVC codec withSpecC SDLD from the very beginning, in this projectwe took the JM 13.0 reference implementation [3],released by Joint Video Team of ISO/IEC MPEG &ITU-T VCEG, as the beginning of this project. Sincethe JM reference implementation is programmed withC language, the first step to create a SpecC model ofH.264/AVC codec in this project, is to modify somesyntax in the JM reference implementation which isnot ANSI-C or SpecC compliant.

In this section, we briefly describe some features ofthe JM reference implementation and the modifica-tion we did toward the reference implementation forsimplicity and SpecC compliant reason.

3.1 Properties of H.264/AVC Video EncoderJM 13.0 Reference Software

The JM reference software of H.264/AVC codec con-sists of both encoder (inlencod directory) and de-coder (inlencoddirectory). In this project, we onlyfocus our work on the modeling of H.264 encoder,and use decoder part of the JM reference implementa-tion for consistency verification. Listing 2 shows the

1 T o t a l number o f t h e s o u r c e f i l e s : 572 T o t a l number o f t h e heade r f i l e s : 513 Number o f t h e s o u r c e code l i n e s : 63K4 Number o f f u n c t i o n s in sou rcecode : 814

Listing 2: Properties of the H.264 JM Reference En-coder

properties of the H.264 JM 13.0 reference encoder.In H.264/AVC video codec, the video is processed

from one picture (frame) to another. Every frame isdivided into one or multiple slices, and every sliceis then divided into several macroblocks in size of16x16 luma pixels. Each macroblock is also di-vided into sub-macroblock partitions for intra and in-ter frame prediction. For example, the size of sub-macroblock partition for inter-frame prediction canbe 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, and 4x4. Thetransform for the difference between original frameand predicted frame is either in size of 8x8 or 4x4.Figure 2 shows the hierarchy of a video sequence inH.264/AVC video codec.

Figure 2: Hierarchy of Video Stream

In the reference C implementation, we can also seethe same hierarchy in the source code. A simplifiedflow chart of the program execution is shown in Fig-ure 3. Frames in the video stream are read out fromthe YUV file one by one by aReadOneFramefunc-tion call and then every slice in the frame is encodedby a encodeoneslice function call in a while loop.The slice is further divided into 16x16-pixel mac-roblocks and every macroblock is then encoded bya encodeonemacroblockfunction call. The codingtools such as inter or intra prediction and spatial trans-form are all executed in theencodeonemacroblock

4

function.Notes that in this JM reference implementation, the

picture frames do not have to be encoded in the dis-play order. According to the JM Reference SoftwareManual, user can decide the order by modifying theHierarchicalCoding setting in the encoding param-eter. Two examples of changing coding order areshown in Figure 4. In the first example, the codingorder for nine consecutive frames is frame 0, 8, 4, 2,6, 1, 3, 5, and frame 7; in the second example, thecoding order for nine consecutive frames is frame 0,8, 2, 4, 6, 1, 3, 5, and frame 7.

Figure 4: Example of Coding Order

3.2 SpecC compliant reference implementa-tion

In this project, we took JM13 version of H.264 videoencoder as reference C implementation, and con-verted it to a hierarchical SpecC model so that we canconduct further design space exploration using SCE[1]. However, the JM reference implementation is notfully ANSI-C compliant and some syntax in the ref-erence C codes will lead to errors when the referencesource codes are compiled with SpecC. The first stepwe have to accomplish is to make some modificationto the reference C implementation so that it can becompiled by SpecC compiler. Besides, some functioncalls, such asfopenor fwrite are ANSI-C compliant,though, they are not adequate to be used in the SpecC

model because they are not hardware-synthesizable.

In this section, we list the modifications we per-formed on the reference C implementation for creat-ing our SpecC model of H.264 encoder.

1. Encoding Parameters Initialization: In the orig-inal reference C implementation, major encoding pa-rameters such frame rate, frame mode selection, andsearch algorithm selection for motion estimation, areinitialized at the beginning of video encoding. Duringinitialization, a configuration file calledencoder.cfgisopened by a function callfopenand a structure namedInputParameters, which stores the encoding param-eters is initialized with the content inencoder.cfg.Sincefopenis not hardware-synthesizable, we eithermove it out of the H.264 SpecC model or just hardcode the parameters in the H.264 SpecC model. Tosimplify the communication between thestimulusbe-havior (in charge of providing test patterns to thede-signbehavior) anddesignbehavior (in charge of im-plementing the major function the target algorithm)here we simply hard-coded the parameters in thede-sign behavior. More details about these two behav-ior and the communication between them will be de-scribed in the next section.

The same situation also happened in quantizationtable initialization. Instead of reading the quanti-zation coefficients from configuration fileq it ma-trix.cfg and q offset.cfg, we hard-coded the coeffi-cients into the tables indesignbehavior.

2. Variables Initialization: This part can be sep-arated into local variables initialization and globalvariables initialization. Unlike C program, in whichvariables can be initialized to the result of certainarithmetic or logical operations, at declaration thevariables can only be initialized to a constant valuesin SpecC SLDL. In local variable initialization, wecan fix this difference by simply manually separatingthe variable declaration and initialization. For globalvariables, since assigning values to variables outsideof function scope is not SpecC compliant, we cre-ated two functions calledsei init andpost2ctxinittofix this problem. At the beginning of encoding pro-cedure, these two functions will be called to initializeglobal variables. Listing 3 shows in general how wemodified the variable initialization.

3. Function Pointer Elimination: In C language,

5

Figure 3: Code Structure

function pointer is very useful for program simplicityand maintenance, which can greatly reduce the useof if or case-switch statements. However, due to itsambiguity for high-level synthesis, function pointersare not adequate in SpecC. In the JM H.264 refer-ence software, the function pointer syntax is widelyused due to various encoding options implementedin the JM reference implementation. To eliminatethe function pointers in the program, we either re-moved the encoding options or replaced them with aswitch statement (even though it makes the programlook more complex). When function pointers are usedfor encoding option selection, for simplicity, we onlykept one encoding option and eliminated the others.Listing 5 and Listing 6 shows how we replaced thefunction pointers with case-switch statement. List-ing 4 shows the existing function pointers in the JMreference implementation.

4. Source files renaming: Since SpecC compiler

only accepts source files with.sc postfix, 53 sourcefiles are renamed with same name and postfix by.sc.

After these modifications, the reference implemen-tation is SpecC compliant. Since the encoding param-eters are hard coded in the program already, except forYUV file, no other input file is needed.

4 Modeling of H.264/AVC Video En-coder Platform

One important feature of SpecC is the separation ofcomputation and communication, which increases thereusability of existing software and hardware imple-mentation. In this section, we also describe our SpecCmodel of H.264 video encoder in these two views.

First, we will describe the functions of three majorbehaviors at the topmost level in every SpecC model:stimulus, design, monitor. Then, in the second sub-

6

1 v a r t y p e g l o b a l v a r 1 = g l o b a l v a r 1 i n i t ;2 v a r t y p e g l o b a l v a r 2 = g l o b a l v a r 2 i n i t ;3 :4 :5

6 r e t u r n v a r t y p e func t i on name ( i n p u t v a r s )7 {8 v a r t y p e l o c a l v a r 1 = l o c a l v a r 1 i n i t ;9 v a r t y p e l o c a l v a r 2 = l o c a l v a r 2 i n i t ;10 :11 :12

13 ( f u n c i t o n d e s c r i p t i o n )14 }

(a) Before Variables Initialization Modification

1 v a r t y p e g l o b a l v a r 1 ;2 v a r t y p e g l o b a l v a r 2 ;3 :4 :5

6 void g l o b a l v a r i n i t ( )7 {8 g l o b a l v a r 1 = g l o b a l v a r 1 i n i t ;9 g l o b a l v a r 2 = g l o b a l v a r 1 i n i t ;10 :11 :12 }13

14 re turn v a r t y p e main ( i n p u t v a r s )15 {16 l o c a l v a r i a b l e d e c l a r a t o i n17 :18 g l o b a l v a r i n i t ( ) ;19 :20 ( f u n c t i o n d e s c r i p t i o n )21 }22

23 r e t u r n v a r t y p e func t i on name ( i n p u t v a r s )24 {25 v a r t y p e l o c a l v a r 1 ;26 v a r t y p e l o c a l v a r 2 ;27 :28 l o c a l v a r 1 = l o c a l v a r 1 i n i t ;29 l o c a l v a r 2 = l o c a l v a r 2 i n i t ;30 :31 ( f u n c i t o n d e s c r i p t i o n )32 }

(b) After Variables Initialization Modification

Listing 3: Variables Initialization Modification

1 void (∗ wr i teMB type In fo )2 void (∗ w r i t e I n t r a P r e d M o d e )3 void (∗ w r i t e B 8 t y p e I n f o )4 void (∗ wr i teRefFrame [ 6 ] )5 void (∗writeMVD )6 void (∗ writeCBP )7 void (∗ wr i t eDquan t )8 void (∗ wri teCIPredMode )9 void (∗ w r i t e F i e l d M o d e I n f o )

10 void (∗ w r i t e M B t r a n s f o r m s i z e )11 i n t (∗WriteNALU )12 i n t 6 4 (∗ g e t D i s t o r t i o n )13 e x t e r n imgpe l ∗ (∗ g e t l i n e [ 2 ] )14 e x t e r n imgpe l ∗ (∗ g e t l i n e 1 [ 2 ] )15 e x t e r n imgpe l ∗ (∗ g e t l i n e 2 [ 2 ] )16 e x t e r n imgpe l ∗ (∗ g e t c r l i n e [ 2 ] )17 e x t e r n imgpe l ∗ (∗ g e t c r l i n e 1 [ 2 ] )18 e x t e r n imgpe l ∗ (∗ g e t c r l i n e 2 [ 2 ] )19 e x t e r n i n t (∗ computeUniPred [ 6 ] )20 e x t e r n i n t (∗ computeBiPred )21 e x t e r n i n t (∗ computeBiPred1 [ 3 ] )22 e x t e r n i n t (∗ computeBiPred2 [ 3 ] )23 void (∗ ge tNe ighbour )24 void (∗ g e t m b b l o c k p o s )

Listing 4: List of function pointers in JM referenceencoder

section, we will describe the communication betweenstimulus& designanddesign& monitor.

4.1 Major Behaviors : Stimulus, Design,Monitor

In HDL implementation, to verify the correctness ofan implementation, a testbench environment includ-ing the function of providing test pattern and functionof output comparison is necessary. This concept isalso applicable to SpecC modeling. At the topmostlevel of a SpecC model, two behaviors namedstimu-lusandmonitorare created to implement the functionof the testbench, and the application we want to modelwith SpecC is encapsulated in a behavior namedde-sign. These three behaviors usually are executed inparallel, and the communication between them areimplemented to guarantee the necessary synchroniza-tion among these three behaviors.

In the following paragraphs, we describe the func-tions of these three behaviors and the roles they playin our SpecC model of H.264/AVC video encoder.

7

1 r e t u r n t y p e ∗ fp ( i n p u t v a r s ) ;2

3 r e t u r n t y p e func 1 ( i n p u t v a r s )4 {5 :6 }7


13 void f p i n i t ( i n i t c o n d i t i o n )14 {15 :16 s w i t c h ( c o n d i t i o n ){17 c o n d i t i o n 1 :18 fp = func 1 ; break ;19 c o n d i t i o n 2 :20 fp = func 2 ; break ;21 d e f a u l t :22 }23 }24

25 r e t u r n t y p e f p c a l l 1 ( i n p u t v a r s )26 {27 :28 fp ( ) ;29 :30 }31

32 r e t u r n t y p e f p c a l l 2 ( i n p u t v a r s )33 {34 :35 fp ( ) ;36 :37 }

Listing 5: Before Function Pointer Elimination


6 r e t u r n t y p e func 2 ( i n p u t v a r s )7 {8 :9 }

10

11 r e t u r n t y p e f p c a l l 1 ( i n p u t v a r s )12 {13 :14 s w i t c h ( c o n d i t i o n ){15 c o n d i t i o n 1 :16 f unc 1 ( i n p u t v a r s ) ; break ;17 c o n d i t i o n 2 :18 f unc 2 ( i n p u t v a r s ) ; break ;19 d e f a u l t :20 }21 :22 }23

24 r e t u r n t y p e f p c a l l 2 ( i n p u t v a r s )25 {26 :27 s w i t c h ( c o n d i t i o n ){28 c o n d i t i o n 1 :29 f unc 1 ( i n p u t v a r s ) ; break ;30 c o n d i t i o n 2 :31 f unc 2 ( i n p u t v a r s ) ; break ;32 d e f a u l t :33 }34 :35 }

Listing 6: After Function Pointer Elimination

8

Main

i_stimulushs_startno_frameshs_frm_nofrm_noimgY_outimgU_outimgV_out

i_designhs_startno_frameshs_frm_nofrm_noimgY_frameimgU_frameimgV_framehs_nlengthqueued_nbytehs_nreceivedhs_finish

frameV

frameY

hs_finish

hs_frm_no

frameU

i_monitordhs_nlengthqueued_nbytedhs_nbyte_receivedhs_finish

hs_nlength

hs_nreceived

hs_start

queued_nbyte

finish

frm_no

no_frames

Figure 5: Top level of H.264 Encoder Model

* BehaviorStimulus:In a SpecC model, this behavior is in charge of pro-

viding input data forDesignbehavior. It can be con-sidered as the test pattern generator for design ver-ification. Depending on the implementation ofDe-signbehavior,Stimulusbehavior sometimes is also incharge of providing configuration information forDe-signbehavior. Since the encoding parameters and thequantization coefficients for H.264 video encoder arehard-coded in ourDesignbehavior,Stimulusbehav-ior does not have to read parameters and coefficientsfrom the configuration file.

In the original JM reference implementation, a vari-able namedFrameNumberInFile, which indicates thenext frame to be encoded, will be updated for eachframe encoding and a function namedReadOneFrameis called.

This function will read the corresponding pictureframe from a opened video raw file namedtest.yuv.

Note that in the JM reference implementation theframe coding order does not have to be the sameas the display order. Therefore,FrameNumberInFilevariable indicating the next picture frame to be en-coded, is necessary. In our model, theStimulusbe-havior is implemented based on theReadOneFramefunction. At the beginning of the encoding processin our SpecC model,Stimulusbehavior will open rawfile test.yuvfor read with functionfopen. After theYUV file is opened,Stimulusbehavior will notifyDesign behavior of the beginning of video encod-ing. Stimulusbehavior then enters a state of waitingfor FrameNumberInfileupdating fromDesignbehav-ior. OnceStimulusbehavior receivesFrameNumber-Infile, it reads the corresponding picture frame fromfile test.yuvand sends frame data toDesignbehav-ior over the communication channels. In this project,for simplicity, we have fixed the sampling format toYUV420 and the frame size at 176x144 for Y frameand 88x72 for U and V frames.

* BehaviorDesign under test (DUT):In our SpecC model, this behavior is in charge of

the implementation of target design, and it is also thebehavior in which we will perform the design spaceexploration in the future.Designbehavior receivesinput data fromStimulusbehavior, performs the ap-plication or algorithm on the received data and thengenerates output results for validation. The content ofDesignbehavior depends on our design goal: we canimplement a specific part of the target application oralgorithm, such as temporal/spatial prediction or spa-tial transform, and only evaluate the execution of thisspecific part; we can also implement the whole tar-get application or algorithm inDesignbehavior andevaluate the execution of the whole algorithm.

In our project, Design behavior implements thewhole H.264/AVC video encoding algorithm. Forevery frame,Designbehavior generates a newFra-meNumberInfileand send it toStimulusbehavior torequest a new set of frame data, and then enters a stateof waiting for corresponding frame data. Once cor-responding Y, U, and V frame data are all received,Design behavior performs the spatial/temporal pre-diction, spatial transform and quantization, and en-tropy coding over the received frame data. The en-coded data is then sent toMonitor behavior over com-

9

munication channels in the form of byte-sequence forgenerating a H.264-format file namedtest.264. Thetransmission of the encoding data is performed onframe basis, i.e., for every frame,Designsends a se-quence of bytes toMonitor behavior. Note that be-cause of the different characteristics between pictureframes, the length of the byte-sequence for encodeddata varies. SinceMonitor behavior is independentof the execution of H.264 video encoding, it doesnot have the information about how long the byte se-quence is. Therefore, except for the encoded data,Designbehavior also has to send the correspondinglength of byte sequence toMonitor behavior for ev-ery encoded frame. Once every picture frame in theinput raw file is encoded,Designissues a notificationto Monitor behavior so thatMonitor behavior knowswhen to terminate the encoding.

In our Design behavior there are two sub-behaviors namedh264encoder and h264writerworking in parallel to achieve the task describedabove:h264encoderis the behavior in charge of theimplementation of H.264 video encoding algorithm,andh264writer deals with the transmission of lengthof byte-sequence toMonitor behavior. When the en-coding of a frame is finished,h264encoderwill sendthe byte-sequence toMonitor through a byte-queueand the corresponding length of byte-sequence is sentto h264writer through a integer-sequence. Once ev-ery frame in the YUV file is encoded,h264encoderwill send out a length of byte-sequence with zerovalue toh264writer. The function ofh264writer isto read the length of byte-sequence from the integer-queue and send the value toMonitor. h264writeralso detects the zero value of the length. When alength of byte-sequence with zero value is detected,h264writer will notify the monitor of the end of en-coding. Figure 6 shows the block diagram ofDe-sign under Testgenerated by System-on-Chip Envi-ronment (SCE).

* BehaviorMonitor :The main function of this module is to generate

output file(s) for verification. Depending on the im-plementation ofDesignbehavior, the content of theoutput files varies. IfDesignbehavior implementsone specific part of the target implementation or al-gorithm, then the content of the output files should

design

i_h264_encoder

i_h264_write

no_frames

hs_start

imgY_frame

imgU_frame

imgV_frame

hs_nlength

queued_nbyte

hs_nreceived

hs_finish

nlength_queue

hs_frm_no

frm_no

i_encoding_init

...g_finished

hs_

sta

rt

..._terminate

nlength_queue

no

_fr

am

es

hs_

frm

_n

o

frm

_n

o

img

Y_

fra

me

img

U_

fra

me

img

V_

fra

me

nle

ng

th_

qu

eu

e

qu

eu

ed

_n

byte

i_encoding_frames

nle

ng

th_

qu

eu

e

hs_

frm

_n

o

i_encode_ip_frame

qu

eu

ed

_n

byte

bit_

slice

_sta

rt_

se

qu

en

ce

frm

_n

on

o_

fra

me

s

img

U_

fra

me

nle

ng

th_

qu

eu

e

nu

m_

fra

me

s

hs_

sig

na

l

i_start_sequence

b_

fra

me

s_

to_

en

co

de

i_encode_b_frames

img

Y_

fra

me

...e_finished

hs_

sta

rt

img

V_

fra

me

qu

eu

ed

_n

byte

qu

eu

ed

_n

byte

hs_

frm

_n

o

b_

fra

me

s_

to_

en

co

de

img

U_

fra

me

frm

_n

o

hs_

sig

na

l

i_encode_one_frame

hs_

frm

_n

o

len

_o

ut

i_WriteNALU

nBitsWrittennlength_queuequeued_nbyte

img

Y_

fra

me

na

lu

tota

l_p

ps

i

hs_

sig

na

l

retu

rn_

va

lue

nle

ng

th_

qu

eu

e

img

V_

fra

me

img

V_

fra

me

i_encode_one_frame

len

qu

eu

ed

_n

byte

BitsW

ritt

en

nle

ng

th_

qu

eu

e

qu

eu

ed

_n

byte

img

U_

fra

me

frm

_n

o

img

Y_

fra

me

nle

ng

th_

qu

eu

e

i_write_PPS

PP

S_

id

i_init_frame

nlength_queue

queued_nbyte

i_WriteNALU

nBitsWrittennlength_queuequeued_nbyte

..._dpb_bhvr

p

en

c_

fra

me

_p

ictu

re_

0

nle

ng

th_

qu

eu

e

na

lu

...e_picture

frame

rd_pass

qu

eu

ed

_n

byte

...ut_picture

pic

nlength_queue

queued_nbyte

..._dpb_bhvr

p

nle

ng

th_

qu

eu

e

i_init_frame

nlength_queue

queued_nbyte

BitsW

ritt

en

frm

_p

ic

qu

eu

ed

_n

byte

...ut_picture

pic

nlength_queue

queued_nbyte

frm

_p

ic

qu

eu

ed

_n

byte

en

c_

fra

me

_p

ictu

re_

0

...e_picture

frame

rd_pass

len

_o

ut

nle

ng

th_

qu

eu

e

len

_in

Figure 6: Design under Test Behavior

10

be the output of this specific part; ifDesignbehaviorimplements the whole application or algorithm, thecontent of the output file is the output of the applica-tion or algorithm. According to the implementation ofDesignbehavior, golden sample(s) are pre-generatedand compared with the output file(s) for consistency.In this projectDesignbehavior implement the wholeH.264/AVC video encoding algorithm, which makesthe generation of gold sample(s) easier: we simplyuse the encoded video stream generated by JM refer-ence software as golden sample.

In original reference C implementation of H.264encoder, when a frame encoding is finished, a func-tion calledWriteNALUwill be called. In this func-tion the encoded data bytes will befwrite to a out-put file test.264. In SpecC model,fwrite operationis moved out fromDesign to Monitor behavior be-causefwrite function is not hardware-synthesizable.The main function ofMonitor behavior in our SpecCmodel is to receive byte-sequence fromDesignbe-havior and write those bytes to output filetest.264.Another function implemented inMonitor is to termi-nate the encoding program withexit(0); which is nothardware-synthesizable either.

To satisfy the requirements described above, thereare two separate sub-behaviors namedmonitor writeand monitor finish which work in parallel inMoni-tor behavior. Themonitor write behavior is in chargeof the byte-sequence reading and writing the bytes tothe output filetest.264; themonitor finishbehavior isin charge of the termination of the whole application.After the initialization of the H.264 video encodingin our SpecC model, these two behaviors are both inwaiting state and wait for the length of byte-sequenceand end of encoding notification, respectively.moni-tor write behavior has to receive the length of next en-coded byte-sequence fromDesignbehavior througha doublehandshake channel, before it fetches corre-sponding number of bytes from a byte queue betweenDesignandMonitor behavior andfwrite the bytes intotest.264. During the video encodingmonitor write isalways in waiting state. When it receives the end ofencoding notification fromDesignbehavior, it closesthe output file and usesexit(0) to terminate the pro-gram.

4.2 Communication between top-level be-haviors

* communication between Stimulus and Design:The communication between stimulus and design

modules is implemented with two double handshakechannels and three frame queues. The first hand-shake channel is used to send encoding initiation sig-nal fromStimulusto Designto start the encoding; thesecond double handshake channel is used to transfervariableFrameNumberInFilefrom Designto Stimu-lus to request frame data. OnceStimulusreceivestheFrameNumberInFilevariable fromDesign, it willread the corresponding frame and send them toDe-sign through frame queue channels. The Y, U, andV frames are transmitted separately through differentframe queue channel. For simplicity, the size of theframe in our encoder is fixed to 176x144 pixels perframe. Therefore, the size of frame queue channel is176x144 bytes for Y-frame queue, 88x72 bytes for Uand V frame. For now, in frame encoding, the depthof these three queues are all set to 1.

Figure 7 shows the comparison between JM refer-ence software and SpecC model, and the communica-tion betweenStimulusandDesignbehaviors as well.

*communication between Design and Monitor:The communication between design and monitor is

implemented with one handshake channel, two dou-ble handshake channels, and one byte-queue chan-nel. The two doublehandshake channels are usedto synchronize the update of length of byte-sequencein h264writer and bytes writing operation inmon-itor writer. To update the length of byte-sequenceh264writer in it Design behavior will read one in-teger from the integer queue betweenh264encoderand h264writer. If the length of byte-sequence iszero,h264writer will notify the Monitor of the end ofencoding through the handshake channel, andMon-itor will terminate the encoding program immedi-ately; if the length of byte-sequence is not zero,h264writer will send the value toMonitor through adoublehandshake channel HSNLENGTH. After thetransmission,h264writer will enter a waiting stateand wait for the reply fromMonitor through anotherdoublehandshake channel named HSNRECEIVED.

11

WhenMonitor finishes the writing operation, it sendsan acknowledge through HSNRECEIVED to notifyh264writer of the accomplishment of writing oper-ation. After receiving the acknowledge,h264writercontinues the update of length of byte-sequence.

Figure 8 shows the comparison between JM refer-ence software and SpecC model, and the communica-tion betweenDesignandMonitor behaviors as well.

5 Exploiting parallelism in H.264Video Encoder

H.264 video codec is a very complex algorithm withheavy a computational load. For some applications,such as video camera which has to encode input videoin real-time, we might have to add parallel structure inthe implementation to accelerate the encoding. Afterinspecting the reference JM source code of H.264 en-coder, we have found some possible parallelism in theencoder.

In this section, we have two parallel structures im-plemented in our H.264 encoder model.

One important feature of SpecC SLDL is its abil-ity to simulate parallel structure. Parallel structure isvery common in hardware description language likeVerilog, but for high level languages like C, it is quitedifficult to simulate parallel structure since programsare usually executed in sequential way. In SpecC,par statement is used when concurrent execution isrequired.

One simple instance of concurrent execution withpar statement is shown in Listing 7. With this fea-ture, designer can explore parallelism in the target ap-plication and verify the functionality without actuallyimplementing it with hardware description language.

Dependency between codes implies sequential ex-ecution order, and in most cases disturbing this or-der leads to functionality error. Therefore, we have tomake sure that there is no dependency between thevariables used in the code before executing two ormore sections of code, which was running sequen-tially originally, in concurrent way. Due to the pre-diction mechanism in the H.264 video encoder, paral-lelism across frames and macroblocks might not be

1 behavior B par2 {3 B b1 , b2 , b3 ;4

5 void main (void )6 {7 par{8 b1 . main ( ) ;9 b2 . main ( ) ;

10 b3 . main ( ) ;11 }12 }13 } ;

Listing 7: Concurrent Execution with par statement

feasible: intra-prediction of a macroblock requiresthe pixel values from the left, up, and up-left mac-roblocks, and inter-prediction of a macroblock usesthe macroblocks in the previous reconstructed framesas references to compute the motion vectors.

Two possible opportunities for parallelism we havefound are concurrent encoding of multiple slices, andparallel execution of encoding procedure inside onemacroblock. In this report, for now we only focuson the parallelism inside macroblock encoding. Thetwo occurrences of parallelism, which we have im-plemented in our H.264 encoder model, are (1) Lumi-nance/Chrominance residual coding and reconstruc-tion, and (2) Motion vector search for multiple refer-ence frames.

5.1 Luminance/Chrominance residual cod-ing and reconstruction

The first potential for parallelism we found in theH.264 encoder is the residual coding of Luma(Y) andChrominance(U/V) pixels in a macroblock. In intra-and inter-prediction encoding, the difference betweenpredicted macroblock and original macroblock, calledresidual, will be transformed with DCT and thenquantized to reduce the size of data. Then the trans-formed and quantized residual will be written intooutput file. Since the quantization in H.264 codec islossy quantization, to make sure the reference framesin the encoder are identical to the reference framein the decoder, the quantized residual in the encoderhas to be inverse-quantized and inverse-transformed

12

so that the residual in the encoder and decoder endare identical. The predicted macroblock then willbe combined with inverse-quantized and transformedresidual to reconstruct the decoded picture. The de-coded picture will be put in decoded picture buffer forfuture reference in inter-prediction. During the resid-ual coding and reconstruction, luminance pixels andchrominance pixels are processed separately. In origi-nalRDCostfor macroblocksfunction, the luma pixelcoding and chroma pixel coding are executed sequen-tially. In our H.264 SpecC model, we managed tocombine the luma and chroma residual coding and re-construction in inter-prediction coding and execute itconcurrently.

The code before and after modification are shownin Listing 8. In the original source code, wecan see that functionChromaResidualCodingwillbe called after luma residual coding except forIPCM mode. Now we have created a new be-haviorLumaChromaResidualCodingbhvr to concur-rently execute luma and chroma residual coding ininter-prediction encoding. In the future, we willadd luma residual coding in intra-prediction encod-ing(mode==I16MB and I8MB) into the concurrentexecution as well. Another potential for parallelismin this part is that the U and V chroma pixel resid-ual coding can be further separated and executed inparallel.

5.2 Motion vector searching for multiple ref-erence frames

In H.264 encoder, the reference pictures are stored intwo decoded picture buffers namedList0 and List1.The predicted macroblock in inter-prediction is ob-tained by comparing the current macroblock with themacroblock in the reference frame and searching forthe reference macroblock with minimal error. Inter-prediction will search reference pictures in decodedpicture buffer and find out the motion vector betweenoriginal macroblock and best-matching reference pic-ture. For P frame encoding, all reference pictures inList0will be searched for inter prediction; for B frameencoding, both decoded picture bufferList0andList1will be searched.

The parallelism we exploited in inter prediction is

1 func RDCost fo r Macrob locks ( . . )2 {3 :4 i f ( mode<P8x8 )5 LumaResidualCoding ( currMB ) ;6 e l s e i f ( mode==P8x8 )7 Se tCoe f fAndRecons t ruc t i on8x8 ( currMB ) ;8 e l s e i f ( mode==I16MB )9 I n t r a16x16 Mode Dec i s i on ( currMB ) ;

10 e l s e i f ( mode==I8MB)11 I n t r a8x8Mac rob lock ( currMB ) ;12 e l s e i f ( mode==IPCM)13 :14

15 i f ( mode !=IPCM)16 ChromaResidualCoding ( currMB ) ;17

18 :19 }

(a) Original sequential Luma/Chroma Residual Coding

1 behavior LumaChromaResidualCodingbhvr ( . . )2 {3 par{4 i LumaRes idua lCod ingbhv r . main ( ) ;5 i Ch romaRes idua lCod ingbhv r . main ( ) ;6 }7 } ;8

9 behavior RDCos t fo r Macrob lock bhv r ( . . )10 {11 :12 i f ( mode<P8x8 )13 i LumaChromaRes idua lCod ingbhvr . main ( ) ;14 e l s e i f ( mode==P8x8 )15 Se tCoe f fAndRecons t ruc t i on8x8 ( currMB ) ;16 e l s e i f ( mode==I16MB )17 I n t r a16x16 Mode Dec i s i on ( currMB ) ;18 e l s e i f ( mode==I8MB)19 I n t r a8x8Mac rob lock ( currMB ) ;20 e l s e i f ( mode==IPCM)21 :22

23 i f ( ( mode !=IPCM)&(mode>=P8x8 ) )24 i Ch romaRes idua lCod ingbhv r . main ( ) ;25

26 :27 } ;

(b) Luma/Chroma Residual Coding in Parallel

Listing 8: Residual Coding Parallelization

13

in the function calledPartitionMotionSearch. Thisfunction in the original C code calculates the minimalerror and finds out the best motion vector for eachreference picture. In our H.264 encoder model, wecreate a parallel behavior to execute the motion vectorsearch for all reference pictures in the decoded picturebuffer. Since the number of reference pictures is user-defined and we set this number to five, there will be atmost five parallel motion vector searches for P frameencoding and ten parallel motion vector searches forB frame encoding in our H.264 model.

The codes before and after modification are shownin Listing 9. Note that the behaviorRefFrameMo-tionSearchbhvr in parallel model is not identical tothe behavior in the sequential model. In parallel ver-sion of motion vector search, everyRefFrameMotion-Searchbhvr will be assigned a list number and ref-erence picture number. Before executing the mo-tion search, these two numbers will be comparedwith variablesnumlist and numref to determine ifthe motion vector search in the corresponding refer-ence picture and list is required. We did this modi-fication to prevent motion vector search from beingexecuted when the reference picture does not exist.For example, if there are only two reference picturesin the decoded picture buffer and the current frameis a P frame, onlyRefFrameMotionSearchbhvr 00and only RefFrameMotionSearchbhvr 01 will exe-cute the motion vector search, and the restRef-FrameMotionSearchbhvrwill be idle.

6 Experiments and Results

In this project, we validate our H.264 SpecC modelby comparing the output video file generated by ourmodel with the output file generated by reference Ccode. We also calculate the run time of the encodingprocess of our H.264 model. A simulation report ofthe H.264 video encoding for a 19-frame video clipwith our H.264 SpecC model is shown in Figure 9.

In the simulation report, some parameters of thisencoder model are shown: the input video format isYUV420, the frame size is 176x144, the number ofreference frames is 5, and so on. We can also see thatthe encoding order is different from displaying orderin this simulation report. The encoding order in this

1 behavior P a r t i t i o n M o t i o n S e a r c hb h v r ( . . )2 {3 f o r ( l i s t =0; l i s t < n u m l i s t s ; l i s t ++)4 {5 f o r ( r e f =0; r e f < num ref ; r e f ++)6 {7 :8 i Re fF rameMot ionSearchbhv r . main ( ) ;9 :

10 }11 }12 :13 } ;

(a) Sequential Motion Vector Searching

1 behavior P a r a l l e l R e f F r a m e M o t i o n S e a r c hb h v r ( . . )2 {3 :4 par{5 i Re fF rameMot ionSearchbhv r 00 . main ( ) ;6 i Re fF rameMot ionSearchbhv r 01 . main ( ) ;7 i Re fF rameMot ionSearchbhv r 02 . main ( ) ;8 i Re fF rameMot ionSearchbhv r 03 . main ( ) ;9 i Re fF rameMot ionSearchbhv r 04 . main ( ) ;

10 i Re fF rameMot ionSearchbhv r 10 . main ( ) ;11 i Re fF rameMot ionSearchbhv r 11 . main ( ) ;12 i Re fF rameMot ionSearchbhv r 12 . main ( ) ;13 i Re fF rameMot ionSearchbhv r 13 . main ( ) ;14 i Re fF rameMot ionSearchbhv r 14 . main ( ) ;15 }16 } ;17

18 behavior P a r t i t i o n M o t i o n S e a r c hb h v r ( . . )19 {20 :21 i P a r a l l e l R e f F r a m e M o t i o n S e a r c hb h v r . main ( ) ;22 :23 } ;

(b) Parallel Motion Vector Searching

Listing 9: Motion Vector Searching Parallelization

14

simulation is I0, P2, B1, P4, B3, P6, B5, P8, B7, P10,B9, P12, B11, P14, B13, B16, B15, P18, and thenB17. The run time of this simulation is also shownafter the encoding is finished.

H.264 video encoder might be the most compli-cated application we have ever modeled with SpecC.In this simulation, there are 19 frames been encoded:one for I-frame and nine for P-frame and B-frame in-dividually. Since the frame rate of this video clip is30 frames per second, the total length of the encodedvideo is about 2/3 second. To encode such a shortvideo clip, it took about 90 seconds to complete theprocess with our H.264 model.

7 Conclusion and Future Work

In this project, we have implemented a system modelof a H.264/AVC video encoder with SpecC SLDL.We first made necessary modifications in the JM ref-erence software to make the reference source codeSpecC compliant. After the reference code is fullySpecC compliant, we separated the reference imple-mentation into three major behaviorsStimulus, De-sign, Monitor in our SpecC model, and assignedproper channels for the communication between thesethree behaviors.

We have also exploited two opportunities for par-allelism in our H.264 encoder model: parallel lumi-nance and chrominance pixel residual coding, andparallel motion vector search for multiple referencepictures.

At this point, our H.264 SpecC model is stillfar away from a perfect synthesizable SpecC model.There is still lots of work to do to make it perfect.Several improvements we plan for the future are listedbelow:

1. Global variables elimination: Global variablesare very convenient for C language programmer andalso makes C programs look concise. Every func-tion in the C program can access the global variableswithout explicit variable transmission. However, itis not appropriate to use global variables in SpecCbecause of the lack of explicit communication. Tolet System-on-Chip Environment (SCE) estimate thecommunication cost in design exploration, we shouldkeep every variable transmission as explicit as pos-

sible. Therefore, the global variables have to be re-placed by local variables and corresponding commu-nications in the SpecC model. Right now the commu-nication between the three major behaviors is totallyglobal-variables free, but we still have to eliminate theglobal variables inDesignbehavior in the future.

2. Memory allocation elimination : In an embed-ded system where memory space is fixed and limited,memory allocation functions such as malloc are nothardware-synthesizable. Therefore, it is better to re-place memory allocation functions with explicit arraydeclarations in the SpecC model.

3. Explore more parallelism inDesignbehavior:We have found two opportunities for parallelism andimplemented them in our H.264 model. There mightbe more parallelism to be exploited in the encoder.For example, since both intra-frame and inter-frameprediction are used in P-slice encoding, we could lookinto this part to find out if it is possible to run thesetwo predictions in parallel. Another potential par-allelism we can explore is to add pipeline structurein our SpecC model. Since the encoding of a mac-roblock is processed in the order of temporal/spatialprediction, transform and quantization, and then en-tropy coding, maybe it is possible to execute thesesteps in pipeline structure.

4. Design space exploration on System-on-ChipEnvironment (SCE) : System-on-Chip Environment(SCE) is a system developing tool in which we can as-sign different combinations of processor(s) and hard-ware(s) to implement the H.264 encoder and estimatethe execution time for those different resource as-signments and operating frequencies. After we cre-ate a ”clean” SpecC model in which all global vari-ables are eliminated and all functions are hardware-synthesizable, we plan to explore the design space ofH.264 video encoder with SCE for different require-ments and applications.

Acknowledgment

This work has been supported in part by fundingfrom the National Science Foundation (NSF) underresearch grant NSF Award #0747523. The authorsthank the NSF for the valuable support. Any opin-ions, findings, and conclusions or recommendations

15

expressed in this material are those of the authors anddo not necessarily reflect the views of the NationalScience Foundation.

References

[1] L. Cai, A. Gerstlauer, S. Abdi, J. Peng, D. Shin,H. Yu, R. Domer, and D. Gajski. System-onchipenvironment (sce version 2.2.0 beta): Manual.Technical Report CECS-TR-03-45, Center forEmbedded Computer Systems, December 2003.

[2] Andreas Gerstlauer, Rainer Domer, Junyu Peng,and Daniel D. Gajski.System Design: A PracticalGuide with SpecC. Kluwer Academic Publishers,2001.

[3] H.264/AVC JM Reference Software.http://iphome.hhi.de/suehring/tml/.

[4] Wikipedia H.264/MPEG-4 AVC.http://en.wikipedia.org/wiki/H.264/MPEG-4AVC.

[5] Gisle Bjontegaard Thomas Wiegand, Gary J. Sul-livan and Ajay Luthra. Overview of the h.264/avcvideo coding standard.IEEE Transactions onCircuits and Systems for Video Technology, 13(7),JULY 2003.

16

http://iphome.hhi.de/suehring/tml/

http://en.wikipedia.org/wiki/H.264/MPEG-4_AVC

Frame Init

Read

One Frame

Encode

Frame Picture

...

...

Encode I/P Frame

...

...

...

Encoding Initialization

Encoding Termination

Read

One Frame

fopen(target)

Double Handshake

Encoding Start Flag

Double Handshake

Frame Number

ReceiverSender

SenderReceiver

Frame Queue

for Y/U/V Frames

Receiver

Design

Stimulus

Encoding Frames

Frame Init

Read

One Frame

Encode

Frame Picture

...

...

Encode I/P Frame

...

...

...

Encoding Initialization

Encoding Termination

Encoding Frames

Reference C

implementation

SpecC

Modeling

Frame Init

Read

One Frame

Encode

Frame Picture

...

...

Encode B Frame

...

...

Frame Init

Read

One Frame

Encode

Frame Picture

...

...

Encode B Frame

...

...

Sender

Receiver

Figure 7: Communication between Stimulus and Design

17

Figure 8: Communication between Design and Monitor

18

------------------------------------- H.264 SpecC Model -------------------------------------

Input YUV file : test.yuv

Output H.264 bitstream : test.264

Output YUV file : test_rec.yuv

YUV Format : YUV 4:2:0

Frames to be encoded I-P/B : 10/9

Freq. for encoded bitstream : 15

PicInterlace / MbInterlace : 0/0

Transform8x8Mode : 1

ME Metric for Refinement Level 0 : SAD

ME Metric for Refinement Level 1 : Hadamard SAD

ME Metric for Refinement Level 2 : Hadamard SAD

Mode Decision Metric : Hadamard SAD

Motion Estimation for components : Y

Image format : 176x144

Error robustness : Off

Search range : 32

Total number of references : 5

References for P slices : 5

List0 references for B slices : 5

List1 references for B slices : 5

Sequence type : I-B-P-B-P (QP: I 28, P 28, B 30)

Entropy coding method : CABAC

Profile/Level IDC : (100,40)

Motion Estimation Scheme : Fast Full Search

Search range restrictions : none

RD-optimized mode decision : used

Data Partitioning Mode : 1 partition

Output File Format : H.264 Bit Stream File Format

---------------------------------------------------------------------------------------------------------

Encoding. Please Wait.

=============== FrameNumberInFile = 0, Type = I_SLICE ===============

=============== FrameNumberInFile = 2, Type = P_SLICE ===============

=============== FrameNumberInFile = 1, Type = B_SLICE ===============

















H264 Encoding Process has finished

Total Run Time: 90.000 seconds

Files test.264 and test.264.gold are identical

Figure 9: Simulation report

19

system level modeling of a h.264 video encoder

Documents