IMAGE AND VIDEO COMPRESSION STANDARDS

Algorithms and Architectures

Second Edition

THE KLUWER INTERNATIONAL SERIES IN ENGINEERING AND COMPUTER SCIENCE

lMAGE AND VIDEO COMPRESSION STANDARDS

Algorithms and Architectures

Second Edition

by

Vasudev Bhaskaran Konstantinos Konstantinides Hewlett-Packard Laboratories

SPRINGER SCIENCE+BUSINESS MEDIA, LLC

Library of Congress Cataloging-in-Publication Data

Bhaskaran, Vasudev, 1956-Image and video compression standards : algorithms and

architectures / by Vasudev Bhaskaran, Konstantinos Konstantinides. -- 2nd ed.

p. cm. -- (The Kluwer international series in engineering and computer science; SECS 408)

Includes bibliographical references and index. ISBN 978-1-4613-7842-6 ISBN 978-1-4615-6199-6 (eBook) DOI 10.1007/978-1-4615-6199-6 1. Image compression. 2. Video compression.

Konstantinos, 1956- . II. Title. III. Series. TA1632.B49 1997 621.36'7 --dc21

1. Konstantinides,

97-17141 CIP

Copyright © 1997 by Springer Science+Business Media New York. Fourth printing 2000. Originally published by Kluwer Academic Publishers in, New York 1997 Softcover reprint of the hardcover 2nd edition 1997

AII rights reserved. No part of this publication may be reproduced, stored in a retrieval system or transmitted in any form or by any means, mechanical, photo-copying, recording, or otherwise, without the prior written permission of the publisher, Springer Science+Business Media, LLC,

Printed on acid-free pa per.

to Achan. Chechi,

Balu, Kishore, Shanti, Paula,

Anastasia, Anneta, and Niki

CONTENTS

PREFACE XIll

I COMPRESSION FUNDAMENTALS 1.1 Introduction J.2 Background 2 1.3 Compression Taxonomy 5 1.4 Issues in Compression Method Selection 12 1.5 To Probe Further 14

2 METHODS AND STANDARDS FOR LOSSLESS COMPRESSION 15 2.1 Introduction 15 2.2 Preliminaries 15 2.3 Huffman Encoding 19 2.4 Huffman Decoding 23 2.5 Huffman Codes with Constrained Length 25 2.6 Constrained-Length Huffman Codes: Ad Hoc Design 32 2.7 Constrained-Length Huffman Codes: The Voorhis Method 34 2.8 Golomb and Rice Coding 38 2.9 Arithmetic Coding 39 2.10 Implementation Issues 44 2.11 Standards for Lossless Compression 46 2.12 New Work in IPEG: IPEG-LS 52 2.13 To Probe Further 59

viii IMAGE AND VIDEO COMPRESSION STANDARDS

3 FUNDAMENTALS OF LOSSY IMAGE COMPRESSION 61 3.1 Introduction 61 3.2 Preliminaries 62 3.3 Basic Coding Schemes for Lossy Compression 68 3.4 OCT -based Coding 74 3.5 Fast Algorithms for the OCT 86 3.6 Rate-Distortion Perfonnance of the OCT 95 3.7 To Probe Further 96

4 FUNDAMENTALS OF LOSSY VIDEO COMPRESSION 99 4.1 Introduction 99 4.2 Video Coding Basics 100 4.3 Rate-Distortion Functions in Video Coding 105 4.4 Motion-Compensated Prediction 117 4.5 Algorithms for Motion Estimation 120 4.6 Motion Estimation Using Low Complexity Matching Cri-

teria 134 4.7 Sub-pixel-Accurate Motion Estimation 140 4.8 Multipicture Motion Estimation 144 4.9 To Probe Further 145

5 THE JPEG STANDARD 147 5.1 Introduction 147 5.2 DCT-based Coding 148 5.3 Processing of Color Images 151 5.4 Design of Quantization Tables 153 5.5 Entropy Coding 157 5.6 JPEG Modes of Operation 162 5.7 Implementation Issues 169 5.8 JPEG Extensions and Applications 173 5.9 To Probe Further 177

6 THE MPEG VIDEO STANDARDS 179 6.1 Introduction 179

Contents ix

6.2 The MPEG-I Video Standard 181

6.3 MPEG·l Implementation Issues 199

6.4 The MPEG-2 Video Standard 200 6.5 The MPEG-4 Coding Standard 216 6.6 Future Directions: MPEG·7 227 6.7 To Probe Further 228

7 VIDEO TELECONFERENCING STANDARDS 231 7.1 Introduction 231

7.2 H.320 Video Teleconferencing 232 7.3 The H.261 Video Coding Standard 234

7.4 Implementation Issues 238 7.5 H.324 Video Teleconferencing 242

7.6 The H.263 Video Coding Standard 243 7.7 Future Directions 252 7.8 To Probe Further 252

8 PROCESSING REQUIREMENTS 253 8.1 Introduction 253 8.2 Measuring Complexity 254 8.3 Distributing the Load 257

8.4 To Probe Further 258

9 RISC AND DSP CORES 261 9.1 Introduction 261 9.2 The RISC Core 261 9.3 The DSP Core 265 9.4 RISC versus nsp 268 9.5 To Probe Further 268

10 ARCmTECTURES FOR THE DCT 271 10.1 Introduction 271 10.2 Vector Processing 272 10.3 Distributed Arithmetic for the ocr 274

x IMAGE AND VIDEO COMPRESSION STANDARDS

10.4 Commercially Available OCT Processors 280 10.5 To Probe Funher 281

11 HARDWARE FOR MOTION ESTIMATION 283 11.1 Introduction 283 11.2 Block Matching Using Linear Arrays 284 11.3 Hardware for One-bit Motion Estimation 291 11.4 Sub-pel Motion Estimation 295 11.5 Implementation Examples 296 11.6 To Probe Further 301

12 HARDWARE FOR ENTROPY CODING 303 12.1 Introduction 303 12.2 Encoder Implementation 304 12.3 Decoder Implementation 307 12.4 Variable-Length Coding in JPEG 312 12.5 To Probe Further 319

13 IMPLEMENTATION OF JPEG PROCESSORS 321 13.1 Introduction 321 13.2 Data VO and Memory Interface 322 13.3 Color Conversion 323 13.4 Commercially Available JPEG Processors 324 13.5 To Probe Further 328

14 INTEGRATED CIRCUITS FOR VIDEO CODERS 329

14.1 Introduction 329

14.2 Video Signal Processors 330

14.3 Multimedia Coprocessors 338

14.4 Dedicated Coders 343 14.5 To Probe Further 345

15 MULTIMEDIA ENHANCEMENTS FOR GENERAL PURPOSE PROCESSORS 347 15.1 Introduction 347

Contents XI

15.2 Generic Operations in Video Processing 348 15.3 Multimedia Enhancements in the HP-PA 349 15.4 Multimedia Enhancements in the UltraSparc 353 15.5 Multimedia Enhancements in the Intel Architecture 356 15.6 To Probe Further 358

16 STANDARDS FOR AUDIO COMPRESSION 359 16.1 Introduction 359 16.2 The G.12X Standards 361 16.3 Fundamentals of Perceptual Coders 362 16.4 MPEG Audio Encoding 364 16.5 MPEG Audio Decoding 370 16.6 The AC-3 Coding Standard 312 16.7. Future Directions in MPEG 376 16.8 Hardware Implementation of Audio Codecs 382 16.9 To Probe Further 384

17 NONSTANDARD COMPRESSION TECHNIQUES 387 17.1 Introduction 387 17.2 Vector Quantization 388 17.3 Subband Coding 393 17.4 Video Coding Schemes for the Desktop 402 17.5 To Probe Further 405

A COLOR TRANSFORMATIONS 409

B ABBREVIATIONS AND SYMBOLS 413

C INTERNET DIRECTORY 419

REFERENCES 423

INDEX 447

PREFACE

This book presents an introduction to the algorithms and architectures that fonn the underpinnings of the image and video compression standards. including JPEG (compression of still-images), H.261 and H.263 (video teleconferenc­ing), and MPEG-I and MPEG-2 (video storage and broadcasting). The next generation of audiovisual coding standards, such as MPEG-4 and MPEG-7. are also briefly described. In addition, the book covers the MPEG and Dolby AC-3 audio coding standards and emerging techniques for image and video compression, such as those based on wavelets and vector quantization.

The emphasis of the book is on the foundations of these standards, namely, techniques such as predictive coding, transform-based coding. motion esti­mation, motion compensation, and entropy coding, as well as how they are applied in the standards. We avoid the implementation details of each standard. However, we do provide all the material necessary to understand the workings of each of the compression standards, including infonnation that can be used by the reader to evaluate the efficiency of various software and hardware implementations confonning to these standards. We place particular emphasis on those algorithms and architectures that have been found to be useful in practical software or hardware implementations.

No prior knowledge of image and video compression theory and architectures is assumed; however, some background in high-level hardware design is expected for a better understanding of the chapters covering the hardware implementation of the standards.

Chapters 1 to 4 review the fundamentals of image and video compression. In Chapter I we cover the key principles of image compression and explain the basic tenninology. We present a compression taxonomy and discuss design issues and tradeoffs in selecting a compression algorithm. In Chapter 2 we review methods and standards for loss less compression, including differential, Huffman, and arithmetic coding. We present several practical algorithms

XIV IMAGE AND VIDEO COMPRESSION STANDARDS

for the design of Huffman codes, and we describe the current lossless JPEG algorithm, work in progress on a new lossless and near-Iossless JPEG standard (JPEG-LS), and the standards for lossless facsimile transmission (Group 3, Group 4, and IBIG).

In Chapter 3 we examine techniques for lossy compression. We review the source-coding theorem and compute the rate-distortion function for several image models relevant to image compression. We place particular emphasis on DCT-based coding. and describe several schemes for the efficient imple­mentation of the OCT. In Chapter 4 we present the fundamentals of video coding, including motion estimation and motion compensation.

In Chapter 5 we describe .lossy JPEG and its application to the compression of still-images. We cover recent extensions to JPEG and the lPEG-based standard for color facsimile. In Chapter 6 we describe the coding algorithms and syntax for the MPEG-l and MPEG-2 video compression standards, the basic video coding aspects of MPEG-4. and objectives and target application areas for MPEG-7. In Chapter 7 we describe the H.320 and H.324 video teleconferencing standards, with emphasis on their video coding parts, namely, H.261 and H.263. Simi.larities and differences among the video compression standards are also presented and reviewed.

Next, we move to the hardware implementation of the standards. In Chapter 8 we define a measure of computational complexity. and we use it to measure the complexity of various algorithms used in the standards. Bearing in mind that most hardware implementations of the standards are based on either RISC or DSP cores, in Chapter 9 we review the principles of RISC and DSP designs and describe their main differences and similarities. Even though we present the latest designs in image and video processors, we place more emphasis on the designs of the core algorithms; that is, the discrete cosine transfonn (Chapter 10), motion estimation (Chapter 11), and entropy coding (Chapter 12). In Chapter 13 we describe architectures for the JPEG standard, and in Chapter 14 we discuss programmable and dedicated video processors for MPEG and videoconferencing standards. Key features of recent VLIW media processors are also described.

In Chapter 15 we review recent developments in the design of multimedia­enhanced general-purpose RISC processors, such as the PA-8000 from Hewlett-Packard, the UltraSPARC from Sun Microsystems. and Intel's MMX-

Preface xv

based Pentium processors. In Chapter 16 we present the key standards on audio coding, and we describe in more detail the MPEG and Dolby AC-3 coding schemes. In Chapter 17 we provide an overview of techniques based on vector quantization and wavelets. Such techniques are used in many proprietary image and video compression schemes and may play a significant role in future standards.

Given the wide use of color imagery as input to image and video coders. Ap­pendix A provides a quick reference to basic color transformations. Appendix B provides a summary of common abbreviations used in the text and can be used as a quick reference for the various standard image fonnats (SIF, CIF, and CCIR 601). Information about sites with public software implementations of the standards is included in Appendix C.

The material in this book is partitioned into the following categories: fun­damenta1s, video standards. architectures. audio standards. and emerging techniques. However. most of the chapters are self contained and allow the reader to easily switch from one category to another.

This book is aimed at the professional who wants to have a basic understanding of the latest developments and applications of image compression standards. It is based on tutorials given by the authors at several forums. including the IEEE International Conference on Acoustics Speech and Signal Processing. the ACM Multimedia Conference, and Hewlett-Packard. It provides a reference for any engineer planning to work in this field, either in basic implementation or in research and development. It is intended for self-study; however, it can also be used as a companion textbook in any course on data compression, video coding. computer architecture. signal or image processing. or the design of signal and video processors.

New topics in the second edition: Since the publication of the first edition. we have seen a number of new developments in standards-related activities. The original JPEG and MPEG standards have been refined, and several new compression standards have emerged. In 10ssless compression, we introduce IPEG-LS. a new standard for lossless and near-Iossless compression of still-images. In video coding, we describe two new MPEG-2 profiles: 4:2:2 and multiview. Activities in two new MPEG standards. MPEG-4 and MPEG-7, are also discussed here. In videoconferencing. we describe the new H.263 standard and compare it with H.261. New developments in fast motion-

XVI IMAGE AND VIDEO COMPRESSION STANDARDS

estimation algorithms. a major component of all MPEG and videoconferencing standards, are also presented. In audio coding, new standards. such as MPEG-2 AAe and G.723.1, are also discussed. Since the flrst edition, there have been significant advances in the hardware support for multimedia and compression standards. We describe enhancements to several general purpose processors, including the Intel MMX and the newly emerging class of media processors. The information on commercially available audio and video coders has also been updated.

Acknowledgments: This book would Dot be possible without the support from the management at Hewlett-Packard Laboratories, especially Fred L. Kitson, Daniel T. Lee, and Ho John Lee. We also thank our col1eagues at Hewlett­Packard for many helpful discussions. Balas K. Natarajan has been particularly helpful from the beginning of this project and we thank him for many valuable discussions. comments. and for reviewing parts of this manuscript. The review of our sections on Golomb-Rice coding and JPEG-LS by Guillermo Sapiro and Gadiel Seroussi is gratefully acknowledged. We thank Bob Rau for his comments on our tutorials and Ruby Lee for providing us with a different perspective on the hardware implementation of the video standards. We thank Nariman Farvardin and David Daut for their comments on the computation of the rate-distortion function. We thank Akio Yamamoto for his help in the example on wavelet coding, Jose Fridman for many valuable comments and suggestions, Masoud Khansari for his comments on the video teleconferencing standards. and Elias Manolakos. Feedback from the readers of the first edition of the book has been extremely valuable in helping us to improve the clarity of the presentation and to select new material. We are grateful to all of them, and especially to Thanos Skordas and Aggelos Katsaggelos, for their suggestions and conections.

We also thank Robert Holland. Je. and his staff from Kluwer Academic Press for their support, and Suzanne M. Rumsey for helping with many typesetting questions. Finally. we dedicate this book to our teachers and to our families, and especially to Achan. Chechi, Balu, Kishore. Shanti, Paula, Anastasia, Anneta, and Niki.