structure and texture filling-in of missing image blocks in

296 IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 12, NO. 3, MARCH 2003

Structure and Texture Filling-In of MissingImage Blocks in Wireless Transmission and

Compression ApplicationsShantanu D. Rane, Guillermo Sapiro, and Marcelo Bertalmio

Abstract—An approach for filling-in blocks of missing datain wireless image transmission is presented in this paper. Whencompression algorithms such as JPEG are used as part of thewireless transmission process, images are first tiled into blocksof 8 8 pixels. When such images are transmitted over fadingchannels, the effects of noise can destroy entire blocks of the image.Instead of using common retransmission query protocols, we aimto reconstruct the lost data using correlation between the lostblock and its neighbors. If the lost block contained structure, it isreconstructed using an image inpainting algorithm, while texturesynthesis is used for the textured blocks. The switch between thetwo schemes is done in a fully automatic fashion based on thesurrounding available blocks. The performance of this methodis tested for various images and combinations of lost blocks. Theviability of this method for image compression, in association withlossy JPEG, is also discussed.

Index Terms—Compression, filling-in, inpainting, interpolation,JPEG, restoration, texture synthesis, wireless transmission.

I. INTRODUCTION

GENERAL purpose images are most commonly com-pressed by lossy JPEG. JPEG divides the image into

blocks of 8 8 pixels and calculates a two-dimensional (2-D)discrete cosine transform (DCT), followed by quantization andHuffman encoding; see [1]. In common wireless scenarios,the image is transmitted over the wireless channel block byblock. Due to severe fading, we may lose an entire block,even several consecutive blocks of an image. In [2] the authorsreport that average packet loss rate in a wireless environmentis 3.6% and occurs in a bursty fashion. In the worst case, awhole line of image blocks might be lost. Note that JPEGuses differential encoding for storing the average (dc) valueof successive pixels. Hence, even if a single block is lost,the remaining blocks in that line (or reset interval) might be

Manuscript received May 1, 2001; revised June 13, 2002. This workwas supported in part by the Office of Naval Research under grantONR-N00014-97-1-0509, the Office of Naval Research Young InvestigatorAward, the Presidential Early Career Awards for Scientists and Engineers(PECASE), a National Science Foundation CAREER Award, and the NationalScience Foundation Learning and Intelligent Systems Program (LIS). Theassociate editor coordinating the review of this manuscript and approving it forpublication was Prof. Timothy J. Schulz.

S. D. Rane and G. Sapiro are with the Department of Electrical and ComputerEngineering, University of Minnesota, Minneapolis, MN 55455 USA (e-mail:[email protected]).

M. Bertalmio was with the Department of Electrical and Computer Engi-neering, University of Minnesota, Minneapolis, MN 55455 USA. He is now withthe Dipartimento de Tecnologia, University of Pompeu-Fabra, 08003 Barcelona,Spain.

Digital Object Identifier 10.1109/TIP.2002.804264

received without their correct average (dc) value. Two commontechniques to make the transmission robust are forward errorcorrection (FEC) and automatic retransmission query protocols(ARQ). Of these, FEC needs extra error correction packets tobe transmitted. As noted in [3], ARQ lowers data transmissionrates and can further increase the network congestion whichinitially induced the packet loss. Instead, we show that it ispossible to satisfactorily reconstruct the lost blocks by using theavailable information surrounding them.1 This will result in anincrease in bandwidth efficiency of the transmission. The basicidea is to first automatically classify the block astexturedorstructured(containing edges), and then fill-in the missing blockwith information propagated from the surrounding pixels. Inthe case of structured blocks, the inpainting algorithm in [4]is used, while for textured regions we follow [5].2 We test theproposed scheme with a variety of images and simulated blocklosses. We also combine this approach with JPEG compressionitself, where the encoder voluntarily skips blocks, and theseare reconstructed at the decoder in the same fashion as in thewireless scenario. This process improves the compression ratio,at little or no quality degradation.

II. PREVIOUS RELATED WORK

Most schemes reported in the literature deal with image trans-mission in error-prone environments using a combination ofsource and channel coding. The authors in [2] describe a pack-etization scheme in which the DCT coefficients array generatedby JPEG is grouped such that bursty (consecutive) packet lossduring transmission is scattered into a pseudo-random loss inthe image domain (i.e., consecutive blocks are rarely lost inthe image domain). The ensuing reconstruction scheme bene-fits because, most frequency components can be recovered fromadjacent blocks. However, large bursts may cause the errorsto cluster in the image, and reconstruction suffers. It shouldbe noted that the packetization scheme proposed in [2], whenused with the reconstruction scheme described in our paper, isexpected to further improve on the results reported here, andprovide satisfactory reconstruction results even for very largebursts.

1The location of lost data, that is, lost image blocks, is known in commonwireless scenarios.

2Other algorithms could be used as well, e.g., [6]–[10], but the ones we usewere shown in the literature to produce state-of-the-art results at an acceptablelow computational cost.

1057-7149/03$17.00 © 2003 IEEE

RANE et al.: STRUCTURE AND TEXTURE FILLING-IN OF MISSING IMAGE BLOCKS 297

Fig. 1. Texture synthesis procedure.

The authors in [3] also note that interleaving the image databefore packetization avoids loss of contiguous areas in an image,facilitating reconstruction. This paper demonstrates reconstruc-tion in the transform domain by expressing the lost data as alinear combination of blocks in the 4-neighborhood of the lostblock. Four optimal weights (coefficients) need to be calculatedper block based on combinations of available adjacent blocks.These weights, which result in a 10% space overhead, are usedlater in reconstruction. Strong diagonal edges are not well re-constructed by this method, as explained in [3].

Additional work on the reconstruction of missing data inblock-based compression schemes is reported in [11], wherethe DCT coefficients of a missing block are interpolated fromthose with the same position in the neighboring blocks.

The novelty of our proposed scheme is in the separation ofthe lost blocks into different classes, followed by the use ofstate-of-the-art image filling-in algorithms for textured andstructured regions. This is done in a complete automatic fashionand without any side information.

III. PROPOSEDALGORITHM

The reconstruction of lost blocks follows three computation-ally efficient steps:3

a) classify lost blocks into texture and structure;b) synthesize blocks which were classified as texture (use

texture synthesis);c) fill in blocks which were classified as structure (use image

inpainting).

We now proceed to describe each one of these components.

A. Block Classification

The first step in the reconstruction is to classify the lost blocksinto texture or structure. This decision is taken at the receiverby querying the region surrounding the lost block. Lost blocksare, of course, excluded from the querying process. (Alterna-tively, we may perform this procedure at the transmitter, and

3Recall that the missing blocks locations are given.

then transmit one bit per block, indicating the presence of tex-ture or structure. This entails the overhead of one extra bit perblock.) At the core of this classification, is the method proposedin [12],4 which inspired the approach here presented. To de-termine whether or not a block has texture (or noise), we usea simple coarseness measure given by the number of local ex-trema in the neighborhood of the lost block. The number of localextrema are simply the pixels which are local row extrema aswell as local column extrema.

Using the method of [12], the number of local extrema in awindow of side is given by

(1)

where and are respectively the upper and lower boundsfor texture coarseness and are selected by the user. These coarse-ness values vary from 0 (no extrema) to 1 (all pixels in the se-lected window are extrema). We have used and

as suggested in [12]. In this implementation, ,giving . Thus, if a 8 8 block has fewer than extrema,it is classified to have structure, else is considered to contain tex-ture (which includes noisy blocks).

The above technique is applied for each available block of8 8 pixels in the immediate neighborhood of the lost block.Even if a single block from this neighborhood contains struc-ture, we first consider a decision in favor of structure. However,reconstruction being our primary goal, this criteria alone mightbe insufficient, as we illustrate now. Consider for example thatwe have lost a block containing an edge between two texturedregions. The edge between two regions is certainly an expres-sion of structure, and needs to be given precedence over tex-ture even if the block in question has more than the necessarycoarseness. The logic behind this will be understood in the nextsection, wherein, we require the textured region surrounding theblock to fill it up. If we were to classify a block containing anedge as texture, we would not be able to reconstruct the edgelater, as will become clear after examining the texture synthesisalgorithm.

To overcome this limitation, we impose an additional con-straint as follows. We consider the 8-neighborhood of a 88block and calculate differences between the average values ofthe blocks on opposite sides of the center block (consideringonly available blocks). If the four resulting differences are abovea threshold, we decide that an edge does indeed pass throughthe textured block. We then designate the block as structure,notwithstanding its high coarseness. This simple additional con-straint has provided a correct classification in all tested images.

B. Texture Synthesis

From the earlier classification, we conclude that when a blockis classified as having texture, the entire 8-neighborhood of thatblock has texture. The missing block is then filled-in with thetexture from its surrounding, following [5].

Let the region to be filled be denoted by. The lost blockwill now be filled, pixel by pixel, in a raster fashion. Let be a

4A similar method was used by the research group at the Navy facilities inChina Lake (C. Schwartz, personal communication).


Fig. 2. Reconstruction results for 1, 2, or 3 contiguous lost blocks.

Fig. 3. Reconstruction results for more drastic losses.

representative template touching the left of a pixel .We proceed to find a from the available neighborhood, suchthat a given distance is minimized. As per [5], is anormalized sum of squared differences (SSD) metric. Once sucha is found, we choose the pixel to the immediate right of,as our candidate for . For stochastic textures, thealgorithm selects at random one of the pixels neighboring.

The template can be a simple seed-block of 33 pixelsas shown in Fig. 1. Then, of all possible 33 blocks in the8-neighborhood, the one with the minimum normalized SSD isfound and a pixel to its right is copied into the current pixel in thelost block, as shown. This algorithm is considerably fast whenusing the improvements in [13], [14].

C. Image Inpainting

Structure in an image can be an edge between two regions ora deterministic change in color or gray value. When the blockclassification algorithm detected a structured block, this is re-stored using the digital inpainting procedure introduced in [4].

Once again let be the region to be filled in (inpainted) andbe its boundary. The basic idea in inpainting is to smoothly

propagate the information surroundingin the direction of theisophotes entering . Both gray values and isophote directionsare propagated inside the region. Denoting bythe image, thispropagation is achieved by numerically solving the partial dif-ferential equation (is an artificial time marching parameter)

where , , and stand for the gradient, Laplacian, and or-thogonal-gradient (isophote direction) respectively. This equa-tion is solved only inside , with proper boundary conditions in

for the gray values and isophote directions.Note that at steady state, , and .

This means that is constant in the direction of theisophotes, thereby achieving a smooth continuation of theLaplacian inside the region to be inpainted.


Fig. 4. Reconstruction misses details smaller than the mask.

For details on the numerical implementation of this in-painting technique, which follows the techniques introducedin [15], [16], as well as numerous examples and applications,see [4].

IV. EXPERIMENTAL RESULTS

A. Application to Wireless Transmission of JPEG CompressedImages

Throughout our experiments, we have assumed, though it isnot completely necessary, that the average (dc) value of a 88block is known at the receiver. This is not needed by the blockreconstruction algorithm, but it is needed for the available sur-rounding blocks due to the differential encoding of dc values inJPEG. In case the dc value is not known, the method proposedin [17] gives a technique to estimate and correct the dc value ofthe lost block and the following blocks in the same line. Retrans-mission of the dc value of a lost block would not be a significantoverhead either, since we just need 1 byte to be retransmitted perblock.

Since we have no control over the fading channel, there isno prior information about the relative locations and number ofblocks that can be lost in the process. We present various exam-ples ranging from low to drastic losses of image information,and demonstrate our proposed technique to restore the lost infor-mation. Figs. 2–4 show the results of reconstruction (from leftto right, transmitted, received, and reconstructed image). Table Ishows the amount of missing data along with the PSNR valuesafter reconstruction. We can make the following observations.

a) When single blocks are missing from the image, they aresatisfactorily reconstructed from the surrounding context. Notehow the reconstructed image is almost identical to the originalone, as expected from the algorithms used for filling-in.

b) When a few contiguous blocks are missing, the algorithmstill reconstructs the blocks so as to be visually unrecognizablefrom the original.

c) As a drastic condition, if an entire line is missing, then,a good reconstruction is not always possible. See Fig. 3. This

TABLE IPERCENTDATA LOST AND PSNR VALUES AFTERRECONSTRUCTION

is mainly due to the fact that most probably such a significantloss will cover entire objects. In general, when feature sizes aresmaller than 8 8 pixels or are totally covered by the missingline, it will be impossible to reconstruct the image correctly. SeeFig. 4. In such cases, we will be forced to request retransmissionof the lost block (or an error block). Such instances will be-come increasingly rare when the image resolution is increased.Further, as explained in Section II, if the packetization in [2] isused, then it is extremely rare to lose an entire line in the imagedomain. In that case, only independent lost blocks would needto be reconstructed from their neighborhood, and as indicatedby Figs. 2 and 3, the above algorithm restores isolated blocksreliably.

B. Application to Image Compression

Since inpainting faithfully reproduces lost edges, and texturesynthesis faithfully grows lost textures, we can afford to volun-tarily remove some blocks containing structure and texture, evenprior to compression. Afterintentionally removing the blocksand replacing them with their dc values, we compress the imageusing lossy JPEG with default settings. Finally, when the imageis decompressed, the above algorithm is used to reconstruct the“lost” blocks.5 This improves the compression ratio providedby lossy JPEG. In this situation, we have the freedom to choosethe blocks which we want to remove, and later reconstruct. A

5As in the wireless scenario, we assume that the positions of the lost blocksare known, since many simple strategies can be devised to automatically detectthis position (e.g., blocks with only dc as nonzero coefficients). We also auto-matically detect the block type, as before, so no overhead is needed at all.


Fig. 5. Stages in compression of 256� 256 Lena image.

common fixed mask to remove blocks from all images is notadvisable of course, since for example, it may totally obliterateimportant details in images where feature sizes are less than 8

8 pixels, and hence cannot be reconstructed later.The objective is to retain only the information whichcannot

be correctly reconstructed (minute but important details) and toremove as much as possible from the remainder of the image.The problem thus reduces to finding the best possible mask fora given image. Following the earlier distinction between textureand structure, our algorithm uses the following general rules toconstruct the mask automatically (see also concluding remarks):

a) For areas of texture, remove as many blocks as possible,since they can be satisfactorily reconstructed from a single seedblock.

b) Remove a few, but not all blocks along an edge, so thatthe direction of the edge is properly preserved. Ideally, alternateblocks along an edge should be removed. Presently, a block con-taining an edge is masked only if the regions on either side ofthe edge are flat, i.e., the gradient is steep on both sides of theblock. This is done, because, while experimenting with variouscases in Section IV-A, it became clear that such edges are bestreconstructed by inpainting.

c) For blocks with smooth variations (i.e., structure withoutedges), remove alternate blocks. Inpainting always restoresthese smooth variations.

Fig. 5 shows how the algorithm is used in conjunction withlossy JPEG. Note that the image quality is very similar to thecase in which JPEG alone is used. Other examples are shown


Fig. 6. Other examples (1)—Result after decompression of 512� 512 peppers image.

Fig. 7. Other examples (2)—Result after decompression of 512� 512 jet image.

TABLE IIIMPROVEMENT IN COMPRESSIONRATIO

in Figs. 6 and 7. The mask is found automatically by the algo-rithm from the image and the output of a Canny edge detector.

The bits per pixel (BPP) and compression ratio (CR) obtainedby lossy JPEG alone, and the compression ratio obtained usingJPEG with the above algorithm, are shown in Table II. Note onceagain that no overhead is needed by the proposed compressionstrategy, and the bitstream it is fully JPEG compliant.

V. CONCLUSIONS ANDFUTURE DIRECTIONS

In this paper we have proposed a new technique for thefilling-in of missing blocks in wireless transmission of JPEG(or block based) compressed images. We have shown that aslong as the features in the image are not completely lost, theycan be satisfactorily reconstructed using a combination of com-putationally efficient image inpainting and texture synthesisalgorithms. This eliminates the need for retransmission of lost


Fig. 8. Using information from blue component to reconstruct lost blocks in red component.

blocks. When the image resolution is increased, the quality ofreconstruction improves and a retransmission request is rarelyrequired, resulting in a better effective data transmission rate.

Further, by intentionally (and automatically) dropping imageblocks, and using this filling-in approach, we can improve thecompression ratio provided by lossy JPEG, without altering theexisting JPEG algorithm. As seen in Table II, the improvementin compression ratio becomes more significant as the image res-olution is increased.

A number of research directions should be taken followingthe results reported here. We have tried to use image-dependentinformation, i.e., texture and structure, to enhance the perfor-mance of JPEG. The compression ratio can be further increasedby finding bettermasks by providing more image information.

In a more general setting, the extension of the approach pre-sented here, to color data needs to be investigated. Since themissing blocks in the different channels need not be in the sameimage position, information from different channels can be usedin the block classification and reconstruction. Adding this to thecurrent neighboring information used is expected to improveeven further the quality of the results. A preliminary exampleis presented in Fig. 8. Further results on this will be reportedelsewhere.

REFERENCES

[1] G. K. Wallace, “The JPEG still picture compression standard,”Commun.ACM, vol. 34, no. 4, pp. 30–44, 1991.


[2] E. Chang, “An image coding and reconstruction scheme for mobile com-puting,” in Proc. 5th IDMS, Oslo, Norway, Sept. 1998, pp. 137–148.

[3] S. S. Hemami, “Digital image coding for robust multimedia transmis-sion,” in Proc. Symp. Multimedia Communications and Video Coding,New York, 1995.

[4] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image in-painting,” in Computer Graphics (SIGGRAPH 2000), July 2000, pp.417–424.

[5] A. A. Efros and T. K. Leung, “Texture synthesis by nonparametric sam-pling,” in IEEE Int. Conf. Computer Vision, Corfu, Greece, Sept. 1999,pp. 1033–1038.

[6] C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera,“Filling-in by joint interpolation of vector fields and gray levels,”IEEETrans. Image Processing, to be published.

[7] T. Chan and J. Shen, “Mathematical models for local deterministic in-paintings,”UCLA CAM Rep., Mar. 2000.

[8] D. Heeger and J. Bergen, “Pyramid based texture analysis/synthesis,” inProc. SIGGRAPH 1995, July 1995, pp. 229–238.

[9] S. Masnou and J. Morel, “Level-lines based disocclusion,” inIEEE Int.Conf. Image Processing, Oct. 1998.

[10] E. Simoncelli and J. Portilla, “Texture characterization via joint statisticsof wavelet coefficient magnitudes,” inProc. 5th IEEE Int. Conf. ImageProcessing, 1998.

[11] Z. Alkachouch and M. Bellanger, “Fast dct-based spatial domain inter-polation of blocks in images,”IEEE Trans. Image Processing, vol. 9, pp.729–732, Apr. 2000.

[12] K. Karu, A. K. Jain, and R. M. Bolle, “Is there any texture in the image?,”Pattern Recognit., vol. 29, no. 9, pp. 1437–1446, 1996.

[13] G. Gorla, V. Interrante, and G. Sapiro. (2001, Feb.) Growing fitted tex-tures, IMA Tech. Rep.. [Online]. Available: http://www.ima.umn.edu.

[14] L. Y. Wei and M. Levoy, “Fast texture synthesis using tree-structuredvector quantization,” inProc. SIGGRAPH 2000, July 2000.

[15] A. Marquina and S. Osher, “Explicit algorithms for a new time depen-dent model based on level set motion for nonlinear debluring and noiseremoval,”UCLA CAM Rep., Jan. 1999.

[16] L. Rudin, S. Osher, and E. Fatemi, “Nonlinear total variation based noiseremoval algorithms,”Phys. D, no. 60, pp. 259–268, 1992.

[17] S. Shirani, F. Kossentini, and R. Ward, “Reconstruction of baseline JPEGcoded images in error prone environments,”IEEE Trans. Image Pro-cessing, vol. 9, pp. 1292–1299, July 2000.

Shantanu D. Ranereceived the B.E degree in in-strumentation engineering from the Government Col-lege of Engineering, Pune University, India, in 1999and the M.S degree in electrical engineering from theUniversity of Minnesota, Minneapolis, in 2001. Atpresent, he is pursuing the Ph.D degree in electricalengineering at Stanford University, Stanford, CA.

He was a Project Student at the National Centerfor Radio Astrophysics, Tata Institute of Funda-mental Research, Pune, in 1998-1999 and workedon the development of a feed-positioning system

for antennas of the Giant Meterwave Radio Telescope. His current researchinterests include coding, transmission and reconstruction of images and video.

Mr. Rane is the recipient of the Gold Medal for the best performance in un-dergraduate studies in Instrumentation Engineering at Pune University (1999).

Guillermo Sapiro was born in Montevideo,Uruguay, on April 3, 1966. He received the B.Sc.(summa cum laude), M.Sc., and Ph.D. degrees fromthe Department of Electrical Engineering at theTechnion—Israel Institute of Technology, Haifa, in1989, 1991, and 1993 respectively.

After postdoctoral research at MIT, he becameMember of Technical Staff at the research facilitiesof HP Labs, Palo Alto, CA. He is currently with theDepartment of Electrical and Computer Engineeringat the University of Minnesota, Minneapolis. He

works on differential geometry and geometric partial differential equations,both in theory and applications in computer vision, computer graphics, medicalimaging, and image analysis. He co-edited a special issue of theJournalof Visual Communication and Image Representation. He has authored andco-authored numerous papers in this area and has written a book published byCambridge University Press, January 2001.

Dr. Sapiro co-edited a special issue of the IEEE TRANSACTIONS ONIMAGE

PROCESSING. He was awarded the Gutwirth Scholarship for Special Excellencein Graduate Studies in 1991, the Ollendorff Fellowship for Excellence in Vi-sion and Image Understanding Work in 1992, the Rothschild Fellowship forPost-Doctoral Studies in 1993, the Office of Naval Research Young InvestigatorAward in 1998, the Presidential Early Career Awards for Scientist and Engineers(PECASE) in 1988, and the National Science Foundation Career Award in 1999.He is a member of SIAM.

Marcelo Bertalmio was born in Montevideo,Uruguay, in 1972. He received the B.Sc andM.Sc degrees in electrical engineering from theUniversidad de la República, Uruguay, in 1996 and1998 respectively, and the Ph.D. degree from theUniversity of Minnesota, Minneapolis, in 2001.

He is currently an Associate Professor at Univer-sidad Pompeu Fabra, Spain.

Dr. Bertalmio was been awarded the Femlab Prize2002 for research done as a student in scientific fieldsrelated to Prof. Stan Osher contributions, the Siemens

“Excellent Student Paper Award” at the 2001 IEEE Workshop on Variational andLevel Set Methods in Computer Vision, the 2001 Programa Ramón y Cajal bythe Spanish Ministry of Science, and several Fellowships and Scholarships inthe U.S. and Uruguay.

structure and texture filling-in of missing image blocks in

Documents