Full Terms & Conditions of access and use can be found athttp://www.tandfonline.com/action/journalInformation?journalCode=tadr20

Advanced Robotics

ISSN: 0169-1864 (Print) 1568-5535 (Online) Journal homepage: http://www.tandfonline.com/loi/tadr20

Range image smoothing and completion utilizinglaser intensity

Shuji Oishi , Ryo Kurazume , Yumi Iwashita & Tsutomu Hasegawa

To cite this article: Shuji Oishi , Ryo Kurazume , Yumi Iwashita & Tsutomu Hasegawa (2013)Range image smoothing and completion utilizing laser intensity, Advanced Robotics, 27:12,947-958, DOI: 10.1080/01691864.2013.797141

To link to this article: https://doi.org/10.1080/01691864.2013.797141

Published online: 10 May 2013.

Submit your article to this journal

Article views: 118

Citing articles: 2 View citing articles

FULL PAPER

Range image smoothing and completion utilizing laser intensity

Shuji Oishia*, Ryo Kurazumeb, Yumi Iwashitab and Tsutomu Hasegawac

aGraduate School of Information Science and Electrical Engineering, Kyushu University, 744 Motooka Nishi-ku,Fukuoka 819-0395, Japan; bGraduate Faculty of Information Science and Electrical Engineering, Kyushu University, Japan;

cKumamoto National College of Technology, Japan

(Received 1 October 2012; accepted 6 November 2012)

In this paper, we propose new denoising techniques for a deteriorated range image taken by a laser scanner. Laser scanneracquires a range value from the scanner to the target by measuring the round-trip time of the emitted laser pulse. At the sametime, they can obtain the strength of the reflected light as a side product of the range value. Focusing on the laser intensity,we propose two denoising techniques for a deteriorated range image utilizing the intensity image: smoothing by extendedbilateral filter, and completion by belief propagation. The extended bilateral filter makes use of laser intensity in addition tothe spatial and range information in order that we can smooth a range image corrupted by noises while the geometric fea-tures such as jump and roof edges are preserved. The range image completion technique with belief propagation restores adeteriorated range image using the adjacent range values and the corresponding intensity values simultaneously. We conductsimulations and experiments using synthesized images and actual range images taken by a laser scanner and verify that theproposed techniques suppress noise while preserving jump and roof edges and repair deteriorated range images.

Keywords: laser scanner; range image; intensity image; extended bilateral filter; belief propagation

1. Introduction

In recent years, low-cost, real-time time-of-flight(RT-TOF) laser sensors have been released, includingSwissRanger SR4100 (MESA Imaging AG), D-imager(Panasonic), and Canesta Vision (Canesta, Inc.).[1] Thesesensors capture range images of approximately 150–200pixels by 150–200 pixels (20–40 thousand pixels) at20–50Hz and are applicable to various applications, suchas robot navigation, vehicle control, and intuitive humaninterfaces. However, range data acquired by RT-TOFsensors contain large errors in distance measurement,and the spatial resolution is also insufficient comparedwith modern digital cameras.

On the other hand, high-precision three-dimensional(3D) laser scanners, such as RIEGL VZ-400 (RIEGLGmbH), Leica Scan Station 2 (Leica Geosystems AG),and TOPCON GLS-1500 (TOPCON) have been widelyused for landscape surveying or digital 3D modeling. Inaddition, low-cost, high-resolution laser measurementsystems using two-dimensional laser scanners (SICKLMS151 (SICK AG) and HOKUYO TOP-URG(HOKUYO)) and a rotary table have been proposed for3D environmental map building for mobile robotnavigation.[2] These LIDAR (light detection and ranging)sensors acquire high resolution and precise range images.However, range images often suffer from noise due to the

reflectance property of objects’ surfaces or electrical andmechanical disturbances. For example, one sigma accu-racy of RIEGL VZ-400 is 3mm per 100 meters, thus, aflat surface is measured as a slightly uneven plate. Metalsurface with a strong specular reflection or black colorcannot be measured by standard laser scanners. There-fore, denoising techniques for range images taken bylaser scanners still remain as a critical problem.

Several approaches can be used to denoise the rangeimages captured by range sensors:

(1) Averaging a series of range images of the samescene (temporal smoothing).

(2) Applying spatial smoothing filters such as aGaussian filter (spatial smoothing).[3–7]

(3) Combining range data with other informationsuch as texture or brightness.[8–11]

Temporal smoothing is an intuitive and fundamentaltechnique for denoising a range image and is widelyused in high-precision laser scanners. However, this pro-cess must be performed for each measured point (pixel)during the measurement and the processing time is pro-portional to the number of images for averaging.

On the other hand, spatial smoothing can be appliedoff-line and is applicable to not only measured pointsbut also structured meshes. In this technique, the range

*Corresponding author. Email: oishi@irvs.ait.kyushu-u.ac.jp

Advanced Robotics, 2013Vol. 27, No. 12, 947–958, http://dx.doi.org/10.1080/01691864.2013.797141

� 2013 Taylor & Francis and The Robotics Society of Japan

values of adjacent points (pixels in a range image or ver-texes in structured meshes) are spatially convolved byapplying a spatial convolution filter such as a median fil-ter or a Gaussian filter.

In the present paper, we propose two denoisingtechniques which can be categorized into the thirdcategory mentioned above by focusing on the laserintensity.[12] When we measure range data by laserscanners, the intensity, which indicates the strength ofthe reflected light, can be obtained as a by-product ofrange data. Note that all pixels in the range image havecorresponding intensity values. In other words, therange image and the intensity image are precisely andfundamentally aligned.

Using the intensity image, we first propose a newsmoothing technique using the extended bilateral filterand intensity as a denoising technique of an image. Inthe proposed method, the extended bilateral filter isapplied for not only the range image but also the corre-sponding intensity image. By taking account of the prop-erties of range and intensity images, the proposedmethod can smooth range images, while preserving geo-metric features such as jump and roof edges. Adelsbergeret al. [13] proposed a similar technique for low-resolu-tion infrared TOF sensors (SwissRanger, MESA-Imag-ing). However, the detailed discussion for denoisingperformance has not been presented.

Next, we propose a new inpainting technique of arange image using an intensity image and belief propaga-tion. In this method, the deteriorated range values in arange image are recovered using not only the adjacentrange values but also the continuity of the intensity image.

In Section 2, an overview of the previous approacheswill be presented. In Sections 3 and 4, we will proposetwo new denoising techniques for range images usingintensity images, that is, range image smoothing by theextended bilateral filter and range image inpainting bybelief propagation. In Section 5, simulations and experi-ments using a laser scanner will be reported for the pur-pose of verifying the performance of the proposedtechniques.

2. Related research

Smoothing techniques for range images are classifiedinto two categories: pixel-based or point-basedtechniques,[7,8,14,15,10,11] and mesh-based techniques.[4–6,9] Raw range data acquired by range sensors iscomposed of a group of 3D points called a point-cloud.Pixel or point-based methods denoise the range image orthe point cloud directly without taking the continuity ofpixels into account explicitly. On the other hand, mesh-based methods are applied to structured meshes such astriangular patches by considering the continuity of thevertexes in the structured meshes.

For the case in which a high-resolution gray-scaleimage and a low-resolution range image are simulta-neously captured from a range sensor, Diebel et al. [8]proposed a technique for estimating high-resolutionrange images by considering the Markov Random Fieldin high- and low-resolution range images and adjustingsmoothing parameters according to the gradient of thehigh-resolution gray-scale image. Crabb et al. [16] andChan et al. [14] also proposed upsampling techniquesusing the joint bilateral filter.[17] Bohme et al. [9] pro-posed the denoising technique for a range image usingthe shape-from-shading technique.[18] They introducedan energy function consisting of the difference of theobserved laser intensity and its estimation based onthe Lambertian reflectance model and the continuity ofthe range image and intensity image. Then, the energyfunction is minimized by the nonlinear conjugate gradi-ent method so that the noise in the range image is sup-pressed. Using the range information from a rangescanner and the information of normal directions fromphotometric stereo, Nehab et al. [10] and Okatani et al.[11] proposed denoising techniques for reconstructing a3D geometric model precisely. Nehab et al. [10] refinesthe bias in the measured normal direction at first, andthen optimizes the 3D model so that the estimated nor-mal direction fits the 3D model. Okatani et al. [11] esti-mates the shape of the object by integrating the surfacenormal and 3D data based on probabilistic framework.

On the other hand, several techniques based on thebilateral filter,[19] which was developed as an edge-preserving filter for gray-scale images, have beenproposed.[5–7,20]. Fleishman et al. [6] proposed a 3Dedge preserving filter by applying the bilateral filter forthe distance from a point to its adjacent points projectedon a tangential plane (tangential component) and the dis-tance from the adjacent points to the tangential plane(normal component). Jones et al. [5] proposed a similartechnique using triangular meshes instead of tangentialplanes. However, these smoothing techniques are appliedafter converting from the point cloud to the meshes andit is difficult to obtain the normal vectors stably frommeshes that contain a great deal of noise. Moreover, insome cases, the construction of structured meshes from anoisy point cloud is not a simple and trivial problem.

Miropolsky [7] proposed the geometric bilateral filterwhich used the distances from the adjacent points andthe difference of normal directions for each point in thepoint cloud. However, a stable solution of normal vectorsfrom a noisy point cloud has not yet been found.

While these methods can be considered as a simpleextension of the bilateral filter for a gray-scale image toa range image, the technique proposed herein uses aintensity image that corresponds one-to-one to the rangeimage for smoothing the range image. Note that althoughwe assume that the range and intensity images have the

948 S. Oishi et al.

same resolution, the proposed method can be applied toimages having various resolutions using the joint bilat-eral filter.[17] On the other hand, similar denoising tech-niques for camera images that extend the bilateral filterto other domains have been proposed.[21–24] Eisemannet al. [21] and Petschnigg et al. [22] proposed flash pho-tography techniques. In their techniques, an image,which was taken under dark illumination and suffersfrom noise or blur, is reconstructed using itself andanother image taken in the same position with flash light.Focusing on the fact that the flash image has betterhigh-frequency information than the no-flash image, theyproposed cross-bilateral filter and joint bilateral filter,respectively. These filters had edge-preserving weightingfunctions based on the differences of pixel intensities inthe flash image in order to obtain the detail shapes.Benett et al. [23] and Weber et al. [24] proposed similarbilateral filtering techniques for each image in a videoand animation. They extended the bilateral filter to timedomain and denoised each image utilizing the informa-tion of adjacent frames.

For the case in which there are several holes in rangedata due to the occlusion or specular or weak reflection,Kawai et al. [25] proposed a completion technique of the3D surfaces. They define an energy function based onthe similarity of shapes and select the best match thatminimizes the energy function to fill in the holes of 3Dgeometry. Becker et al. [26] proposed a completionmethod using an additional color image of the samescene from a different viewpoint. Xu et al. [27] also pro-posed a technique which estimated missing geometry bylearning the association of surface normals to imagepatches in calibrated images.

On the other hand, several inpainting techniquesbased on belief propagation have been proposed.[28,29].Pedro et al. [28] proposed an image completion methodwhich took the continuity of pixels into account byapplying belief propagation. Komodakis et al. [29]proposed an exemplar-based inpainting technique usingbelief propagation. They introduced the Priority-BP thatextended standard belief propagation for priority-basedmessage scheduling and dynamic label pruning.

3. Smoothing range image using the extendedbilateral filter

In this section, we propose a new technique for smooth-ing a range image using the extended bilateral filter andan intensity image.

As mentioned above, the conventional smoothingtechniques for range data are mainly applied for a rangeimage directly. On the other hand, we focus on an inten-sity image that is acquired as a by-product of the rangeimage for most of the range sensors. By taking the prop-erties of both the range and intensity images into

account, the proposed technique can suppress noise in arange image while preserving geometric features such asjump and roof edges.

In the following sections, we introduce the conven-tional bilateral filter and the intensity image captured byrange sensors. The extended bilateral filter using inten-sity image is then described in detail.

3.1. Intensity image

Optical range sensors, such as a laser scanner, obtainrange data by measuring the round-trip time of a laserpulse reflected by an object. Figure 2(a) shows an exam-ple of a range image acquired by a 3D laser scanner(Figure 1[2]). On the other hand, most optical range sen-sors can measure the strength of the reflected laser pulse(laser intensity).

Laser intensity value indicates an intensity on thesurface point of the target under a single-frequency lightsource. Based on the dichromatic reflection model, theintensity value consists of a diffuse and specular reflec-tion. Assuming that the specular component can beignored and Lambertian reflection is used as a model fordiffuse reflection, the intensity value is expressed asfollows [9]:

I ¼ kdIqr2cosa (1)

where kd is the diffuse reflection coefficient, Iq is thepower of the light source, r is the distance of the lightsource toward the target, and a is the incident angleagainst the surface normal. Figure 2(b) shows an inten-sity image that depicts intensity values as a gray-scaleimage. As mentioned above, a unique intensity value isdetermined for each pixel in the range image. In otherwords, the range image and the intensity image are pre-cisely and fundamentally aligned.

3.2. Extension of the bilateral filter to a range image

The bilateral filter [19] is an edge-preserving smoothingfilter that extends the Gaussian filter so that not only thespatial relation but also the variation of the pixel inten-sity is considered. Let us apply the bilateral filter for agray-scale image to a range image. The bilateral filter fora range image can be defined as follows:

gi ¼P

j wx(xi; xj)wf ( fi ; fj)fiPj wx(xi; xj)wf ( fi; fj)

(2)

wx(xi; xj) ¼ 1ffiffiffiffiffiffiffiffiffiffi2prx

p e�jxi � xjj2

2r2x (3)

Advanced Robotics 949

wf ( fi ; fj) ¼ 1ffiffiffiffiffiffiffiffiffiffi2prf

p e�jfi � fj j22r2

f (4)

where gi and fi are new and original range values atpixel i, and wx(xi; xj) and wf ( fi; fj) are Gaussian functionsfor two-dimensional spatial and range information withvariances of r2x and r2f , respectively.

3.3. Extended bilateral filter with a intensity image

The method mentioned above is a straightforward exten-sion of the bilateral filter to a range image. However, asshown in Figure 2(a), although abrupt changes of rangevalues such as a jump edge are easily detected in a range

image, moderate changes such as a roof edge are quitedifficult to detect. Miropolsky [7] introduced the direc-tional variation of normal vectors in order to emphasizethese moderate changes in the range image. However, ifwe observe the intensity image shown in Figure 2(b), itis easy to see that these moderate changes are clearlydetected in the intensity image.

Based on the above consideration, we propose a newfilter that uses intensity and range images simultaneouslyfor smoothing a range image as follows:

gi ¼P

j wx(xi; xj)wf ( fi ; fj)wd(di; dj)fiPj wx(xi; xj)wf ( fi; fj)wd(di; dj)

(5)

wx(xi; xj) ¼ 1ffiffiffiffiffiffiffiffiffiffi2prx

p e�jxi � xjj2

2r2x (6)

wf ( fi ; fj) ¼ 1ffiffiffiffiffiffiffiffiffiffi2prf

p e�jfi � fj j22r2

f (7)

wd(di; dj) ¼ 1ffiffiffiffiffiffiffiffiffiffi2prd

p e�jdi � djj2

2r2d (8)

where fi and di are the range and intensity values inpixel i, and wx(xi; xj), wf ( fi; fj), and wd(di; dj) areGaussian functions in the two-dimensional spatial, range,and intensity domains with variances of r2x , r

2f , and r2d ,

respectively.

Rotating table

LMS 200, SICK

2D Laser range finder

Figure 1. Acquisition system of a panoramic range image.[19]

(a) Range image

(b) Intensity image

Roof edge

Jump edge

Roof edge

Jump edge

Figure 2. Range and intensity images.

950 S. Oishi et al.

The filter given by Equation (5) takes into accountthree kinds of information in range and intensity imagesfor smoothing a range image. In other words, it is anextension of the bilateral filter for images so that it takesthe variation of the laser intensity into consideration forrange image smoothing. Thanks to a variety of propertiesin range and intensity information, the proposedextended bilateral filter enables not only jump edges butalso roof edges to be preserved in a range image, andthe extended bilateral filter is expected to have higherperformance for edge preservation than the simpleexpansion of the bilateral filter given by Equation (2).

Consequently, the proposed smoothing technique fora range image is summarized as follows.

(1) Acquire range and intensity information by aTOF range sensor.

(2) Create range and intensity images in which thevalues of each pixel in range and intensityimages are proportional to the range and inten-sity values.

(3) Apply the extended bilateral filter given byEquation (5) using range and intensity imagesand obtain a smoothed range image.

(4) Construct a 3D model consisting of meshesfrom the smoothed range image.

4. Range image inpainting by belief propagation

In the previous section, we proposed a range imagesmoothing technique using the extended bilateral filterand intensity image. Although this method is effectivefor range images that are corrupted by noise, a deterio-rated range image that is missing part of the originalimage due to specular reflection or weak reflectivity ofthe laser pulse is difficult to repair. For recovering arange image that is missing part of the original image,this section proposes an image inpainting techniqueusing belief propagation and intensity image.

4.1. Loopy belief propagation

Let us consider a graph P consisting of multiple nodesconnected by multiple arcs. We assign label fp to node pso that the following energy function is minimized.

(a) Grayscale image (b) Range image

Figure 3. Synthesized images used in the simulation experiment.

(a) 1% noise in depth values (b) Gaussian filter

(c) Bilateral filter (d) Extended bilateral filter

Figure 4. Denoised images by Gaussian filter, bilateral filter, and extended bilateral filter.

Advanced Robotics 951

E( f ) ¼Xp2P

Dp( fp)þX

(p;q)2NW ( fp; fq) (9)

where Dp( fp) is a cost term for assigning label fp to nodep, and W ( fp; fq) is a penalty term if labels fp and fq areassigned to nodes p and q, respectively. Here, N indi-cates the neighbor nodes of node p .

In the framework of belief propagation, the followingmessages are repeatedly exchanged between the adjacentnodes in order to determine the optimum label fp thatminimizes the energy function:

mtp!q( fq) ¼

minfp

Dp( fp)þW ( fp; fq)þX

s2N (p)nqmt�1

s!p( fp)

!(10)

After T iterations, optimum label f �q is determined so asto minimize the following cost function:

bq( fq) ¼ Dq( fq)þXp2N (q)

mTp!q( fq) (11)

4.2. Range image inpainting using a intensity image

We apply belief propagation to a deteriorated rangeimage and repair the image using the intensity image.When we measure range data using a laser scanner, itoften occurs that part of the range image is lost due tothe saturation of reflectivity by specular reflection or aweak laser pulse reflected on a black surface. In mostcases, not only the range information but also the inten-sity information in this region is lost. The proposedinpainting technique for the range image consists of twosteps. First, we repair the intensity image by belief prop-agation in Section 4.1 because the intensity image clearlycontains roof and jump edges, and the restoration of theintensity image is easier than the restoration of the rangeimage. Then, we apply belief propagation to the rangeimage using the repaired intensity image. In Section 5,this two-step algorithm is demonstrated to be able toinpaint the range image more precisely than directlyapplying belief propagation to the range image.

Since belief propagation requires a huge memory andlarge calculation cost, the range and intensity images arefirst converted into a 256-level gray-scale image. There-fore, the number of labels to be assigned is 256, asexpressed by integers from 0 to 255.

We define the cost term Dp( fp) for assigning label fpto pixel p as

Dp( fp) ¼ 0 (12)

for lost regions and

Dp( fp) ¼ j fp � Lpj (13)

Table 1. RMS error.

RMS (mm)

Original image 45.8Gaussian filter 17.8Bilateral filter 14.1Extended bilateral filter (proposed) 11.7

Figure 5. Experimental setup.

Figure 6. Range image and intensity image.

952 S. Oishi et al.

for other regions where Lp is the original label of pixelp. In addition, we consider the four-neighbor q of pixelp and define the cost function for assigning labels fp andfq as

W ( fp; fq) ¼ g(rp; rq)( fp � fq)2 (14)

where rp and rq are the values of pixels p and q in theintensity image and g(rp; rq) is a gain term that indicatesthe effect of the intensity image.

g(rp; rq) ¼ ae�b(rp�rq)2

(15)

Equation (14) indicates that the neighboring pixelwhich has a similar intensity value is preferentiallyselected to repair a lost pixel in the deteriorated rangeimage. In contrast, a pixel having an intensity value thatis changed discontinuously affects the repair of the rangeimage only slightly.

5. Experiment

This section introduces the results of the preliminaryexperiments for image smoothing using the extendedbilateral filter and image inpainting by belief propagationusing simulated and actual range images. We conductedexperiments with various parameters selected manuallyand determined parameters used for the followingexperiments.

5.1. Range image smoothing by the extended bilateralfilter

5.1.1. Simulation using a synthesized image

First, we performed the simulation experiments using thesynthesized image shown in Figure 3, which is a sceneof a square box having sides of one meter in a room. Agray-scale image (Figure 3(a)) is used instead of inten-sity image, and we added a random noise of 1% of therange value to the range image.

Figure 4 shows the results obtained using the Gauss-ian filter, bilateral filter, and extended bilateral filter,respectively. Table 1 shows the RMS errors of the rangeimages after applying these filters. In the experiment, thekernel size of each filter is 9� 9 pixels, and the rangesof the range data and the intensity data are 13,293–17,128mm and 0–255, respectively. The variances areset as rx ¼ 4:0, rf ¼ 0:4, and rd ¼ 6 .

As shown in Table 1, the RMS error of the proposedextended bilateral filter is the smallest and the proposedextended bilateral filter is verified to have high perfor-mance for range image smoothing and edge preservation.

5.1.2. Experiments with LIDAR

Next, we performed the experiments using the 3D lasermeasurement robot CPS-V shown in Figure 1.[2] Therobot enables the surrounding range data to be capturedby rotating the laser scanner (SICK, LMS151) by meansof a rotary table. The image size is 760� 1133 pixels.

(a) Original range image

(b) Bilateral filter with range image

(c) Bohme’s method[9] with range and intensity images

(d) Extended bilateral filter with range and intensity images

Figure 7. Experimental results for a stone monument.

Advanced Robotics 953

Figure 5 shows the experimental condition: a stonemonument with a Kanji inscription. Figure 6 shows therange image and intensity image of the scene capturedby the measurement robot.

The results for the scene (Figure 5) are shown inFigure 7. In this experiment, we set the kernel size ofthe filters to 9� 9 pixels, and the ranges of the rangedata and the intensity data are 443–49,726mm and

0–255, respectively. The variances are rx ¼ 3, rf ¼ 0:3,and rd ¼ 13 for the normalized range image. As shownin Figure 7(b), the bilateral filter blurs the edges of thecharacters, while smoothing the surfaces of the stonemonument. Figure 7(c) shows the result after applyingBohme’s technique [9] that estimates a Lambertianreflection coefficient and a normal direction on each partof the surface simultaneously by minimizing the energy

(a) Original range image (b) Deteriorated range image

(e) Inpainted range image without intensity image

(c) Deteriorated intensity image (d) Inpainted intensity image

(f) Inpainted range image using intensity image

(g) Standard BP (h) Proposed method

Figure 8. Range image inpainting by Belief Propagation.

954 S. Oishi et al.

function. Since this technique utilizes the differencebetween intensity and a normal direction in each pixeland their averages of its neighbor pixels as a smoothingterm, the obtained range image also fails to conserve thedetailed geometric features as is the case with the meanfilter and Gaussian filter. On the other hand, the extendedbilateral filter preserves the geometric features success-fully suppressing noises in the range image as shown inFigure 7(d).

5.2. Range image inpainting by belief propagation

5.2.1. Simulation using a synthesized image

We performed the simulation for range image inpaintingby belief propagation as described in Section 4. In theexperiment, we prepared deteriorated intensity and rangeimages which have a small missing region. The size ofthe image is 320� 240 pixels and the size of the miss-ing part is 20� 20 pixels. In this experiment, we usea ¼ 0:75 and b ¼ 1:0 .

Figure 8(a) and (b) show the original and deterioratedrange images and Figure 8(c) and (d) show the deterio-rated and repaired intensity images. The inpainted rangeimages after applying belief propagation 30 times areshown in Figure 8(e) and (f). These images are repairedwith and without the intensity image, respectively. TheRMS errors for these repaired images are compared inTable 2. From these results, the range image inpainting issuccessfully carried out using the two-step algorithm withbelief propagation and intensity image.

5.2.2. Experiments with LIDAR

Next, we performed the experiments using actual rangeand intensity images taken by the laser scanner on theCPS-V robots (Figure 1). In this experiment, we usea ¼ 0:75 and b ¼ 0:5. Figure 9 shows the 3D modelsrestored by two techniques, that is, the simple beliefpropagation for range images and the proposed two-stepalgorithm using laser intensity. Missing parts are recov-ered appropriately by both techniques and there is no bigdifference in terms of the image quality. To emphasizethe difference of the two techniques, we prepared deteri-orated range and intensity images manually by cutting apart of a wall, and applied simple belief propagation andthe proposed technique. Figure 9 shows the restored wallafter applying these techniques and each RMS error isshown in Table 3.

As shown in Table 3, the RMS error of the proposedtechnique is the smallest, and the proposed technique isverified to fill-in a hole in the range image successfully.

Table 2. RMS error for range image inpainting.

RMS (mm)

Number of iteration Without intensity With intensity

12 36.24 29.0120 30.15 14.1630 28.99 10.6840 29.47 10.4750 29.44 10.64

(a) Original 3D mesh model with a missing region

(b) The 3D mesh model after applyingbelief propagation

(c) The 3D mesh model after applyingour two-step algorithm

Figure 9. Experimental results for the performance evaluation.

Advanced Robotics 955

5.3. Range image smoothing and completion utilizinglaser intensity

Finally, we performed the experiment to verify ourdenoising techniques using actual range and intensityimages taken by the laser scanner(LMS511) on the CPS-Vrobots (Figure 1). In this experiment, we set the kernel sizeof the filters to 9� 9 pixels and the variances are rx ¼ 3,rf ¼ 0:3, and rd ¼ 9 for the extended bilateral filter. Wealso use a ¼ 0:75, b ¼ 2:0, and T ¼ 100 for our two-stepalgorithm. Figure 10(a) is a 3D model constructed fromthe original range image. Several unexpected bumpsappear on the surfaces of the walls and objects due to thenoise in the range image. In addition, several holes can beseen due to the occlusion or specular, or weak reflection.Figure 10(b) and (c) show the 3D model restored by thetwo-step inpainting algorithm and its smoothed 3D modelby using the trilateral filter, respectively. The two-stepalgorithm repairs the missing region appropriately bytaking the continuity of laser intensity into account. More-over, the extended bilateral filter smoothes the range imagesuccessfully, while preserving the geometric features suchas jump and roof edges appropriately.

6. Conclusion

In the present paper, we proposed two denoisingtechniques of range images using laser intensity, namely,range image smoothing by the extended bilateral filterand range image inpainting by belief propagation. Bytaking into account the properties of range and intensityimages, the proposed extended bilateral filter can sup-press noises in range images, while preserving geometricfeatures such as jump and roof edges. Belief-propaga-tion-based range image inpainting was also proposed torecover deteriorated range images using not only theadjacent range values but also the continuity of the inten-sity image. We conducted experiments using a synthe-sized image and actual range images and verified thatthe proposed denoising techniques successfully suppressnoises and repair deteriorated range images.

Since the intensity image is obtained as a by-productof range data, the proposed method has several advanta-ges. For example, no additional measurements or instru-ments are required and, unlike conventional cameraimages, the intensity image is not affected by lightingconditions. In addition, a range image has the advantageof detecting jump edges, and the intensity image is suit-able for detecting roof edges. Therefore, it is expectedthat the proposed technique has higher performance inedge preservation than the techniques that uses range orintensity information only.

In future, we will discuss the optimum parametersfor the proposed technique and perform quantitativeevaluation for a variety of scenes.

Table 3. RMS error in experimental results for theperformance evaluation.

RMS (mm)

Without intensity 7.68With intensity (2 step) 1.44

(a) Original 3D mesh model

(b) The 3D mesh model after applying our two-step algorithm

(c) The 3D mesh model after applying our two-step algorithm andproposed extended bilateral filter

Figure 10. Experimental results for the performanceevaluation.

956 S. Oishi et al.

AcknowledgmentsThis work was supported in part by JSPS KAKENHI GrantNumber 246404 and a Grant-in-Aid for ChallengingExploratory Research (24656173).

Notes on contributorsShuji Oishi received his MS degree fromthe Graduate School of InformationScience and Electrical Engineering, KyushuUniversity, in 2012. He is a research fellowof the Japan Society for the Promotion ofScience. He is currently working on theextension of capabilities of 3D laserscanning as a PhD student of the GraduateSchool of Information Science and Electrical

Engineering, Kyushu University.

Ryo Kurazume received his ME and BEdegrees from the Department of MechanicalEngineering Science, Tokyo Institute ofTechnology, in 1991 and 1989, respectively.His PhD degree was from the Departmentof Mechanical Engineering Science, TokyoInstitute of Technology, in 1998. He is aprofessor at the Graduate School ofInformation Science and Electrical

Engineering, Kyushu University. His current research interestsinclude multiple mobile robots, 3D modeling, manipulator,walking robots, and medical image analysis.

Yumi Iwashita received her MS degree andPhD degree from the Graduate School ofInformation Science and electricalEngineering, Kyushu University in 2004and 2007, respectively. She was a researchfellow of the Japan Society for thePromotion of Science from 2006 to 2007.In 2007, she was a postdoc at ImperialCollege London under Professor Maria

Petrou. Currently she is an assistant professor at the GraduateSchool of Information Science and electrical Engineering,Kyushu University. From 2011 to 2013, she is an academicvisitor, Jet Propulsion Laboratory. Her current research interestsinclude robot vision, biometrics, and medical image processing.

Tsutomu Hasegawa received his BEdegree in Electronic Engineering and hisPhD degree both from the Tokyo Instituteof Technology, in 1973 and 1987,respectively. He was associated with theElectrotechnical Laboratory of the JapaneseGovernment from 1973 to 1992, where heperformed research in robotics. From 1981to 1982, he was a Visiting Researcher at the

Laboratoire d’Automatique et d’Analyses des Systemes(LAAS/CNRS), Toulouse, France. From 1992 to 2013, he wasa Professor with the Department of Intelligent Systems,Graduate School of Information Science and Electrical

Engineering, Kyushu University. He is currently Director ofKumamoto National College of Technology. His researchinterests are in manipulator control, geometric modeling andreasoning, motion planning, and man–machine interaction. Hereceived the Franklin V. Taylor Memorial Award from theIEEE Systems, Man and Cybernetics Society in 1999.

References

[1] Kolb A, Barth E, Koch R, Larsen R. Time-of-flight cam-eras in computer graphics. Comput. Graphics Forum.2010;29:141–159.

[2] Kurazume R, Noda Y, Tobata Y, Lingemann K, IwashitaY, Hasegawa T. Laser-based geometric modeling usingcooperative multiple mobile robots. In: Proc. IEEE Inter-national Conference on Robotics and Automation; 2009;Kobe, Japan. p. 3200–3205.

[3] Taubin G. A signal processing approach to fair surfacedesign. In: SIGGRAPH ‘95: Proceedings of the 22ndannual conference on Computer graphics and interactivetechniques; New York: ACM; 1995. p. 351–358.

[4] Desbrun M, Meyer M, Schro”der P, Barr AH. Implicit fair-ing of irregular meshes using diffusion and curvature flow.In: SIGGRAPH ‘99; 1999; Los Angeles. p. 317–324.

[5] Jones TR, Durand F, Desbrun M. Non-iterative, feature-preserving mesh smoothing. In: SIGGRAPH ’03: ACMSIGGRAPH 2003 Papers; New York: ACM; 2003.p. 943–949.

[6] Fleishman S, Drori I, Cohen-Or D. Bilateral mesh denois-ing. ACM Trans. Graphics. 2003;22:950–953.

[7] Miropolsky A, Fischer A. Reconstruction with 3D geo-metric bilateral filter. In: SM ’04: Proceedings of the ninthACM symposium on Solid modeling and applications;Aire-la-Ville, Switzerland: Eurographics Association;2004. p. 225–229.

[8] Diebel J, Thrun S. An application of Markov randomfields to range sensing. Proceedings of conference on neu-ral information processing systems (NIPS); Cambridge,MA: MIT Press; 2005.

[9] Bohme M, Haker M, Martinetz T, Barth E. Shading con-straint improves accuracy of time-of-flight measurements.Comput. Vis. Image Underst. 2010;114:1329–1335.

[10] Davis J, Nehab D, Rusinkiewicz S, Ramamoorthi R.Efficiently combining positions and normals for precise3D geometry. ACM Trans. Graphics (Proc. ACMSIGGRAPH 2005). 2005;24:536–543.

[11] Takayuki Okatani and Koichiro Deguchi. Optimal integra-tion of photometric and geometric surface measurementsusing inaccurate reflectance/illumination knowledge.CVPR2012; 2012; Providence, Rhode Island.

[12] Ryo Kurazume, Ko Noshino, Zhengyou Zhang, KatushiIkeuchi. Simultaneous 2D images and 3D geometricmodel registration for texture mapping utilizing reflectanceattribute. In: Proc. of Fifth Asian Conference on ComputerVision (ACCV); 2002; Melbourne, Australia. p. 99–106.

Advanced Robotics 957

[13] Adelsberger R, Ziegler R, Levoy M, Gross M.Spatially adaptive photographic flash. ETH Zurich:Department of Computer Science, Technical Report No.612; 2008.

[14] Chan D, Buisman H, Theobalt C, Thrun S. A noise-awarefilter for real-time depth upsampling. In: Workshop onMulti-Camera and Multi-Modal Sensor Fusion Algorithmsand Applications; 2008; Marseille, France.

[15] Lindner M, Lambers M, Kolb A. Sub-pixel data fusionand edge-enhanced distance refinement for 2D/3D images.Int. J. Intell. Sys. Technol. Appl. 2008;5:344–354.

[16] Crabb R Tracey C, Puranik A, Davis J. Real-time fore-ground segmentation via range and color imaging. In:Proc. of CVPR Workshop on Time-of-flight ComputerVision; 2008; Anchorage, Alaska. p. 1–5.

[17] Kopf J, Cohen M, Lischinski D, Uyttendaele M. Jointbilateral upsampling. ACM Transactions on Graphics.2007 Jul;26(3). Article 96.

[18] Durou J, Falcone M, Sagona M. Numerical methods forshape-from-shading: a new survey with benchmarks.Comput. Vis. Image Underst. 2008;109:22–43.

[19] Tomasi C, Manduchi R. Bilateral filtering for gray andcolor images. In: ICCV ’98: Proceedings of the SixthInternational Conference on Computer Vision; Washing-ton, DC: IEEE Computer Society; 1998. p. 839–846.

[20] Yoshizawa S, Belyaev A, Seidel H-P. Smoothing byexample: mesh denoising by averaging with similaritybased weights. In: Proceedings of the IEEE InternationalConference on Shape Modeling and Applications; IEEE;2006; Matsushima, Japan. p. 38–44.

[21] Eisemann E, Durand F. Flash photography enhancementvia intrinsic relighting. ACM Trans. Graphics (Proc.SIGGRAPH 04). 2004;23(3):673–678.

[22] Petschnigg G, Szeliski R, Agrawala M, Cohen M, HoppeH, Toyama K. Digital photography with flash and no-flashimage pairs. ACM Trans. Graphics (Proc. SIGGRAPH 04).2004;23(3):664–672.

[23] Benett EP, Mcmillan L. Video enhancement usingper-pixel virtual exposures. ACM Trans. Graphics (Proc.SIGGRAPH 05). 2005;24(3):845–852.

[24] Weber M, Milch M, Myszkowski K, Dmitriev K, RokitaP, Seidel H-P. Spatio-temporal photon density estimationusing bilateral filtering. In: Computer Graphics Interna-tional 2004 (CGI’04); 2004; Crete, Greece. p. 120–127.

[25] Kawai N, Sato T, Yokoya N. Surface completion by mini-mizing energy based on similarity of shape. In: Proc.IEEE Int. Conf. on Image Processing (ICIP2008); 2008;San Diego, California, USA. p. 1532–1535.

[26] Becker J, Stewart C, Radke RJ. LiDAR inpainting from asingle image. In: Proceedings of the IEEE InternationalWorkshop on 3-D Digital Imaging and Modeling; 2009;Kyoto, Japan.

[27] Xu S, Georghiades A, Rushmeier H, Dorsey J, McMillanL. Image guided geometry inference. In: Proc. 3rd Int.Symp. on 3DPVT; 2006; Chapel Hill, USA. p. 310–317.

[28] Felzenszwalb PF, Huttenlocher DP. Efficient belief propaga-tion for early vision. Int. J. Comput. Vis. 2006;70(1):41–54.

[29] Komodakis N, Tziritas G. Image completion using globaloptimization. In: Proc. IEEE Conf. on Computer Vision andPattern Recognition; 2006; New York, USA. p. 442–452.

958 S. Oishi et al.