A NOVEL APPROACH TO ENVIRONMENT RECONSTRUCTION IN LIDAR AND HSIDATASETS

Dejan Nikic, Jason Wu ∗

The Boeing CompanyApplied Physics Lab

1420 Henderson St. Seattle, WA 98124

Paul Pauca, Robert Plemmons, Peter Zhang

Wake Forest UniversityDepartment of Computer Science and Mathematics

P.O. Box 7388 Winston Salem, NC 27109

ABSTRACT

A combination of shape and material data presented in a com-mon data structure provides a platform for effective environ-ment reconstruction and target detection. Lidar and hyper-spectral image (HSI) data offer a vast amount of information,which can be redundant, that can completely reconstruct sur-faces that generate reflections. These reflections are then cap-tured by respective sensors. This paper discusses a physi-cal approach, similar to electric field calculation of electronclouds, which can reconstruct the shapes captured by lidardata, while a subspace representation is used to best approx-imate materials present in HSI data. The two data sets aremerged by assigning the appropriate materials to the recon-structed surfaces. In this paper some of the benefits of thisapproach are discussed and experimental data is used to showthe compression capability.

Index Terms— lidar, hyperspectral images, implicit ge-ometry, subspace representation, nonnegative matrix factor-ization, sparcity, classification, detection, spectral mixtureanalysis.

1. INTRODUCTION

Lidar and HSI data aggregation can be a powerful tool intarget detection of remotely sensed environments. Unfortu-nately, both data types come with their own set of challenges.With lidar data the sheer number of 3D points collected (ap-proximately 1 million) presents a daunting processing task,even for modern computers. Traditional methods of Eucle-dian distance calculation between the points would requiretrillions of operations and are impractical for most caseswhere speed and efficiency are needed. HSI data has a verylarge number of spectral combinations (on the order of 256)that can be mixed, requiring a careful reconstruction of spec-tra. Because of the atmospheric effects and large stand-offdistances of HSI sensors, noise and blur are presented inmost HSI datasets. Algorithms for processing HSI data must

∗RESEARCH SUPPORTED BY NATIONAL GEOSPATIAL INTELLI-GENCE AGENCY UNDER CONTRACT HM1582-10-C-0011

account for noise and blur and correct the data before pro-cessing. These algorithms must also efficiently unmix andinterpret the resulting HSI spectral signatures. Because ofthe difference in resolution between lidar and HSI data, thereneeds to exist a method that co-relates the two data sets. Amethod for classification and segmentation of the environ-ment can be used to co-register the data. The data is basedon the extents of the objects in the scene, allowing for largespectrally mixed HSI pixel integration on top of a higher res-olution lidar data set. In the model presented, the HSI data isnot limited to the top surface of the object, but is also gatheredalong the height of the scanned surface. With this method, acombination of HSI sensors can be used (e.g., ground-based,different flight path) to generate material data for otherwiseoccluded surfaces.

2. ENVIRONMENT RECONSTRUCTION

In order to faithfully represent the physical properties of a re-motely sensed environment, a model is developed to representthe shape of the objects present and apply spectral responsesto the surfaces that make up the shape.

2.1. Shape reconstruction

Using lidar data, a shape is reconstructed using the implicitgeometry (IG) method [1]. Three functions are applied to therectangular mesh that contains all of the data points. Eachfunction provides a unique field, each with its own set of ad-vantages for different applications. Let Ω denote the observedspace or test site. Given a set of LiDAR sampling pointsP ⊂ Ω collected, three different types of fields can be cal-culated using the functions described below.

2.1.1. Population function

Computationally fastest of all, O(N), the population func-tion, counts the number of lidar data points belonging to aspecific mesh cell. A field that is generated by the popula-tion function is localized to an individual cell. This functiondoes not depend on the lidar points that lie outside of the IG

cell. The population function is advantageous in applicationswhere rapid processing is required and where a distribution oflidar point cloud is important.

2.1.2. Distance function

The distance function d(w) for any w ∈ Ω to the data pointset P is defined as the solution of the Eikonal equation (1).

|∇d(w)| = 1

d(w) = 0 if w ∈ P(1)

Unlike the population function, distance function valuesare computed by looking at the lidar points contained in aregion of interest defined by a radius. In general, the distancefunction offers a superior rendering capability when com-pared to the population function. The computation time fordistance function is O(N), the same as the population func-tion but there is an additional computational penalty causedby extra calculation of the surrounding mesh cells. In ourstudies, the distance function has been 5 to 10 times slowerthan population function.

2.1.3. Validity function

For any given point w ∈ Ω, the validity of this position isdefined in (2), where p is a data point in P and f could bea hardware-dependent point-spread function (PSF) or a usersupplied distribution function.

v(w) =∑p∈P

f(w − p) (2)

For our model, the validity function closely resembles theelectric field calculation because the imposed PSF resembles1r or 1

r2 depending on the application and raw data proper-ties, where r is the distance between the cell node and lidarpoint. Similar to the distance function, the lidar data pointscontained in neighboring cells that are defined by a radiusaffect the validity function. The computation time of the va-lidity function is comparable to that of the distance function.

2.1.4. Surface rendering

FFor all three functions the underlying shape of the sensedobject can be reconstructed by rendering an isosurface of theIG field [2] . For a given threshold T, we then compute theisosurface S ⊂ Ω such that for any w ∈ S, v(w) = T . Thevalue of T is chosen based on the function used to generate theIG field. The rendering process is same for all three functionsdescribed above, but the choice and span of T varies based onthe function. We implemented our renderings based on theMarching Cubes method [3].

Surface rendering is used only for display purposes andhas no direct effect on the algorithms that analyze the data.Surface rendering is just one of the many applications that theIG fields can be used for.

2.2. Material reconstruction

Given a HSI data cube f(x, y, λ) we seek a sparse representa-tion of the two spatial dimension shown in (3), where s(λ) isthe spectral signature and u(x, y) is the characteristic function[4].

f(x, y, λ) =

L∑i=1

ui(x, y)si(λ) (3)

Using this type of representation the HSI data is com-pressed, unmixed and easier to interpret.

2.2.1. Total variation (TV) model

A piecewise constant segmentation model (4), where fj is thej-th slice of the hyperspectral cube, sij is the reflectance ofi-th spectrum at j-th wavelength and τ is the regularizationparameter, can be extended to HSI data. Applying the totalvariation model to HSI data combines all slices while mini-mizing the total variation of the membership function, whichonly resides on the two spatial dimensions, independent fromspectral slices.

E(u, s) =

L∑i=1

∫Ω

|∇ui|dxdy+

+

L∑i=1

τ

2

∫Ω

1

m

m∑j=1

(fj − sij)2u2i dxdy

(4)

One major advantage of the total variation model is that inthe presence of a noise or blurring, the model can be coupledwith a deblurring and denoising model (5), where f0 is the ob-served HSI data and γ is the regularization parameter [5]. Forthe cases where the blurring operator h is well characterized,the blur can be eliminated completely.

E(u, s) =

m∑j=1

∫Ω

|∇fj |dxdy +γ

2

∫Ω

(h · f − f0)2dxdy

+

L∑i=1

∫Ω

|∇ui|dxdy

+

L∑i=1

τ

2

∫Ω

1

m

m∑j=1

(fj − sij)2u2i dxdy

(5)

2.3. Combination of shape and material information

Given the IG approach, various properties can be assigned toindividual cells. A single spatial location in the IG mesh canhold population, validity, and distance field values alongsidethe HSI data characteristic function coefficient.

Before assigning HSI characteristic function coefficientsto their respective cells, a data structure containing spectralsignatures needs to be generated. Once the spectral signaturesare connected with the IG cells by characteristic function co-efficients, a full representation of geometry and material datais presented. A final data structure (6) will contain the IG fieldvalues as well as the characteristic function coefficients, andadditional data can be added as well. In the current imple-mentation, available RGB data can be added.

IG(x, y, z) = p, d, v, u (6)

To consolidate HSI and lidar data, each mesh is assignedvalues of the three IG functions (population, distance, andvalidity) and the HSI coefficient ui. Spectral signature si(λ)is stored in a data structure accessible by the IG mesh. SinceBecause the IG mesh is a 3D structure and the characteristicfunction ui(x, y)is in 2D space, we project ui over the IGmesh. Depending on the application, the ui can be assignedjust to the bottom row of mesh cells, all mesh cells belongingto particular (x,y) location, or any other combination.

Currently, data from only one pose is available. But withthe meshed domain a variety of sensors can be combined togive data from different viewing angles. This increases thefidelity and fills in areas that might be occluded. For instance,ground-based HSI and lidar sensors can obtain data for thesides of objects that is generally not available from sensorsthat are flown directly above the scene.

3. APPLICATION TO GULFPORT LIDAR AND HSISCENE

A data set was acquired by a team led by Professor PaulGader from the University of Florida Department of Com-puter and Information Science and Engineering. The datawas captured over the campus of the University of SouthernMississippi (USM) in Gulfport, Mississippi. Lidar and HSIdata were acquired concurrently and coregistered. The HSIdata, with bands from 0.453 µm to 0.719 µm, was acquiredby an OpTech International, Inc. sensor. Out of the origi-nal 72 bands ranging from 0.4 µm to 1.0 µm, 58 bands wereselected for higher signal to noise ratio.

From the data selected, a section that spans 320 by 360meters was segmented for testing. This area encompasses acenter of the USM campus and shows numerous urban fea-tures (e.g., buildings, roads, trees, vehicles), as shown in Fig.1.

A lidar data set with approximately 183,000 lidar pointswas used to generate the IG validity field (Fig. 2) A regularrectangular mesh with one-meter resolution was used. The to-tal computation time of the validity field was approximately0.5 seconds. The isosurface was generated using the IG va-lidity field (Fig. 3). The cut-off value of the isosurface waschosen manually to generate a plot that best represented theUSM campus scene.

Fig. 1. Areal view of USM campus in Gulfport, MS [6]

Fig. 2. LiDAR data of the USM campus, with false colorbased on height.

Fig. 3. Isosurface plot of IG validity field, with false colorbased on height.

Fig. 4. Spectral returns of the sparse representation of HSIdata for 7 spectral signatures.

Fig. 5. Combination of IG validity field isosurface and char-acteristic function, where colors represent the different spec-tral returns shown in Fig. 4

HSI portion of the data was processed using a TV methodwith m=7 as defined in (4). Figure 4 shows a subset of repre-sentative spectral returns. The characteristic function u(x,y)that was obtained using the total variation method was com-bined with the validity field. The combination of the charac-teristic function and the validity field resulted in a full repre-sentation of the USM scene (Fig. 5).

4. FUTURE WORK

Using the described data fusion model one can extract fuseddata and use it for object detection, classification, and cluster-ing. Our future work will center on interrogating this fuseddata.

An extension of the model presented here will be used toeffectively classify and cluster mixed lidar and HSI data. Ap-plication of a mixed dataset should provide superior classifi-cation capabilities with reduced false alarms and an increased

probability of detection.Similarly, our future object detection work will use rule

sets to interrogate the scene and find patterns that define aparticular object. We will investigate the suitability of this ap-proach for complex objects such as trees, vehicles, and build-ings.

5. CONCLUSION

The described algorithms and methods for combining lidarand HSI data have provided great utility in combining thedatasets. After combining, the new data structure can storemultiple datasets from different sensing platforms and posesnecessary to obtain a complete environment reconstruction.

6. REFERENCES

[1] B. Merriman, M. Kang, H.K. Zhao, and S. Osher., “Im-plicit and nonparametric shape reconstruction from unor-ganized data using a variational level set method.,” Com-puter Vision and Image Understanding, vol. 80, pp. 295–314, 2000.

[2] R. Fedkiw and S. Osher., “Level set methods: anoverview and some recent results,” Journal of Compu-tational Physics, vol. 169, pp. 463–502, 2001.

[3] Harvey E. Cline and William E. Lorensen, “March-ing cubes: A high resolution 3d surface construction al-goirthm,” Computer Graphics, vol. 21, pp. 4, 1987.

[4] R. Plemmons, S. Prasad, F. Li, M. Ng, and Q. Zhang.,“Hyperspectral image segmentation, deblurring and spec-tral analysis for material identification,” in SPIE Sympo-sium on Defense, Security, and Sensing, 2010.

[5] D. Kittle, D. Brady, Q. Zhang, R. Plemmons, andS. Prasad., “Joint segmentation and reconstruction of hy-perspectral data with compressed measurements,” Ap-plied Optics, vol. 50, pp. 4417–4435, 2011.

[6] GOOGLE EARTH, “3021′06.19”N 8908′07.59”W ,”Gulfport, MS, 12/15/2011.