point cloud processing: estimating normal vectors and curvature indicators using eigenvectors

1 Challenge the future

Basic Point Cloud Processing

Estimating Normal Vectors and Curvature Indicators

Ir. Pirouz Nourian

PhD candidate & Instructor, chair of Design Informatics, since 2010

MSc in Architecture 2009

BSc in Control Engineering 2005

Geo1004, Geomatics Master Track Directed by Dr. ir. Sisi Zlatannova


How do we make a computer ‘see’ what we understand from this?

What do you make of this?

What is this all about?


Data to Information: LIDAR to LOD2

How do we make sense out of such big data?

Image courtesy of GEOCONNEXION

?


SOFTWARE APPLICATIONS

• FME, Safe Software

• LASTools, rapidlasso

• Point Cloud Library (PCL)

• Cloud Compare

• MeshLab

• LP360 QCoherent

A NEW VERSION OF TOIDAR! YES!

http://opentopo.sdsc.edu Information from:

http://opentopo.sdsc.edu/


FME, Safe Software

transform, convert, translate, extract, integrate, automated, repeatable


LASTools, rapidlasso

filtering, clipping, reprojecting, compression, classification, DSM, DTM, TIN, contours bare-earth

de facto standards for point cloud data:

.las .laz


LP360, QCoherenet

LIDAR, Classification, Breaklines, Visualization, Extraction, Automatic


Point Cloud Library PCL

point clouds, visualization, processing, segmentation, filtering, feature estimation, registration

Using this library in Rhino?


Cloud Compare

Implements PCL and more methods, handy to use for point cloud processing

Image from software.informer.com


MeshLab

Has some surface reconstruction methods, handy to have when working with point clouds.

Image from https://danieleferdani.wordpress.com/


Brainstorm

A sample building extracted from the AHN2 dataset.

?

?


Part 1 finished

We will continue with one specific way of dealing with point

clouds and estimating normal vectors and curvatures…


An idea!

A sample building extracted from the AHN2 dataset.


A Curvature-Based Approach to Point Cloud

Segmentation & Feature Detection

Removing

Outliers

Elevation

Classification

Slope

Classification

Aspect

Classification

Region-Growing

Segmentation

Point Cloud

of a Building

Noise

reduction

Forming

Neighbourhoods:

e.g. KNN or Range

Forming

Covariance

Matrices for

Neighbourhoods

Using PCA, Estimating:

Normal Vectors &

Curvature Indicators

Normal

Vectors

Curvature

Indicators

Eigen

System

Triangulation Mesh

Surface


Different Approaches for Finding Fitting Planes/Edges

1. Using curvature values computed (estimated) by eigenvalues of covariance matrices, to run a segmentation algorithm based on region growing.

2. Using Octree Voxels, to detect edge voxels, remove them and create segments.

3. Using Hough transform on a 2.5D point cloud, converting to the parameter space and using DBScan clustering method to find clusters of parameters, each of which correspond to a segment in a point cloud.

4. And a few more in the literature…


Different Approaches for Finding Fitting Planes/Edges

Region growing based on normals and curvature Segmentation using Octree voxels Hough transformation


Different Approaches for Finding Fitting Planes/Edges:

our experience!

Curvature based region

growing

Voxel based region

growing

Hough transform

Noise management Fair Good Good

Density variation

management

Fair Fair Good

Efficiency Good Good Poor

Ambiguity of vantage

point management

Fair Good Good

Avoiding priori

knowledge

Good Good Poor

Ease of further

processing

Good Poor Poor


Neighbourhoods of Points

Fixed Distance Neighbors (FDN) and K-Nearest Neighbors (KNN)

• Rabbani, T., van den Heuvel, F., & Vosselmann, G. (2006). Segmentation of point clouds using smoothness

constraint. International Archives of Photogrammetry, Remote Sensing and Spatial Information

Sciences, 36(5), 248-253.

• Pauling, Frederick, Michael Bosse, and Robert Zlot. "Automatic segmentation of 3d laser point clouds by

ellipsoidal region growing." Australasian Conference on Robotics and Automation. 2009.

The idea is to form neighborhoods based on [squared] distance These two options are usually provided. Apart from normal vector estimation KNN can be useful also for removal of outliers.


Fitting Planes to Neighbourhoods

The idea is that the underlying surface is a 2-manifold; therefore it

resembles a 2D plane locally

Least Square Plane Fitting Estimation Problem

Principal Component Analysis Problem

Intuition: Defining plane as the locus of lines that have direction vectors perpendicular to a normal vector; considering an origin for the plane in questions, we can consider the plane as the locus of points A such that A-O is perpendicular to N.




resembles a 2D plane locally, we look at ellipsoids showing local

dispersions

Images courtesy of Olga Sorkine


How to estimate normals using PCA


resembles a 2D plane locally

• Pauly, Mark, Markus Gross, and Leif P. Kobbelt. "Efficient simplification of point-sampled surfaces." Proceedings of the conference on Visualization'02. IEEE Computer Society, 2002.

• Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., Stuetzle, W. Surface reconstruction from unorganized points. SIGGRAPH 92, 1992

• Shaffer, E., Garland, M. Efficient Adaptive Simplification of Massive Meshes. IEEE Visualization 01, 2001



We form a covariance matrix for each neighborhood, showing how

neighbors are dispersed around their average (centroid).

This will be a 3

by 3 symmetric

matrix!



We form a covariance matrix for each neighborhood, showing how

neighbors are dispersed around their average (centroid).

This will be a 3

by 3 symmetric

matrix! Form an Eigen system for this matrix using a linear algebra library, and the first eigenvector corresponding to least eigenvalue will be the normal vector at each neighbourhood. Explanation follows…

PCL implementation: http://pointclouds.org/documentation/tutorials/normal_estimation.php

http://pointclouds.org/documentation/tutorials/normal_estimation.php

http://pointclouds.org/documentation/tutorials/normal_estimation.php


How to estimate curvature using PCA

The idea is to use an indication of change along the normal vector

Jolliffe, I. Principle Component Analysis. Springer-Verlag, 1986


How to do all this in code? • We try not to reinvent the wheel; the idea is to use free open

source libraries like Math.NET and Accord.NET

• We are not the first people dealing with such issues, these are

generally matters of data mining and machine learning

• We can find KNN neighbourhoods using Accord.NET http://accord-framework.net/

• We can find eigenvalues and eigenvectors using MetaNumerics.dll, MathNet.dll or Accord.NET

• You will receive example code implementing MetaNumerics

http://accord-framework.net/





How do we minimize the error in fitting a plane to the

mentioned neighbourhood? First we define it…

Images courtesy of Olga Sorkine


An explanation after Olga Sorkine:

• Input points:

• Centroid:

• Vectors from the centroid:

m


Continued…

m m minimizes SSD and it can be shown that m is the centroid C

We can rewrite the problem as:


Continued…

m

m minimizes SSD, formally it is the “arg min” of the following function (to be minimized), i.e. the argument that makes it reach its minimum. and it can be shown that m is the centroid C

Solving this turns out to be equal to

solving an Eigen system for the

covariance matrix; you can see how…


Long story short…

We find eigenvalues and eigenvectors of the covariance matrices…


Long story short…

We find eigenvalues and eigenvectors of the covariance matrices…

What was this all about? Segmenting the point cloud taking into account [underlying] surface variations, in search of smooth patches, made disjoint by edges (where we find high curvature/variation). Therefore we can use the above estimated vectors and values for such a segmentation.


Explanation from lecture notes of Olga Sorkine…


Continued from lecture notes of Olga Sorkine…



This S is our covariance

matrix, remember?


•Constrained minimization – Lagrange multipliers


Given this equality in

differentiation of vector

valued functions




maximize f(x, y) subject to g(x, y) = c. Lagrangian: The prblem is transformed to finding a ‘’staionary point for the Langrangian at which the partial derivatives of Lagrangian function are zero.




This one means that the

normal vector is

normalized, meaning its

length is equal to 1; why,

because the dot product

of a vector by itself gives

the squared length


How to form covariance matrices in code?

VB.NET (confirm and debug for yourself)

If(Pts.Count > 3) Then Dim CovMatrices As New list(Of Meta.Numerics.Matrices.SymmetricMatrix) For j As Integer=0 To Pts.Count - 1 Dim CovM As New Meta.Numerics.Matrices.SymmetricMatrix(3) Dim Neighbors As List(Of Integer) = DirectCast(Neigh(j), List(Of Integer)) Dim Centroid As New Point3d Dim NPts As New List(Of Point3d) For Each neighbor As Integer In Neighbors Centroid = Centroid + Pts(neighbor) NPts.Add(Pts(neighbor)) Next Centroid = Centroid / Neighbors.Count Dim CiCBar As New RectangularMatrix(3, Neighbors.Count) For k As Integer=0 To Neighbors.count - 1 Dim Diff As point3d = Pts(Neighbors(k)) - Centroid CiCBar(0, k) = Diff.X CiCBar(1, k) = Diff.y CiCBar(2, k) = Diff.z Next CovM = CiCBar.MultiplySelfByTranspose() CovM = (1 / (Neighbors.count - 1)) * CovM CovMatrices.Add(CovM) Next A = CovMatrices(0).ToArray() C = CovMatrices End If


How to form covariance matrices in code?

C#.NET (confirm and debug for yourself)

if ((Pts.Count > 3)) { List<Meta.Numerics.Matrices.SymmetricMatrix> CovMatrices = new List<Meta.Numerics.Matrices.SymmetricMatrix>(); for (int j = 0; j <= Pts.Count - 1; j++) { Meta.Numerics.Matrices.SymmetricMatrix CovM = new Meta.Numerics.Matrices.SymmetricMatrix(3); List<int> Neighbors = (List<int>)Neigh(j); Point3d Centroid = new Point3d(); List<Point3d> NPts = new List<Point3d>(); foreach (int neighbor in Neighbors) { Centroid = Centroid + Pts(neighbor); NPts.Add(Pts(neighbor)); } Centroid = Centroid / Neighbors.Count; RectangularMatrix CiCBar = new RectangularMatrix(3, Neighbors.Count); for (int k = 0; k <= Neighbors.count - 1; k++) { point3d Diff = Pts(Neighbors(k)) - Centroid; CiCBar(0, k) = Diff.X; CiCBar(1, k) = Diff.y; CiCBar(2, k) = Diff.z; } CovM = CiCBar.MultiplySelfByTranspose(); CovM = (1 / (Neighbors.count - 1)) * CovM; CovMatrices.Add(CovM); } A = CovMatrices(0).ToArray(); C = CovMatrices; }


Part 2 finished

Some hints on previous assignment follow…


Grid Surface Reconstruction

Pseudo-Code

Define a new Mesh with a list of Vertices and a list of Faces Mesh.Vertices=Points For j as Integer=0 to V-2//choose only bottom-left corners as pivot points For i As Integer=0 To U - 2 define n0,n1,n2,n3 As Integer n0 = j * U + I //bottom-left n1 = n0 + 1 //bottom-right n2 = n0 + U //top-right n3 = n1 + U //top-left Define face As new MeshFace(n0, n1, n3, n2) Mesh.Faces.AddFace(face) Next Next



VB.NET

Dim M As New Mesh M.Vertices.AddVertices(P) If Not C Is Nothing Then M.VertexColors.AppendColors(C.toArray) For j As Integer=0 To V - 2 For i As Integer=0 To U - 2 Dim n0,n1,n2,n3 As Integer n0 = j * U + i n1 = n0 + 1 n2 = n0 + U n3 = n1 + U Dim face As MeshFace = New MeshFace(n0, n1, n3, n2) M.Faces.AddFace(face) Next Next

with two nested loops



VB.NET

Dim M As New Mesh M.Vertices.AddVertices(P) If Not C Is Nothing Then M.VertexColors.AppendColors(C.toArray) For i As Integer=0 To U * (V - 1) - 1 If (i Mod u) < u - 1 Then Dim n0,n1,n2,n3 As Integer n0 = i n1 = n0 + 1 n2 = n0 + U n3 = n1 + U Dim face As MeshFace = New MeshFace(n0, n1, n3, n2) M.Faces.AddFace(face) End If Next

with one loop



C#.NET

Mesh M = new Mesh(); M.Vertices.AddVertices(P); if ((C != null)) M.VertexColors.AppendColors(C.toArray); for (int i = 0; i <= U * (V - 1) - 1; i++) { if ((i % u) < u - 1) { int n0 = 0; int n1 = 0; int n2 = 0; int n3 = 0; n0 = i; n1 = n0 + 1; n2 = n0 + U; n3 = n1 + U; MeshFace face = new MeshFace(n0, n1, n3, n2); M.Faces.AddFace(face); } }

with one loop


Simple Estimation of Normal

Vectors Pseudo-Code

Form an empty list of normal vectors Define deviation as a double For each point as Point3d in the point cloud find neighbors fit a plane to neighbors Get the normal of this plane and put it out as the normal of the point form a vector from the vantage point VP to point=VP-point and call it dir if this normal.dir>0 then Add the normal to the list of normals Else Add –normal to the list of normals End Next


Estimation Normal Vectors

C#.NET

List<Vector3d> Normals = new List<Vector3d>(); Point3dList PCList = new Point3dList(); PCList.AddRange(x); double Dev = MD; foreach (Point3d point in PCList) { dynamic Neighbors = PCList.FindAll(V => V.DistanceTo(point) < D); plane NP = default(Plane); Plane.FitPlaneToPoints(Neighbors, NP, Dev); if (NP.Normal * (VP - point) > 0) { Normals.Add(NP.Normal); } else { Normals.Add(-NP.Normal); } } A = Normals; B = PCList.FindAll(VT => VT.DistanceTo(x(654)) < D);


Estimation Normal Vectors

VB.NET

Dim Normals As New list(Of Vector3d) Dim PCList As New point3dlist PClist.AddRange(x) Dim Dev As Double = MD For Each point As point3d In PClist Dim Neighbors = PClist.FindAll(Function(V) V.distanceto(point) < D) Dim NP As plane Plane.FitPlaneToPoints(Neighbors, NP, Dev) If NP.Normal * (VP - point) > 0 Then Normals.Add(NP.Normal) Else Normals.Add(-NP.Normal) End if Next A = Normals B = PClist.FindAll(Function(VT) VT.DistanceTo(x(654)) < D)


Basics of Scripting in Rhino + GH

• VB.NET (Sub[routine], Function, ByVal, ByRef)

• C#.NET (Void, [Functions], [val],ref)

• Python (defs)

Basic concepts of systems and modules, Inputs & Outputs

Private Sub RunScript(ByVal x As Object, ByVal y As Object, ByRef A As Object) End Sub '<Custom additional code> '</Custom additional code>

Function plus(A As Integer, B As Integer) As Integer Return A + B End Function

private void RunScript(object x, object y, ref object A) { } // <Custom additional code> // </Custom additional code>

public int plus(int a, int b) { return a + b; }

def plus(a,b): return a+b

Nothing visible.


Basics of Scripting in Rhino + GH

•Rhinocommon •Rhinoscript •Grasshopper Kernel

Rhinocommon

Library of

Geometry

Operations VB.NET. C#.NET &

Python

Grasshopper Kernel

Library of Some

Special Geometry

Operations VB.NET. C#.NET &

Python

Rhionscript (with [Iron]Python)

Operations as in Rhino Command Line

Python

http://4.rhino3d.com/5/rhinocommon/ Rhinocommon SDK:

Grasshopper SDK: [Rhino]Command: GrasshopperGetSDKDocumentation

Rhinoscript Syntax SDK: [GH Python]Help: Rhinoscript syntax help

http://4.rhino3d.com/5/rhinocommon/


Questions? [email protected]

• I will give you the tools we have developed so far as open source;

• You will make them better using the following libraries; • Using either Math.NET or MetaNumerics is a must; • Using Accord.NET is a big plus! • If you are using Python, use similar libraries such as SciPy • If you want to do something else let us discuss it now!

mailto:[email protected]