TKK 12.4.2007

STRS system combining LiDAR and multiple images:

Multi-scale template matching and LS-adjustment of a parametric crown model with lidar data in 3D tree top positioning and estimation of the crown shape

Species recognition in aerial images

Ilkka Korpela (Morten Larsen)

Contents (Demos included?)

Single-tree remote sensing (STRS)

Photogrammetric 3D reconstruction and object recognition

Airborne lidar-based 3D reconstruction and object reconstruction

Coupling allometric constraints to the STRS problems

3D treetop positioning with template matching (TM)

“ multi-scale TM

Species recognition

LS-adjustment of crown models with lidar points

Conclusions and outlook

STRS Single-Tree Remote Sensing

• Air- or spaceborne; active and/or passive sensing

• 2D or 3D = with or without tree height

• “Direct estimation” of tree/crown position and species; indirect model-based estimation of height, dbh

• Restrictions: tree discernibility due to scale (detectable object size), occlusion and shading.

• Alternative or complement to A) field inventory, B) area-based remote sensing

STRS Single-Tree Remote Sensing

• Accuracy restricted by “allometric noise” alike in volume functions → tree and stand level bias, tree level impresicion. dbh ~ 10-12 %.

• Measurements subject to bias: DTM-errors, lidar does not hit apexes, Dcr underestimation

• Nothing can be known about quality, only quantity

• Unsolved issues: species recognition, regeneration stands, calibration and validation in the field, aggregated crowns result in fused trees.

Photogrammetric 3D reconstruction and object recognition

Usually from N>1 images (multiscopic)

Correspondence problem - ill-posed, perspective errors, reflectance, occlusion, scene complexity

Texture in the image functions needed

Relies on accurate geometry (camera interior, exterior), ray-intersection

Photogrammetric 3D reconstruction and object recognition

Digital revolution (2000→)

• Aerial images upto 2GB, manageable (I/O, analysis, storage, transfer)

• Automatic methods in orientation, incl. DSO with GPS/INS, reduction in GCPs.

• Digital cameras with MS images 2005→ multiple images per target, better radiometry and geometry

• Automatic DSM production

Photogrammetric 3D reconstruction and object recognition

Automation

• Laborious orientation tasks ±solved

• DSMs, DTMs using image matching

• e.g. Building extraction - semiautomatic

3913, 3914, 3915 (triplet matching) → 1946, 1962Demo 1

Photogrammetric 3D reconstruction and object recognition

Photogrammetric STRS

• scene and object complexity

• occlusion & shading

• scale: h = 0..40 m, Dcr 0..10 m

• “BDRF”-effects

• → automation challenging

Photogrammetric 3D reconstruction and object recognition

Demo – manual STRS - NLS 04403 (19)

treetop 3D, height, Dcr, Sp

dbh = f(Sp, h, Dcr) + epsilon

Image matching fails for treetop positioning unless we use a feature detector for treetops.

Airborne lidar-based 3D reconstruction and object reconstruction

Impivaaran_lidar_MV.avi

Impivaaran_Lidar_siksak.avi

A pulse of short duration (~ 3 m, 1064 nm)

Observing returned signal. Discrete data. Upto 128 samples?.

Signal is reconstructed into points or samples for later waveform analysis. Intensity of the return/echo.

Pros: No texture needed, active → no shading, “real ease of 3D”

Cos: discrete sampling, high sampling rates costly, difficult to reconstruct “high-frequency relief”.

Airborne lidar-based 3D reconstruction and object reconstruction

Automation

DTMs – manual assistance needed – high accuracy – even under canopy

Volume estimation of trees – “automatic” – e.g. using regression between lidar features and field observations (e.g. Naesset 2004. Suvanto et al. 2005)

Lidar and STRS

Algorithms that process point clouds directly or interpolated DSMs (CHMs)

Underestimation of heights (footprint size, density, crown shape, equipment sensitivity)

Species not obtained (so far)

Coupling allometric constraints to the STRS tasks

If we know dbh, we have an idea of the height

If we know dbh, species and age, we have better idea of height

If we know dbh, species, and height we have a good idea of volume

If we know dbh, species, height and height of crown, we have a better idea of volume.

If we know height, species and crown width we can estimate dbh and volume

If we know species we have an idea of the shape of the “crown envelope”.

If we know species and height, can we set limits for the variation of crown width f(sp, h) => [Min, Max] of Dcrm, and assume a basic shape?

Coupling allometric constraints to the STRS tasks

If we know species can we have an idea of the shape of the “crown envelope”?

Timo Melkas

Dcrm (min,max) / Shape | (Sp, height) →

- Consistency of measurements (rule out impossible observations)

- Initial approximations for iterative approaches in finding true Dcrm & crown shape

E.g Short trees have small crowns(adjust search space accordingly, or look for small crowns from a low height)

Coupling allometric constraints to the STRS tasks

3D treetop positioning with template matching (TM)

Demo 04402 - 3D treetop positioning using TM

1) Use, for each of the N views, a model image (template) of a crown.

2) Compute N normalized cross-correlation images (template matching).

3) Form a Cartesian 3D grid in the canopy – in the search space.

4) Aggregate 3D correlation to the grid points.

5) Process the 3D correlation into “hot-spots” – 3D treetop positions.

Fine, but not invariant to object variation.

Cartesian grid / Search space in the upper canopy

Correlation > threshold

Multi-scale TM – Treetop positioning

Can we assume that the optical properties and the “relative shape” of trees are invariant to their size? I.e. small trees appear as scaled versions of large trees in the images? (inside one species and within a restricted area)

Multi-scale TM – Treetop positioning

Maxima at different scales, take global → (X,Y,Z)

Multi-scale TM – Crown size

Demo 04403_19, 18, 20, 06214_3900, 3901

Near-nadir views are best for manual measurement of Dcr (crown width)

Species recognition

Spectral valuesTexture

Variation:

- Phenology- Tree age and vigour- Image-object-sun geometry=> reliable automation problematic => bottleneck

LS-adjustment of a crown model with lidar pointsAssume that

1) Photogrammetric Multi-scale TM 3D treetop position is highly accurate2) Trees have only moderate slant3) Crowns are ± rotation symmetric4) We know tree height and species which give a reasonable approximaion of the crown size and shape

→ LiDAR hits are “observations of crown radius at a certain height below the apex”

Assume a rather large crown and collect lidar hits in the visinity of the 3D treetop position, down to relative height of apprx. 60 %.

Use LS-adjustment to find best set of parameters for the crown model.

“LiDAR hits are observations of crown radius at a certain height below the apex?”

LiDAR hits are observations of crown radius at a certain height below the apex – what if crowns are interlaced?”

Example - a 19-m high spruce:

Solution in three iterations.

Final RMSE 0.31 m

Note apex! LiDAR did not hit the apex and the “crown width at treetop” (constant term) is negative.

Example - a 22-m high birch:

Solution in six iterations.

Final RMSE 0.47 m

For some reason RMSEs are larger for birch in comparison to pine and spruce.

Convergence?

Conclusions and outlook A

- Multi-scale TM works in a manual semi-automatic way for treetop positioning

Possible to automate? Computation costs? (NCC now tried everywhere)

- Multi-scale TM in crown width estimation needs comprehensive testing (Image scales, required overlaps)

-Species recognition was “overlooked” here, still I think that good 3D treetop positions can be used for the purpose.

- Matching LiDAR points using LS-adjustment works only if the exact treetop position is known. Aggregated crowns are problematic, but these cases are known from the tree map

Conclusions and outlook B

- If we have a system that can be operated so that a tree measurement takes 2-3 seconds and the measurement inaccuracies are:

h ~ 0.6 m Dcr ~ 10% d13 ~ 10-12 %XY-position ~ 0.3 mSp ~ 95%

Is this fast and accurate enough for sample-plot basedSTRS? Can we afford the images and LiDAR?

Calibration and validation of results?

• Presenting method and early results in ISPRS workshop in Hannover, June 1, 2007

• Presentation at Silvalaser, Espoo September 2007?

• Article about Multi-Scale TM in 3D treetop positioning (ISPRS JPRS?)

• Article about Multi-Scale TM combined with crown modeling using LS-adjustment and allometric constraints (Silva Fennica 100-yr issue?)

• Species recognition study remains of the Academy 3-yr resrach project to be completed before VIII/2008