geomatics and water resources research group seminars autumn term 2007

1

Geomatics and Water Resources Research Group Geomatics and Water Resources Research Group SeminarsSeminars

Autumn Term 2007Autumn Term 2007

Dr. Fernando J. Aguilar TorresDr. Fernando J. Aguilar Torres

Department of Agricultural Engineering, University of Almeria, SpainDepartment of Agricultural Engineering, University of Almeria, Spain

A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT A METHODOLOGICAL PROPOSAL FOR ACCURACY ASSESSMENT OF LIDAR DERIVED DIGITAL ELEVATION MODELSOF LIDAR DERIVED DIGITAL ELEVATION MODELS

Newcastle, September 2007

2

1.1. Introduction.Introduction.

2.2. Accuracy assessment of DEMs.Accuracy assessment of DEMs.

3.3. Reference Standards (official guidelines). Are they enough?Reference Standards (official guidelines). Are they enough?

4.4. Do we really know the reliability of our DEM accuracy measures?Do we really know the reliability of our DEM accuracy measures?

5.5. Our methodological proposal in the case of LiDAR derived DEMs.Our methodological proposal in the case of LiDAR derived DEMs.

6.6. Modelling LiDAR error. Preliminary results. Modelling LiDAR error. Preliminary results.

7.7. Conclusions.Conclusions.

ScheduleSchedule

A METHODOLOGICAL PROPOSAL FOR ACCURACY A METHODOLOGICAL PROPOSAL FOR ACCURACY

ASSESSMENT OF LIDAR DERIVED DEMsASSESSMENT OF LIDAR DERIVED DEMs


3

1. 1. IntroductionIntroduction




DEM? What is a DEM?DEM? What is a DEM?

Z = f(x,y)

A DEM is a digital and mathematical representation of an existing or virtual terrain by means of storing the land elevations (void of vegetation and manmade features) usually at regularly spaced intervals in x and y directions.

4





DEM?DEM?

5





Applications of DEMsApplications of DEMs

Hydrological and erosion modelsHydrological and erosion models Viewshed analysis and visual impactViewshed analysis and visual impact Flood risk analysisFlood risk analysis Planning of land development. Suitability models (GIS)Planning of land development. Suitability models (GIS) Civil Engineering (cut and fills calculation)Civil Engineering (cut and fills calculation) Relief description and geomorphology (slopes, aspects and so on)Relief description and geomorphology (slopes, aspects and so on) Topographic correction of remote sensing imagery, insolation and Topographic correction of remote sensing imagery, insolation and

shadowing modelsshadowing models 3D visualisation and virtual environments3D visualisation and virtual environments Orthoimages generationOrthoimages generation

6





Orthoimages generationOrthoimages generation

2D environment 3D environment

7

OffNadir View Angle (degrees)

DE

M a

ccur

acy

(m)

RMSEortho (m)0.71.01.31.61.92.22.52.83.13.4

3 6 9 12 15 18 21 240123456789

10





Error propagation from DEM to the final productError propagation from DEM to the final product

Aguilar et al. Annual International Conference ADM and INGEGRAF. Perugia, Italy, June 2007.

8

2. 2. Accuracy assessment of Accuracy assessment of DEMsDEMs




Why?Why?

A responsible DEM user must be A responsible DEM user must be able to answer the following able to answer the following questions (planning):questions (planning):

What precisely is the application for What precisely is the application for the DEM?the DEM?

What type of DEM will best meet What type of DEM will best meet these needs?these needs?

How do I know that I am getting How do I know that I am getting what I ordered?what I ordered?

USER

PRODUCER

9





How to compute it? Statistical inference (Sampling How to compute it? Statistical inference (Sampling theory)theory)

Check points selection (finite sample N)

Differences between z DEM and z from an independent source of higher accuracy

RMSE and ME calculation

DEM quality evaluation

Check Points Error (ZDEMi -ZCPi = ei)

10





Types of errorsTypes of errors

1.1. Blunders or OutliersBlunders or Outliers

2.2. Systematic (bias) errors (constant Systematic (bias) errors (constant offset) offset)

3.3. Random errors (random fluctuations Random errors (random fluctuations in the measurements)in the measurements)

11





Types of errorsTypes of errors

• Systematic errors (A and B)

• Spatially autocorrelated errors (C)

• Random errors with no spatial autocorrelation (D)

P. Fisher and N. Tate, 2006. Causes and consequences of error in DEMs. Progress in Physical Geography 30(4): 467-489

Ground truth

DEM

12

3. 3. Reference StandardsReference Standards




National Standard for Spatial Data Accuracy (NSSDA, National Standard for Spatial Data Accuracy (NSSDA, US)US)

> 20%

> 20%

> 20%

> 20%

Federal Geographic Data Commitee U.S., 1998

Minimum distance between check points >0,10 diagonal

> 20 check points

Assumption of a normal distribution of residuals and the absence of systematic errors

13





National Standard for Spatial Data Accuracy (NSSDA, National Standard for Spatial Data Accuracy (NSSDA, US)US)

95% confidence level

Compiled to meet ...... meters vertical accuracy at 95% confidence level

Check points selection (finite sample N>20)

Differences between z DEM and z from an independent source of higher accuracy

RMSE calculationVertical accuracy = 1,96.RMSE

14





But is it enough? Some questions arise...But is it enough? Some questions arise...

It is assumed that residuals at check points follow a normal distribution and It is assumed that residuals at check points follow a normal distribution and systematic errors have been “reasonably” removed (no bias), which is known systematic errors have been “reasonably” removed (no bias), which is known as the “strong assumption”. as the “strong assumption”.

We need at least 20 check points. But it is supposing error normal We need at least 20 check points. But it is supposing error normal distribution. If not, how many check points do we need? 30, 50, maybe 100?distribution. If not, how many check points do we need? 30, 50, maybe 100?

Who controls the reliability of the accuracy

assessment process?

15





ASPRS Guidelines. Vertical accuracy reporting for ASPRS Guidelines. Vertical accuracy reporting for LiDARLiDAR

Non-open terrain

Non-open terrainOpen

terrain

Flood, M., 2004. http://www.asprs.org/society/divisions/ppd/standards/Lidar%20guidelines.pdf

Fundamental accuracy (NSSDA protocol)

Supplemental accuracy

16





ASPRS Guidelines. Supplemental accuracyASPRS Guidelines. Supplemental accuracy

Flood, M., 2004. http://www.asprs.org/society/divisions/ppd/standards/Lidar%20guidelines.pdf

Residuals

+

-

95th percentile = vertical accuracy at 95% confidence level

Maybe used regardless of whether or not the errors follow a normal distribution and whether or not errors qualify as outliers. 5% of the errors will be of larger value.

17

4. Reliability of DEM accuracy 4. Reliability of DEM accuracy measures?measures?




Bringing ReliabilityBringing Reliability

We need to quantify which is the error we are committing when we say “the RMSE of this DEM resulted to be ..... meters”

That error should depend on the number of check points used and somehow the “quality” of the sample from which we have computed the total error (RMSE).

18





The Li’s modelThe Li’s model

Li, Z., 1991. Effect of check points on the reliability of DTM accuracy estimates obtained from experimental tests. PE&RS 57(10): 1333-1340.

10012

1

)()(

NSdR

11

2

N

ee

Sd

N

ii )(

How many check points do we need to evaluate the error at a confidence level of 90% (R=10%):

puntosSdR

NSdR 5112

110

2

)(,)(

Hypothesis: normal distribution of errors and no bias

19





The Aguilar’s modelThe Aguilar’s model

Aguilar F.J. et al., 2007. A theoretical approach to modelling the accuracy assessment of DEMs. PE&RS 73(12): to be published in December.

Error population

NxxxxX ,...,,, 321

1

443

2

100100

2

21

222

2

)((%)

nR

RMSE

RMSESd

x

x

2 = standardised kurtosis = (4/4)-3

1 = skewness = 3/3

Any assumption, any restriction to use it

20




Morphology (cm) (cm) Skewness (1)

Kurtosis (2)

Mountainous 0.02 11.16 0.39 12.18

Rolling 1 0.01 2.72 0.84 13.20

Flat 0.01 2.08 0.64 23.99

Steep rugged hillside

0.48 41.01 0.60 21.55

Highly rugged -0.87 135.02 0.12 31.95

Slightly mountainous

0.12 6.36 1.12 21.12

Rolling 2 -0.08 1.84 -0.37 29.66


Residuals datasets (raw data)Residuals datasets (raw data)

Leptokurtosis

2>0

Platykurtosis

2<0

21





Residuals datasets (corrected data using 3-sigma rule)Residuals datasets (corrected data using 3-sigma rule)

Morphology (cm) (cm) Skewness Kurtosis Residuals removed (%)

Mountainous -0.12 8.20 -0.04 3.07 2.32

Rolling 1 -0.13 1.88 0.41 3.79 2.73

Flat -0.01 1.35 -0.04 4.15 2.16

Steep rugged hillside 0.22 30.34 0.04 3.16 1.90

Highly rugged -1.51 87.57 0.02 6.05 2.25

Slightly mountainous 0.03 4.42 0.10 4.39 2.49

Rolling 2 -0.03 1.11 -0.25 5.11 2.15

22





Model validation using Monte Carlo simulation methodModel validation using Monte Carlo simulation method

Reliability predicted (%)

Rel

iabi

lity

obs

erve

d (%

)

0 10 20 30 400

10

20

30

40


Rel

iabi

lity

obs

erve

d (%

)

0 20 40 60 800

20

40

60

80


Rel

iabi

lity

obs

erve

d (%

)

0 10 20 30 400

10

20

30

40


Rel

iabi

lity

obs

erve

d (%

)

0 10 20 30 40 50 600

10

20

30

40

50

60

Aguilar et al., 2007 (raw data) Aguilar et al., 2007 (filtered data)

Li, 1991 (raw data)

R2=97.32% R2=99.28%

R2=58.07% R2=82.61%

Li, 1991 (filtered data)

23





Visualisation of theoretical modelVisualisation of theoretical model

N (number of check points)

Kurtosis

Rel

iabi

lity

(%

)

0 50 100150200250300350400 010

2030

40

010203040506070

24

5. Accuracy assessment of LiDAR derived 5. Accuracy assessment of LiDAR derived DEMsDEMs




Estimating LiDAR vertical accuraciesEstimating LiDAR vertical accuracies

Non open terrain

25





Model overviewModel overview

Error population

NxxxxX ,...,,, 321

Non-parametric approach using Estimating Functions Theory for computing mean error

confidence intervals

Statistical inference from N check points

(sample size)

Godambe, V.P., 1991. Estimating functions. Oxford University Press, Oxford, 356 pages.

26

2

122

422

1

2122

2

1

2

1

2

N

t

xm

mmm

m

m

m

m

upper

))((





Estimating LiDAR vertical accuraciesEstimating LiDAR vertical accuracies

2

122

422

1

2122

2

1

2

1

2

N

t

x

mmm

m

m

m

m

lower

))((

and Being 22m

11m NN

Sd tx upperupper

Sd tx lowerlower

Aguilar, F.J. and Mills, J.P. Accuracy assessment of LiDAR derived digital elevation models. The Photogrammetric Record, under review.

27





Datasets from EuroSDR project on laser scannerDatasets from EuroSDR project on laser scanner

7 datasets with 15 reference data

28





Datasets from EuroSDR project on laser scannerDatasets from EuroSDR project on laser scanner

TerrascanTM last pulse data filtering

Comparison with

reference data

Error datasets for

non open terrain

29





Error datasets from EuroSDRError datasets from EuroSDR

Samples Points Mean (m) Sd (m) γ1 γ2 % data outliers

11 16995 0·16 0·44 3·30 11·82 3·5

12 25203 0·04 0·16 6·26 54·41 0·5

21 9742 0·02 0·05 1·83 4·77 3·1

22 21193 0·04 0·13 5·12 32·39 1·7

23 10871 0·05 0·20 5·06 31·26 1·4

24 3695 0·04 0·17 5·31 45·81 1·6

31 15315 0·01 0·04 1·29 4·16 1·4

41 1626 0·25 1·11 5·05 24·41 1·6

42 11743 0·02 0·07 2·98 13·22 2.0

51 13701 0.00 0·06 0·26 5·32 1·1

52 17368 0·08 0·30 2·52 8·41 1·7

53 24702 0·16 0·71 6·64 57·08 1·6

54 3863 0.00 0·08 2·79 19·27 2·1

61 31057 0·01 0·10 3·84 24·07 1·5

71 12517 0·02 0·11 2·33 10·80 2·1

30




0

0.15

0.3

0.45

0.6

0.75

0.9

1.05

1.2

1.35

1.5

0 20 40 60 80 100 120

Number of check pointsA

ccur

acy

(m)

0

10

20

30

40

50

60

70

Rel

iabi

lity

(%)

Fundamental vertical accuracy

Supplemental vertical accuracy

truth accuracy (95th method)

Reliability for fundamental accuracy calculation

Reliability for supplemental accuracy calculation


Results for EuroSDR error datasetsResults for EuroSDR error datasets

75

80

85

90

95

100

0 20 40 60 80 100 120

Number of check points

Per

cent

age

of e

rror

s w

ithin

co

mpu

ted

inte

rval

s

0

0.5

1

1.5

2

2.5

3

3.5

4

4.5

5

Rel

iabi

lity

(%)

Aguilar and Mills model Reliability

Results corresponding to dataset 1, sample 1

31

-1-0.8-0.6-0.4-0.2

00.20.40.60.8

11.21.4

0 20 40 60 80 100 120

Number of check points

Err

or v

alue

(m)

upper bound lower bound population mean

NSSDA upper NSSDA lower





Results for EuroSDR error datasetsResults for EuroSDR error datasets

32

6. Modelling error for LiDAR derived 6. Modelling error for LiDAR derived DEMsDEMs



2222filteringgriddingSDEtotal 222

griddingSDEtotal


Outlining the approachOutlining the approachNon-open terrainOpen

terrain

33





Outlining the approachOutlining the approach

222griddingSDEtotal

Computation at N check points on open terrain 22864602 03181 ILSDEgridding M ..

IDW method power to 2

34

0

1

2

3

4

5

6

7

8

9

0 1 2 3 4 5 6 7 8 9

Sd predicted (m)

Sd

ob

se

rve

d (

m)





Calibrating the empirical component (Information Loss)Calibrating the empirical component (Information Loss)

499870973070275840 .... DSlopeIL

29 morphologies of 4 has with average slopes ranging from 3% to up to 82%

R2 = 0.9856

35





Validating the modelValidating the model

33 GPS-obtained check points

Dataset of LiDAR data captured by Riegl Q560 sensor in August 2006 over Bristol area (Ordnance Survey project). Average density > 0.5 points/m2

With the permission of the Ordnance Survey

Sd = 0.124 m

max error = 0.37 m

min error = -0.17 m

mean error = 0.04 m

36





Validating the modelValidating the model

00.10.20.30.40.50.60.70.80.9

0 0.01 0.02 0.03 0.04 0.05 0.06

Lidar points density (points/m2)

Err

or

(m)

estimated error

observed error

spacing 4.4 m

spacing 23.5 m

37





Behaviour of the modelBehaviour of the model

0

1

2

3

4

5

6

0 0.02 0.04 0.06 0.08 0.1 0.12

D (poins/m2)

To

tal e

rro

r (m

)

0.2 slope

0.6 slope

0.8 slope

1 slope

spacing 3.1 m

spacing 4.1 m

SDE = 0.15 m

spacing 7.1 m

38

7. Conclusions7. Conclusions




Very little work has been done to determine the minimum data requirements for specific applications of DEMs, although there is a increasing tendency to collect larger volumes of elevation data. In the majority of the cases it is preferable to have an optimised DEM adapted to our needs rather than to have a vast amount of data, which will be more difficult to handle.

The reference standards methods for accuracy assessment of DEMs are based on hypothesis very restrictive and sometimes not according to reality, above all in the case of LiDAR data un non open terrain.

The tools expound in this talk are seeking to establish more general protocols for testing the quality of the product delivered from the part of the producer or even checking the quality of the own control quality, if there was.

39

8. That’s all8. That’s all




Thank you very much Thank you very much for your kind attention for your kind attention

geomatics and water resources research group seminars autumn term 2007

Documents

case of lidar derived

dem accuracy measures

spaina methodological

modelling lidar error

ya dem

type of dem

responsible dem user

error propagation