108 PHOTOGRAMMETRIC ENGINEERING

TABLE 3

COMPARISON OF ELEVATION ACCURACY \\'lTH

AND ''''lTHOUT GROUND CONTROL

say, they represent a shift in the datum forelevations, and could be removed if a singlevertical-control point were present. Two con­trol points at the X limits of the stereo-modelwould permit removal of the total errorcaused by camera altitude. These effects aresummarized in Table 3.

It is apparent from Table~3-that auxiliary

Standard error Contour intervalControl

Auxiliary dataOne pointTwo pointsComplete control

17.1 ft.9.8 ft.5.5 ft.2.2 ft.

56 ft.33 ft.18 ft.7 ft.

data, at the present state of accuracy, is notan adequate substitute for ground-control.The suitable contour interval is increased by afactor of 8 above that obtained with com­plete ground-control. The addition of a singlecontrol point reduces the factor to 5, and twocontrol points reduce it to less than 3.

This investigation has been concerned onlywith a single stereo-model. Quite obviouslythe errors propagated from the auxiliary datacan be reduced somewhat if the stereo-modelsare triangulated between reasonably spacedcontrol-points. However, at the moment, noadequate means exists for utilizing all theauxiliary data in an aerial triangulation. Inthe future this may be accomplished by ananalytical solution in which the auxiliarydata are imposed as constraints with weightsinversely proportional to their variances.

"Accuracy and Precision zn Photogrammetry"*

BERTIL P. HALLERT,

Inst. for PhotogrammetriKungl Tekniska Hdgskolan

Stockholm 70, Sweden

ABSTRACT: The main task of photogrammetry is to measure geometrical datawith the aid of photographs. The measurements must be made with sufficientgeometrical quality at lowest possible costs and within the shortest possible time.

Since the quality of the measurements is of fundamental importance for theapplication of photogrammetry, the methods to determine this quality in actualcases and the terminology to express the results must be of great interest.

In photogrammetry as well as in other sciences of measurement, a certain con­fusion concerning the terminology for quality is frequently found. There aremany possible interpretations of common expressions for quality and in mostcases the reader does not know the real meaning behind expressions like "theresults were obtained with an accuracy (precision) of ...." In this paper somepoints of view of a highly desirable standardization of the terminology forquality in photogrammetry are given.

INTRODUCTION

I N THE literature on measurements, not onlyconcerning photogrammetry and geodesy,

but other sciences as well, the geometricalquality of basic data and of final results issometimes expressed in vague and unclearterms. In reading a paper or an advertise­ment, or when listening to an oral presenta­tion on measu remen ts or instru men ts, expres­sions of the following type will sooner or laterappear:

The measurements have (have been made

with, can be made with). Or the instrumenthas....

An accuracy of ... (for instance 1 micron),a precision of ... (± 1 micron).

Sometimes we also find expressions like:a reliability of ... (10 feet), a geometricalquality of ... , an uncertainty of ... , orsimply, an error of ... , and in other casesaccuracy within ... , precision within ... ,accurate to ... , precise to....

All of these expressions may refer to one ormore of the following proper concepts and

* Presented at St. Louis ACSM-ASP Convention, Sept. 10, 1962.

"ACCURACY AND PRECrSION IN PHOTOGRAMMETRY" 109

terms which are found in various textbooksand other pu blications on statistics and theoryof errors for measurements:

Standard deviation (referred tu onc singlemeasurement or to the mean of several re­peated or replicated measurements), standarderror, probable error, standard error of uni tweight, root mean square error, mean error,average error, discrepancy, root mean squarevalue of discrepancies, closing error, standardclosing error, variance, sigma, two-or-threesigma, the range, the semi-interquartile range,various ki nds of con fidence Ii mi ts on certai nand different confidence levels, tolerancelimits, maximum errors, maximum deviations,etc., etc.

It is certainly not easy tu find one's waythrough this jungle of expressions and tointerpret correctly quality information asgi ven by the terms: accu racy of ... , preci­sion of ... , accurate to ... , and preciseto.... I t is not easy ei ther to interpret themagnificent expression "exact results of me­dium accuracy" which is given in a goodmodern textbook of photogrammetry.

In many cases the consequences of suchinexact expressions might not be too serious.But sometimes7)t is of basic importance toknow what the author really means. In par­ticular, for investigations of the error propaga­tion from the actual observations, and for thedetermination of tolerances for instrumentsand procedures, it is absolutely necessary toknow the real meaning of the quality expres­sions.

It is for instance not satisfactory to char­acterize the geometrical quality of an instru­ment with the expression that it has an ac­curacy of one micron because there are di ffer­ent opinions as to what such an expressionreally means. For a definite reason an investi­gation concerning the interpretation of thisexpression was made among prominent sci­entists in the field of technology in Europe,many of whom are responsible for teaching.Out of twenty persons: as to what an "ac­curacy of one micron" means to them:

Five interpreted the expression as stand­ard deviation of one measurement.

Eleven understood it to mean maximumerror.

One proposed its meaning to him as prob­able error but proceeded to define this con­cept incorrectly.l

Three said they did not know. Theirs weredou btless the best answers.

I It should be noted that the "probable error" isnot most probable to occur.

Also in this country a number of scientists"'ere asked about their interpretation. A verycom mon answer is that an "accuracy of onemicron" means "probable error."

In summary, the existing jungle of conceptsand terms for geumetrical quality of measure­ments needs to be cleared, and some kind ofstandardization is doubtless highly desirable.This has also been noted by the In ternationalSociety for Photogrammetry \\·ho made somegeneral statements in the resolutions from theLondon Congress in 1960. The resolutions 2aand 4 from Commission II are as follows:

"All information on accuracy should beexpressed in clear and well-defined terms.This information should be combined withdata about its reliability, e.g. by giving thenumber of redundant observations."

"In order to compare the results of differ­ent theoretical and practical investigationsinto instru men ts and methods, it is sug­gested that the observations are adjustedaccording to the method of least squares."

A committee was also appointed for furtherinvestigations into this difficult and contro­versial problem.

What can now be done?

First it is obvious that a close cooperationbetween statistics, geodesy, photogrammetry,and other measuring sciences is very desirable.This has also been proposed. It is probablyimpossible to find a terminology which willsatisfy all of these branches cumpletely be­cause of the great variety of the measuringmethods and the basic data. In geodesy, forinstance, the basic data-angles and distancesare generally directly measlll'ed and the basicmeasurements can often be repeated orreplicated arbitrarily, for the purpose ofcorrection of possible regular or systematicerrors before further computations, and inorder to increase the geometrical quality ofthe averages.

In photogrammetry, the basic data are theimage coordinates, which always are functionsof at least nine parameters, namely the ele­ments of the interior and exterior orientation,which, however, can compensate each otherto a certain extent. In addition there areregular errors, for instance radial and tangen­tial distortion, affinities, etc. The imagingprocedure usually takes place during a veryshort time and cannot be replicated. Conse­quently regular or systematic errors of theimage coordinates must playa yery importantrole in the basic photogrammetric data.Calibration procedures of cameras and instru-

110 PHOTOGRAMMETRIC ENGINEERING

La Lv'=na-La=Oii= --

n

2 Replicated measurements are made at one placeand at one period of time.

Repeated measurements ~re mad~ at differentplaces and at different penods of time. (Kendalland Buckland)

If the geometrical quality of the measureddata is known, the geometrical quality of xcan be determined from the special law oferror propagation. For the determination ofthe geometrical quality of the angles, twoprinciples can be utilized namely:

1. Replicated2 or repeated measuremen ts ofeach angle and statistical study of the devia­tions between the individual measurementsand the average. This gives the precision.

2. Comparison between the sum of thethree angles with the condition that this sumshall be 1800 or 200g

• The accuracy can bedetermined from the discrepancy.

1. THE PRECISIO:-<

The measurements of each angle are repli­cated or repeated an arbitrary number oftimes n and the averages of the measurementsare to be used as the final angles. The preci­sion of the measurements of the angles and ofthe averages can then be determined as fol­lows: The measurements of the angle a havegiven the following results:

Correction,sto each angle

j1JJeasured with respectA ngles to the

average

V/ = ii - at

V2' = ii - 0':2

v,/ = ii - an

+ VI' = a+ v2' = a+ v,/ = iian

al

which repeated observations conform tothemsel ves.

A similar definition has been given by H. C.Mitchell in a paper: "Precision and Accu­racy," published in Military Engineer, ov.:­Dec., 1950. This paper refers to another publI­cation: "Definitions of Terms Used in Geo­detic and Other Surveys," also by H. C.Mitchell and published by the Coast and Geo­detic Survey in 1948 as a special publicationNo. 242. As an illustration of the two concepts,measu remen ts and checks of the angles inatriangle will be shown, see Figure 1.

Assume the side x of the triangle ABC to becomputed from the measured side a and thethree angles a, {3 and "y:

a sin {3x=~~

B

FIG. 1.

c

x = a 510,8sin c(

A

ments are therefore of fundamental importancefor the photogrammetric procedure but mustalso be made under real operational conditionsin addition to laboratory work.

Regular or systematic errors do not lendthemselves to "probabilistic" treatment andare therefore not particularly observed instatistics. In fact, the determination of thegeometrical quality of calibr~tion proc~d~res

is given very little treatment 111 .the statIs.tlcalliterature and the terminology IS not entIrelysatisfacto'ry for the photogrammetric prob­lems.

In order to survey the opinion amongAmerican photogrammetrists concerning pos­sible terms to be used, a draft containing someexisting concepts and terms ~~s been dis­tributed with a request for cntlcal remarksand positive proposals. The ans\~ers.showverydifferent opinions and clearly 1I1dlcate thatthere is a real need for a well-defined ter­minology. The work which is b~ing perf?rmedin this respect within the Amencan SocIety ofPhotogrammetry and within the I~terna­

tional organizations is therefore very I mpor­tanto Close cooperation is obviously muchdesired.

In this paper only two terms will b~ ~urther

discussed, namely accuracy and prec1swn be­cause these are so widely used and seem to beof particular importance.. . .

Clear definitions are given In the statIstIcalliterature of the two concepts. The book-"ADictionary of Statistical Terms"-by Kend.alland Buckland, published in 1957, conta1l1sthe following defini tions:

A ccuracy in the general statistica! sensedenotes the closeness of computatIOns orestimates to the exact or true values.

Precision is a quality associated with a cla~s

of measurements and refers to the way 111

"ACCURACY AND PRECISION IN PHOTOGRAMMETRY" 111

e = a + i3 + i' - 180°

Sa = 1~~12n - 1

The standard det'iation of the AVERAGE IS

then found as

The preCISion of the measurements of theangle a can now, according to the definition,be determined from the conformi ty of theobservations to themselves i.e. from the cor­rections VI' to vll '. According to well-knownformulas the standard deviation of ONE observa­tion is

2. THE ACCURACY

Disregarding possible inAuence of sphericalexcess, the discrepancy e can be regarded as anindication of the accuracy of the measuredangles because it denotes the closeness of thecomputation (a+~+-Y) to the correspondingexact or true value 180°.

For determining a formal expression for theaccuracy according to usual procedures, thediscrepancy is distributed to the measureddata i.e. the angles a, ~ and 'Y. Assuming the

3 The corrections to measured data are denotedv according to established practice. v is the firstletter in the German word Verbesserung. Note thedifference between v' and v.

4 Because there is only one redundant quantitythe determination of the accuracy is weak.

geometrical quality of the angles to be mu­tually equal-that is they are regarded ashaving the same weights-the discrepancy isdi vided inlo three equal parts e/3 and theseare interpreted as the error of each angle. Thecorrection3 to each angle is consequently-e/3 = v. The su m of the squares of the cor­rections is

e2

.L v2 =-3

There is only one redundant angle or onedegree of freedom in this adjustment, and thestandard error of unit weight becomes conse­quently according to usual procedures

So = I.L~=~3 - 2 yl3

This is the expression for the basic acCitracyof each of the three angles, obtained after anadjustment of the discrepancy in the condi­tion, in fact, according to the method of leastsquares.4 More sources of errors are includedin this determination of the geometricalquality (the accuracy) of the angles than fromrepeated measurements only, which gives theprecision of one measurement and of theaverage.

The precision expressed in terms of stand­ard deviation of the average may be regardedas the Ii mi ti ng val ue of the accuracy deter­mined as standard error of unit weight fromthe adjustment of discrepancies in suitableconditions, which should be as reliable or"tight" as possible. In photogrammetry thereare many similar relations, sometimes morecomplicated than the simple example dis­cussed. The replicated measurements ofparallaxes can give good information aboutthe precision of individual measurements andof the average of replicated measurements,but the accuracy of the measured parallaxesmust be determined from conditions, as forinstance those of the relative orientation thatall pairs of rays shall intersect, or from eleva­tion differences, determined by geodeticmeasurements. For determining the geo­metrical quality of a measuring operation orfor the study of the error propagation througha measuring procedure-for instance photo­grammetry-the most reliable information isobtained as accuracy, expressed as standarderrors in terms of functions of standard errors

/ .L V'2

1/--n(n - 1)

Sas;;=-=

yin

A very important consequence of thisprinciple is that the standard deviation of theaverage decreases with the square root of thenumber of the observations, and that at leasttheoretically the standard deviation of theaverage can be arbi trarily decreased, or thatthe precision of the average can be arbitrarilyincreased, in repeating or replicating theobservations a sufficient number of times.This increase is in practice limited by the factthat not all errors may show up in the devia­tions between the average and the individualobservations.

Errors in the centering of the theodoliteand signals or side refraction may affect theindividual angles with equal amounts and willtherefore affect the averages also. Such errorsmay be interpreted as correlations. The cen­tering of signals and theodolites, and theleveling of the instruments etc., are in factalso measuring operations which never can bemade entirely free from errors. Even if thenature of these errors may be of irregularcharacter, the effect upon the angles can be ofconstant or systematic character. Therefore,even if the individual angles are mcasured aninfinite number of times it cannot be expectedthat the su m of the angles will become ex­actly 180° or 200g

• l\ discrepancy e in thiscondition is to be cxpected according to theformula:

112

c1

PHOTOGRAMMETRIC ENGINEERING

y

r1------------o14

'If----~7---;t---C)---p--=--. x

FIG. 2. Determination of the basic accuracy of elevation measurementsfrom least squares adjustment of indirect observations.

Least square adjustment with one parameter dhoregarded as a constant error.

v, + e, = dho; v, = dlto - e,.............v" + e" = dlto; Vn = dlto - en

Normal equation:

ndho = I:eI:e

dho=­n

(I:e)2I:v2 = I:e2- ~~­

n

So = .lI:v2

'V n - 1

Sos"o= -----

v2(n - 1)

50= Standard error of unit weight from one-parameter

adjustment.

of unit weight of the basic observations.5 Forsuch purposes the method of least squares is ofgreat importance, in particular because a welldefined, unique procedure is given which usu­ally leads to very simple computations, seeFigure 2. It is a special case of the famousMaximum likelihood method from statistics.

It must, however, be emphasized that themethod of least squares is no magic procedurewhich can convert poor observations intoresults of high geometrical quality. But forinvestigating the geometrical quality of thebasic observations and in particular for thedistinguishing between regular and irregular

5 In order to increase the reliability of the de­termination of the standard error of unit weight,the number of redundant measurements shouldbe high.

Least squares adjustment with three parametersdho, d7l, and df

vl(+el) = dho+ X,d71 + Yld~ - e,. .v"(+e") = dho+ Xnd71 + Y"d, - en

Normal equations:

ndlto = I:ed'lI:x2 + d~I:XY = I:xe

d7lLXY +d~LY2 = LYe

Lv2 = Le2- dhoI:e - d'lLXe - d~I:Ye

So = . iI:v2

'V n - 3

SoS"o= ----~

v2(n - 3)

So = Standard error of unit weight from 3-parameteradjustment.

errors of such observations-primarily inconnection with calibration procedures-themethod is the most powerful tool that we haveand we should use it a great deal more.

In applying the method of least squares tosuch problems the need for a clear terminol­ogy becomes evident. In particular it is de­sirable that the terms accuracy and precisionbe used in qualitative sense only, and not inconnection with quantitative information aboutthe geometrical quality.

Theory of errors and adjustment proceduresbecome more and more important in all meas­uring sciences with the increased demand forhigher geometrical quality; also in photogram­metry. It is therefore necessary that the basicconcepts and terms be well defined and clearlyexpressed. Otherwise the entire developmentmay become a giant on feet of clay.