on the measurement of intangibles. a principal eigenvector

On the Measurement ofIntangibles. A PrincipalEigenvector Approach toRelative MeasurementDerived from PairedComparisonsThomas L. Saaty

IntroductionNearly all of us have been brought up to believethat clear-headed logical thinking is our only sureway to face and solve problems. But experiencesuggests that logical thinking is not natural to us.Indeed, we have to practice, and for a long time,before we can do it well. Since complex problemsusually have many related factors, traditionallogical thinking leads to sequences of ideas sotangled that the best solution cannot be easilydiscerned.

For a very long time people believed and arguedstrongly that it is impossible to express the intensityof human feelings with numbers. The epitome ofsuch a belief was expressed by A. F. MacKay whowrites [12] that pursuing the cardinal approachesis like chasing what cannot be caught. It wasalso expressed by Davis and Hersh [5]: “If youare more of a human being, you will be awarethere are such things as emotions, beliefs, attitudes,dreams, intentions, jealousy, envy, yearning, regret,longing, anger, compassion and many others.These things—the inner world of human life—can never be mathematized.” In their book [11]

Thomas L. Saaty holds the Chair of Distinguished Uni-versity Professor at the University of Pittsburgh, withappointments in several departments. His email address [email protected].

DOI: http://dx.doi.org/10.1090/noti944

Lawrence LeShan and Henry Margenau write: “Wecannot as we have indicated before, quantifythe observables in the domain of consciousness.There are no rules of correspondence possiblethat would enable us to quantify our feelings.We can make statements of the relative intensityof feelings, but we cannot go beyond this. Ican say: I feel angrier at him today than I didyesterday. We cannot, however, make meaningfulstatements such as, I feel three and one half timesangrier than I did yesterday. . . . The physicists’schema, so faithfully emulated by generations ofpsychologists, epistemologists and aestheticians,is probably blocking their progress, defeatingpossible insights by its prejudicial force. Theschema is not false—it is perfectly reasonable—butit is bootless for the study of mental phenomena.”

The Nobel Laureate Henri Bergson [4] writes:“But even the opponents of psychophysics do notsee any harm in speaking of one sensation asbeing more intense than another, of one effort asbeing greater than another, and in thus settingup differences of quantity between purely internalstates. Common sense, moreover, has not theslightest hesitation in giving its verdict on thispoint; people say they are more or less warm, ormore or less sad, and this distinction of more andless, even when it is carried over to the region ofsubjective facts and unextended objects, surprisesnobody.”

192 Notices of the AMS Volume 60, Number 2

Figure 1. Decomposition of the problem into a hierarchy.

If we were to ask what in practical terms mea-surement means, one would most likely propose ascale that is applied to measure objects: a set ofnumbers, a set of objects and a mapping from theobjects to the numbers. Then we can agree thatappropriate judgment must be used to interpretthe scale readings and use them in practice. Sojudgment is also essential. But there is anotherway to think of a scale.

Henri Lebesgue, who was concerned with ques-tions of measure theory and measurement, wrote[10]:

“It would seem that the principle of econ-omy would always require that we evaluateratios directly and not as ratios of measure-ments. However, in practice, all lengths aremeasured in meters, all angles in degrees,etc.; that is we employ auxiliary units and,

February 2013 Notices of the AMS 193

Table 0. Fundamental scale of absolute numbers.

as it seems, with only the disadvantage ofhaving two measurements to make insteadof one. Sometimes, this is because of exper-imental difficulties or impossibilities thatprevent the direct comparison of lengths orangles. But there is also another reason.

“In geometrical problems, one needs tocompare two lengths, for example, and onlythose two. It is quite different in practicewhen one encounters a hundred lengths andmay expect to have to compare these lengthstwo at a time in all possible manners. Thusit is desirable and economical procedureto measure each new length. One singlemeasurement for each length, made asprecisely as possible, gives the ratio of thelength in question to each other length.This explains the fact that in practice,comparisons are never, or almost never,made directly, but through comparisonswith a standard scale.”

Lebesgue did not go far enough in examining whywe have to compare. When we deal with intangiblefactors, which by definition have no scales ofmeasurement, we can compare them in pairs.Making comparisons is a talent we all have. Not

only can we indicate the preferred object, but we canalso discriminate among intensities of preference.The philosopher, Arthur Schopenhauer [25], said,“Every truth is the reference of a judgment tosomething outside it, and intrinsic truth is acontradiction.”

A common kind of decision problem we face issomething like this: which house to buy? The differ-ent houses being considered have some attributesor criteria in common that are important to thedecision maker. If one house were best on everycriterion, the choice would be easy, but usually thehouse that is the best on one criterion (e.g., cost) isworst on another (e.g., size). How should one makethe tradeoff? We describe and discuss a mathemat-ical model, the Analytic Hierarchy Process (AHP),not a prescriptive (normative) but a descriptivepsychophysical process that can be used to makesuch decisions by dealing with the measurementof intangibles using human judgment. Intangiblescan be nonphysical influences that are passingand very transient. No conceivable instrument canbe devised to measure them other than the minditself, which must also interpret their meaning.Intangibles leave an impact on our minds, whichare biologically endowed to respond to influences,


Table 1. The family’s pairwise comparison matrix for the criteria.

Table 2. Pairwise comparison matrices for the alternative houses.

Table 3. Distributive and ideal synthesis.


by making comparisons both consciously andsubconsciously. It is a way of measurement thattook place long before Nicole Oresme, Pierre deFermat, and finally René Descartes more rigor-ously introduced general coordinate systems forphysical measurements and assumed that theywere extensible from zero to infinity by usingan arbitrarily chosen unit applied uniformly overthe entire range of measurement. Taking ratiosremoves the arbitrariness of the unit and createsrelative absolute scales, invariant under the iden-tity transformation. The example below is morestaid than dealing with evanescent phenomena likepolitical decisions, but it serves to illustrate theideas whose mathematical foundations have beendeveloped in a separate paper and in books.

Choosing the Best HouseConsider the following (hypothetical) example: afamily wishing to purchase a house identifies eightcriteria that are important to them. The problem isto select one of three candidate houses. The firststep is to structure the problem into a hierarchy(see Figure 1). On the first (top) level is the overallgoal of Satisfaction with House. On the second levelare the eight criteria that contribute to the goal, andon the third (bottom) level are the three candidatehouses that are to be evaluated by considering thecriteria on the second level.

The criteria important to the family are:1. Size of House: Storage space; size of rooms;

number of rooms; total area of house.2. Transportation: Convenience and proximity

of bus service.3. Neighborhood: Degree of traffic, security,

taxes, physical condition of surrounding buildings.4. Age of House: How long ago the house was

built.5. Yard Space: Front, back, and side space and

space shared with neighbors.6. Modern Facilities: Dishwasher, garbage dis-

posal, air conditioning, alarm system, and othersuch items.

7. General Condition: Extent to which repairs areneeded; condition of walls, carpet, drapes, wiring;cleanliness.

8. Financing: Availability of assumable mortgage,seller financing or bank financing.

The next step is to make comparative judgments.The family assesses the relative importance ofall possible pairs of criteria with respect to theoverall goal, Satisfaction with House, comingto a consensus judgment on each pair; andtheir judgments are arranged into a matrix. Thequestions to ask when comparing two criteria are,which is more important and how much moreimportant is it with respect to satisfaction with ahouse?

The matrix of pairwise comparison judgmentson the criteria given by the home-buyers inthis case is shown in Table 1. The judgmentsare entered using the fundamental scale of theAnalytic Hierarchy Process (AHP) Table 0: a criterioncompared with itself is always assigned the value1 so the main diagonal entries of the pairwisecomparison matrix are all 1. We are permitted tointerpolate values between the integers, if desired.Reciprocal values are automatically entered in thetranspose position, so the family must make atotal of twenty-eight pairwise judgments.

We have assumed that an element with weightzero is eliminated from comparison because zerocan be applied to the whole universe of factors notincluded in the discussion.

The foregoing integer-valued fundamental scaleof response used in making paired comparisonjudgments can be derived from the logarithmic re-sponse function of Weber Fechner in psychophysicsas follows. For a given value of the stimulus, themagnitude of response remains the same until thevalue of the stimulus is increased sufficiently largein proportion to the value of the stimulus, thuspreserving the proportionality of relative increasein stimulus for it to be detectable for a newresponse. This suggests the idea of just noticeabledifferences (jnd), well known in psychology. Thus,starting with a stimulus s0, successive magnitudesof the new stimuli take the form [2]

s1 = s0 +∆s0 = s0 + ∆s0s0s0 = s0(1+ r),

s2 = s1 +∆s1 = s1(1+ r) = s0(1+ r)2 ≡ s0α2,...

sn = sn−1α = s0αn (n = 0,1,2, . . . ),∆s0s0= ∆s1s1= ∆s2s2= · · · .

We consider the responses to these stimuli tobe measured on a ratio scale (b = 0). A typicalresponse has the form Mi = a logαi , i = 1, . . . , n,or one after another they have the form

M1 = a logα, M2 = 2a logα, . . . ,Mn = na logα.

We take the ratios Mi/M1, i = 1, . . . , n, of theseresponses in which the first is the smallest andserves as the unit of comparison, thus obtainingthe integer values 1,2, . . . , n of the fundamentalscale of the AHP. It appears that numbers areintrinsic to our ability to make comparisons andthat they were not an invention by our primitiveancestors. We must be grateful to them for thediscovery of the symbolism. In a less mathematicalvein, we note that we are able to distinguishordinally between high, medium, and low at onelevel and for each of them in a second level belowthat also to distinguish between high, medium, and


low, giving us nine different categories. We assignthe value one to (low, low), which is the smallest,and the value nine to (high, high), which is thehighest, thus covering the spectrum of possibilitiesbetween two levels and giving the value nine for thetop of the paired comparisons scale as comparedwith the lowest value on the scale. The scaleconsists of absolute numbers, which unlike thefamiliar numbers that belong to a ratio scale thatare invariant under a similarity transformation(multiplication by a positive number) are invariantunder the identity transformation.

In his book The Number Sense: How the MindCreates Mathematics [6], the mathematician andcognitive neuropsychologist Stanislas Dehaenewrites more than twenty-five years after we derivedthis scale that “Introspection suggests that wecan mentally represent the meaning of numbers 1through 9 with actual acuity. Indeed, these symbolsseem equivalent to us. They all seem equally easy towork with, and we feel that we can add or compareany two digits in a small and fixed amount oftime like a computer. In summary, the inventionof numerical symbols should have freed us fromthe fuzziness of the quantitative representation ofnumbers.”

It can be shown that when comparisons involvegreater contrast than 9, the elements can beaggregated into clusters, with a pivot element fromone cluster to an adjacent one, again applyingthe same kind of comparisons within the nextcluster using the scale 1–9. One then divides bythe weight of the pivot element in the secondcluster and multiplies by its weight from the firstcluster, and now the priorities in the two clustersare comparable and can be put together and soon. This type of clustering is called homogeneousclustering [22]. In addition, as we shall see later,for stability of the priorities with respect to smallchanges in judgment, each cluster must not containmore than a few elements: about seven [20].

In the AHP model, the vector of priorities forthe criteria is obtained by computing the principaleigenvector, the classical Perron vector, of thepairwise comparison matrix. The Perron vector isa necessary condition for obtaining the prioritieswhen the judgments are inconsistent. As we shallsee below, consistency and near consistency areimportant concepts in our considerations. Becausethe pairwise comparison matrix has positive entries,Perron’s theorem ensures that there is a uniquepositive vector (denoted by w) whose entries sumto one that is an eigenvector of the pairwisecomparison matrix, associated with an eigenvalue(denoted by λmax) of strictly largest modulus. Thateigenvalue, the Perron eigenvalue, is positive andalgebraically simple (multiplicity one as a root ofthe characteristic equation) [14]. Consistency of

the family’s set of judgments is measured by theconsistency ratio (C.R.), which we explain later.

Table 1 shows that size dominates transporta-tion strongly since a 5 appears in the (size,transportation) position. In the (finance, size) po-sition we have a 4, which means that finance isbetween moderately and strongly more importantthan size. The priority vector shows that financingis the most important criterion to the family asthe entry of w corresponding to finance has thelargest value, 0.345.

Consistency, which was alluded to previously, isan elaboration of the common sense view expressedin this statement: if you prefer spring to summerby 2, summer to winter by 3, and spring to winterby 6, then those three judgments are consistent.

The family’s next task is to compare the housesin pairs with respect to how much better (moredominant) one is than the other in satisfying eachof the eight criteria. There are eight 3-by-3 matricesof judgments since there are eight criteria andthree houses are to be compared for each criterion.The matrices in Table 2 contain the judgments ofthe family. In order to facilitate understanding ofthe judgments, we give a brief description of thehouses.

In Table 1 an element on the left of the matrix iscompared for dominance over another at the top.

House A: This house is the largest. It is locatedin a good neighborhood with little traffic and lowtaxes. Its yard space is larger than that of eitherhouse B or C. However, its general condition is notvery good, and it needs cleaning and painting. Itwould have to be bank financed at a high interestrate.

House B: This house is a little smaller than houseA and is not close to a bus route. The neighborhoodfeels insecure because of traffic conditions. Theyard space is fairly small, and the house lacks basicmodern facilities. On the other hand, its generalcondition is very good, and it has an assumablemortgage with a rather low interest rate.

House C : House C is very small and has fewmodern facilities. The neighborhood has high taxesbut is in good condition and seems secure. Itsyard is bigger than that of house B but smallerthan house A’s spacious surroundings. The generalcondition of the house is good, and it has a prettycarpet and drapes. The financing is better than forhouse A but poorer than for house B.

In Table 2 both ordinary (distributive) and ide-alized priority vectors of the three houses aregiven for each of the criteria. The idealized priorityvector is obtained by dividing each element of thedistributive priority vector by its largest element.The composite priority vector for the houses isobtained by multiplying each priority vector bythe priority of the corresponding criterion, adding


across all the criteria for each house and thennormalizing. When we use the (ordinary) distribu-tive priority vectors, this method of synthesis isknown as the distributive mode and yields A = .345,B = .369, and C = .285. Thus house B is preferredto houses A and C in the ratios: .369/.346 and.369/.285, respectively.

In Table 2 again an element on the left of eachmatrix is compared for dominance over another atthe top. If the top element is dominant, a fraction isentered and its inverse, a whole number, is enteredin the reciprocal position.

When we use the idealized priority vector,the synthesis is called the ideal mode. This yieldsA = .315, B = .383, C = .302 and B is again the mostpreferred house. The two ways of synthesizing areshown in Table 3. They need not yield the sameranking. In general, the ideal mode is used whenrating the alternatives one at a time (see later)with respect to the criteria and when the criteriapriorities are independent of the alternatives. Theideal mode is used to force rank preservation whennew houses are added to the collection by onlycomparing them with respect to the ideal and notthe other alternatives. But always preserving rankis not always desirable [16].

The Pairwise Comparison MatrixIn comparing pairs of criteria with respect to thegoal, one estimates which of the two criteria ismore important and how much more. The result ofthese comparisons is arranged in a positive matrixA = [aij] whose entries satisfy the reciprocalproperty aji = 1/aij .

We start with a positive reciprocal matrix suchas Table 1. In an n-by-n table, n(n−1)/2 judgmentsmust be made, which is why the house-buyingfamily had to make (8 × 7)/2 = 28 judgments.These judgments are made independently, but theyare not really “independent”. If the family feels thatfinancing is twice as important as size and thatsize is twice as important as age, for consistencyof judgments we should expect them to feel thatfinancing is four times as important as age. Themathematical expression of our expectation is theset of identities

aij = aik/ajk for all i, j, k = 1, . . . , n

among the entries of a consistent pairwise com-parison matrix A = [aij]. Of course, real-worldpairwise comparison matrices are very unlikely tobe consistent, and we address the consequencesof that reality next [16], [19].

Why the Principal Eigenvector?

Suppose a positive square matrix A = [aij] isconsistent. ThenAmust have unit diagonal entries,since aii = aik/aik, for all i, k = 1, . . . , n. Moreover,

A must be reciprocal since aijaji = 1 means thataij = 1/aji . Such a matrix has a very simplestructure since aik = ai1a1k = ai1/ak1 for all i, k.Thus the entries in the first column of A determineall other entries! For convenience, write αi ≡ ai1,so that A = [aij] = [αi/αj]. If we define the twopositive n-by-1 vectors x ≡ [αi] and y ≡ [(αi)−1],then it is clear that A = xyT has rank one. Thus,the positive matrix A has one nonzero eigenvalueand n − 1 zero eigenvalues. It is easy to checkthat Ax =

∑nj=1(αi/αj)αj = [nαi] = nx, so the

nonzero eigenvalue of A (its Perron eigenvalue,which we have denoted by λmax) is n, and anassociated positive right eigenvector is x ≡ [αi]. Ifwe set c ≡ α1 + · · · + αn, the Perron vector of A(its unique positive eigenvector whose entries sumto one) may be written as w ≡ x/c = [αi/c] ≡ [wi].The Perron vector determines all the entriesof A : A = [aij] = [αi/αj] = [(αi/c)/(αj/c)] =[wi/wj]. When a matrix is consistent, right andleft eigenvectors have reciprocal correspondingentries.

We know that an n-by-n positive consistentmatrix A = [aij] has a unique positive eigenvectorw ≡ [wi] (its Perron vector) whose entries sumto one and whose corresponding eigenvalue (itsPerron eigenvalue) is n. Moreover, the ratios ofthe entries of w are precisely the entries ofA : aij = wi/wj . If we think of A as a matrix of(perfectly) consistent pairwise comparisons for ngiven elements, then the n values wi are a naturalset of priorities that underlie the set of pairwisejudgments: aij = wi/wj . There are several ways toprove that the principal eigenvector is necessarywhen the judgments are inconsistent [18], [21].

The foregoing discussion is intended to motivatethe central and critical choice of the Perron vectoras the means to extract a vector of prioritiesfrom a given pairwise comparison matrix in theAHP model. If humans made perfectly consistentjudgments all the time, the model would be perfect.But they do not, so we must now face the questionof assessing the deviation from consistency ofan actual pairwise comparison matrix and theconsequences of inconsistency for the quality ofdecisions made according to the AHP model. Inpassing, we observe that if humans were alwaysperfectly consistent, they would not be able to learnnew things that modify or change the relationsamong what they knew before and they wouldbe like robots. But there is a level of tolerableinconsistency that we must allow beyond whichthe judgments would appear to be uninformed,random, or arbitrary.

When we compare things, unlike assigningthem numbers independently of one another, theirpriorities always depend on what other thingsthey are compared with. Were we to assume that


the universe is interdependent (the subject of theAnalytic Network Process (ANP) [17], [18]) with acomplicated field of influences in every aspect, ourtraditional way of assigning numbers from scalesof measurement would not be “the natural way” todetermine their importance. More and more we arefinding with this relative way of thinking that theworld is different (has different rank orders) than wethink we understand it to be today. Another usefulobservation is that comparisons are necessaryfor comparing criteria to derive their prioritiesbecause there are no scales for their measurementand also because their importance varies fromdecision to decision. In the ANP the criteria arecompared with respect to each alternative and thealternatives are compared with respect to eachcriterion to derive their interdependent priorities.

Regrettably, the three laws of thought (identity,excluded middle, and contradiction), known toPlato and Aristotle and even to Leibniz, that arestrict requirements we all adhere to in language,logic, science, and mathematics have precludedcomparisons, our biological heritage. Withoutcomparisons nothing can be known in and of itselfwithout also knowing other things with which itis compared, including knowing it at an earliertime, so we can ensure it is the one we have inmind (the law of identity), thus recognizing it.Arthur Schopenhauer, who was not equipped todevelop a mathematical theory to use comparisons,listed the laws of thought by adding a fourthone in his On the Fourfold Root of the Principle ofSufficient Reason. (1) a subject is equal to the sumof its predicates, or a = a; (2) no predicate can besimultaneously attributed and denied to a subject,or a 6=∼ a; (3) of every two contradictorily oppositepredicates one must belong to every subject;and (4) truth is the reference of a judgment tosomething outside itself as its sufficient reasonor ground.

We are all familiar with the arbitrarily imposedaxiom in logic and mathematics that if A dominatesB and B dominates C, then A must dominate C.But in the real world team A beats team B, B beatsC but C beats A, contradicting theory. It appearsthat theory needs to be changed to accommodatereality.

When is a Positive Reciprocal MatrixConsistent?Let A = [aij] be an n-by-n positive reciprocalmatrix, so all aii = 1 and aij = 1/aji for all i, j =1, . . . , n. Let w = [wi] be the Perron vector of A, letD = diag(w1, . . . , wn) be then-by-n diagonal matrixwhose main diagonal entries are the entries of w ,and set E = D−1AD = [aijwj/wi] = [εij]. Then E issimilar toA and is a positive reciprocal matrix sinceεij = ajiwi/wj = (aijwj/wi)−1 = 1/εij . Moreover,

all the row sums of E are equal to the Perroneigenvalue of A:

n∑j=1

εij =∑jaijwj/wi = [Aw]i/wi

= λmaxwi/wi = λmax.

The computation(1)

nλmax =n∑i=1

n∑j=1

εij

= n∑i=1

εii +n∑

i,j=1i 6=j

(εij + εji)

= n+n∑

i,j=1i 6=j

(εij+ε−1ij ) ≥ n+ 2(n2 − n)/2 = n2

reveals that λmax ≥ n. Moreover, since x+ 1/x ≥ 2for all x > 0, with equality if and only if x = 1, wesee that λmax = n if and only if all εij = 1, which isequivalent to having all aij = wi/wj .

The foregoing arguments show that a positivereciprocal matrix A has λmax ≥ n, with equalityif and only if A is consistent. As our measure ofdeviation of A from consistency, we choose theconsistency index

µ ≡ λmax − nn− 1

.

We have seen that µ ≥ 0 and µ = 0 if and only ifA isconsistent. These two desirable properties explainthe term “n” in the numerator of µ; what about theterm “n−1” in the denominator? Since trace A = nis the sum of all the eigenvalues of A, if we denotethe eigenvalues of A that are different from λmax

by λ2, . . . , λn+1, we see that n = λmax +∑ni=2 λi , so

n − λmax =∑ni=2 λi and µ = − 1

n−1

∑ni=2 λi is the

negative average of the non-Perron eigenvalues ofA.

It is an easy, but instructive, computation toshow that λmax = 2 for every 2-by-2 positivereciprocal matrix:[

1 αα−1 1

][1+α

(1+α)α−1

]= 2

[1+α

(1+α)α−1

].

Thus, every 2-by-2 positive reciprocal matrix isconsistent.

Not every 3-by-3 positive reciprocal matrix isconsistent, but in this case we are fortunate to haveagain explicit formulas for the Perron eigenvalueand eigenvector. For

A =

1 a b1/a 1 c1/b 1/c 1

,we have λmax = 1+ d + d−1, d = (ac/b)1/3, and

(2)

w1 = bd/(1+ bd + c

d

), w2 = c/d

(1+ bd + c

d

),

w3 = 1/(

1+ bd + cd

).


Table 4. Random index.

Figure 2. Plot of random inconsistency.

Figure 3. Plot of first differences in randominconsistency—7 is critical.

Note that λmax = 3 when d = 1 or c = b/a, whichis true if and only if A is consistent.

In order to get some feel for what the consistencyindex might be telling us about a positive n-by-n reciprocal matrix A, consider the followingsimulation: choose the entries of A above themain diagonal at random from the seventeenvalues {1/9,1/8, . . . ,1/2,1,2, . . . ,8,9}. Then fill inthe entries of A below the diagonal by taking

reciprocals. Put ones down the main diagonal andcompute the consistency index. Do this 50,000times and take the average, which we call therandom index. Table 4 shows the values obtainedfrom one set of such simulations for matrices ofsize 1,2, . . . ,10.

Figure 2 is a plot of the first two rows ofTable 4. It shows the asymptotic nature of randominconsistency.

The third row of Table 2 gives the differencesbetween successive numbers in the second row.Figure 3 is a plot of these differences and showsthe importance of the number seven as a cutoffpoint beyond which the differences are less than0.10, where we are not sufficiently sensitive tomake accurate changes in judgment on severalelements simultaneously.

Since it would be pointless to try to discern anypriority ranking from a set of random comparisonjudgments, we should probably be uncomfortableabout proceeding unless the consistency index ofa pairwise comparison matrix is very much smallerthan the corresponding random index value inTable 4. The consistency ratio (C.R.) of a pairwisecomparison matrix is the ratio of its consistencyindex (C. I.) to the corresponding random indexvalue in Table 4.

As a rule of thumb, we do not recommendproceeding if the consistency ratio is more thanabout 0.10 forn > 4. Forn=3 and 4 we recommendthat the C.R. be less than 0.05 and 0.09, respectively.Thus in general when asked, we require that C.R.not exceed 0.10 by much. How do we explain thisoutcome in general?

The notion of order of magnitude is essentialin any mathematical consideration of changes inmeasurement. When one has a numerical valuesay between 1 and 10 for some measurement andone wants to determine whether a change in thisvalue is significant or not, one reasons as follows: achange of a whole integer value is critical because itchanges the magnitude and identity of the originalnumber significantly. If the change or perturbationin value is of the order of a percent or less, it wouldbe so small (by two orders of magnitude) andwould be considered negligible. However, if thisperturbation is a decimal (one order of magnitudesmaller), we are likely to pay attention to modifythe original value by this decimal without losing the


Table 5. A family’s housebuying pairwise comparison matrix for the criteria.

Table 6. Partial derivatives for the house example.

significance and identity of the original number aswe first understood it to be. Thus in synthesizingnear-consistent judgment values, changes thatare too large can cause dramatic change in ourunderstanding, and values that are too small causeno change in our understanding. We are left withonly values of one order of magnitude smallerthat we can deal with incrementally to changeour understanding. It follows that our allowableconsistency ratio should be not more than about.10. The requirement of 10% cannot be madesmaller, such as 1% or 0.1%, without trivializing theimpact of inconsistency. But inconsistency itself isimportant because without it new knowledge thatchanges preference cannot be admitted. Assumingthat all knowledge should be consistent contradictsexperience, which requires continued revision ofunderstanding.

If the C.R. is larger than desired, we do threethings: (1) find the most inconsistent judgment inthe matrix; (2) determine the range of values towhich that judgment can be changed correspondingto which the inconsistency would be improved; (3)ask the family to consider, if they can, changingtheir judgment to a plausible value in that range.If they are unwilling, we try with the second mostinconsistent judgment, and so on. If no judgmentis changed, the decision is postponed until betterunderstanding of the criteria is obtained. In our

house example the family initially made a judgmentof 6 for the a37 entry in Table 1 and the consistencyindex of the set of judgments was C.I. = (9.669−8)/7 = 0.238. But C.R. = .238/1.40 = 0.17 is largerthan the recommended value of 0.10. If we aregoing to ask the family to reconsider, and perhapschange, some of their pairwise comparisons, whereshould we start?

Three methods are plausible for this purpose.All require theoretical investigation of convergenceand efficiency. The first uses an explicit formulafor the partial derivatives of the Perron eigenvaluewith respect to the matrix entries.

For a given positive reciprocal matrix A = [aij]and a given pair of distinct indices k > l, defineA(t) = [aij(t)] by akl(t) ≡ akl + t , alk(t) ≡ (alk +t)−1, and aij(t) ≡ aij for all i 6= k, j 6= l, soA(0) = A. We use a linear function of t becausemultiplying by t when t is zero or close to zerocan make the reciprocal very large, and thuswe want t to be bounded away from zero. Also,we don’t want t to be very large because thejudgments would be too widespread, violating therequirement of homogeneity. Thus our assumptionon the functional relationship is reasonable. Letλmax(t) denote the Perron eigenvalue ofA(t) for allt in a neighborhood of t = 0 that is small enoughto ensure that all entries of the reciprocal matrixA(t) are positive there. Finally, let v = [vi] be the


Table 7. εij = aijwj/wiεij = aijwj/wiεij = aijwj/wi .

Table 8.

Table 9. Modified matrix in the a37a37a37 and a73a73a73 positions.

unique positive eigenvector of the positive matrixAT that is normalized so that vTw = 1. Then aclassical perturbation formula [4, Theorem 6.3.12]tells us that

dλmax(t)dt

∣∣∣∣t=0= v

TA′(0)wvTw

= vTA′(0)w

=∑k6=lvkwl −

1

a2klvlwk.

We conclude that

∂λmax

∂aij= viwj − a2

jivjwi for all i, j = 1, . . . , n.

Because we are operating within the set of positivereciprocal matrices,

∂λmax

∂aji= − 1

a2ij

∂λmax

∂aijfor all i and j.

Thus, to identify an entry of A whose adjustmentwithin the class of reciprocal matrices would resultin the largest rate of change in λmax, we shouldexamine the n(n − 1)/2 values {viwj − a2

jivjwi},i > j , and select (any) one of largest absolute value.This is the method proposed for positive reciprocalmatrices by Harker [8]. Ergu et al. [7] proposeanother method for dealing with consistency inthe ANP. Here is how Harker’s method is appliedto our house example with the initial judgments inTable 1 replaced by a37 = 6, a73 = 1/6 to make itmore inconsistent.

Table 6 gives the array of partial derivatives forthe matrix of criteria in Table 1.

The (4, 8) entry in Table 5 (in bold print andunderlined) is largest in absolute value. Thus,the family could be asked to reconsider theirjudgment of 1/8 for age vs. finance which indicates


that finance is very strongly to extremely moreimportant than age. One needs to know how muchto change a judgment to improve consistency,and we show that next. One can then repeat thisprocess with the goal of bringing the C.R. within thedesired range. If the indicated judgments cannotbe changed fully according to one’s understanding,they can be changed partially. Failing the attainmentof a consistency level with justifiable judgments,one needs to learn more before proceeding with thedecision. Actually, the values used in the originalexample were a37 = 1/2, a73 = 2, derived in thesimpler approach described next.

Two other methods, presented here in orderof increasing observed efficiency in practice, areconceptually different. They are based on the factthat

nλmax − n =n∑

i,j=1i 6=j

(εij + ε−1ij ).

This suggests that we examine the judgment forwhich εij is farthest from the number 1, that is,an entry aij for which aijwj/wi is the largest, andsee if this entry can reasonably be made smaller.We hope that such a change of aij also resultsin a new comparison matrix that has a smallerPerron eigenvalue. To demonstrate how improvingjudgments works, take the house example matrix inTable 1. To identify an entry ripe for consideration,construct the matrix εij = aijwj/wi (Table 7). Thelargest value in Table 7 is 5.32156, which focusesattention on a37 = 6.

How does one determine the most consistententry for the (3,7) position? Harker [8] has shownthat when we compute the new eigenvector w afterchanging the (3,7) entry, we want the new (3,7)entry to be w3/w7 and the new (7,3) entry tobe w7/w3. On replacing a37 by w3/w7 and a73 byw7/w3 and multiplying by the vectorw , one obtainsthe same product as one would by replacing a37

and a73 with zeros and the two correspondingdiagonal entries with two (see Table 8).

We take the Perron vector of the latter matrixto be our w and use the now-known values ofw3/w7 and w7/w3 to replace a37 and a73 in theoriginal matrix. The family is now invited tochange their judgment towards this new valueof a37 as much as they can. Here the value wasa37 = 0.102/0.223 = 1/2.18, approximated by1/2 from the AHP fundamental scale, and wehypothetically changed it to 1/2 to illustrate theprocedure (see Table 9). If the family does notwish to change the original value of a37, oneconsiders the second most inconsistent judgmentand repeats the process.

One by one, each reciprocal pairaij andaji in thematrix is replaced by zero and the correspondingdiagonal entries aii and ajj are replaced by 2. The

principal eigenvalue λmax is then computed. Theentry with the largest resultingλmax is identified forchange as described above. This method is in usein the ANP software program SuperDecisions [26].The SuperDecisions software is used in teaching thesubject. Here is the link to the webpage from whichthe SuperDecisions software can be downloaded,and it is free to educators and researchers: www.superdecisions. com. Incidentally, the name of thesoftware is borrowed from its use of a matrix whoseentries are matrices, the supermatrix, and is notan attempt to sound like something extraordinary.

Alternatives in a decision may be comparedin pairs or if there are many, they can be ratedone at a time by assigning them numbers fromappropriate priority scales developed for eachcriterion such as (high, medium, low), (excellent,outstanding, very good, good, poor and very poor)that are then compared in pairs and their prioritiesderived, with each scale divided by the largestderived eigenvector component, making the largestvalue equal to one and the rest proportionatelysmaller. Comparisons yield a more accurate rankingof alternatives than rating them one at a timebecause rating involves memory of an ideal thatis likely to vary among different people. If thealternatives vary widely, then the scales developedmust reflect different orders of magnitude that areappropriately linked together [22].

The Normalized Priority Vector is UniqueTo choose the best alternative in a decision,the priorities must be unique. There is more tothe concept of priority. When A = [wi/wj] isconsistent, Ak = nk−1A. This says how much acriterion represented by a row of A dominatesother criteria through chains of k arcs, uniquelydetermined by the single arc chains representedby the rows of A itself. But this is not true when Ais inconsistent.

Criterion i is said to dominate criterion j inone step if the sum of the entries in row i of Ais greater than the sum of the entries in row j .It is convenient to use the vector e = (1, . . . ,1)Tto express this dominance: criterion i dominatescriterion j in one step if (Ae)i > (Ae)j . A criterioncan dominate another criterion in more thanone step by dominating other criteria that inturn dominate the second criterion. Two-stepdominance is identified by squaring the matrixand summing its rows, three-step dominance bycubing it, and so on. Thus, criterion i dominatescriterion j in k steps if (Ake)i > (Ake)j . Criterioni is said simply to dominate criterion j if entry i ofthe vector

(3) limm→∞

1m

m∑i=1

Ake/eTAke


Figure 4. Arrow’s four conditions.

is greater than its entry j . But this limit of averagescan be evaluated: the Perron-Frobenius Theoremensures that Ak/λkmax → wvT as k→∞, so

Ake/(eTAke) ≈ λkmaxw(vTe)/λkmax(eTw)(vTe)= w as k→∞.

(4)

Since (3) is a limit of averages of terms of asequence that converges to the Perron vector wof A, according to Cesaro summability [17], (3) isactually equal to w .

More simply, a priority vector w can be usedto weight the columns of its matrix and sum theelements in each row to obtain a new priority vector.Such ambiguity is eliminated if we require thata priority vector satisfy the condition Aw = cw ,c > 0. The constant c is needed because w isderived from absolute scale entries (invariantunder the identity transformation) and thus innormalized form also belongs to an absolute scale.It is easy to prove, using the biorthogonality ofleft and right eigenvectors [7] that c and w mustbe, respectively, the Perron value and vector of A.With the concept of dominance, we have provedthat the Perron vector is necessary for derivingpriorities.

Should one be concerned about the often inexactform of the judgments? Wilkinson [27] has studiedthe stability of an eigenvector of a square matrixwith real coefficients. Perturbing the matrix byadding to it the perturbation matrix ∆A yieldsthe following perturbation ∆w1 in the principaleigenvector w1. The expression below involves allthe eigenvalues λi of A and all of both its left (vj)and right (wj) eigenvectors:

∆w1 =n∑j=2

(vTj ∆Aw1/(λ1 − λj)vTj wj)wj .

The left and right eigenvectors v and w are innormalized form. T indicates transposition.

The eigenvector w1 is stable when the followinghold:

(1) The perturbation ∆A is small as observingthe consistency index would ensure.

(2) λj is well separated from λ1; when A isconsistent, λ1 = n, λj = 0.

(3) The product of left and right eigenvectorsis not too large, which is the case for a consis-tent (and near-consistent) matrix if the elementsare homogenous (compared here on the relativedominance scale of the absolute values 1–9) withrespect to the criterion of comparison.

(4) The number of their entries is small (henceperhaps why inconsistency becomes problematicas to which element causes it the most for n > 7,[13]).

The conclusion is that n must be small, and onemust compare homogeneous elements, which is inharmony with the axioms of the AHP [18].

Synthesis of Individual Judgments into aRepresentative Group JudgmentKenneth Arrow’s Impossibility Theorem, for whichhe received the Nobel Prize in 1972, stated thatit was not possible to find a representative groupjudgment from the judgments of individualsusing ordinal preferences. However, if one allowscardinal preferences and uses the geometric meanto combine individual judgments as we do in theAHP, it is possible. In 1983 we proved, in a papercoauthored with Janos Aczel, that the unique wayto combine reciprocal individual judgments intoa corresponding reciprocal group judgment is byusing their geometric mean [1].

Arrow proved in his impossibility theorem,using ordinal preferences (either A is preferredto B or it is not) that there does not exist asocial welfare function that satisfies all four con-ditions listed in Figure 4, at once. We showedin April 2011 [24] in a journal of which Ar-row is an editor that with cardinal intensities ofpreference and the geometric mean to combinethe individual judgments into a representativegroup judgment, a social welfare function existsthat satisfies these four conditions. Thus we havea possibility theorem.

Validation and Diverse UsesHow do we test for the validity of the process?One of the things we can do is to get judgmentsfrom many people, even those who may not beexperts in decision making but who are expertsin what they do. Should the answer always matchthe data available? What if the data themselves areincorrect? What if we don’t know enough to createa very complete structure for a decision? Thesequestions have been examined in the literature, but


at best one needs to apply the process in a numberof decisions to develop confidence in its reliability.We have provided three examples to illustrate itsaccuracy when used with systematic knowledgeand understanding, of both the decisions to which itis applied and of its limitations that revolve aroundthe adequacy of the structure used to represent thedecision, and the experience necessary to developsound and accurate judgments used in makingcomparisons.

Relative Consumption of Drinks

Table 10 shows how an audience of about thirtypeople, using consensus to arrive at each judgment,provided judgments to estimate the dominanceof the consumption of drinks in the U.S. (whichdrink is consumed more in the U.S. and how muchmore than another drink?). The derived vectorof relative consumption and the actual vector,obtained by normalizing the consumption given inofficial statistical data sources, are at the bottomof the table.

Optics Example

Four identical chairs were placed on a line from alight source at the distances of nine, fifteen, twenty-one, and twenty-eight yards. The purpose was to seeif one could stand by the light and look at the chairand compare their relative brightness in pairs, fillin the judgment matrix, and obtain a relationshipbetween the chairs and their distance from thelight source. This experiment was repeated twicewith different judges whose judgment matrices areshown in Table 11.

The judges of the first matrix were the author’syoung children, ages five and seven at that time,who gave their judgments qualitatively. The judgeof the second matrix was the author’s wife, alsoa mathematician and not present during thechildren’s judgment process. In Table 12 we givethe principal eigenvectors, eigenvalues, consistencyindices, and consistency ratios of the two matrices.

First and second trial eigenvectors of Table12 have been compared with the last column ofTable 13 calculated from the inverse square lawof optics. How close are the eigenvectors to theactual result for physics? We use a compatibilityindex that we developed for that purpose. We takethe Hadamard product of a matrix of ratios ofthe entries of one vector with the transpose ofa second matrix of the other vector. If the twovectors are identical, each entry of the Hadamardproduct would be equal to one and the sum of allresulting entries would be equal to n2. Otherwise,one divides the resulting sum by n2 and ensuresthat the ratio is about 1.01. It is interesting andimportant to observe in this example that the

Table 10. Relative consumption of drinks.

Table 11. Pairwise comparisons of the fourchairs.

Table 12. Principal eigenvectors andcorresponding measures.

numerical judgments have captured a natural law.It would seem that they might do the same in otherareas of perception or thought, like the one onestimating chess championship outcomes that weshow in the next example, and, more generally, incontinuous versions of these ideas.

Note the sensitivity of the results as the closestchair is moved even closer to the light source,for then it absorbs most of the value of therelative index and a small error in its distancefrom the source yields great error in the values.What is noteworthy from this sensory experimentis the observation or hypothesis that the observedintensity of illumination varies (approximately)inversely with the square of the distance. The morecarefully designed the experiment, the better theresults obtained from the visual observations.


Table 13. Inverse square law of optics.

Figure 5. Criteria, factors, and players in chesscompetition.

World Chess Championship Outcome Validation:The Karpov-Korchnoi Match

The following criteria (Table 14) and hierarchy(Figure 5) were used to predict the outcome of worldchess championship matches using judgments often grandmasters in the former Soviet Union andthe United States who responded to questionnairesmailed to them. The predicted outcomes thatincluded the number of games played, drawn, andwon by each player either was exactly as theyturned out later or adequately close to predictthe winner. The outcome of this exercise wasofficially notarized before the match took place.The notarized statement was mailed to the editorof the Journal of Behavioral Sciences along with thepaper later published in May 1980. The predictionwas that Karpov would win by six to five gamesover Korchnoi, which he did.

The AHP, the name of the decision processdescribed above, has been used in various settingsto make decisions.

This approach to prioritization provides the op-portunity to help focus attention on the importantissues in the world and allocate resources to themaccordingly.

• Since its early development, the AHP hasbeen used to predict correctly, a fewmonths before the elections, the nextcandidate to be elected for president. Thefactors involved varied from election to

election depending on the domestic andinternational circumstances prevailing atthe time.

• In 1986 the Institute of Strategic Studiesin Pretoria, a government-backed organiza-tion, used the AHP to analyze the conflictin South Africa and recommended actionsranging from the release of Nelson Man-dela to the removal of apartheid and thegranting of full citizenship and equal rightsto the black majority. All of these recom-mended actions were quickly implementedby the white government.

• A company used it in 1987 to choose thebest type of platform to build to drill foroil in the North Atlantic. A platform costsaround three billion dollars to build, butthe demolition cost was an even moresignificant factor in the decision.

• Xerox Corporation has used the AHP toallocate close to a billion dollars to itsresearch projects.

• IBM used the process in 1991 in designingits successful mid-range AS 400 computer.IBM won the prestigious Malcolm Baldrigeaward for Excellence for that effort. Thebook about the AS 400 project has achapter devoted to how AHP was used inbenchmarking.

• The AHP has been used since 1992 instudent admissions and prior to that in mil-itary personnel promotions and in hiringdecisions.

• The process was applied to the U. S. versusChina conflict in the intellectual propertyrights battle of 1995 over Chinese individ-uals copying music, video, and softwaretapes and CD’s. An AHP analysis involvingthree hierarchies for benefits, costs, andrisks showed that it was much better forthe U. S. not to sanction China. Shortly afterthe study was completed, the U. S. awardedChina most-favored nation trading statusand did not sanction it.

• In sports it was used in 1995 to predictwhich football team would go to the SuperBowl and win (correct outcome, Dallas wonover my hometown, Pittsburgh). The AHPwas applied in baseball to analyze whichPadres players should be retained.

• British Airways used it in 1998 to choosethe entertainment system vendor for itsentire fleet of airplanes

• The Ford Motor Company used the AHP in1999 to establish priorities for criteria thatimprove customer satisfaction.


Table 14. Definitions of chess factors.

• In 2001 it was used to determine the bestsite to relocate the earthquake-devastatedTurkish city of Adapazari.

• A comprehensive analysis as to whetherthe Unites States should develop an anti-nuclear missile (estimated in the 1990sto cost sixty billion dollars and stronglyopposed by scientists as technically in-feasible) was presented to the NationalDefense University (NDU) in February 2002.In December of that year President Bushdecided to go for it. The U.S. actually devel-oped prototypes and tested them in stagessuccessfully.

• An application by Professor Wiktor Adamusof Krakow University convinced the primeminister of Poland in 2007 not to adopt theEuro for currency until many years later.

• An AHP application, known to the militaryat the Pentagon, showed that occupyingor bombing Iran in terms of benefits,opportunities, costs, and risks is not thebest option for security in the Middle East.

• The AHP was used to assist the Green BayPackers to hire the best players, perhapspartly the reason why they won the Su-per Bowl football championship in 2011by beating the Pittsburgh Steelers. Otherteams, including hockey and baseball, arealso using it.

• In 1991, 2001, and 2009, AHP was used inthree studies by economists to determinethe turn-around dates of the U.S. economyand the strength of recovery. These studieswere uncannily accurate.

• The latest application made in August2011 was to the Israeli-Palestinian conflict


when five top people from each side usedthe AHP to reach an agreement called thePittsburgh Principles. One of them wrote: “Ihad been in hundreds of meetings betweenIsraelis and Palestinians where we tried toreach a joint statement but failed becausein most of the cases each side was tryingto score points and court his own publicopinion rather than being objective andtrying to be real and responsible.”

• The AHP is used by many organiza-tions, including the military, to prioritizetheir projects and allocate their resourcesoptimally according to these priorities.

Since the AHP helps one organize one’s thinking,it can be used to deal with many decisions thatare often made intuitively. At a minimum, theprocess allows one to experiment with differentcriteria, structures, and judgments and also totest the sensitivity of the outcome to changesin both the structure [19] and the judgments. Itappears that if we know how to measure things inrelative terms according to the criteria that theyshare, we can measure anything that way, and thatkind of measurement includes, as a special case,the normalized measurement that we make in ascientific field, in which we always have to interpretthe significance of the measurements obtained byusing expert knowledge and judgment in that field.

This work on the AHP was developed indepen-dently of the Theory of Perron although I refer tohim abundantly. Consistent matrices automaticallysatisfy Perron’s conditions, lead to his results,and generalize to acceptably inconsistent matricesthrough perturbation arguments some of whichwere developed by J. H. Wilkinson [27]. We hopethat we can have another opportunity to show thereader how the ANP works and how the discretemathematics of comparisons has been generalizedto the continuous case involving Fredholm’s equa-tion whose solution produces results associatedwith neural firing and synthesis.

References[1] J. Aczel and T. Saaty, Procedures for synthesizing

ratio judgments, Journal of Mathematical Psychology27 (1983), 93–102.

[2] E. Batschelet, Introduction to Mathematics for LifeScientists, Springer-Verlag, New York, 1971.

[3] R. A. Bauer, E. Collar, and V. Tang, The SilverlakeProject, Oxford University Press, New York, 1992.

[4] H. Bergson, The Intensity of Psychic States, Chapter1 in Time and Free Will: An Essay on the ImmediateData of Consciousness, translated by F. L. Pogson, M.A.,George Allen and Unwin, London, 1910, pp. 1–74.

[5] P. Davis and P. T. Hersh, Descartes’ Dream, HarcourtBrace Jovanovich, New York, 1982.

[6] S. Dehaene, The Number Sense: How the Mind CreatesMathematics, Oxford University Press, USA, 1997.

[7] Daji Ergu, G. Kou, Yi Peng, and Yong Shi, A simplemethod to improve the consistency ratio of the pair-wise comparison matrix in ANP, European Journal ofOperational Research 213 (2011), no. 1, 246–259.

[8] P. Harker, Derivatives of the Perron root of a pos-itive reciprocal matrix: With applications to theanalytic hierarchy process, Applied Mathematics andComputation 22 (1987), 217–232.

[9] R. A. Horn and C. R. Johnson, Matrix Analysis,Cambridge University Press, New York, 1985.

[10] H. Lebesgue, Leçons Sur L’integration, second ed.,Gauthier-Villars, Paris, 1928.

[11] L. LeShan and H. Margenau, Einstein’s Space and VanGogh’s Sky, Macmillan, 1982.

[12] A. F. MacKay, Arrow’s Theorem: The Paradox of SocialChoice, Yale University Press, 1980.

[13] M. Ozdemir and T. L. Saaty, The unknown in decisionmaking: What to do about it, European Journal ofOperational Research 174 (2006), 349–359.

[14] Y. Peng, G. Kou, G. Wang, W. Wu, and Y. Shi, En-semble of software defect predictor: An AHP-basedevaluation method, International Journal of Informa-tion Technology & Decision Making 10 (2011), no. 1,187–206.

[15] T. L. Saaty, The Analytic Hierarchy Process, McGrawHill, New York, 1980. Reprinted by RWS Publications,available electronically free, 2000.

[16] , Fundamentals of Decision Making, RWSPublications, 2006, 478 pp.

[17] , Theory and Applications of the AnalyticNetwork Process, RWS Publications, 2006, 352 pp.

[18] , Principia Mathematica Decernendi, sub-titled Mathematical Principles of Decision Mak-ing, RWS Publications, 2010, 531 pp. http://[email protected].

[19] T. L. Saaty and E. Forman, The Hierarchon, a col-lection of nearly 800 hierarchies in all kinds of life,many actual, applications made by people, and The En-cyclicon, three volumes of nearly 900 pages of severalhundred network decision applications, all publishedby RWS Publications.

[20] T. L. Saaty and M. Ozdemir, Why the magic numberseven plus or minus two, Mathematical and ComputerModelling 38 (2003), 233–244.

[21] T. L. Saaty and K. Peniwati, Group Decision Mak-ing: Drawing Out and Reconciling Differences, RWSPublications, 2008, 385 pp.

[22] T. L. Saaty and J Shang, An innovative orders-of-magnitude approach to AHP-based mutli-criteriadecision making: Prioritizing divergent intangible hu-mane acts, European Journal of Operational Research214 (2011), no. 3, 703–715.

[23] T. L. Saaty and L. G. Vargas, Hierarchical analy-sis of behavior in competition: Prediction in chess,Behavioral Sciences 25 (1980), 180–191.

[24] , The possibility of group choice: Pairwise com-parisons and merging functions, Social Choice andWelfare, April 2011.

[25] A. Schopenhauer and K. Hillebrand, On the Four-fold Root of the Principle of Sufficient Reason, an essay,Kindle eBook, 2011.

[26] Superdecisions. http://www.superdecisions.com,until 2013.

[27] J. H. Wilkinson, The Algebraic Eigenvalue Problem,Clarendon Press, Oxford, 1965.


http://www.superdecisions.com

http://[email protected]

http://[email protected]

on the measurement of intangibles. a principal eigenvector

Documents