patrick s. chen peter chu michelle lin centmi abstractarchive/pdf/e_mag/vol.45_03_233.pdfa sound...

Journal of the Operations Research Society of Japan

Vol. 45, No. 3, September 2002

ON VARGAS'S PROOF OF CONSISTENCY TEST FOR 3 x 3 COMPARISON MATRICES IN AHPI

Patrick S. Chen Peter Chu Michelle Lin CentmI Police University Independent Researcher

(Received October 13, 2000; Revised March 4, 2002)

Abstract In this paper we have raised, discussed, and solved the problems relating the Vargas's verification of consistency test for 3 x 3 comparison matrices in the Analytic Hierarchy Process (AHP). In the AHP community, assessing attribute weights by using a consistency ratio of 0.1 or less has been widely practiced. Though Vargas did study the problem, the method he adopted might lead to certain questionable results. By trying to clarify such a critical situation, we pointed out the problems arose form the solutions of two inequality systems within the proof process of Vargas. We proved that the solution for the first system was an interval, not a point as he had indicated. After we solved the first inequality system, we found that the criterion for the solution interval of the first system would satisfy the second inequality system. We have then justified the inherent problem in Vargas's verification procedure.

1. Introduction Numerous books and papers concerning the theory and applications of the Analytic

Hierarchy Process (AHP) have been written since Saaty [lo] first presented this decision model. Due to the practical nature of the method, which is suitable for solving complicated and elusive decision problems, it has been widely applied in diverse areas. Many researchers have made their efforts to improve the theoretical foundations of AHP. Among the most interesting discussions is the verification of consistency of pairwise comparison matrices. Since uncertainties in our judgements and in many real-world cases are unlikely to avoid, many studies have been made toward this direction. Sajjad's study [13] has shown how such uncertainties could be incorporated within the framework of AHP and how the result- ing uncertainties in the related priorities of the decision alternatives could be computed. Aupetit and Genest [l] mentioned that some useful properties of the Perron eigenvalue of a positive reciprocal matrix could be applied in the context of AHP. For the problem of retrieving weights from judgment matrices, the entries are the related important ratios of alternatives. Cook, Golany [6] and Barzilai [2] argued that the only solution satisfying consistency axioms would be the geometric mean. For the problem of retrieving weights from inconsistent additive judgement matrices, Barzilai and Golany [3] proved that the only solution satisfying consistency axioms would be the arithmetic mean. Some researches, such as Ra [9], Finan and Hurley [7], Monsuur [8], or Xu and Wei [22], only to mention some of them, raised and tried to modified AHP in eliciting and evaluating subjective judgments. The judgments include the constant-sum measurement scale (1-99 scale) for comparing two elements, the logarithmic least squares met hod for computing normalized values, the sum of inverse column, the sums for measuring the degree of inconsistency, and the sensitivity

'This research is partially supported by the National Science Council under the contract number NSC 91-2213-E015-003

© 2002 The Operations Research Society of Japan

234 P. S. Chen, P. Chu & M. U n

analysis of pairwise comparison matrices. When making decisions on several alternatives of a matter, we consider good properties

of acceptable solutions for the problem of deriving weights from pairwise comparisons should include the immunity to rank reversals, the independence of description of the problem, the independence of scale inversion, the left-right eigenvector asymmetry, the uniqueness, the independence of order of operations and inter-level consistency, the preservation of algebraic structure of the problem, the extensibility to the additive case, the related optimization models and the related error measures (Barzilai [Z]).

However, several researchers, like Watson and Freeling [20] and [21], Belton and Gear [4] and [5], Schoner and Wedley [15], Schoner Wedley and Choo [16] and others, have criticized AHP in various respects. Among them, the ambiguity of the questions to be answered by decision-makers, the scale used to measure the intensity of preference, the principle of synthesizing priorities, rank reversal, etc. These criticisms make AHP theory not only more mature, but more applicable as well. In 1994, besides the reiterated fundamentals of AHP, Saaty [12] also raised topics regarding the scales of measurement, judgments, consistency and the eigenvector. In this paper, we have focused the discussion on the consistency test for assessing attribute weights.

Zahedi [23] pointed out that, after Saaty's developing of the eigenvalue method for estimating local relative weights, several techniques for estimating the attribute weights have been suggested. The techniques can be roughly grouped into two categories:

A deterministic approach: The method has an underlying assumption that the preferences would be known with certainty and the only source of error would be inconsistency. In this paradigm, the error exists only in the elicitation of the preferences. In other words, inconsistency is due to measurement errors and can be easily reduced. A statistical approach: The method is inherently probabilistic. In other words, the preferences probably contain errors. Therefore, no consistency checks existed to avoid the happening of errors. Under the circumstances, developing a measure of inconsistency is meaningless. The major effort in this paradigm will be in identifying what statistical model could be the best explanation of the variability in the input data.

Some considered that the upper bound of consistency index should be verified by using an actual sample concerning subjective issues, based on the size of consistency index (CI) with regard to the size of pairwise comparison matrix and on rank reversal problem. However, we have concentrated ourselves on the problems mentioned in the first category instead. If there is an evaluator existed to imply the actual relative weights of n judgements, the matrix of the pairwise comparisons would be

In this case, the relative weights could be trivially obtained from each of the n rows of the matrix A. In other words, matrix A has rank 1; and the following holds:

where W = (wl) wa, . . . , wn)T is the vector of actual relative weights, and n is the number of elements. In linear algebra, n and W in (1.2) are called the eigenvalue and right eigenvector of matrix A respectively.

Copyright © by ORSJ. Unauthorized reproduction of this article is prohibited.

On Vargas's Proof of Consistency Test in AHP 235

AHP posits that the evaluator does not know W and, therefore, is not able to pro- duce the pairwise relative weights of A accurately. Thus, the observed matrix A contains inconsistencies. The estimation of W (denoted as G) could be obtained, similar to (1 -2))

If A is the observed matrix of pairwise comparisons, A,̂ , is the largest eigenvalue of A, and is its right eigenvector. @ contains the estimation of W. In (1.3), Am= may be considered as the estimation of n in (1 -2). Saaty [lo] has shown

that Amax is always greater than or equal to n. The closer the value of computed Amax is to n, the more consistent are the observed values of A. This property has led to the construction of the consistency index p as

and the consistency ratio 0 as

that XF is the distribution of the average index of randomly generated weights for n x n matrices (Saaty [Ill). As a rule of thumb, a consistency ratio of 0.1 or less is considered acceptable. Otherwise, it is recommended that A should be reconstructed to resolve inconsistencies in pairwise comparisons. Salo and Hamalainen [14] have demonstrated that the election of the threshold level for the consistency ratio (such as 0.1) is contingent on the choice of the 1-to-9 scales, even though many other scales have also been proposed. It casts doubts on the utilization of the 0.1 levels, and suggests that a more robust, scale-independent consistency measures would be preferred.

Vargas [19] tried to verify the consistency ratio 0 that should be no more than 0.1. He made the verification with the help of 3 x 3 matrices. The threshold is widely accepted by many practitioners of AHP method. Since he is able to validate that the consistency ratio valued at 0.1 is reasonable, his achievement is beyond doubt and his verification procedure deserves more careful examination.

In order to prove that the consistency ratio 0 of a comparison matrix should be no more than 0.1, we intend to show that, under certain circumstances, the threshold can be, for example, 0.12. Therefore, when assessing attribute weights, the ratios under 0.1 are consistent, and the ratios between 0.1 and 0.12 rarely meet the needs. Hence, those matrices with the ratios between 0.1 and 0.12 become negligible.

A sound proof of the consistency test should derive from an interval between 0 and 0.12. We then have a good reason to take the threshold value as 0.1. The comparison matrices with consistency ratio less than 0.1 can be accepted as consistent, and not too many comparison matrices with consistency ratio between 0.1 and 0.12 are rejected.

Based on the supposition above, we found out that there are problems contained in Vargas's verification of the consistency test. The problems are listed as follows: (1) The solution of an inequality system in the proof is an interval instead of a point as he

indicated, and (2) His procedure of testing the reliability contains a structural problem. We will discuss the main points of the out-finding in the following sections: In the next section, we will describe the origin of the problem. In section 3, we will deal with mathematical formulation, in which a series of lemmas and main theorem are proven. In the concluding section, we will summarize the contributions of this paper.


2. Discussion on Related Work AHP provides a useful analytic criterion for the measurement of the consistency test for

a comparison matrix. The reliability of a reciprocal matrix is measured by taking the ratio of the consistency index of the matrix, p, to the average random consistency ~ ( n ) . If the ratio is less than or equal to a threshold 0, we will accept the results as being sufficiently consistent. Let us now review the procedure of the Vargas's and examine how he proved that the threshold value 0 should be 0.1:

Firstly, Vargas rewrote a 3 x 3 reciprocal matrix X and, then decomposed it as follows:

1 x12x23 3 wl 1 wl if a = , w2 = a-, w3 = -- and wl is any positive number. When a = 1, it

xl2 a w13 shows that ~12x23 = xi3 and X is a consistent matrix.

Vargas multiplied the first column of X by xi3 and the second column by x23, then he obtained the matrix

Vargas found that the geometric mean of the rows of X* is the vector

He also found out the arithmetic mean of the rows of X* is the vector

Vargas compared (2.3) and (2.4), then he wanted to solve the following inequalities

1 as the principal eigenvalue of the reciprocal 3 x 3 matrix, Amax, satisfied Amax = 1 + a + -.

a (Note that Tummala and Wan [8] has a close-form solution for the largest eigenvalue of a

A--3 3 x 3-matrix.) Moreover, the consistency index, p defined as p =

2 , and an average

consistency index, XÃ§ as a truncated normal distribution N (- p(3), %). Using the (43) result of Vargas [19], it indicated that p(3) = 0.5381 and - = 0.0433. If we construct a "500



(1 - a)% confidence interval, then the acceptance threshold equals to 0.4532, with a = 5%. Vargas examined the reason why 6 should be 10%. For a given reciprocal matrix, one could accept the hypothesis that the entries of the given reciprocal matrix were reliable

P 1 if - < 6. Through solving the inequalities of (2.5)) Vargas proclaimed a w 1 + -, so XTT 3 r-

1 "max 3 Amax = 1 + a + - = 3.0833 and p =

2 = 0.0417. Thus, he obtained

a

^ - 0.0417 --- 0.0919 < 6 (= 0.1 by Vargas). XF 0.4532

Vargas implicitly predicted that taking 6 = 0.1 was valid. His procedure was the only existing analytic method to discuss the verification of the consistency ratio. Hence, this particular method deserves a more careful examination. In the following we will (1) Show that there are omissions in the solution of Vargas for the inequalities of (2.5); (2) Prove the claim of Vargas for 6 = 0.1 through analytical technique; (3) Solve the inequalities of (2.5) through analytical method for abstract 8; (4) Offer a criterion such as the solution interval of (2.5) will satisfy (2.6); (5) Indicate that the structural weakness within the procedure of Vargas for testing the

reliability of consistency ratio.

3. Mathematical Formulation 1 1 + 2~~

Recall (2.5), let f( t ) = - 1 for t > 0 and g(s) = - 3 3

( s2 j - 1 for s > 0. Given 9 = 0.1, then we need to solve' 1 f ( t ) 1 < 0.1 for t > 0 and lg(s) 1 < 0.1 'for s >

Here, we give the outline of our proof procedure. We need to display the solution for the system of two inequalities (2.5)) and test the solution for the inequality (2.6). Since the system of (2.5) contains two inequalities, we will offer the individual solutions, which are intervals, and find their intersection. Because certain relations among boundary points of the solution intervals exist, we express the solution of (2.5) in a compact form. For the second inequality (2.6), we discover the criterion for determining whether or not inequality (2.6) holds. Hence, the structural weakness in the paper of Vargas becomes obvious.

Firstly, using the first and second derivatives of the respective functions and f (I) = g(1) = 0, we can prove that the solution for the inequality system is an interval, but not a point.

Lemma 1. (a) The solution of 1 f(t)l < 0.1 is an interval, say (tl,t2), with 0 < t i < 1 < t2. (b) The solution of b(s)l < 0.1 is an interval, say (sl, s2) with 0 < sl < 1 < sz. From Lemma 1, we conclude that the solution for those two inequalities is the intersection of two intervals, (ti, t2) ("1 (sl, s2), not a point of value 0.0919 as asserted by Vargas [19]. For further investigation, we are going to abstract the value of 6 from the specified value 0.1 to any positive value 6

Lemma 2. Given a positive value 6, the solution of 1 f (t) 1 < 6 is an interval, say (ti ( O ) , t2(6)), with f(tl(6)) = 6 = f(t2(6)) and 0 < ti($) < 1 < t2(6). Moreover, the solution of lg(s)l < 6 is also an interval, say (sl (6), s2(6)), with f ( ~ ~ ( 6 ) ) = 6 = f (s2 (6)) and 0 < si (0) < 1 < ~ ~ ( 6 ) .

2 + t 2 s + 1 s - 1 2

'Since / ( t ) = ( t - 1)' and g(s) = , both / ( t ) and g(s) are positive and it is, in fact, unnecessary to consider their absolute values.


238 P. S. Chen, P. Chu & M. Lin

Now, we will show some relations among t i , t2, sl and s2. Lemma 3. Given a positive value 0, then s1(0)t2(0) = s2(0)t1(0) = 1, t1(0)t2(0) < 1, s1(0)s2(0) > 1, s1(@) > ti(@) and t 2 ^ ) < s2^). Proof: We reconsider the equation / ( t ) = 0, as t3 -3(l +0)t +2 = 0. Let F ( t , 0 ) = -3(l+ 9)t + 2, for 0 > 0 and -00 < t < 00. We know that F( t , 0 ) is a cubic polynomial in t and evaluate lim F ( t , 0 ) = oo, F ( l , 0 ) = -39 < 0, F(0, 0 ) = 2 > 0, and lim F(t , 6 ) = -00. By

tÃ‘Ã̂ t+00

the Intermediate Value Theorem [17], without loss of generality, we suppose that F(t ,0) = 0 has three solutions: to ($) , t l (0) and t@), such that to($) < 0 < ti (0) < 1 < t2(6).

Here, we will show to(0) < -2 and t1(0)t2(0) < 1. Since F(-2, 0 ) = 60 > 0, we obtain to($) < -2.

Using F ( t ,0) = (t - to($)) (t - tl (0 ) ) (t - t2 (9)) and comparing the constant terms on both sides, we induce 2 = -to(0}ti(0)t2(0), from to(@) < -2. It follows that

In the same fashion mentioned previously, from g(s) = 0, we suppose that for -00 < s < oo and 0 > 0 , G ( s , 0 ) = 2s3 - 3(1 + 0)s2 + 1, there are three roots for G ( s , 0 ) that are so (@) , sl(0) and ~ ~ ( 0 ) with so(0) < 0 < sl(0) < 1 < ~ ~ ( 0 ) .

In the remaining proof of Lemma 3, 6 is a fixed positive number. To simplify the notation, we use t l , t2, s1 and 5 2 to denote t l (0) , t2(0) , s l (0) and ~ ~ ( 0 ) .

1 2+t3 Using / ( t l )=0and0=g(s2) ,wehave- - - I = -

3 ( t l ) ; ( l^)- l . I t fo"ws

Consequently, from (3.2) we obtain 2 -t:-

From 0 < ti < 1 < s2, we ascertain

= -sf1s2) ($ + t l ) , then

From (3.3) and (3.4), we conclude t1s2 = 1. Similarly, we can deduce sit2 = 1. Therefore,

t1s2 = 1 = sit2. (3.5) 1

Putting (3.1) and (3.5) together, we obtain ti < - = sl. We have shown t 2

Using the same method, we obtain

Here, we have derived the desired properties among the boundary points for the simultaneous system of inequalities. Q.E.D.

Since ltl (9) > t2(9))n(si{0), s2(6)) = (max{t1(0), s1 ( f l ) } , min{t2(6), ~ 2 ( 9 ) } ) = (s1(@), t 2 ( 6 ) \ we have the next lemma.



Lemma 4. The solution for the simultaneous system of inequalities lf(t) 1 < 0 and b(s) 1 < 0 is (s1(0),t2(9)).

We are now in the place to go back to the special case of 0 = 0.1. After we learned the solutions for the inequalities, say (sl (0. I) , ti (0. I)), we could answer the following question:

'̂ max(x) - 3 1 If x ? (sl (0. I) , t2 (0.1)) and p(x) = P(XI , with Amax (x) = 1 + x + - , then is -

2 x 0.4532 less than o r equal to 0.1 ?

tlw < 0.1 if and only if Lemma 5. Each x ? ( s l ( O . l ) , t2(0.1)) satisfies - ^t2(0.1)) < 0.4532 0.4532

Moreover, we obtain ^2(0.1)) < 0.1. 0.4532

Proof: To simplify the notation, let c = 0.4532, sl = si(O.l), and t2 = t2(0.1). From Amax(x) is concave-up with an absolute minimum point at x = 1, then

By (3.5), we know sit2 = 1, then p(sl) = p(t2). Hence, we can reduce (3.8) as

c ^^ 0.1. ^ 5 0.1, for x 6 (s1,t2) w -

c

Recall f (t2) = 0.1, then

Using (3.10), we compute

With the help of a computer program, we get

f(1.34) = -3.954 x and f(1.35) = 1.327 x

3 know 1.34 < t2 < 1.35, with - - 2 = 1.3098. Hence, using (3.11), we prove -

2c ^ < 0.1.

c Q.E.D.

Combining with Lemma 4 and Lemma 5, we conclude that using the solutions for the two inequalities under the restriction of 0 = 0.1, the ratio between the consistency index and the average consistency index will be less than 0.1. In other words, we have proven the claim of Vargas for 0 = 0.1 through analytical technique.

Finally, we reconsider the structure of verification by Vargas. We will point out the structural weakness within the procedure of Vargas for testing the reliability. If we replace 0.1 by a given 0, we can show that

Lemma 6. t2(0) is a strictly increasing function from 1 to oo and there exists a unique 0 iJ

point, 0#, with - - 2 = ta(O#), where c = 0.4532. 2c


240 P. S. Chen, P. Chu & M. Lin

Proof: Recall w 6 ) , 6) = 0, we have

t2(6)13 - 3(1+ e)t2(e) + 2 = o with t2(6) > 1. Using the implicit differentiation, we obtain that

d ([t2(Q)I2 - (1 + 6))-,zt2(e) = t2(6).

Since 6 > 0, t̂ O} > 1 and (3.12), we know

[t2(6)J3 - (1 + e)t2(6) = 2(1+ 6)t2(6) - 2 > 0,

d From (3.14), we know -t2 (6) = t2 (6)

d0 > 0. Using (3.12), lim t@) is bounded

lt2(@)12 - (1 + 0) 8++0 (for example, lim tz (6) < 3). We take the limit of 6 -> +0 on (3.12), then

0++0

Therefore, we compute lim t2(8) = 1 based on (3.15). Again, by (3.14), we obtain e++o lim t2(6) = oo. Hence, for 6 6 (0, oo), t2(0) is a strictly increasing function from 1 to

@++oo

3 oo. So, there is a unique point, say 0:#, with - - 2 = t^O^}. Q.E.D.

2c We extend Lemma 5 from 6 = 0.1 to arbitrary positive number to derive our final result,

Theorem 1. 3

Theorem 1. Every x E (si(6), t2(6)) satisfies /-w < 6 if and only if - - 2 $ t2(8), where c 2c

c = 0.4532.

Proof: By the same method used to derive (3.8), (3.9), (3.10) and (3.11), we get

4 x 1 3 - $ 6, for x e (si(0),t2(6)) - 2 5 t2(6). c Q.E.D.

3 3 By Lemma 6, if 0 < 0 < 6#, then - - 2 > t2(6), and if 6 > 6#, then - - 2 < t2(6).

2c 2c That means that using Theorem 1, 0# is the greatest lower bound for those 0 such that the solutions of (3.12) will satisfy (2.5). With the help of a mathematical program, we can estimate 6# 3 0.08084.

Therefore, even if we change 6 from 0.1 to any positive number, 6, with 6 > 6#, the procedure of Vargas still verify that 0 is a proper choice for testing the consistency of the reciprocal matrix. We prove that the criterion of Theorem 1 will generate an interval such as (0#, oo) not as it was expected to be (0, portal-value) with 0.1 < portal-value and 0.1 is very close to the portal-value. So, the verification structure of Vargas to prove 6 = 0.1 indeed has some inherent faults.



4. Conclusion Since assessing attribute weights is often practiced in AHP community, the proof of the

assertion that the consistency ratio should be no more than 0.1 is of great importance. In this paper, we have pointed out the problems existing in the verification of Vargas. Because the solution to the inequality system (2.5) constitutes the essential part of the proof, we have shown that the solution is an interval, not a point as Vargas indicated. We have also pointed out an anomalous phenomenon in his analytic structure: based on his proof structure, we can extend the threshold 0 from 0.1 to any positive value greater than 0.1 that leads to a serious problem.

One of the slight imperfections in AHP studies is that quite a few statistical verifications of the upper bound of consistency index have been done since Vargas's paper was published 20 years ago. Therefore, it is meaningful to apply AHP to experiments concerning subjective issues and verify the upper bound of consistency index in our future study.

Acknowledgements The authors would like to extend their tremendous appreciations toward the honorable

referees for their valuable advices that have greatly polished the quality of this paper.

References

B. Aupetit and C. Genest: On some useful properties of the Perron eigenvalue of a positive reciprocal matrix in the context of the analytic hierarchy process. European Journal of Operational Research, 70 (1993) 263-268.

J. Barzilai: Deriving weights from pairwise comparison matrices. Journal of the Oper- ational Research Society, 48 (1997) 1226-1232.

J. Barzilai and B. Golany: Deriving weights from pairwise comparison matrices the additive case. Operations Research Letters, 9 (1997) 407-410.

V. Belton and A. E. Gear: On a shortcoming of Saaty's method of analytic hierarchies. Omega, 11 (1983) 228-230.

V. Belton and A. E. Gear: The legitimacy of rank reversal - a comment. Omega, 13 (1985) 143-144.

W. D. Cook and B. Golany: Consistent weights for judgments matrices of the relative importance of alternatives. Operations Research Letters, 6 (1987) 131-134. J. S. Finan and W. J. Hurley: Analytic hierarchy process: does adjusting a pairwise comparison matrix to improve the consistency ratio help? Computers & Operations Research, 24 (1997) 749-755.

H. Monsuur: Intrinsic consistency threshold for reciprocal matrices. European Journal of Operational Research, 96 (1997) 387-391. J. W. Ra: Hierarchical decision process. Proceedings of the 1991 Portland International Conference on Management of Engineering and Technology, PICMET '91, (1991) 595- 599.

T. L. Saaty: A scaling method for priorities in hierarchy structures. Journal of Math- ematical Psychology, 15 (1977) 234-281.

T. L. Saaty: The analytic hierarchy process, (McGraw-Hill, New York, 1980).

T. L. Saaty: Highlights and critical points in the theory and application of the analytic hierarchy process, European Journal of Operational Research, 74 (1994) 426-447.


242 P. 5. Chen, P. Chu & M. fin

[13] Sajjad and M. Zahir: Incorporating the uncertainty of decision judgements in the analytic hierarchy process. European Journal of Operational Research, 53 (1991) 206- 216.

[14] A. A. Salo and R. P. Hamalainen: On the measurement of preferences in the AHP. Re- search Report. Systems Analysis Laboratory. Helsinki University of Technology (1993).

[15] B. Schoner and W. C. Wedley: Ambiguous criteria weights in AHP: consequences and solutions. Decision Sciences, 20 (1989) 462-475.

[16] B. Schoner, W. C. Wedley and E. U. Shoo: A unified approach to AHP with linking pins. European Journal of Operational Research, 64 (1989) 384-392.

[I?] C. B. Thomas and R. L. Finney: Calculus and Analytic, (Addison-Wesley Publishing, 1992).

[18] V. M. R. Tummala and Y. W. Wan: On the mean random inconsistency index of analytic hierarchy process (AHP). Computers & Industrial Engineering, 27 (1994) 401- 440.

[19] L. G. Vargas: Reciprocal matrices with random coefficients. Journal of Mathematical Modeling, 3 (1982) 69-81.

[20] S. R. Watson and A. N. S. Reeling: Assessing Attribute weights. Omega, 10 (1982) 582-585.

[21] S. R. Watson and A. N. S. Reeling: Comment on: assessing attribute weights by ratios. Omega, 11 (1983) 13.

[22] Z. Xu and C. Wei: Consistency improving method in the analytic hierarchy process. European Journal of Operational Research, 116 (1999) 443-449.

[23] F. Zahedi: The analytic hierarchy process - a survey of the method and its applications. Interfaces, 16 (1986) 96-108.

Patrick S. Chen Department of Informat ion Management, Central Police University 333 Taoyuan, TAIWAN E-mail: chenQsun2. cpu. edu . tw


patrick s. chen peter chu michelle lin centmi abstractarchive/pdf/e_mag/vol.45_03_233.pdfa sound...

Documents