a note on precision of qualitative data

ISQC2013

2013-08-22 Tomomichi Suzuki, Tokyo University of Science

/41

A note on precision of qualitative data

Tomomichi Suzuki, Tokyo University of Science [email protected]

Yusuke Tsutsumi, Mitsubishi Tanabe Pharma Corporation

Natsuki Sano, Tokyo University of Science

1

ISQC2013


/41

Introduction of myself

• I focus on “Statistical Data Analysis” that will bridge the gap between theory and practice.

• I am attending ISQC for the fourth time• Warsaw 2004, “Effective Dynamic Process Control of Assembly

Processes”– statistical control of assembly process with time dependent noise

• Beijing 2007, “A Study on Adaptive Paired Comparison Experiments”– design of experiment for paired comparison– proposal on the Swiss tournament system

• Seattle 2010, “Improving Taguchi’s linear graphs for split-plot experiments”– proposed new linear graphs for expressing interaction of whole-plots

and sub-plots

2

ISQC2013


/41

Outline of Today’s Talk

• Introduction

• Precision for Quantitative Data

• Precision for Qualitative Data

• Comparison

• Conclusions

3

ISQC2013


/41


• Introduction



• Comparison

• Conclusions

4

ISQC2013


/41

Introduction

• ISO 5725 accuracy (trueness and precision) of measurement methods and results

• Tests performed on presumably identical materials in presumably identical circumstances do not, in general, yield identical results.

5

ISQC2013


/41

Introduction

• ISO 5725 accuracy (trueness and precision) of measurement methods and results

• Trueness:– refers to the closeness of agreement between

the arithmetic mean of a large number of test results and the true or accepted reference value.

• Precision:– refers to the closeness of agreement between

test results.

6

ISQC2013


/41

ISO/TC 69/SC 6

• ISO/TC 69 (Application of Statistical Methods)/SC 6 (Measurement Methods and Results)/WG1 (Accuracy of measurement methods and results) is preparing a document (TR: Technical Report) on precision of qualitative data. Now in Preliminary Work Item.

• Reviewed existing methods and established methods.

7

ISQC2013


/41

ISO/TC 69/SC 7

• ISO/TC 69 (Application of Statistical Methods)/SC 7 (Six Sigma) published ISO TR 14468 “Selected illustrations of attribute agreement analysis”

• This is based on kappa coefficient approach.

8

ISQC2013


/41

ISO/TC 34/SC 9

• ISO/TC 34 (Food products) /SC 9 (Microbiology) produced ISO 16140 “Microbiology of food and animal feeding stuffs — Protocol for the validation of alternative methods” in 2003.

• It includes method by Langton et al. (2002)• ISO/TC 34/SC 9 is revising ISO 16140

“Microbiology of food and animal feed — Method validation — Part 2: Protocol for the validation of alternative (proprietary) methods against a reference method”.

• It includes method by Wilrich.

9

ISQC2013


/41

AOAC InternationalISO/TC 34/SC 16

• ISPAM (International Stakeholder Panel on Alternative Methods) produced a document on “Guidelines for Validation of Qualitative Chemistry Methods” which is based on POD model proposed by P. Wehling et al. (2011)

• This is the main part of the ISO/TC 34 (Food products) /SC 16 (Horizontal methods for molecular biomarker analysis) document. “Validation Scheme for Qualitative Analytical Methods” (possible alternative title: "Performance characteristics and validation of binary measurement methods")

10

ISQC2013


/41

IMEKO/TC 21

• IMEKO (International Measurement Confederation) / TC 21 (Mathematical Tools for Measurements) hold SIG (Special Interest Group) “Precision evaluation in non-quantitative measurements”.

11

ISQC2013


/41

Purpose

• Many methods are proposed for qualitative data, but their effectiveness and statistical properties are not so clear.

• This paper introduces the methods to evaluate precision for qualitative data, then proposes a method using logit model. The proposed method is compared with existing methods.

12

ISQC2013


/41


• Introduction



• Comparison

• Conclusions

13

ISQC2013


/41

Collaborative Assessment Experiment

Laboratory Run 1 ... Run k ... Run n

1 y11 y1k y1n

2 y21 y2k y2n

:

:

i yi1 yik yin

:

:

L yL1 yLk yLn

• Every laboratory measures the identical test item number of times.

14

ISQC2013


/41

Precision for Quantitative Data

• Repeatability:– is the precision under repeatability conditions– where independent test results are obtained with the same

method on identical test items in the same laboratory by the same operator using the same equipment within short intervals of time.

– Repeatability indicates the smallest variation for a particular measurement method.

• Reproducibility:– is the precision under reproducibility conditions– where test results are obtained with the same method on

identical test items in different laboratories with different operators using different equipment.

– Reproducibility indicates the largest variation for a particular measurement method.

15

ISQC2013


/41


• Model used in ISO 5725

y = m + B + e

– y is the measurement result– m is the general mean (expectation)– B is the laboratory component of bias under

repeatability conditions (variance L2)

– e is the random error in every measurement under repeatability conditions. (variance e

2)16

ISQC2013


/41


• Repeatability variance r2

r

2 = e2 , or

• Reproducibility variance R2

R2 = L

2 + r2 = L

2 + e2 , or (1)

• The estimates of repeatability variance and

reproducibility variance are calculated from interlaboratory studies or collaborative assessment experiments.

)(eVr

)()( eVBVR

17

ISQC2013


/41


• Gauge R & R• Many objects are measured (there are variation in products)

• Gauge Repeatability = Repeatability in ISO 5725 (r2)

• Gauge Reproducibility ≠ Reproducibility in ISO 5725

• Gauge Reproducibility = Between Laboratory Variance in ISO 5725 (L

2) R

2 = L2 + r

2 (1)

18

ISQC2013


/41


• Introduction



• Comparison

• Conclusions

19

ISQC2013


/41

Precision for Qualitative Data

• Non-quantitative measurements– binary data, categorical data, ordinal data, etc.

• In this paper, the methods to evaluate precision for binary data are considered.

• Methods compared are– Accordance and concordance (Langton’s)– Attribute agreement analysis (Kappa)– van Wieringen’s method– Proposed method

20

ISQC2013


/41

Precision for Quality Data

Laboratory Run 1 ... Run k ... Run n

1 y11 y1k y1n

2 y21 y2k y2n

:

:

i yi1 yik yin

:

:

L yL1 yLk yLn

• The value of yik is either 0 (negative, non-detect, fail, etc.) or 1 (positive, detect, pass, etc.).

Table 1

21

ISQC2013


/41

Qualitative Methods0. ISO Based Method

ijiij ey Wilrich’s model

eBmy ISO5725 model

i

ije

： general mean

： laboratory component of bias： random error

Repeatability Variance

Inter-laboratory Variance Component

Reproducibility Variance

L

iiir Ln

n

1

2 ˆ1ˆ1

1ˆ

L

i

L

iiiiL LnL 1 1

22 ˆ1ˆ1

1

1ˆˆ

1

1,0maxˆ

222 ˆˆˆ LrR

： Estimate of probability of detecting for lab i (i=1, 2, …, L)

i

n ： number of repetitions (measurements)

22

ISQC2013


/41

Qualitative Methods1. Accordance and Concordance

Accordance１

１

１

１

０

Lab i

matching probability Ai

)1(

)1)(()1(

nn

xnxnxxA iiii

i

L

iiA

LA

1

1

iA

A

： Accordance for laboratory i (i1, 2, ..., L)

： Accordance

nix ： number of ‘detect’ for lab i (i1, 2, ..., L)： number of repetitions (measurements)

23

ISQC2013


/41


)1(

)1(12

211

LLn

nnLAnLnLnLxx

Ci

L

ii

L

ii

i

L

iiC

LC

1

1

iA

C ： Concordance

iC

Concordance １１

１

１

０

lab i

１１

１

１

１

lab i’matching probability Ci

１０

０

１

１

lab i’’

： Accordance for laboratory i (i1, 2, ..., L)

： Concordance for laboratory i (i1, 2, ..., L)

nix ： number of ‘detect’ for lab i (i1, 2, ..., L)： number of repetitions (measurements) 24

ISQC2013


/41


• Relation between ISO based method and accordance, concordance

25725:ˆ

2

1basedISOr

A

25725:ˆ

2

1basedISOR

C

AccordanceA : eConcordancC :

,

25

ISQC2013


/41

Qualitative Methods2. Van Wieringen et al(2008)

・ binary 1: detect, pass 0: no-detect, fail

Sensitivity

Specificity

hj

iijhilmiii YXPYXXXP,

,,2,1,1,1, ,...,,lhmjni ,...,1,...,1,...,1 　　

items appraisers repetitions

model

ijhX

iY： measurement

： true value

)1( iYP

yYXPy iijhj 1

true probability of ‘pass’

1j

01 j

xj

xjijh xXP 1)0(1)0(1 x

jx

j 1)1(1)1(

where

latent class model

26

ISQC2013


/41


l

h ijhij XR1

Tmm 0,...,0,1,...,1, 11 Likelihood when

n

i

Rj

Rlj

m

j ij

ijij

R

lRL

1 1

0011;

ijij R

jRl

j

m

j ijR

l111

1

Maximum Likelihood Estimate using EM algorithm Maximum Value:

where

n

i

Rj

Rmlj

i

ii

R

mlRL

1

0011;

i

Rj

Rmlj R

mlii 111

Likelihood when )0()0(1 m )1()1(1 m and

RRL

Maximum Likelihood Estimate using EM algorithm Maximum Value: RL27

ISQC2013


/41


Repeatability

Reproducibility

28

ISQC2013


/41

Qualitative Methods3. Attribute Agreement Analysis

Fleiss’ Kappa Statistic

e

eo

P

PPˆ1

ˆˆˆ

n

i

M

jijo nmx

mnmP

1 1

2

)1(

1ˆ

M

jje pP

1

2ˆ

oP

eP

jp

n

m

M

ijx

： Probability that results actually matched： Probability that results match by chance

： number of item i categorized as j

： number of items： number of appraisers

： Ratio of category j ： number of categories

11 complete agreement the same as chance (no correlation)complete non-agreement

within appraisers between appraisers 29

ISQC2013


/41

Qualitative Methods4. Proposed Method

• We propose the method of estimating repeatability and reproducibility using the logit transformation.

• When the number of positive results xi follows a binomial distribution with parameters n and pi, then logit transformation of pi

* asymptotically follows a normal distribution as follows.

(4)

where and

iii

ii n

NL

1

1,

1ln~

*1

*ln*

i

iii LogitL

1

5.0*

n

xii

30

ISQC2013


/41

Qualitative Methods4. Proposed Method

• When we consider xi as the measurement result of laboratory i, the variances can be estimated by means of one-way ANOVA as shown below.

• Repeatability Variance

(6)• Reproducibility Variance

(7)

L

i iir nL 1

2

ˆ1ˆ11

ˆ

LLLL

L

ii

L

iiR

2

11

22

1

1

31

ISQC2013


/41


• Introduction



• Comparison

• Conclusions

32

ISQC2013


/41

Comparison of methods

• Methods are compared using the same set of data in order to clarify the relation of among the methods.– Accordance and concordance,– Attribute agreement analysis (Kappa)– van Wieringen’s method– Proposed method

• We compared the methods by averaging the obtained precision measures in the case for Langton's method and the proposed method.

• The parameters are set based on van Wieringen's method.

33

ISQC2013


/41

Comparison of methods

• Values of parameters (of figures next page)– overall (probability of an item being

conforming): 0.5– number of items: 200– number of raters L: 3– number of repetitions for each rater n: 3– probability of evaluating conforming item as pass:

0.99, 0.95, 0.90– probability of evaluating nonconforming item as

pass: 0.01, 0.05, 0.10(those probabilities are for raters 1 to 3

respectively)34

ISQC2013


/41

Comparison of methodsResults

• Repeatability(above) and Reproducibility(below)

• Strong relation among the methods. But not identical.

1000

1050

1100

1150

1200

1250

1300

0.83 0.84 0.85 0.86 0.87 0.88 0.89

van

Wie

ringe

n(r)

Accordance

0.67

0.68

0.69

0.70

0.71

0.72

0.73

0.74

0.75

0.76

0.83 0.84 0.85 0.86 0.87 0.88 0.89

κ w

ithin

rate

rs

Accordance

10.1

10.2

10.3

10.4

10.5

10.6

10.7

10.8

10.9

11.0

0.83 0.84 0.85 0.86 0.87 0.88 0.89

logi

t(r)

Accordance

0

20

40

60

80

100

120

140

0.82 0.83 0.84 0.85 0.86 0.87 0.88 0.89

van

Wie

ringe

n(R)

Concordance

0.66

0.68

0.70

0.72

0.74

0.76

0.78

0.82 0.83 0.84 0.85 0.86 0.87 0.88 0.89

κ be

twee

n ra

ters

Concordance

6.95

7.00

7.05

7.10

7.15

7.20

7.25

7.30

7.35

7.40

7.45

0.82 0.83 0.84 0.85 0.86 0.87 0.88 0.89

logi

t(R)

Concordance

35

ISQC2013


/41

Comparison of methodsResults

• Accordance and concordance, attribute agreement analysis and the proposed method gave very similar results.

• The method proposed by van Wieringen also gave similar results but the relationship was not as strong.

• The reason for giving different precision measures should be investigated.

36

ISQC2013


/41


• Introduction



• Comparison

• Conclusions

37

ISQC2013


/41

Conclusions

• The method utilizing logit transformation is proposed.• The proposed method and other existing methods are

compared using the same set of data.• Accordance and concordance, attribute agreement

analysis and the proposed method gave very similar results.

• The method proposed by van Wieringen also gave similar results but the relationship was not as strong.

• Other methods (POD models etc.) should also be compared. How to compare is the problem.

• It would be expected that these findings contribute to standardization of evaluating precision of binary measurements. 38

ISQC2013


/41

References• Danila O., Steiner, S. H., and Mackay R. J. (2008). Assessing a

Binary Measurement System. Journal of Quality Technology, 40, 310-318.

• Fleiss, J. L. (1981). Statistical Methods for Rates and Proportions. 2nd edition, John Wiley & Sons.

• Horie K., Tsutsumi Y., Suzuki T. (2008). Calculation of Repeatability and Reproducibility for Qualitative Data. Proc. 6th ANQ Congress, 12 pages (CDROM).

• ISO 5725 (1994). Accuracy (trueness and precision) of measurement methods and result – Part 1 : General principles and definitions.

• ISO 5725 (1994). Accuracy (trueness and precision) of measurement methods and result – Part 2 : Basic methods for the determination of repeatability and reproducibility of a standard measurement methods.

• ISO/TR 14468 (2010). Selected illustrations of attribute agreement analysis.

• Langton, S.D., Chevennement, R., Nagelkerke N., and Lombard B. (2002). Analysing collaborative trials for qualitative microbiological methods: accordance and concordance. International Journal of Food Microbiology, 79, 175-181.

39

ISQC2013


/41

References• Mandel, J. (1997). Repeatability and Reproducibility for Pass/Fail

Data. Journal of Testing and Evaluation, 25, 151-153.• Van der Voet, H. and van Raamsdonk L. W. D. (2004). Estimation of

accordance and concordance in inter-laboratory trials of analytical methods with qualitative results. International Journal of Food Microbiology, 95, 231-234.

• Wehling, P., LaBudde, R.A., Brunelle, S. L., and Nelson, M. T. (2011). Probability of Detection (POD) as a statistical model for the validation of qualitative methods. Journal of AOAC International, 94, 335-347.

• Van Wieringen, W. N., and de Mast, J. (2008). Measurement System Analysis for Binary Data. Technometrics, 50, 468-478.

• Wilrich, P.-Th. (2010). The determination of precision of qualitative measurement methods by interlaboratory experiments. Accreditation and Quality Assurance, 15, 439-444.

40

ISQC2013


/41

Thank you for your attention!

41

a note on precision of qualitative data

Documents