statistics & graphics for the laboratory 93 content overview interactive part factors that...

Statistics & graphics for the laboratory 1

Content overview

Interactive part• Factors that influence the interpretation of a method comparison: qualitative

use of a difference (Bland & Altman) plotFinal remark

• When to use regression-based interpretation

Exercises• Case studies 1 - 5

Content


Factors that influence the interpretation of a method comparison

Qualitative use of a difference (Bland & Altman) plot

How it works – The plot & the task

How it works – Example

Interactive part

PASSED!PASSED!

FAIL!FAIL!

DOUBT!DOUBT!

FOCUS: Total ErrorDecisions by use of the percentage of differences outside specifications for total error (TEspec):–Passed: 5% "Out" (95% "In": within ±TEspec)–Fail: >5% "Out" (<95% "In")–Doubt: no decision can be made: too close to the 5% criterium

Decision criterium Equals for pure random error"5% out" ["95% in"] 1.96 SDDifferences ≤TEspec

>Normal distribution: 95% of values are within ±1.96

-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e) n = x Total # of points

TEspec ± 6.3%

• Task –Count # of points "out"–Convert them into % of total–Decide (5% criterium): Pass, Fail, Doubt

• Plot

"Out"

-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e)

Decision criterium"5% out" ["95% in"]

n = 40 Total # of points

Total error limit ± 6.3%"1 Out"5%!

PASSED!PASSED!


Glucose "biological criteria": Maximum deviation: 6.3%

Glucose “CLIA criteria”: Maximum deviation: 0.33 mmol/l or 10%

Glucose "Glucometer-criteria": Maximum deviation = 20%

1st observationData were the same for all Specifications were 6.3, 10, 20%

May depend on specification!

Interactive part

PASSED!PASSED!

FAIL!FAIL!

Decision………………………………………………………………………

Decision………………………………………………………………………

Decision………………………………………………………………………


-20

-15

-10

-5

0

5

10

15

20

0 5 10 15 20 25

Reference (mmol/l)

Ro

utin

e (

% d

iffe

ren

ce

) n = 50

-20

-15

-10

-5

0

5

10

15

20

0 5 10 15 20 25

Reference (mmol/l)

Ro

utin

e (

% d

iffe

ren

ce

) n = 50

-3

-2

-1

0

1

2

3

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(a

bs

. dif

fere

nc

e)

n = 50

-3

-2

-1

0

1

2

3

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(a

bs

. dif

fere

nc

e)

n = 50

-25-20-15-10

-505

10152025

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e)

n = 50

-25-20-15-10

-505

10152025

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e)

n = 50


The quality of the comparison method

Cave: Total variance of a method comparison!CVTot = SQRT[CV2

Ref + CV2Rout]

Specifications are for CVRef = 0! Otherwise: Expand specifications!

Errors in the comparison methodExpand specifications!

Start specification: S (10.4%)

The comparison method has a bias B (2%)• The new specification becomes:

Snew = S + B (10.4 + 2 = 12.4%)

The comparison method has an imprecision (2%) that cannot be neglected in comparison to the specification (understood as "2 SD"-limit)

• The new specification becomes:Snew = 2 • [(S/2)2 + CVComp

2]0.5

(2•[5.2•5.2+2•2] = 11.1%)

In case of combination of both error types• The new specification becomes:

Snew = B + 2 • [(S/2)2 + CVComp2]0.5 (2+11.1 = 13.1%)

Interactive part


The quality of the comparison ("Referral") method

2nd observation

May depend on the quality of the comparison method!


3rd observation

May depend on the distribution of the outlying differences!

Interactive part

Potassium specification:10.4% ("biology")

Without error in the comparison method

New specification: 13.1%Because of error in the

comparison method, i.e. SE & RE 2%, REtest = 2%)

G (Potassium)

2,5

3,5

4,5

5,5

6,5

2,5 3,5 4,5 5,5 6,5

SPLIT 2: Referral

SP

LIT

1

n = 109

FAIL!FAIL!

G (Potassium)

2,5

3,5

4,5

5,5

6,5

2,5 3,5 4,5 5,5 6,5

SPLIT 2: ReferralS

PL

IT 1

PASSED!PASSED!

PASSED!PASSED!

FAIL!FAIL!

PASSED!PASSED!

FAIL!FAIL!

Decision………………………………………………………………………

-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e) n = 80

-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e) n = 80





Interactive part

Decision………………………………………………………………………

Decision………………………………………………………………………

Decision………………………………………………………………………


-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e) n = 40

-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e) n = 40

-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e) n = 40

-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e) n = 40

-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e) n = 40

-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e) n = 40


Statistics behind2-sided 95% confidence limits of SD and sample size n

4th observation

May depend on the sample size

Summary – Interactive part• Decisions are influenced by

– "Quality" of specification– Quality of comparison method– Distribution at "critical" analyte values– Sample size (95% CL of, e.g., "2s")

Interactive part

PASSED!PASSED!

FAIL!FAIL!


PASSED!PASSED!

-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

diff

ere

nc

e)

-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

diff

ere

nc

e)

PASSED!PASSED!

DOUBT!DOUBT!

-15

-10

-5

0

5

10

15

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

diff

ere

nc

e)

FAIL!FAIL!

They are all from the same population!

(simulations with CVdiff 3.2%)

It's statistics!

The less samples you use,

the more uncertain your outcome is!

0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0 20 40 60 80 100

n (from n = 4)

2-s

ide

d 9

5% C

L (

SD

un

its)

Upper limit

Lower limit0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0 20 40 60 80 100

n (from n = 4)

2-s

ide

d 9

5% C

L (

SD

un

its)

0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0 20 40 60 80 100

n (from n = 4)

2-s

ide

d 9

5% C

L (

SD

un

its)

Upper limit

Lower limit

"True" CVdiff,true 3.2% Point estimate n = 40: the

CVdiff,exp can be

= 1.28 • 3.2 = 4.1%

= 0.82 • 3.2 = 2.6%0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0 20 40 60 80 100

n (from n = 4)

2-s

ide

d 9

5% C

L (

SD

un

its)

Upper limit

Lower limit0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0 20 40 60 80 100

n (from n = 4)

2-s

ide

d 9

5% C

L (

SD

un

its)

0,0

0,5

1,0

1,5

2,0

2,5

3,0

3,5

4,0

0 20 40 60 80 100

n (from n = 4)

2-s

ide

d 9

5% C

L (

SD

un

its)

Upper limit

Lower limit

"True" CVdiff,true 3.2% Point estimate n = 40: the

CVdiff,exp can be

= 1.28 • 3.2 = 4.1%

= 0.82 • 3.2 = 2.6%


Method comparison S-cholesterol

Comparison method• Isotope dilution - gas chromatography/mass spectrometry (ID-GC/MS)• Negligible measurement error construct the x-axis of the difference plot by use of the ID-GC/MS, only.

Test method• Routine• Bias = 2.3%, CV = 3%.

Sample sizes• 80, 40, and 20

Plot• %-differences

Specifications• SEspec = 3% (NCEP) and TEspec = 10% (CLIA).

n = 80 n = 40

n = 20

The Bland & Altman approach

-12

-10

-8

-6

-4

-2

0

2

4

6

8

10

12

3 4 5 6 7 8 9

ID-GC/MS (mmol/l)

Ro

uti

ne

- ID

-GC

/MS

(%

)

+TE: 10%

-TE: -10%

+SE: 3%

-SE: -3%

+1.96s

-1.96s

Mean

-12

-10

-8

-6

-4

-2

0

2

4

6

8

10

12

3 4 5 6 7 8 9

ID-GC/MS (mmol/l)

Ro

uti

ne

- ID

-GC

/MS

(%

)+TE: 10%

-TE: -10%

+SE: 3%

-SE: -3%

+1.96s

-1.96s

Mean

-12

-10

-8

-6

-4

-2

0

2

4

6

8

10

12

3 4 5 6 7 8 9

ID-GC/MS (mmol/l)

Ro

uti

ne

- ID

-GC

/MS

(%

)

+TE: 10%

-TE: -10%

+SE: 3%

-SE: -3%

+1.96s

-1.96s

Mean

Observation…………………………………….…………………………………….

Observation…………………………………….…………………………………….

Observation…………………………………….…………………………………….

Conclusion• Easy visual inspections from the extended Bland&Altman plot.• Importance of the sample size• Account for the sample size via the statistical estimates


Regression-based interpretation …

Thienpont LM, Van Nuwenborg JE, Bland JM, Altman DG. Stöckl D. Clin Chem 1998;44:849-57 Stat Meth Med Res 1999;8:135-60

… when systematic errors are related to the concentration!

Checklist

Consider• Sort of plot (absolute, %, "extended") • "Quality" of specification• Quality of comparison method• Distribution at "critical" analyte values• Sample size (95% CL of estimates, e.g., "2s")• Regression-based interpretation (systematic errors related to the concentration)

• Sample quality (clinical relevance)


-6

-4

-2

0

2

4

3,5 4,0 4,5 5,0 5,5

IC (mmol/L)

(

%)

Ro

uti

ne

- IC

-4

-2

0

2

4

0 1 2 3 4 5 6 7Average fat content (g/100 ml)

Dif

fere

nc

e (

g/1

00 m

l)

-6

-4

-2

0

2

4

3,5 4,0 4,5 5,0 5,5

IC (mmol/L)

(

%)

Ro

uti

ne

- IC

-4

-2

0

2

4

0 1 2 3 4 5 6 7Average fat content (g/100 ml)

Dif

fere

nc

e (

g/1

00 m

l)


References

Altman DG, Bland JM. Measurement in medicine: the analysis of method comparison studies. Statistician 1983;32:307-17.

Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet 1986;i:307-10.

Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res 1999;8:135-60.

Dewitte K, Fierens C, Stöckl D, Thienpont LM. Application of the Bland-Altman plot for the interpretation of method-comparison studies: a critical investigation of its practice. Clin Chem 2002;48:799-801.

Stöckl D, Rodríguez Cabaleiro D, Van Uytfanghe K, Thienpont LM. Interpreting method comparison studies by use of the bland-altman plot: reflecting the importance of sample size by incorporating confidence limits and predefined error limits in the graphic. Clin Chem 2004;50:2216-8.

Mantha S, Roizen MF, Fleisher LA, Thisted R, Foss J. Comparing methods of clinical measurement: reporting standards for Bland and Altman analysis. Anesth Analg 2000;90:593-602.

Pollock MA, Jefferson SG, Kane JW, Lomax K, MacKinnon G, Winnard CB. Method comparison - a different approach. Ann Clin Biochem 1992;29:556-60.Stöckl D. Beyond the myths of difference plots [letter]. Ann Clin Biochem 1996;33:575-7.

Stöckl D. Difference versus mean plots [reply]. Ann Clin Biochem 1997;34:571.Hyltoft Petersen P, Stöckl D, Blaabjerg O, Pedersen B, Birkemose E, Thienpont L, Flensted Lassen J, Kjeldsen J. Graphical interpretation of analytical data from comparison of a field method with a reference method by use of difference plots [opinion]. Clin Chem 1997;43:2039-46.

National Cholesterol Education Program. Recommendations for improving cholesterol measurements. US Department of Health and Human Services publication NIH 90-2964. Washington, DC: National Institutes of Health, 1990.Clinical Laboratory Improvement Amendments of 1988; Final Rule. Fed Reg February 28 1992;57: 7001-288.



Graphical interpretation of a method comparison

By use of integrated specifications

Data not within or at the limit of the specifications Interpret with graphical and statistical techniques

Method comparison exercises

InterpretationThe comparison shows visually that the quality of the test method is well within the specification. There is no need for further investigation (for, e.g., linearity, bias, Sy/x, etc.)

ConclusionInterpret the method comparison in first instance visually against the selected specification(s).

0

5

10

15

20

25

0 5 10 15 20 25Reference (mmol/l)

Glu

co

me

ter

XY

Z (

mm

ol/l

) AA

BB

C

C

D D

E

E

-20

-15

-10

-5

0

5

10

15

20

0 5 10 15 20 25

Reference (mmol/l)

Ro

uti

ne

(%

dif

fere

nc

e) Specification (6.3%)!

not Bland & Altman 2s

Glucose “biological criteria”:Max. deviation: 6.3%

Sort of specification:Proportional error

Choice: % bias plot


Interpretation strategy

“When the data are not within or at the limit of the specifications”

General• Reconsider the quality of the samples (number, matrix) and the measurement protocol (incl. the comparison method) • Reconsider the internal quality control• Judge the comparison visually (concentration range, outliers) (consider all possible graphics)

Judge the data for linearity• Direct (if more appropriate, in the logarithmic plot) or• Use a “residuals plot” and check the sequence of the signs (+/-) of the residuals• If x and y indeed are linearly related, perform correlation analysis• If x and y are not linearly related, perform non-linear regression (only for the purpose of calibration)

Correlation analysis• If r <0.99 (broad range) or <0.975 (small to medium range), check again for outliers• When, thereafter, r sufficiently increases, perform linear regression without the outliers• Perform Deming regression (DR) (or Passing Bablok regression (PBR))• When r does not sufficiently increase, calculate SDy/x of OLR (note: use

alternatively SD of the differences yi - xi (SDdiff)) and compare with the total

imprecision of the method comparison due to both SDax and SDay

• a) If SDy/x or SDdiff ~ total imprecision, reduce the latter (e.g., by performing

multiple measurements)• b) If SDy/x or SDdiff >>> total imprecision

there is a substantial analytical difference between the methods due to sample-related effects

Deming (PB) regression• Use the regression parameters to find the cause of the poor quality:

–SE proportional (slope sign. 1)–SE constant (intercept sign. 0)–Matrix-related RE (SDy/x or SDdiff >>> total SDa)

• Improve the method (perform, if more appropriate, an in-depth evaluation of the "elements" of the method: calibration, recovery, interference studies, ...).

Adapted from: Stöckl D, Dewitte K, Thienpont LM. Clin Chem 1998;44:2340-6



Case study 1

Analyte: Estradiol in serum

Samples• 24 (range: 15 - 3000 pmol/l)• 4 series of 6 samples mixed from 4 native pools and 1 "stripped" serum ("6"/0

ml; 5/1; 4/2; 3/3; 2/4; 1/5)IQC: 3Design: 1 series, duplicatesMethod x: GC/MS, CV 6%: confirmed from IQC and duplicatesMethod y: Immunoassay, CV 6%: confirmed from IQC and duplicatesCVtot,expected: 8.5%

Graphical judgement: no irregularitiesCorrelation/regression data:

• r = 0.998; y = 1.03 (± 0.05) x - 10 (± 15) pmol/l, • Sy/x (%): 9%

Conclusion: Is the immunoassay equivalent to GC/MS?

We check all elements of the method comparison studyIQC: 3Design: 1 series, duplicates Method x: GC/MS, CV 6%: confirmed from IQC and duplicates Method y: Immunoassay, CV 6%: confirmed from IQC and duplicates CVtot,expected: 8.5% Graphical judgement: no irregularities Correlation/regression data:

• r = 0.998 ; y = 1.03 (± 0.05) x - 10 (± 15) pmol/l, • SDy/x (%): 9%

Samples• 24 (range: 15 - 3000 pmol/l)• 4 series of 6 samples mixed from 4 native pools and 1 "stripped" serum ("6"/0

ml; 5/1; 4/2; 3/3; 2/4; 1/5)

Conclusion: Decision not possible, wrong design! No native samples!

Strategy, General• [Re]consider the quality of the samples (matrix)



Case study 2

Case analogous to an example in literatureAnalyte: HDL-cholesterol in serum

Samples: 100 native serum samplesDesign: multiple series, singlicates

Method x: "reference method" (validated with the official reference method of CDC): indirect method (phosphotungstic acid/MgCl2)

Between-day CV: 3%

Method y: direct method (detergent + enzymatic)Between-day CV: 3.8%

CVtot,expected: 4.8%

Specifications (NCEP)SD < 0.044 mmol/l, or CV < 4%

Interpretation of combined absolute & %-specifications (select the highest)Example: 0.044 mmol/l and 4%

• Consider the measurement range and calculate at which concentration 4% = 0.044 mmol/l• Here: at 1.1 mmol/l is 4% = 0.044 mmol/l > 1.1 mmol/l, use the 4% specification, 1.1 mmol/l, use the 0.044 mmol/l specification

Specifications case 2SD < 0.044 mmol/l (up to 1.1 mmol/l), or CV < 4% (> 1.1 mmol/l) But, specifications are expressed in terms of RE

Transformation of specifications for RE to TEwith the formula: TE = SE + k • RETE = RE • k (k = 2, Westgard “classical”; k = 3, Ehrmeyer & Laessig; k = 4, Westgard, with IQC)We select: k = 2 for a method comparison TE = 2 • RE (specification) = 8%

Note: Only valid when the comparison method (x) is error free.Here: CV(x) = 3%, which cannot be neglected in comparison tothe CV of the test method (3.8%) Expand the specification


Basic

introduction-participant

Datasets-MethComp


0

1

2

3

0 1 2 3HDL-indirect (mmol/l)

HD

L-d

ire

ct

(mm

ol/l

)

Original specification (TE) = 8%Suppose SE = 0The CV(x) = 3%, which cannot be neglected in comparison to the CV of the test method (3.8%) and the specification.

The new specification becomes:Snew = 2 • [(S/2)2 + CVComp

2]0.5

= 2•[4•4+3•3] = 10.0% or= TE(abs) = 0.11 mmol/L

Interpretation of case study 2Graphical presentation&Specifications• Absolute: <0.11 mmol/l (up to 1.1 mmol/l)• Proportional: <10% (>1.1 mmol/l)

“When the data are not within or at the limit of the specifications”General

• Reconsider the quality of the samples (number, matrix) and the measurement protocol (incl. the comparison method)

• Reconsider the internal quality control • Judge the comparison visually (concentration range, outliers) (consider all

possible graphics)

Judge the data for linearity!-Direct (if more appropriate, in the logarithmic plot) or-Use a “residuals plot” and check the sequence of the signs (+/-) of the residuals

Something unusual?

0

1

2

3


HD

L-d

ire

ct

(mm

ol/l

)

Expansion of specifications



Judge the data for linearity• If the data are not linearly related

–Perform non-linear regression analysis for the purpose of calibration

Calibration-Linearisation and correction for SE: via the reverse plot and the trend line Note: The success of recalibration has to be checked with a new set of data!

0

1

2

3


HD

L-d

ire

ct

(mm

ol/l

)

Case study 2 (ctd.)


Something unusual?

y = 1,1729x - 0,1518r = 0,9764

0

1

2

3

0 1 2 3

HDL-indirect (mmol/l)

HD

L-d

ire

ct

(mm

ol/l

)

0

0,5

1

1 ,5

2

2 ,5

3

3 ,5

0 0,5 1 1 ,5 2 2 ,5 3 3 ,5

HDL-in d ire c t (mmo l/l)

0

0,5

1

1 ,5

2

2 ,5

3

3 ,5

0 0,5 1 1 ,5 2 2 ,5 3 3 ,5

HDL-in d ire c t (mmo l/l)

-1

0

1

0 1 2 3


H

DL

-dir

ec

t (m

mo

l/l) Judgement of linearity:

Residual-plot!

y = 0,1137x2

+ 0,4731x + 0,3907

0

1

2

3

0 1 2 3

HDL-direct (mmol/l)

HD

L-in

dir

ec

t (m

mo

l/l)

0

1

2

3


HD

L-d

ire

ct

(mm

ol/l

) Transformed data

y' = 0,1137*y2 + 0,4731*y + 0,3907


Case study 3

Case analogous to an example in literatureAnalyte: HDL-cholesterol in serum

Samples: 100 native serum samplesDesign: multiple series, singlicatesMethod x: indirect method (phosphotungstic acid/MgCl2)

Between-day CV: 3%Method y: direct method (detergent + enzymatic)Between-day CV: 3%CVtot,expected: 4.2%

Specifications (NCEP)SD < 0.044 mmol/l ( 1.1 mmol/l), or CV < 4% (>1.1 mmol/l)

Note: The new specification for TE = 2•[4•4+3•3] = 10.0% orTE(abs) = 0.11 mmol/L

Visual interpretation -Too many points outside specifications-Otherwise no irregularities Further statistical, analytical investigations needed

General• Reconsider the quality of the samples (number, matrix) and the measurement protocol (incl. the comparison method) • Reconsider the internal quality control • Judge the comparison visually (concentration range, outliers) (consider all possible graphics)


Basic

introduction-participant

Datasets-MethComp

0

1

2

3


HD

L-d

ire

ct

(mm

ol/l

)

-1

0

1

0 1 2 3

[HDL-ind. + HDL-direct]/2 (mmol/l)

H

DL

-dir

ec

t (m

mo

l/l)


Judge the data for linearity• Direct (if more appropriate, in the logarithmic plot) or • Use a “residuals plot” and check the sequence of the signs (+/-) of the residualsIf x and y indeed are linearly related, perform correlation analysis• If x and y are not linearly related, perform non-linear regression (for the purpose of calibration)

Correlation analysisr = 0.9803 (small range!) • If r <0.99 (broad range) or <0.975 (small to medium range), check again for outliers• When, thereafter, r sufficiently increases, perform linear regression without the outliers• When r does not sufficiently increase, calculate SDy/x of OLR or SDdiff and

compare with the total SDa of the method comparison

• a) If SDy/x or SDdiff ~ total SDa, reduce the SDa of the method comparison (e.g.,

by performing multiple measurements)• b) If SDy/x or SDdiff >>> total SDa, there is a substantial analytical difference

between the methods due to sample-related effects Perform Deming regression (DR) (or Passing Bablok regression (PBR))

Use the regression parameters to find the cause of the poor quality• SE proportional: slope (1.010 ± 0.045) • SE constant: intercept (-0.03 ± 0.058)

No problems indicated by regression.

> Look at the random differences

y = 1,010x - 0,03r = 0,9803

0

1

2

3

0 1 2 3


HD

L-d

ire

ct

(mm

ol/l

)

Case study 3 (ctd.)


Linear regression95% CLs of intercept: –0.061 to 0.055of slope 0.965 to 1.055 Note: SDy/x (OLR) = 0.116 mmol/l


Compare SDy/x from OLR (0.1164 mmol/l) with the total SDa of the method

comparison:SDy/x

2 = SDay2 + b2 SDax

2 (CVy/x = 2 CVay)

0.1164 >>> 0.0528 mmol/l (7.8% >> 4.2%)There is a substantial analytical difference between the methods due to sample-related effects

Alternatively: SD of the differences yi – xi can be used, here SDdiff = 0.1159 mmol/l

Note: "RE" in method comparison. Not only consider the value, but compare the spread of all points in a graphic

Observation:Too many points outside specifications due to "random" spread.

ConclusionImprove the method.If appropriate, perform an in-depth evaluation of the “elements” of the method: interference studies, specificity, method principle, ... WHICH ONE? X or Y?

Experimental investigations, done after the methodcomparison, according to literature

Interference studies• Hemolysis• Lipemia (caused most of the problems)• Bilirubinemia

-30

-20

-10

0

10

20

30

0 1 2 3

[HDL-ind. + HDL-direct]/2 (mmol/l)

H

DL

-dir

ec

t (%

) +2CV

-2CV

Case study 3 (ctd.)


Note:Bland-Altman "2s" (2 CVdiff) = 15.6%

CVtot = 2 * 4.2% = 8.4%


Case analogous to an example in literatureAnalyte: Troponine-I in serum

Samples: >200 native serum samplesDesign: multiple series, singlicatesMethod x: immunoassay 1, CV ~ 12%Method y: immunoassay 2, CV ~ 10%CVtot,expected: ~ 16%

Specifications None: Select 2 • CVtot,expected = 32%

The big slope and the high number of outliers• Have both methods specificity problems?• Is the slope only caused by a difference in calibration?

Can we be sure that both methods measure the same analyte?

-40-30-20-10

010203040

0 5 10 15 20 25 30 35

Troponine-I [1+2]/2 (µg/l)

Tro

po

nin

e-I

[2

-1]

(µg

/l)

0

10

20

30

40

50

0 10 20 30 40 50Troponine-I 1 (µg/l)

Tro

po

nin

e-I

2 (

µg

/l) Without outliers:y = 3,3 x + 1,2r = 0,9597With outliers:y = 2,1 x + 5,8r = 0,7134

Case study 4


Graphical interpretation(incl. regression)

Two problems apparent:- the outliers- the big slope

Bias plotNote that the concentration range (x-values)(in particular of the "1" method) cannot longer be recognized>Problem of the large slope


Case study 5

Case analogous to an example in literatureAnalyte: Potassium in serumAim: eventual recalibration

Samples: 60 native serum samplesMethod x: IC-reference methodBetween-day CV: 1.5% (for a design of 4 measurements per sample = 0.75%)Method y: ISEWithin-day CV: 1.1% (singlicates)

Specification (CLIA)TE = 0.48 mmol/l (= 10% at 4.8 mmol/l)

Regression in a method comparison

Judgement of the 95% CLs of the regression parameters in comparison to specifications

Uncertainty of the slope alone ca. 17%A recalibration via the method comparison exceeds the error budget!


y = 1.16 x – 0.86 95% CLs of slope = ±0.1795% CLs of intercept = ±0.74 mmol/l

SpecificationCLIA limit: 10% at 4.8 mmol/l

y = 1.16x - 0.86r = 0.871

3

4

5

6

3 4 5 6Reference (mmol/L)

Ro

uti

ne

(mm

ol/

L) ..

.


Notes

Notes

statistics & graphics for the laboratory 93 content overview interactive part factors that...

Documents