using commutable materials to assess bias by a clinical

AACB Harmonisation Workshop September 12, 2016

Using commutable materials toassess bias by a clinical laboratory

Greg Miller, PhDVirginia Commonwealth University

Richmond, Virginia, [email protected]

Financial disclosures

Siemens Healthcare Diagnostics, consultant

Roche Molecular Diagnostics, speakinghonorarium and travel expenses, focus group

Abbott Diagnostics, research grant

Outline

What is bias

What is commutability

How to assess bias usingcommutable materials

-4 -3 -2 -1 0 1 2 3 4 5

Num

ber o

f obs

erva

tions

ExpectedValue

Measured valuePrecision

ObservedValue

BIAS

Types of Bias

Bias vs a reference measurement procedure

Bias vs other measurement procedures in EQA

Bias vs other measurement procedures in ahealthcare system

Bias within a measuring system when changingreagent or calibrator lots

Commutability (Commutable)

Property of a reference material demonstratedby the closeness of agreement betweenresults for a reference material and results forclinical samples

.

(Derived from VIM 3: 2008)

0

2

4

6

8

10

0 2 4 6 8 10

Measurement procedure 1

Mea

sure

men

t pro

cedu

re2

Patient samplesReference orEQA/PT materials

Commutable: RM has same relationshipas observed for patient samples

0

2

4

6

8

10

0 2 4 6 8 10


Mea

sure

men

t pro

cedu

re2


Non-commutable: RM has different relationshipthan observed for patient samples

0

2

4

6

8

10

0 2 4 6 8 10


Mea

sure

men

t pro

cedu

re2


Apparent agreement for non-commutable RMmeans patient samples DO NOT agree

Terminology

Correct terminology:

A reference material is commutable withpatient samples among a group ofmeasurement procedures

.

Incorrect terminology: patient sample resultsare commutable among a group ofmeasurement procedures

Correct terminology: patient sample resultsare equivalent (i.e. harmonized) among agroup of measurement procedures

o Equivalent means within limits suitable for use inmedical care decisions

.

Commutability can be different with differentreagent lots or batches

Patient sample results and QC sample results werecompared for 18 QC materials before and after 1496reagent lot changes for 83 analytes using 7 differentinstrument platforms

41% of the reagent lot changes had a significantdifference between the results for QC samplescompared to those for patient samples

Miller et al. Clin Chem 2011;57:76-83.

Internal QC assesses relative bias

Assesses bias relative to a calibration condition

Does not assess bias vs a reference system

Must adjust the expected value of a QC materialwhen influenced by a reagent lot change

Measure patient and QC samples withboth current and new reagent lots

New reagent lot is acceptablefor patient testing

Agreementfor patientsamples isacceptable

Investigate new reagent lot; donot report patient results using

new reagent lot

Yes

No

Protocol for assessing QC for a reagent lot change

Step 1

Miller, QC chapter in Henry’s Clinical Diagnosis and Management by Laboratory Methods, 23rd ed. 2016

Measure patient and QC samples withboth current and new reagent lots

Agreementfor QC is

acceptable

New reagent lot is acceptablefor patient testing

Agreementfor patientsamples isacceptable

Investigate new reagent lot; donot report patient results using

new reagent lot

Yes

No

Yes

No

Continue with sametarget value

• Update QC target value• Do not change SD

Protocol for assessing QC for a reagent lot change

Step 1

Step 2


25

30

35

40

45

50

0 5 10 15 20 25 30 35

QC sequence

QC

val

ue

Target

Rule limit

Rule limit

New Reagent LotMeaninglessQC failures

Should beQC failures

Reagent lot change influence on QC


25

30

35

40

45

50

0 5 10 15 20 25 30 35

QC sequence

QC

val

ue

Target

Rule limit

Rule limit

AcceptableQC values

Real QCfailures

Adjust QC target value for meaningful assessment


Calibrator lot change

Commutability is a property influenced byreagents

QC results will correctly assess bias caused by anew lot of calibrator when the reagents do notchange at the same time

Outline

What is bias

What is commutability

How to assess bias usingcommutable materials

How can a laboratory estimate bias

1. Use commutable certified reference materials

2. Use commutable EQA materials

3. Use a set of individual patient samples

4. Use pooled patient samples (validate orassume commutability)

Bias estimation

Observed bias = biasP + biasM

Where:

biasP = bias that affects patient results

biasM = matrix-bias not observed in patientresults (non-commutability bias)

Bias estimation

Observed bias = biasP + biasM

Where:

biasP = bias that affects patient results

biasM = matrix-bias not observed in patientresults (non-commutability bias)

CommutableRMs have nomatrix bias

CLSI C37 Commutable Frozen Human Serum Pools

<15 min.

Clot 3-4 hours room temp

Pool Filter Aliquot Freeze -70C

CLSI C37 includes assessment of commutabilityof the pooled serum

1. Prepare aliquots of the individual donors

2. Measure concentration in pool and in individualdonors

3. Commutable if value for pool = mean value ofindividual donors proportional to their volumefraction

Pooled patient samples may not be commutable

Pooling can cause artifacts

Validating commutability is recommended

CDC pooled serum EQA for HDL-C Pools were prepared according to CLSI C37 protocol

Commutability was lot and criteria dependent

Korzun et al. Clin Chem 2015;61:1107-13.

Supplemented pools may be commutablePercent of SY.X <3 for EQA vs panels of patient samples

Cobbaert et al. Clin Chim A 2012;414:234-40.

What bias question does a laboratory want to know


Relative bias vs other measurement procedures in EQA

Relative bias vs other measurement procedures in ahealthcare system


Practical:

1. Use a commutable certified reference material2. Use EQA with commutable samples and values

assigned by reference measurement procedures

Not practical:

3. Use individual patient samples with values assigned bya reference measurement procedure

4. Use pooled patient samples with values assigned by areference measurement procedure

Commutable certified reference materials

Not all CRMs have had commutability validated

Many CRMs are intended for use with referencemeasurement procedures, not with clinical laboratorymeasurement procedures

JCTLM did not require commutability data prior to 2014for CRMs intended for use with clinical laboratorymeasurement procedures

WHO has not required commutability validation

Many WHO RMs for immunoassay are not commutable

Examples, commutable CRMs

NIST SRM 967 Creatinine in frozen human serum

NIST SRM 965 Glucose in frozen human serum

NIST SRM 956 Electrolytes in frozen human serum

JRC ERM DA451/IFCC Cortisol reference panel offresh frozen human sera

JRC ERM DA470k/IFCC Human serum proteins

JRC ERM DA471/IFCC Cystatin C

JRC ERM C-reactive protein

EQA with commutable samples and valuesassigned by reference measurement

procedures

Perfect when available

EQA may assume commutable samples basedon how they were prepared

o May be a reasonable assumption

o May be the only practical approach available

EQA for HbA1c correlates with NGSP certification status

NGSP.org.

CAP Accuracy Based Creatinine Survey

Used with permission from the College of American Pathologists

Value assign a pool to be traceable toa commutable CRM (or EQA)

Residual specimens

POOL

Commutable RM

Assay pool and RM Same run Replication Multiple instruments

Pool value = Value RM xMeasured poolMeasured RM

Possibleaggregates; filter

or centrifuge

Pool values should be similarto those of the RM

Value assign a pool to be traceable toa commutable CRM (or EQA)

Assumes the pool is commutable

Reasonable assumption when valueassigned using the same measuringsystem(s) for which the pool will beused to assess bias in a laboratory

MP 1 MP 2

CRM traceablepool value 95 110

What if the pool value is different for two differentmeasurement procedures?

The pool is not commutable

The pool is still useful because the assigned valuescorrect for the non-commutability bias for thespecific measurement procedures used




Relative bias vs other measurement procedures

Bias vs other measurement procedures in EQA

Only when commutable EQA materials

Clin Chem 2011;57:1670-80.

Influence of reagent lot on peer group mean

Clin Chem 2016;62:708-15.

Peer group mean

EQA samples were commutable when reagent lot H was used

Clin Chem 2016;62:708-15.

Clin Chem 2016;62:708-15.

Commutability with the next reagent lot is unknown


Absolute bias vs a reference measurement procedure


Relative bias vs other measurement procedures

Bias vs other measurement procedureswithin a health delivery system

Several instruments in different locations

o Some could be different methods

Responsibility of laboratory is that results reflectphysiology not differences in methods orinstruments

Different model instruments from the same manufacturerSame reagent and calibrator lots

-50

-40

-30

-20

-10

0

10

0 5 10 15 20 25 30

Trop

onin

I Di

ffere

nce

(ED-

Mai

n) n

g/m

L

Troponin I Concentration, ng/mL

Mfr Calib

-50

-40

-30

-20

-10

0

10

0 5 10 15 20 25 30

Trop

onin

I Di

ffere

nce

(ED-

Mai

n) n

g/m

L

Troponin I Concentration, ng/mL

Mfr Calib

CorrectionFactor

Result x 1.65

Different model instruments from the same manufacturerCorrection factor used to achieve agreement in results

CLSI EP 31

Verification of Comparability of PatientResults Within One Health Care System

Uses residual patient samples or commutablematerials

Criteria for agreement is based on a criticaldifference derived from medically allowable errorand precision performance of measurementprocedures

Lab can value assign a pool for usewithin the laboratory to assess bias

Residual specimens

POOL

Assay pool Same run Replication All methods

Pool value =

Possibleaggregates; filter

or centrifuge

Average of all methods

Selected comparativemethod

Lab can validate the commutability of a pool

Pool

Individual Specimens

Pool commutable

Measure withmethod 1 and

method 2 Pool not commutable

Do for allcombinations of

different methods

See CLSI EP-14 for details

Internal Round RobinPatient sample measured on several instruments

Criteria +/- Inst 1 Inst 2 Inst 3 Inst 4 Min Max

NA 4 mmol/L 138 139 137 136 136 139 Pass

K 0.3 mmol/L 4.5 4.6 4.5 4.2 4.2 4.6 Fail

CL 4 mmol/L 102 103 103 100 100 103 Pass

CO2 4 mmol/L 27 25 27 26 25 27 Pass

GLU 6 mg/dL or 5%* 118 119 119 116 116 119 Pass

UREA 3 mg/dL or 10%* 23 24 23 26 23 26 Pass

CREAT 0.2 mg/dL or 10%* 0.98 0.97 0.94 0.98 0.94 0.98 Pass

* which ever is higher

Within instrument vs. between instrument variability

SDa

4 instruments; same IVD• 3 Main lab• 1 ED lab

Within instrument vs. between instrument variability

SDa

For Change in Troponin

Between instrument variability dominates

Summary: bias assessment

It’s all about the matrix - use commutablesamples

Commutable CRMs or EQA samples

Pooled patient specimens Be aware of possible commutability limitations

Value assign a pool to a commutable CRM orEQA; or within lab to verify consistency

Individual patient specimens are always suitable

Bibliography

CLSI EP 30, Characterization and qualification of commutablereference materials for laboratory medicine

CLSI EP 14, Evaluation of commutability of processed samples

CLSI EP 31, Verification of comparability of patient results withinone health care system

CLSI EP 26, User evaluation of between reagent lot variation

CLSI C 37, Preparation and validation of frozen human serumpools as secondary reference materials for cholesterolmeasurement procedures

using commutable materials to assess bias by a clinical

Documents