commutable certified reference materials for next

Laboratoire national de métrologie et d’essais

Dr Vincent DELATOUR, LNE Paris

Commutable Certified Reference Materials for Next Generation

Accuracy-based EQA Schemes

1

Reliability of medical tests is a major public health challenge

1) Health point of view : risk of inappropriate medical decisions

2) Economic point of view : repetition of measurements = waste of money

25 to 30% of costs are due to test repetitions, prevention and errordetection instead of diagnostic itself (15-30 billion $ / year in the US)

3) Science point of view : lack of reliable data for epidemiological studies

and clinical trials hampers understanding of pathologies & discovery

of new treatments

❖ 60 to 70% of medical decisions are based on an in vitro diagnostic test

❖ Results are not always traceable to internationally recognized references

Results will not always be comparable depending on what method is used!

How to assess and improve reliability

and comparability of clinical measurements?

2Labquality Days 2019 – Commutable CRMs for Next Generation Accuracy-based EQAS

Importance of reference methods

Assess and improve

reliability and comparability

of medical tests

Target values

to quality control

materials

Target values

to CRMs & calibration

materials

Reference methodsReference methods


Regulatory drivers

New EU regulation 2017/746 on in vitro diagnostic medical devices

”Metrological traceability of values assigned to calibrators and/or control

materials shall be assured through suitable reference measurement

procedures and/or suitable reference materials of a higher metrological order”

Reform of medical biology in France

By 2020, accreditation according to ISO 15189 will become mandatory for ALL medical laboratories (both public and private)

✓ Development of higher order reference methods for

the main biomarkers measured in medical laboratories

✓ Certification of commutable Certified Reference Materials

✓ Assignment of target values to EQA materials

LNE’s mission : help medical labs & IVD industrials to meet

regulatory requirements regarding metrological traceability of results


Traceability chains in laboratory medicine

Miller et al. Ann Lab Med. 2014;34(3):187-197.

Metrology Institutes

& Reference Labs

IVD Manufacturers

Medical

laboratories


JCTLM

❖ JCTLM = Joint Committee for Traceability in Laboratory Medicine

❖Review teams of experts regularly review Reference Materials, Reference

Methods and Reference Laboratories for entry into a public database

http://www.bipm.org/jctlm/

Reference methods & CRMs are available only for a limited number of biomarkers


http://www.bipm.org/jctlm/

Prioritizing projects

Criteria to prioritize projects : need for a metrological input, prevalence

of the concerned pathology, potential partners, analytical challenge


❖ Diabetes Mellitus : Glucose & HbA1c

❖ Kidney Disease : creatinine, cystatin C, urea

❖ Cardio-vascular Disease : total cholesterol,

triglycerides, HDL-C, LDL-C, non-HDL-P, LDL-P

❖ Alzheimer’s Disease : A & Tau

❖ Scepsis : Procalcitonin

❖ Gout, metabolic syndrome : uric acid

❖ Eletrolyte disorders : Na, K, Cl, Ca

❖ Heavy metal intoxication : Pb, As, Hg, Cd

❖ Anemia and iron-related disorders :

Serum Iron, total hemoglobin, Hepcidin

Covered pathologies & associated biomarkers


Pillars of standardization

Certified Reference Materials for

standardization & post market vigilance

Reference Method

Target ValuesStability Homogeneity Commutability


Quality Controls in Laboratory Medicine

Internal

Quality

Controls

Externalized

Internal Quality

Controls

External

Quality

Controls


Commutability of EQA materials


Different types of External Quality Assessment Schemes

❖ EQAS relying on non-commutable materials don’t make it possible to

assess comparability of results between different peer groups

❖ EQAS relying samples which target values have not been value assigned

with a reference method don’t make it possible to assess absolute bias


Importance of commutability in EQA schemes

❖Use of a non-commutable control

material lead to the conclusion

that dry chemistry (used by 149

laboratories) had a bias of -50%

agaist IDMS reference method !

❖Agreement between results of the

different peer groups could not

be accurately estimated

❖ Fully independant post-market

vigilence could not be performed

Jaffe Enzymatic

Dry chemistry

IDMS


What is commutability?

VIM definition



When the same 10 clinical specimens (fresh serum samples) are measured with

2 different methods, Method 2 produces results about 10% higher than Method 1



When candidate CRM 1 is measured with the same 2 methods, the same behaviour

is observed : results from Method 2 are about 10% higher than those from Method 1

candidate CRM 1 is considered commutable



When candidate CRM2 is measured with the 2 same methods, a differentrelationship

is observed : results from Method 2 are about 20% lower than those from Method 1.

For the candidate CRM 2, the relationship between results from the 2 methods is

different than the relationship observed for the 10 clinical specimens.

Candidate CRM 2 is considered non commutable



❖ VIM definition

❖ Simplified definition

Property of a RM that indicates how well it mimics the characteristics

of (a set of) typical clinical specimens for a given method and for a

stated measurand


LNE-LABAC 2013 commutability study

✓ LNE CRM BIO 101a

2 pools of Human Frozen Serum (CLSI C37A)

Certified for Glucose, Creatinine, TCh, LDLc, HDLc & TG

Recognized as higher order RM by the JCTLM

✓ 9 control materials from various EQAS

- 5 EQA materials from the French mandatory EQAS (Lyophilized sera)

- 3 EQA materials from an EQAS in Singapore (Frozen sera - CLSI C37-A)

- 1 EQA material from a French EQA Provider (Frozen serum, « home made »)

Objectives : Assess commutability of 7 CRMs and 9 EQA materials

for 6 parameters : glucose, creatinine, TCh, LDLc, HDLc, TG

Commutability assessed for all (most popular) methods

37 medical labs involved : 7 Roche Cobas, 6 Siemens Vista,

6 Abbott Architect, 5 Beckman DxC, 3 Beckman AU, 2 Siemens

Advia, 3 Ortho-CD Vitros, 2 Roche Modular, 2 Thermo KoneLab

A material could be commutable for a given method but not for another one!


Commutability total cholesterol

LNE CRMs

frozen sera

CLSI C37A

ANSM EQA samples

lyophilized seraCTCB

Frozen sera

« Home made »

frozen sera

CLSI C37AControl

Fresh

patient

samples


Trueness total cholesterol

Measured bias depend on the quality

(commutability) of control materials!


Trueness total cholesterol

❖Much better agreement with commutable materials only (frozen sera, C37A)

❖ Very strong tendency to overestimate TCh conc. (mean bias 5%)

(Maximum bias recommended by the NCEP = 3%)

❖Case study where the concensus value is biased and where reference

method target values will bring true added value !

3,6mM 6,0mM 4,0mM 6,1mM 2,9mM


Décret du 26 janvier 2016 relatif à la biologie médicale

Target values associated to materials used in EQAS should

periodically be determined with a reference method, if available


Commutability TG

LNE CRMs

Frozen sera

CLSI C37A

ANSM

EQA samples

lyophilized sera

CTCB

Frozen sera

« Home made »

Frozen sera

CLSI C37A

Control

Fresh

patient

samples


Commutability direct LDL-C

LNE CRMs

Frozen sera

CLSI C37A

ANSM

EQA samples

lyophilized sera

CTCB

Frozen sera

« Home made »

Frozen sera

CLSI C37A

Control

Fresh

patient

samples


Commutability HDL-C

LNE CRMs

Frozen sera

CLSI C37A

ANSM

EQA samples

lyophilized sera

CTCB

Frozen sera

« Home made »

Frozen sera

CLSI C37A

Control

Fresh

patient

samples


2016 French National Quality Control

❖ For the first time in France, the mendatory National Quality Control organized

by ANSM relied on Certified Reference Materials of proven commutability

❖ 2 pools of frozen human serum produced according to CLSI C37A guidelines

❖ 6 parameters : glucose, creatinine, TCh, LDLc, HDLc, TG

❖ Target values assigned in LNE using higher order reference methods listed

in JCTLM database

❖ Samples distribued to all 1100 French medical laboratories in Nov. 2016

❖Results published on ANSM web site


LNE-LABAC 2016 commutability study

32 fresh, unadultarated clinical specimens

Ortho-CD

Vitros

Abbott

Architect

Beckman

DxC

Roche

Cobas

Beckman

AU

Siemens

EXL

Siemens

Vista

Siemens

Advia

❖ 30-33 FRESH, unadulterated clinical specimens selected so as to bracket

concentration of CRMs & EQA samples whose commutability is assessed

❖Clinical specimens collected and aliquoted into 8 fractions on Monday,

shipped on Tuesday and measured in triplicate on Wednesday / Thursday

in the same analytical sequence as CRMs & EQA materials

❖Commutability assessed for the 8 most popular methods in France :

Roche Cobas, Siemens Vista, Abbott Architect, Ortho CD Vitros,

Beckman AU, Beckman DxC, Siemens Advia, Siemens EXL

GOAL : Assess commutability of 8 CRMs (C37A) and 22 EQA materials

(6 frozen + 16 lyophilized) for 14 parameters : glucose, creatinine,

TCh, LDLc, HDLc, TG, uric Acid, urea, Ca, Na, Cl, K, Fe, albumin


Commutability of ANSM samples

Control materials were found commutable for most methods and parameters

Possibility to assess comparability between the different methods

Possibility to assess trueness of the different methods, bearing in mind

that 2 samples are probably not enough, especially for HDLc and LDLc

(trueness should be evaluated using panels of commutable samples)


Conclusions

➢ Commutability of frozen materials and especially those produced according

to CLSI C37A guidelines is generally better that that of lyophilized materials,

especially for HDLc

➢ HOWEVER, all frozen materials are not always commutable for all methods

and for all parameters while some lyophilized materials behave very well!

(non)commutability can’t be assumed a priori!

➢ Very few materials are commutable for all analytes: some materials are more

appropriate for some groups of parameters (electrolytes, lipids, metabolites)

➢ Very few materials are commutable for all methods, which make it difficult to

rigorously estimate agreement between the different peer-groups

➢ Some methods seem to be more affected than others by matrix effects

(eg. dry chemistry, Siemens Vista for uric acid) commutable materials

needed to trueness of these methods

Ideally, commutability of EQA materials should be

evaluated beforehand but doing it afterwards still

helpful to determine how far data analysis can go


How to assess commutability?



When candidate CRM2 is measured with the 2 same methods, a differentrelationship

is observed : results from Method 2 are about 20% lower than those from Method 1.

For the candidate CRM 2, the relationship between results from the 2 methods is

different than the relationship observed for the 10 clinical specimens.

Candidate CRM 2 is considered non commutable


Questions

- Would the conclusion be the same if other methods would have been used?

If different methods are available, how should the methods be selected?

- Would the conclusion be the same if another set of patient samples would

have been used? Are 10 patient samples enough or could we measure fewer

samples? Can any patient sample be used? How to select the number and

characteristics of the patient samples?

- In the given example, it was concluded that a -20% difference between the

2 methods for candidate CRM 2 was unacceptable : would +5% have been

acceptable? How should acceptance criteria be chosen?

- Is commutability evaluation a qualitative assessment or should some formal

statistical analysis be performed?


IFCC WG on commutability

IFCC WG-C Chair : Greg Miller

❖ TF1: Selecting patient specimens for inclusion in a commutability

study; Vincent Delatour (chair), Chris Burns, Angie Caliendo, Neil

Greenberg.

❖ TF2: Qualification of measurement procedures for inclusion in a

commutability study

Ingrid Zegers & Heinz Schimmel (co-chairs), Mauro Panteghini.

❖ TF3: Criteria to make a determination that a RM is commutable

Bob Rej (chair), Ferruccio Ceriotti, Cas Weykamp, Göran Nilsson.

❖ TF4: Statistical designs to assess commutability

Göran Nilsson (chair), Jeff Budd, Ramon Durazo, Greg Miller


Recommendations of IFCC WG on commutability

36Labquality Days 2019 –

Commutable CRMs for Next

Generation Accuracy-based EQAS

What are the different steps in a commutability study?

A commutability assessment requires the following steps:

(1) Obtain Reference Material(s) (RMs) to be evaluated;

(2) Obtain representative clinical specimens (CSs);

(3) Select methods for which commutability should be characterized;

(4) Measure the RM(s) and CSs using the selected methods;

(5) Define acceptance criteria for commutability;

(6) Perform a statistical analysis of results to evaluate commutability of the RM(s) in

relation to the results for the CSs.

(7) Make a conclusion for commutability of the RM(s) for each method involved


What is a difference in bias?

DRM = Difference in bias

= Mean Bias measured on RM – Mean Bias measured on CS


What is a difference in bias?

DRM = Difference in bias

= Mean Bias measured on RM – Mean Bias measured on CS

= (Mean result of Method y on RM - Mean result of Method x on RM)

- (average difference between results of Method y and Method x on all CS)

Difference

in bias

= - 0,13 mM

Mean


How to select patient samples?

Fresh single donations (individual clinical specimens) represent ideal samples to

establish the relationship between different methods. The clibical specimens (CSs)

may come from healthy volunteers, patients visiting a clinic or diseased patients.

Clinical specimens (CSs) should be excluded that are known to contain interfering

substances or unusual molecular forms, such as found in less common pathologic

conditions, when these affect all or most of the methods.

The interval of concentrations of the measurand in CSs must include that of the

RM(s) but don’t need to cover the entire measuring interval for the methods included.

Although individual CS are preferred, sufficient volumes of individual CSs cannot

always be obtained to enable aliquotting for distribution and measurement by all

involved methods. In this case, using pooled CSs instead of single donations is a

practical solution that may reduce the cost and complexity of a commutability study.

A preliminary experiment should be performed to demonstrate that modified CSs

(eg. pooled or frozen) are likely to be commutable with fresh individual CSs


How many patient samples are needed?

- The number of CSs needed vary with the experimental design (eg. nb of replicates)

and the performance characteristics of the methods involved (eg. precision).

- Although no fixed number can be given, a usual practice is to include at least 20-25 CSs

- Sourcing more CSs than the minimum needed for statistical assessment is

recommended to ensure that enough useable data will be available to meet the

statistical requirements for a study.

- For measurands that have large differences in measuring interval for different clinical

uses, commutability assessment may be restricted to one of the intended use

intervals or may require separate experiments for each interval. In this case,

different sets of clinical specimens are needed (eg. 25 CSs covering low and

intermediate concentrations+25CSscoveringhighconcentrations=50 CSs in total !)


How to select methods?

- It is desirable to include as many different methods and analytical measurement

principles as possible in a commutability assessment.

- However, it may not be possible to include all existing methods. Including the most

representative groups of methods will improve the likelihood of a RM being suitable

for use with methods not included in the initial assessment. Considerations for

inclusion include market share for commercially available methods and types of

analytical measurement principles.

- Includedmethodsshouldmeasure the same entity over similar concentration intervals

- Methods included in a commutability assessment should have adequate

performance characteristics, typically sufficient precision and selectivity

- Qualification of methods to be included in a study should be done prior to setting

requirements for exclusion of CSs so that methods with inadequate selectivity will

not compromise the usefulness of the evaluation of commutability of a RM.


How should measurements be conducted?

- Clinical speciems and RM(s) which commutability is being characterized should be

measured in the same conditions, ie. in the same analytical sequence

(same analyzer, same reagent lot, same calibration lot, same period of time)

- Possible existence of position effects should be investigated

- A sufficient number of replicates should be performed depending on methods’

precision : the more, the better!


How should acceptance criteria be chosen?

- Acceptance criteria for commutability depend on the intended use of the considered

RM(s): it is expected that more stringent acceptance criteria will be used to evaluate

commutability of a RM used in a calibration traceability hierarchy than for trueness

verifiers or EQA materials

- The criterion for a RM intended for use in a calibration traceability hierarchy should

be a fraction of the allowable bias for an individual CS result

- The criterion for commutability should be a fraction of the bias component of the

acceptance limits for evaluating an EQA or trueness control result.

- The criterion for a conclusion that a RM is commutable should be established at the

beginning of a commutability assessment

- In some cases, practical limitations in study design (e.g. limited number of replicates

or limited number of CSs) and / or the performance capability of all or many MPs

(e.g. poor precision or selectivity) could produce large uncertainties that could limit

the ability to make a decision about suitability of a RM. In such situations, using less

stringent acceptance criteria taking into account the study design and performance

capability of MPs can be considered. In this situation, the claimed use of the RM

should be reconsidered accordingly.


How should acceptance criteria be chosen?


How to decide whether a RM is commutable or not?

- A number of methods will be included in the assessment of commutability of a RM.

Assessment of commutability needs to consider what fraction of methods for which a

RM is considered commutable.

- For example, involvement of 7 methods (without any reference method) results in

21 pair-wise comparisons. One or more pairs of methods may be identified for which

a given RM is not commutable.

- There are no simple recommendations for the fraction of methods for which a RM

must be commutable that would qualify a RM as being fit for purpose. Considerations

include the market share for methods and consequently the number of tested

individuals who may be affected.


Conventional approaches

(CLSI C53A & EP14)

➢ Linear regressions

➢ 95% prediction intervals

➢ Uncertainties are neglected : the

hypothesis of non-commutability is

tested on the 50% level of significance!

➢ Acceptance criteria don’t take the

intended use into account

The difference in bias appoach

(IFCC WG-C)

➢ Difference plots

➢ Uncertainties are considered : some

assessments will be inconclusive

➢ Acceptance criteria defined as

function the intended use :

IQC<EQA<Trueness verifier <Calibrator

How to perform statistical analysis of results?


Séminaire HEGP 03/06/2015

doi:10.1373/clinchem.2015.240861


48Labquality Days 2019 –

Commutable CRMs


- Currently available procedures to assess commutability use criteria based only on

the statistical distribution of differences in results for CSs between two methods and

do not allow criteria based on the influence of differences between RM and CSs on

medical decisions made using the laboratory test results. Latest recommendations

established by IFCC WG-C present commutability assessment procedures involving

criteria based on medically relevant differences in results between RM and CSs

- Criteria based on the intended medical use of laboratory test results are preferred

but need to be established with consideration of methods performance capability.

- Criteria based on statistical distribution of CS results between methods are less

desirable and not recommended because they can produce different criteria for

different combinations of MPs for the same measurand. Criteria based on statistical

distribution of CS results may be unreasonably small or large compared to the

intended medical use of laboratory test results. Criteria based on statistical limits

expect some fraction of RM-MP combinations to fail to meet the criteria that could

cause an erroneous conclusion regarding the RM being fit for purpose. However, a

criterion related to achievable performance of available MPs may be acceptable as

long as the same criterion is used for all MPs in a commutability assessment,


sx and sy : pooled

SDs from replicates

performed on all CSs

with methods x & y

q : nb of CSs

r : nb of replicates

k : coverage factor

(usually k=2)



Challenges associated with commutability assessment

❖ Source a sufficient number of fresh clinical specimens which

concentrations bracket those of candidate CRMs / EQA materials

access to clinical specimens through cooperation with hospitals

possibility to use frozen materials but freeze thaw study needed

❖ Source sufficient amounts of clinical specimens so that samples can

be measured by ALL routine methods need to collect several tubes

ethics to be considered!

❖ Recrute medical labs that use the most popular routine methods

(eg. Roche Cobas, Beckman AU, Siemens Vista, Abbott Architect, …)

access to clinical specimens through cooperation with medical labs

❖ Logistics : if fresh clinical specimens are used all measurements

should be synchronized and conducted within 1 week

➢ Methods sometimes have different selectivities, making it difficult to

establish relationships between methods : ensure that methods

actually measure the same thing! Adequate precision is also needed

(otherwise, rate of inconclusive results will increase)


JCTLM WP-TEP : 2 Webinars on commutability

https://www.jctlm.org/resources/


Perspectives : Predicting non-commutability

➢ Preparation of pools : number, properties

and storage of single donations (inclusion

of « atypical » single donations?)

➢ Manufacturing process : freezing / lyophilization,

addition of preservatives and/or exogenous

substances (spiking), …

➢ Sample properties : pH, turbidity, presence of

interfencing substences or some compounds

at some concentrations (eg. TG, total protein?)

Commutability assessment of candidate CRMs & EQA materials for :

➢ Tau : EMPIR projects NeuroMet 1 (2016-2019) & NeuroMet 2 (2019-2022)

➢ PCT : EMPIR projects AntiMicroResist (2016-2019) & SeptiMet (2019-2022)

➢ Apolipoproteins, BNP & Troponin I : EMPIR project CardioMet (2019-2022)

Upcoming commutability studies organised by LNE

Predicting non-commutability


Acknowledgements : IFCC WG on PCT standardization


Acknowledgements : IFCC WG on CSF Proteins

55Labquality Days 2019 – Commutable CRMs for Next

Generation Accuracy-based EQAS

Acknowledgements : IFCC WG on CSF Proteins


Acknowledgements : IFCC WG on commutability

IFCC WG-C Chair : Greg Miller

❖ TF1: Selecting patient specimens for inclusion in a commutability

study; Vincent Delatour (chair), Chris Burns, Angie Caliendo, Neil

Greenberg.

❖ TF2: Qualification of measurement procedures for inclusion in a

commutability study

Ingrid Zegers & Heinz Schimmel (co-chairs), Mauro Panteghini.

❖ TF3: Criteria to make a determination that a RM is commutable

Bob Rej (chair), Ferruccio Ceriotti, Cas Weykamp, Göran Nilsson.

❖ TF4: Statistical designs to assess commutability

Göran Nilsson (chair), Jeff Budd, Ramon Durazo, Greg Miller


Acknowledgements : LABAC


Thank you for your attention!

Contact : [email protected]


commutable certified reference materials for next

Documents