metodologia de validacion hkas

Method Validation

HKAS Training

27 October, 2011

Dan Tholen, M.S.

Method Validation

• Validation – general discussion

• Adoption of accepted methods

– Medical: CLSI EP15

– General: ISO 21748

Method Validation

• Development of new (modified) methods

• ISO 5725: Accuracy (trueness and precision) of measurement methods

• Medical applications, CLSI

– EP5: Precision of quantitative methods

– EP17: Limits of detection and quantitation

• Food microbiology: ISO 16140

Verification and Validation

• VIM3:

– 2.44 verification: provision of objective evidence that a given item fulfills specified

requirements, taking any measurement uncertainty into consideration

– 2.45 validation: verification, where the specified requirements are adequate for a stated use

‘validation’ is subordinate to ‘verification’

VIM3: Verification

Examples:

a) Confirmation that a given reference material as claimed is homogeneous for the quantity and measurement procedure concerned…

b) Confirmation that stated performance properties or legal requirements of a measuring system are achieved.

c) Confirmation that a stated target measurement uncertainty can be met.

NOTES (verification)

1 — The item may be, e.g., a process, measurement procedure, material, compound, or measuring system.

VIM3: Validation

2.45 validation: verification, where the specified requirements are adequate for a stated use

Example:

A measurement procedure, ordinarily used for the measurement of nitrogen concentration in water, may

be validated also for the measurement of nitrogen concentration in human serum.

What’s the difference?

When a measurement method is verified, there is objective evidence that the repeatability and reproducibility (r&R) of the method can be determined for specific measurands and matrices. That is, these statistics have a mean and variance that follow an assumed distribution

When a method is validated for a particular use, there is objective evidence that the r&R are adequate for the matrices and measurands.

ISO/IEC 17025

• 5.4.5.1 Validation is the confirmation by examination and the provision of objective evidence that the particular requirements for a specific intended use are fulfilled.

ISO/IEC 17025

NOTE 1 (to 5.4.5.2) Validation may include procedures for sampling, handling and transportation.

NOTE 1 (to 5.4.5.3) Validation includes specification of the requirements, determination of the characteristics of the methods, a check that the requirements can be fulfilled by using the method, and a statement on the validity.

ISO/IEC 17025 Question

• Does a laboratory need to validate a standard method?

• Does a laboratory need to verify that a standard method works in their laboratory?

ISO/IEC 17025

• 5.4.5.2 The laboratory shall validate non-standard methods, laboratory-designed / developed methods, standard methods used outside their intended scope, and amplifications and modifications of standard methods to confirm that the methods are fit for the intended use. The validation shall be as extensive as is necessary to meet the needs of the given application or field of application.

Validating Methods

• 5.4.5.2 NOTE 2 The techniques used for the determination of the performance of a method should be one [or more] of the following:

– calibration using … reference materials;

– comparison of results achieved with other methods;

– interlaboratory comparisons;

– systemmatic assessment of the factors influencing the result;

– assessment of the uncertainty of the results based on scientific understanding of the theoretical principles of the method and practical experience.

ISO 15189 Question

• Does a laboratory need to validate a standard method?

• Does a laboratory need to verify that a standard method works in their laboratory?

ISO 15189

• 5.5.1 The laboratory shall use examination procedures, including those for selecting/taking sample portions, which meet the needs of the users of laboratory services and are appropriate for the examinations. Preferred procedures are those that have been published in established / authoritative textbooks, peer-reviewed texts or journals, or in international, national or regional guidelines. If in-house procedures are used, they shall be appropriately validated for their intended use and fully documented.

ISO 15189

• 5.5.2 The laboratory shall use only validated procedures for confirming that the examination procedures are suitable for the intended use. The validations shall be as extensive as are necessary to meet the needs in the given application or field of application. The laboratory shall record the results obtained and the procedure used for the validation.

ISO 15189

• 5.5.2 (continued)

The methods and procedures selected for use shall be evaluated and found to give satisfactory results before being used for medical examinations. A review of procedures by the laboratory director or designated person shall be undertaken initially and at defined intervals. Such a review is normally carried out annually. These reviews shall be documented.

15189 and 17025 Questions

• What needs to be validated?

– Standard methods?

– Clarifications of standard methods?

– Revisions of standard methods?

– Laboratory-developed methods?

ISO 5725 Accuracy (trueness and precision) of measurement methods and results

• Six part series, published 1994-1998– Part 1: General principles and definitions

– Part 2: Basic method for the determination of repeatability and reproducibility of a standard measurement method

– Part 3: Intermediate measures of the precision of a standard measurement method

ISO 5725 Accuracy (trueness and precision) of measurement methods and results

• Six part series (continued)– Part 4: Basic methods for the determination of the

trueness of a standard measurement method

– Part 5: Alternative methods for the determination of the precision of a standard measurement method

– Part 6: Use in practice of accuracy values

ISO 5725 General Principles

• Repeatability (r, or sr) – all conditions of measurement are the same

• Reproducibility (R, or sR) – all conditions of measurement are different, except method

• r and R represent the extremes of precision (smallest and largest)

ISO 5725 General Principles

• sr and sR are characteristics of the measurement method

• A laboratory could have repeatability different than sr, but ‘reproducibility’ is not defined for a single laboratory– Single laboratories have ‘intermediate’ measures

of precision

ISO 3534-2 Definitions

• Repeatability conditions: where independent test/measurement results are obtained with the same method on identical test/measurement items in the same test or measuring facility by the same operator using the same equipment within short intervals of time.

– A single laboratory could have smaller repeatability than is obtained in a 5725 collaborative study


• Reproducibility conditions: where independent test / measurement results are obtained with the same method on identical test/measurement items in different test or measurement facilities with different operators using different equipment

– ‘reproducibility’ is not defined for a single laboratory, but could be defined for a network of laboratories


• Intermediate precision conditions: where test results or measurement results are obtained with the same method, on identical test / measurement items in the same test or measurement facility, under some different operating condition

– NOTE 1 There are four elements to the operating condition: time, calibration, operator and equipment.

– NOTE (me) – the changed conditions must be identified when reporting intermediate precision

ISO 5725 Statistical Model

• Basic Statistical Model for any test or measurement result, y

y = m + B + e where

m is the general mean (expectation);

B is the laboratory component of bias under repeatability conditions;

e is the random error occurring in every measurement under repeatability conditions.

ISO 5725 Statistical Model: Lab Bias

• Laboratory bias (B) is a combination of both systematic and random components

var(B) = s2L

• e is the random error occurring in every test or measurement result

s2r = var(e) = s2

W

ISO 5725 Statistical Model: Lab Bias

• Reproducibility variability is a combination of laboratory and random components

sR = √(s2L + s2

r)

Or in an experiment, replace s with s:

s2R = s2

L + s2r

ISO 5725-2 Precision Experiment

• Balanced, uniform level experiment, with

q materials (each a different level)

p laboratories

n replicate results on each material

• Assume that materials are homogeneous

• Assure laboratories are competent

• Replicates are measured under repeatability conditions (not necessary for different levels)


• Statistical analysis – general steps

1. Review data for irregularities

• Missing data (can replace limited number)

• Outlying laboratories (remove, consult with lab)

• Outlying results (Grubbs or Cochran)

2. Calculate preliminary estimates of precision and means for each level separately

3. Calculate final estimates of precision, including a linear relationship between level and each type of imprecision (if appropriate)


• Statistical analysis – conventional approach

• For each level separately:

– Calculate means and SDs for each laboratory

– Calculate pooled within-laboratory standard deviation (sr for each level)

– Calculate between-laboratory standard deviation, then derive sL.

– Combine sr and sL to get sR for each level


• Statistical analysis – conventional approach

• Examine sr sL and sR across levels to see if there is a relationship between precision and level

– Could be a relationship with relative precision

– Could use regression analysis, or weighted regression

ISO 5725-3 Intermediate Precision

• When operator, calibration, equipment, or time varies within a laboratory

• Different analysis depending on the number of factors that vary

• Different analysis depending on degree of nesting, random factors, and fixed factors

• Further details, see ISO 5725-3 or standard statistical texts

ISO 5725-4 Trueness Study

• Appropriate only when it is possible to conceive of a ‘true’ value for the property being measured

• When ‘true’ values do not exist, can use an ‘accepted reference value’

• Applies to two measures of trueness

– Bias of a measurement method

– Laboratory bias

• 5725-4 applies to only one level at a time

– Not appropriate if bias varies with level

ISO 5725-4 Trueness Study

• Requires use of materials with known properties

– CRM

– RM with known properties

– Referring to a reference method

Trueness Study – Statistical Model

• Basic statistical model:

m is replaced by m+d

d = bias of a measurement method

m = true value, or accepted reference value

When d is of interest:

y = m + d + B + e

Trueness Study – Statistical Model

• When lab bias is of interest

d + B is replaced by D

D = bias of a laboratory

When laboratory bias is of interest:

y = m + D + e

Bias of a Measurement Method

• Study design similar to ISO 5725-2

– Use many laboratories

– Use CRMs or accepted reference material as test material

– Calculate sr and sR as for 5725-2

• Bias estimated as the grand mean minus the accepted reference value

• If bias d < 2sR then probably no bias

• Varies by number of labs, replicates, sR/sr

Laboratory Bias

• Only one laboratory

• Calculate D as the average of the results minus the reference value

D = ȳ - m and

sD = sW/√n n = number replicates

Laboratory Bias

• Compare D to 2/√n * sr

If D < 2/√n * sr then not significant bias

Medical Applications

• Precision for medical applications in the USA are produced by CLSI – Clinical and Laboratory Standards Institute.

• CLSI EP5 A2 (2004): Evaluation of precision performance of quantitative measurement methods

• CLSI EP9 A2 (2002): Method comparison and bias estimation using patient samples

– ‘Interim revision’, 2010 – no protocol changes

CLSI EP5-A2 - Scope

• “… for manufacturers of in vitro diagnostic (IVD) devices and developers of clinical laboratory measurement methods who wish to establish the precision capabilities of their methods. It is also for the users of those methods who wish to measure their own precision.”

– EP5-A2 also has procedures for verification of manufacturer claims

CLSI EP5-A2 - History

• EP5 (1999) – took 18 years to prepare (original work proposal, 1981).

• Revised in 2002-2004

• Currently under revision (since 2005)

– First attempt at revision timed out in 2008

– New revision convened by industry statistician

• CLSI Standards are heavily influenced by IVDD Industry

CLSI EP5-A2 Revision

• Objective for revision was to update terminology from initial version and to prepare for harmonization with ISO 5725-2

– More than one instrument

– More than one laboratory

– Precision across measuring interval

• Revision was successful, recommendations were made

• Next revision is controversial

CLSI EP5-A2 - Protocol

• EP5-A2 uses a simple protocol

– At least 2 levels of material

– At least 20 operating days

– 2 runs per day

– 2 samples per run

– 2 replicates per sample

• Recommends including other sources

– calibration

– instrument

– laboratory

CLSI EP5-A2 Protocol

• No requirement for interlaboratory comparison study

• No requirement for description of precision across measuring interval

• Includes considerations for more than one device or more than one laboratory

• Includes recommendation for describing precision across measuring interval

CLSI EP5-A2 Components

• The main objective of the precision evaluation experiment is to estimate the precision of the device or measurement method as used on a single instrument in a single laboratory.

• Components estimated:

– Repeatability

– Between run; within day; between day

– Within laboratory

CLSI EP5-A2 – Data analysis

• Remove outliers

– Replicate outliers only

diff between replicates > 5.5sr (not Cochran)

• Components estimated by conventional statistical procedures

CLSI EP5-A2 – Data analysis

• Compare repeatability and within-laboratory estimates with manufacturer’s claims

– Chi-Square statistic

CLSI EP9 - Scope

• Objective is an “independent evaluation of bias performance by individual laboratories”

• “The user is free to compare these performance estimates with either the manufacturer’s claims or the user’s own internal criteria.”

• No reference to trueness

CLSI EP9-A2 - History


• Revised in 2002


– ‘Interim revision’ in 2010 to provide new introduction by CLSI marketing department

– No substantive changes

• Not really validation

CLSI EP9-A2 - Protocol

• Simple protocol

– At least 5 operating days

– At least 40 patient samples

– Samples tested with reference and comparative methods

– 2 replicates per analysis

CLSI EP9-A2 - Analysis

• Outlier check for replicates

• Visual comparison

• Linear regression

• Bias estimate is the difference between methods at medical decision points

CLSI EP17

• EP17 (2004): Protocols for determination of limits of detection and limits of quantitation

CLSI EP17 - Scope

• “… recommendations for determining the lower limit of detection of clinical laboratory methods, for verifying claimed limits, and for the proper use and interpretation of the limits. It also provides guidance for determining lower limits of quantitation based on a laboratory’s goals for performance at low-levels.”

CLSI EP17 - History


– Based on ISO 11843 series “Capability of detection” from ISO TC69/SC6


• Applied to all IVD methods with LOD claims, since 2004

Limit of Detection - Conventional

• Repeated measurements on a blank sample or very low level sample

• Calculate SD

– LOD = 3SD; LOQ = 7 SD

– LOD = 5 SD; LOQ = 10SD

– Etcetera…no consensus

– ‘Signal to Noise Ratio’ > 4 (or 3 or 7 or ??)

• No consideration for what happens for samples that have low level positive

CLSI EP17 - Protocol

• EP17 uses a nonparametric procedure (based on ranks)

• Assumes that the distribution of results on blank samples is different than the distribution of samples with small amounts of the measurand


• EP17 uses a nonparametric procedure

– Analysis of 60 “blank” samples

– Determine “Limit of Blank” (LOB) (critical value, 95th percentile)



• Analysis of > 60 samples with level > LOB

– LOD = level where >95% of results > LOB



• LOQ based on analytical goals for error (target uncertainty)

CLSI EP17 Protocol

• All in one laboratory - no requirement for interlaboratory comparison study

• Could be manufacturer or test laboratory

Medical Applications - Verification

• In US (CLIA’88) laboratories must ‘verify’ that a method works according to manufacturer specification, prior to using the method for patient examinations.

• CLSI EP15 (2005): User verification of performance for precision and trueness

EP15 A2 (2005)

• Experiment to verify precision claim

– 5 days

– 2 different levels

– 1 run per day

– 1 sample for each level

– 3 replicates per sample

• Estimates obtained:

– Repeatability

– Within laboratory precision

EP15 A2 (2005)

• Compare repeatability and within-laboratory precision estimates with manufacturer claim

– Chi-square tests

EP15 A2 (2005)

• Experiment to verify trueness claim

– 20 patient samples (across range)

– Each sample tested in duplicate

– Need reference method

• Not really trueness, unless a CRM or definitive reference method is used

• Can compare with manufacturer claim only if the same reference method is used

– Use t test on differences

Food Microbiology

• ISO 16140: Microbiology of food and animal feeding stuffs -- Protocol for the validation of alternative methods

• Originally a European Standard, moved to ISO TC34

• In Food Microbiology there is a reference method for every organism. This standard applies to ‘alternative methods’ for the organisms…often commercial methods

ISO 16140:2003

• The need for the food industry to rapidly assess the microbiological quality of raw materials and finished products and the microbiological status of manufacturing procedures, has led to the development and refinement of alternative microbiological methods of analysis that are quicker and/or easier to perform than the corresponding reference method; some can also be automated.

• Among these alternative methods, some can yield results that are equivalent to those provided by the reference method, while others can lead to results that differ appreciably.

• (Note, some ‘alternative methods’ can be better)

ISO 16140:2003

• Normative reference to ISO 3534 (Terminology)

• Normative reference to ISO 5725 series

• All terms and procedures are predicated on the assumption that the reference method is correct

• Problem is that sometimes the reference method is not correct and alternatives can be better than the reference, for some strains of microorganisms

New ISO 16140 Series

• Part 1: Terminology

• Part 2: Protocol for the validation of alternative methods against a reference method

– Assumes that ‘alternative methods’ are proprietary

metodologia de validacion hkas

Documents

validating methods

developed methods

nonstandard methods

validation vim3

specified requirements

particular requirements

legal requirements

specific intended use