performing a self-evaluation when proficiency testing is not ......good laboratory practices, cap...

Performing a Self-Evaluation When Proficiency Testing is Not Graded

1 Role of PT in Promoting Laboratory Quality ................................................ 2

2 When PT Self-Evaluation is Required ........................................................... 3

3 Choosing a Method for PT Self-Evaluation .................................................. 4

4 Examples of PT Self-Evaluation Methods .................................................... 5• Quantitative Tests ..............................................................................................5• Qualitative Tests ...............................................................................................6• Challenges Intended to be Educational ..................................................................7

5 Documenting PT Self-Evaluation .................................................................. 8

6 Review of Method/Instrumentation ............................................................. 10

7 Relevant Regulatory and Accreditation Requirements ..............................11

Table of Contents

Performing a Self-Evaluation When Proficiency Testing is Not Graded College of American Pathologists 2

Role of PT in Promoting Laboratory QualityLaboratory quality is promoted by a robust internal quality management program overseen by laboratory management, supplemented by inspections of laboratory operations by external organizations.

Laboratory internal quality management programs have many elements such as regular instrument maintenance, instrument calibration and calibration verification, training and competency assessment of testing personnel, verification or validation of new test methods, monitoring of quality control for shifts and trends, a prescribed laboratory management structure that includes a laboratory director and other key roles, and other components.

External inspections of laboratory quality include periodic inspection of laboratory practices and procedures against a set of published requirements, and PT to assess the accuracy of laboratory testing.

Graded PT provides an external check on many aspects of a laboratory’s quality management program. PT can identify when a laboratory’s assay has a quality problem when compared to its peer group.

There are many types of problems that PT has revealed: PT can uncover when laboratory staff do not follow the PT provider’s instructions or pick the appropriate method/instrument code, resulting in being placed in the wrong peer group and ultimately failing an analyte. If a laboratory continues to use the same reagent/method code while it had switched to a third-party reagent, the results will not match with its peer group resulting in failure. PT has disclosed instances when a laboratory has not addressed the issues uncovered from a previous mailing by examining the standard deviation index for a positive or negative bias.

When grading of PT indicates an individual laboratory result is unsatisfactory, laboratory management is required to investigate the issue. Sometimes, investigation reveals a clerical error reporting PT results, or some other issue that is unique to PT and would never impact a patient. In other instances, an underlying issue that could affect patient care is revealed, and management is required to address the problem. It is encouraged that the laboratory management review all PT results regardless of performance.

External laboratory inspectors frequently review laboratory management’s response to PT issues. In some instances, when a laboratory demonstrates multiple unacceptable PT results, the laboratory can be prohibited from performing a particular test until the problem is corrected to the satisfaction of regulatory agencies.

The CAP believes that all types of laboratory testing present risks, and that no test is so foolproof or inconsequential that it can be performed in a clinical setting without the safeguards provided by PT.

This guide is designed for laboratories enrolled in CAP proficiency testing (PT) programs that have submitted results to the CAP, but did not receive a grade for one or more results.


When PT Self-Evaluation is RequiredThe CAP is the largest PT provider in the world, with more than 22,700 subscribers and 700 different PT programs.

The CAP strives to grade as many PT results as possible, but occasionally a laboratory’s PT results are not graded. This can occur for several reasons, but most commonly has one of the following explanations:

• Insufficiently sized peer group. Most PT materials contain an artificial matrix that preserves sample stability. Various instrument and reagent combinations can be differentially affected by the matrix in the PT material. To ensure that laboratories are evaluated fairly, CAP only grades challenges when there are at least 10 laboratories in a peer group that use the same instrument and reagent combinations, or when CAP has reason to believe that several instrument/reagent peer groups can be combined together without penalizing laboratories.

• Lack of participant or referee consensus. Sometimes, fewer than 80% of participants or referees (for Clinical Laboratory Improvement Amendments of 1988 [CLIA] regulated analytes) agree on what is the correct response for a challenge. In this case, the challenge is not graded. CAP does not believe it is reasonable to hold laboratories accountable for interpreting challenges that did not meet the expected criteria.

• Educational challenge. CAP sometimes includes PT specimens that assess the ability of laboratory staff to make difficult distinctions, to deal with special interferences or circumstances, or that may challenge the routine capabilities of many well-run laboratories. In these cases, the PT specimen is not graded by design. In some situations, an entire PT program may consist of educational, ungraded specimens designed to help subscribers improve their skills and capabilities.

Good laboratory practices, CAP Laboratory Accreditation Program (LAP) requirements, and (in some cases) governmental regulations require that laboratories perform a PT self-evaluation when PT is not graded by the PT provider. Self-evaluation must be documented.

Self-evaluation is required for ungraded PT specimens even when they are intended to be educational. For example, if a graded microbiology PT program includes an ungraded educational specimen, self-evaluation and documentation of the educational PT specimen is required.

The CAP also offers some programs such as the Laboratory Preparedness Exercise which is an educational exercise and not PT. Laboratories are encouraged to perform a self-evaluation by comparing results published in the participant summary report.

See the next section (Choosing a Method for PT Self-Evaluation) for more information about options for performing a self-evaluation.


Choosing a Method for PT Self-EvaluationThere are different methods for performing a PT self-evaluation. The laboratory director is responsible for choosing a self-evaluation method appropriate for the laboratory’s individual circumstances. The laboratory director may choose to delegate this task to others within the laboratory.

The selection of a method for self-evaluation is important:

• Any PT results falling outside the laboratory’s self-established criteria for acceptable performance must be investigated and corrective action taken, as would be done for graded PT results.

• An external laboratory inspector may review whether the method selected by laboratory management is appropriate. Although uncommon, the external inspector may challenge the self-evaluation method selected by laboratory management.

The PT self-evaluation method should be selected to maximize the likelihood that laboratory problems will be detected, while minimizing the number of “false positives” — incorrectly flagging a result as unacceptable when the assay, in fact, is performing within acceptable tolerances. In other words, laboratory directors should not aim to select the least stringent or the most stringent method of self-evaluation. Instead, the focus should be on selecting a method that is appropriate and practical.

Once a self-evaluation method is selected, that method should be used for all ungraded challenges in a PT program for an analyte. It is not acceptable to use different methods for self-evaluation for challenges in the same PT program and same analyte, simply to improve a laboratory’s grade.1 In general, when a laboratory changes its method for self-evaluation, a reason for making the change should be documented.

The next section provides examples of different PT self-evaluation methods and discusses some of the advantages and drawbacks of each method.

1 An exception to this rule may be made when the laboratory changes its testing system or new information

becomes available that suggests another approach to grading is more appropriate.


Examples of PT Self-Evaluation MethodsIn this section, we list methods that may be used to perform a PT self-evaluation when test results are quantitative, when test results are qualitative, and for educational challenges. For each method of self-evaluation, we explain some of its advantages and disadvantages, and illustrate how to use the method in practice.

Quantitative TestsUsing an alternative peer group. This is the most popular and easiest method for performing a self-assessment when PT of a quantitative test is not graded. It is most commonly used when a laboratory’s peer group is too small for grading (fewer than 10 laboratories reporting PT results for a method; what the CAP calls a “Code 20” situation).

If the laboratory director believes the method used in the laboratory is scientifically comparable to another method used by a greater number of laboratories, the director can elect to self-evaluate PT results against the results from a larger peer group that was graded. In some situations, the director may determine that the laboratory’s results can be reasonably compared to the results from all methods combined (“all-method group,” because the results from different methods do not differ significantly.)

The participant summary (PS) report that is supplied with PT results contains the mean value and high/low values reported for large peer groups that are graded, as well as the target value and high/low values from all methods combined if provided.

For example, your laboratory’s reported PT result of 13.8 g/dL, but the peer group was too small for the CAP to grade. A similar method that your laboratory director believes to be comparable has group mean of 13.77 g/dL and an acceptable range of +7%. Your laboratory director can decide to use 13.77 g/dL as a target value, and self-evaluate your PT result using the same range of acceptability that was used for graded results (±7% in this example):

Your Result 13.8 g/dLNot graded by the CAP, due peer group size <10 laboratories.

Target Value for a Similar Method

13.77 g/dLFound in the Participant Summary

Range of Acceptability 12.8-14.8 g/dL= Target Value for Similar Method +7%. The +7% threshold is found in the PS.

Self-Evaluation AcceptableYour result falls within range for acceptability

In a similar fashion, if the PS includes data for groups of 3-9 laboratories, the laboratory director can decide to perform a self-evaluation using the median, low and high values provided.


• If no method for self-evaluation is determined to be appropriate, an alternative approach to performance assessment is recommended. Alternative performance assessment does not make use of CAP PT material but ensures that the laboratory assay is still subjected to an external check on the accuracy of results, beyond routine quality control testing and calibration verification. It is the responsibility of the laboratory director to define such alternative performance assessment procedures, as applicable, in accordance with good clinical and scientific laboratory practice. Tolerance limits should be defined to determine acceptable performance. A few approaches of alternative performance assessments are listed below:

• Split sample analysis with another laboratory• Split sample analysis with an established in-house method• Split sample analysis with an assayed material

Qualitative TestsSurmising the correct response from other participant’s results. If a qualitative result is not formally graded due to lack of participant or referee (for CLIA regulated analytes) consensus (what the CAP calls a “Code 27” situation), laboratory management can often self-evaluate results by comparing the laboratory’s response to results in the PS. Usually, the majority of referees or laboratories reported the intended, correct response, even though consensus (80%) was not achieved. Transfusion medicine requires 100% referee or 95% participant consensus. For example, a PT specimen for Gram stain might not be graded due to lack of consensus (77% reported gram-negative, 23% reported gram-positive or gram-variable), but a large majority of laboratories were able to agree on a negative gram stain reaction.

The scientifically defensible approach for reasonable consensus for self-evaluation is at the discretion of the laboratory director when participants and/or referees achieved below the 80% required by the CAP for formal grading.

If laboratory management uses this approach for self-evaluation, the evaluator should refer to the majority response (gram-negative in the case above) and indicate that the laboratory’s response was “acceptable” or “unacceptable” using the majority response as the correct value. If the laboratory director determines that the result was unacceptable, corrective action should be taken.

Surmising the correct response from the PT provider’s intent. This method of self-evaluation is like the previous method but relies on background information provided by the CAP to determine the correct response. For example, a PS might indicate that an organism provided for gram staining was Pseudomonas aeruginosa, a gram-negative bacillus. In this case, the laboratory can self-evaluate its reported PT result and consider a response of “gram-negative” to be acceptable, given the organism that was used to formulate the challenge.

The scientifically defensible approach for reasonable consensus for self-evaluation is at the discretion of the laboratory director when participants and/or referees achieved below the 80% required by the CAP for formal grading.

Sometimes, this approach and the preceding approach can be combined. There are instances in which the required 80% consensus is not achieved and the CAP does not grade a challenge, and the intended correct response is also spelled out in the participant summary (eg, naming the organism used for a gram stain challenge).


If it is not possible to determine a correct/intended response, the laboratory should consider performing an alternative assessment method.

Challenges Intended to be EducationalEducational PT challenges are designated with an evaluation code of 26. These challenges are not formally graded, and laboratories should utilize data in the PS to perform a self-evaluation using the guidance provided above.


Documenting PT Self-EvaluationGood laboratory practice, the CAP LAP, and (in some cases) governmental regulations require that laboratory staff document PT self-evaluation.

Written Laboratory Procedure Regarding PT Evaluation. The following is an excerpt from the CAP Accreditation Program checklist regarding PT procedures for handling, analysis, review, and reporting of PT.

COM.01000 PT Procedure Phase II

The laboratory has written procedures for proficiency testing (PT) sufficient for the extent and complexity of testing done in the laboratory.

NOTE: The laboratory must have written procedures for:

• Proper handling, analysis, review and reporting of PT materials.• Investigation of each unacceptable PT result to evaluate the impact

on patient sample results and to correct problems identified in a timely manner.

REFERENCE

Clinical and Laboratory Standards Institute (CLSI). Using Proficiency Testing and Alternative Assessment to Improve Medical Laboratory Quality. 3rd ed. CLSI guideline QMS24. Clinical and Laboratory Standards Institute. Wayne, PA; 2016.

Specific Ungraded Challenges. The laboratory should create documentation of its self-evaluation for each PT challenge that was not graded. In general, this documentation should include the following:

1. Specify the process for self-evaluation (ie, similar peer group, 3-9 peer data, all method). In general, the same process for self-evaluation should be used for every ungraded challenge for an analyte.

2. Document whether the laboratory’s PT result was acceptable or unacceptable, using the selected process.

3. If the result is not acceptable, investigate the issue, document the result of the investigation, and document any corrective action taken.

4. Sign and date documentation, in conformance with general laboratory procedure.

COM.01700 PT and Alternative Performance Assessment Result Evaluation

Phase II

There is ongoing evaluation of proficiency testing (PT) and alternative performance assessment results with appropriate corrective action taken for each unacceptable result.


NOTE: Each unacceptable PT or alternative performance assessment result (any result or sample not meeting defined acceptability criteria) must be evaluated. It is recommended that the laboratory investigate acceptable results that show significant bias or trends.

Primary records related to PT and alternative performance assessment testing are retained for at least two years (five years for transfusion medicine). These include all instrument tapes, work cards, computer printouts, evaluation reports, evidence of review, and records of follow-up or corrective action.

For laboratories outside the US, PT failures relating to problems with shipping and specimen stability should include working with local customs and health regulators to ensure appropriate transit of PT specimens.

Evidence of Compliance: Records of ongoing review of all PT reports and alternative

performance assessment results by the laboratory director or designee AND

Records of investigation of each “unacceptable” PT and alternative performance assessment result including records of corrective action appropriate to the nature and magnitude of the problem

REFERENCES1. Department of Health and Human Services, Centers for Medicare

and Medicaid Services. Clinical laboratory improvement amendments of 1988; final rule. Fed Register. 1992(Feb 28):7173 [42CFR493.1407(e)(4)(iv)].

2. Steindel SJ, et al. Reasons for proficiency testing failures in clinical chemistry and blood gas analysis. A College of American Pathologists Q-Probes study in 655 laboratories. Arch Pathol Lab Med. 1996;120:1094-1101.

3. Clinical and Laboratory Standards Institute. Using Proficiency Testing and Alternative Assessment to Improve Medical Laboratory Quality; Approved Guideline. 3rd ed. CLSI Document QMS24-ED3. Clinical and Laboratory Standards Institute. Wayne, PA; 2016.

4. Shahangian S, et al. Toward optimal PT use. Med Lab Observ. 2000;32(4):32-43.

5. Zaki Z, et al. Self-improvement by participant interpretation of proficiency testing data from events with 2 to 5 samples. Clin Chem. 2000;46:A70.

6. Stavelin A, Riksheim BO, Christensen NG, Sandberg S. The Importance of Reagent Lot Registration in External Quality Assurance/Proficiency Testing Schemes. Clin Chem. 2016;62(5):708-15.


Review of Method/InstrumentationWhen a specimen is not graded because there were less than 10 laboratories reporting (code 20), it could be because the laboratory is either using an older version/model, or conversely, it is also possible that the analyte/method is so new that it has not been widely adopted.


Relevant Regulatory and Accreditation Requirements This final section is a reference. For the convenience of CAP PT subscribers, we have included references from the United States CLIA and the CAP LAP that relate to PT and PT self-evaluation.

CLIA regulations may or may not apply to your laboratory, depending on your jurisdiction.

Department of Health and Human Services, Centers for Medicare and Medicaid Services. Clinical laboratory improvement amendments of 1988; final rule. Fed Register. 1992(Feb 28) [42CFR Subpart H-Participation in Proficiency Testing for Laboratories Performing Tests of Moderate Complexity (Including the Subcategory), High Complexity, or Any Combination of These Tests]

CAP Accreditation Program participants only:

CAP Checklist: All Common – COM.01000 – COM.01950

performing a self-evaluation when proficiency testing is not ......good laboratory practices, cap...

Documents