qcp start-up kit worksheets for manual...

Developing Quality Control PlansIn a Medical Laboratory

QCP Start-up KitWorksheets for Manual Applications

Developed by Westgard QC, Inc.Sponsored by

Version 1.4, February 23, 2015

Directions for use: "Print entire workbook" to produce worksheets for your application

D-Index

Directions for Use of QCP Startup Kit

1 Worksheets 1 through 5 are intended for use in identifying those tests and test systems that may require development of a risk-based "Individualized QC Plan" (IQCP).

2

Worksheet 1 should be used to review current options for complying with the CLIA QC regulations. It is recommended that the laboratory director be involved in this review, along with a manager and a laboratory scientist or analyst. The technical review (WS-2) and calculation of quality on the sigma scaled can be delated to the manager and laboratory scientist (WS-3, 4, or 5).

3

Worksheet 2 should be used to provide a technical review of a manufacturer's directions for use, claims for performance, and instructions for QC. It is particularly important to clarify whether the manufacturer requires the analysis of "two levels of controls per day" which is the CLIA default QC procedure. If so, the laboratory should focus on development of a Total QC Plan; if not, and if the laboratory does not want to add 2 levels of controls per day, the laboratory must develop an Individualized QC Plan.

4

Worksheets 3, 4, and 5 may be used to determine quality on the sigma scale. Worksheet 3 makes use of a manufacturer's performance claims. Worksheet 4 makes use of method validation data. Worksheet 5 makes use of routine QC and PT data. Sigma is the best indicator of risk! A high sigma-metric indicates a low risk testing process, whereas a low sigma-metric indicates high risk.

5

After reviewing the information in worksheets 1 through 5, you must make a decision on whether to develop a Total QC Plan that includes at least 2 levels of controls per day; if you cannot analyze at least 2 levels of controls per day, you must develop an Individualized QC Plan

6

Worksheets 6 through 12 should be used for development of a Total QC Plan. This work can be assigned to a small project group, or even to an individual who has expertise with that test and test system. Follow the order of the worksheets to implement the process and assemble a Total QC Plan.

7Worksheets 13 through 21 should be used for development of a risk-based Individualized QC Plan. This project should be assigned to a team whose members have knowledge about all phases of the Total Testing Process. Follow the order of the worksheets to implement the process and assemble an Individualized QC Plan.

Flowchart describing the steps and worksheets for developing QC plans

Determine Sigma Quality of Test System (WS 3, 4, or 5)

Develop Total QC Plan

(TQCP) (WS 6-12)

Develop Individualized QC Plan (IQCP)

(WS 13-21)

Analyze 2 levels of

controls/day?

Yes No

Review Manufacturer’s QC Instructions and Performance Claims

for Test System (WS 2)

Audit Tests and Test Systems for Compliance with CLIA QC

Requirements (WS 1)

Review Performance, Quality and Safety (QA Plan)

Make Improvements in QC Plan

Worksheet Description and PurposeAudit/Review of CLIA QC CompliancePurpose: Identify current SQC and EQC test applicationManufacturer's Claims and Instructions for Intended UsePurpose: Identify manufacturer's safety characteristics and QC requirementsAssessment of Sigma Quality from Manufacturer's Performance ClaimsPurpose: Assess sigma-quality from manufacturer's dataAssessment of Sigma Quality from MV DataPurpose: Assess sigma-quality from in-house method validation dataAssessment of Sigma Quality from SQC&PT DataPurpose: Assess sigma-quality from in-house SQC and PT dataProject Plan/Activities for Developing Total QC Plan (TQCP)Purpose: Plan and schedule for activities"Westgard Sigma Rules" Selection ToolsPurpose: Select controls rules and Ns based on sigma-metricsAudit/Review SQC PracticesPurpose: Standardize operations for SQC proceduresReview and Prioritize Available Control MechanismsPurpose: Identify implementable control mechanisms and prioritize applications.Selection of Control Mechanisms for Total QC PlanPurpose: Identify pre-analytic, analytic, and post-analytic controlsFormulation of a Total QC PlanPurpose: Summarize controls, frequency, recovery, and disclosure for TQC PlanQuality Assessment Plan for Total QC PlanPurpose: Identify quality indicators for assessment and improvementProject Plan/Activities for Developing Individualized QC Plan (IQCP)Purpose: Plan and schedule of activitiesHazards Checklist for Identification of Failure Modes that Need ControlPurpose: Review CMS list of failures to identify those that need controls in IQCPRisk Assessment of Targeted Failure ModesPurpose: FMEA table for evaluation of risk factorsRisk Mitigation ProceduresPurpose: Identification of controls to mitigate risks from targeted failure modesAssessment of Residual RisksPurpose: Assess effectiveness of risk controls and acceptability of residual riskFormulation of an Individualized QC PlanPurpose: Summarize controls, frequency, recovery, and disclosure for IQCPAssessment of Variability for Repeat Patient Test (RPT) ControlPurpose: Calculate averages and differences for duplicate test resultsControl Chart for Plotting Repeat Patient Test (RPT) ResultsPurpose: Provide a visual display of RPT resultsQuality Assessment Plan for Individualized QC PlanPurpose: Identify quality indicators for assessment and improvement

20

21

14

15

16

17

18

19

13

1

2

6

4

5

7

9

10

11

12

3

8

WS-1. Audit/Review of CLIA QC ComplianceOrganization:

Laboratory:

Director:

Manager:

Scientist/Analyst:

Default QC1 EQC2 Waived

1 - Specify # of controls/run, #/shift, or #/day2 - Specify # of controls/week or #/month

Test Test SystemQC Compliance

WS-2. Manufacturer's Claims and Instructions for Intended Use Organization

Laboratory/Location

Manager

Scientist/Analyst

Test/Test SystemManufacturer

Information for use Manufacturer's claims and instructions Source

Intended clinical use

Intended specimen types

Stated limitations

Precision Level 1 (mean/SD/%CV)



Accuracy (regression equation)

Accuracy (calculated bias at Xc1)

Accuracy (calculated bias at Xc2)

Reference range(s)

Stated interferences

Calibration (frequency)

System internal checks

System internal controls

Recommended SQC (# levels)

Recommended SQC (frequency)

Recommended maintenance

Education

Training

Analyst/operator qualifications

Safety characteristics

QC instructions

D-SigCalcDirections for Calculating Quality on Sigma-Scale

1Select WS-3 for use with manufacturer's performance claims, WS-4 for use with in-house method validation data, or WS-5 for use with in-house QC and PT data. Use of in-house data is recommended to satisfy CMS IQCP requirements.

2Define the allowable Total Error, TEa, at an appropriate medical decision concentration(s) (Xc), on basis of CLIA's criteria for acceptable performance in Proficiency Testing surveys. If not specified by CLIA, consider other sources and recommendations in the literature, then document the appropriate reference.

3

Find the precision data, usually a replication experiment in the manufacturer's claims or laboratory's method validation study or a laboratory's routine QC data. Extract the results for the number of measurements, observed mean, observed SD, and calcluated %CV (SD*100/mean)

4

Find the accuracy or bias data, using a comparison of methods experiment in the manufacturer's claims or laboratory's method validation study, or the results of PT/EQA samples in a survey program. If data for estimating bias is not available, calculate sigma assuming bias is zero.

4aThe statistical results from comparison studies are often presented in the form of a regression equation, Y = a + bX, where a is the y-intercept and b is the slope of the regression line.

4b Calculate bias as Yc-Xc, where Xc is a critical decision concentration. Express bias in the same units as TEa, either concentration units or percentage [Bias*100/Xc].

5Calculate the sigma-metric, making sure all terms are in the same units, either concentration units or percentages.

6Select the sigma-metric for the most critical Xc or the average sigma for the relevant Xc's, whichever is most appropriate for the clinical intended use of the test.

7Assess risk based on the observed sigma-metric. Low risk for sigma ≥ 5.5, moderate risk for sigma between 5.5 and 3.5, and high risk for sigma ≤ 3.5.

WS-3. Assessment of Sigma Quality from Manufacturer's ClaimsOrganization:

Laboratory:

Manager/Analyst:

Test (Units)

Test System/Device

Source of information

LOW Xc MID Xc High Xc

TEa at Xc %TEa

Source/Reference

SD

Mean

%CVProtocol

Regression Eq.

Bias @ Xc

%BiasComparative method/material

Protocol

(%TEa)/%CV

(%TEa - %Bias)/%CVCritical and/or Average Sigma Metric

Assessment of Risk (High, Med, Low)

Sigma-Metric

Quality Required for Intended Use Med Dec Conc (Xc)

Manufacturer's claims for precision from replication experiments

Manufacturer's claims for accuracy from comparison of methods data

WS-4. Assessment of Sigma Quality from Method Validation DataOrganization:

Laboratory:

Manager/Analyst:

Test (Units)

Test System/Device

Supplier of Reagents

Time Period of Validation Studies


TEa at Xc %TEa

Source/Reference

SD

Mean

%CVProtocol

Regression Eq.

Bias @ Xc

%BiasComparative method/material

Protocol

(%TEa)/%CV

(%TEa - %Bias)/%CVCritical and/or Average Sigma Metric

Assessment of Risk (High, Med, Low)

Precision Replication Exp.

Sigma-Metric


Comparison of Methods/Materials

WS-5. Assessment of Sigma Quality from QC and PT(EQA) DataOrganization:

Laboratory:

Manager/Analyst:

Test (Units)

Test System/Device


Supplier of Control Materials


TEa at Xc %TEa

Source/Reference

# Measurements

Mean

SD

%CVSource/Reference Precision Data

Target Value

Lab Result

Difference (L-T)

Bias

%Bias

Average Abs %BiasSource PT/EQA Data

(%TEa)/%CV

(%TEa - %Bias)/%CVCritical and/or Average Sigma-Metric

Assessment of Risk (Hi, Med, Lo)

Sigma-Metric

PT, EQA, or Peer Data


QC Data

D-TQCPDirections for Developing a Total QC Plan

1Identify project test/test system and organize an appropriate project group. For a TQC Plan, the project group can be small, even a single individual who is knowledgeable about the Total Testing Process. Review WS-6 to organize your activities.

2 Review the manufacturer's instructions for QC and identify control mechanisms that are built into the test system. See WS-2.

3 Review the determination of quality on the sigma-scale, or perform those calculations to get started. See WS-3, 4, or 5.

4 Right-size the Statistical QC procedure by selecting appropriate SQC rules and the total Number of control measurements to detect medically important errors. Use the "Westgard Sigma Rules" graphic tools in WS-7.

5Audit the laboratory SQC practices to review and standardize current practices. WS-8

6 Review and prioritize other control mechanisms that can readily be implemented in the laboratory. See WS-9.

7 Select pre-analytic, analytic, and post-analytic controls to monitor possible failure-modes. See WS-10.

8 Assemble the identified controls and organize into a Total QC Plan. WS-11

9 Prepare a Quality Assurance Plan to monitor the TQC Plan. WS-12

10 Document your Total QC Plan in a format that is useful in your laboratory.

Flow Chart for Developing A Total QC Plan

Organize Project Team to Develop Total QC Plan (WS-6)

Determine Sigma Quality of Test/Test System (WS-3, 4, or 5)

Right-size SQC Procedure Using Westgard Sigma Rules (WS-7)

Review Available Control Mechanisms (WS-9)

Add Pre-analytic and Post-analytic Controls (WS-10)

Formulate/Assemble a Total QC Plan (WS-11)

Prepare QA Plan to Monitor TQCP (WS-12)

Review Manufacturer’s QC Instructions (WS-2)

Audit and Standardize SQC Practices (WS-8)

Document Total QC Plan

WS-6. Project Plan/Activities for Developing Total QC PlanOrganization:

Laboratory/Location: Manager:

Test/Test System

Project Team Leader

Team Members

Schedule Activity for Developing Total QC Plan WS Done

Organizational meeting 1,6

Review manufacturer's instructions for QC 2

Determine sigma-quality 3,4,5

Confirm decision to develop Total QC Plan

Right-size SQC procedure 7

Audit/review/standardize SQC practices 8

Review and prioritize available control mechanisms 9

Select additional analytic controls 10

Select pre-analytic and post-analytic controls 10

Formulate Total QC Plan 11

Document SOP for Total QC Plan

Formulate Quality Assessment Plan 12

WS-7. "Westgard Sigma Rules" Selection Tools

Organization/Laboratory/Location: Manager/Analyst/Project Group (Date):

Test/Test SystemAverage or Critical SigmaWestgard Sigma Rules selectedNumber control measurements/RunNumber of runs/Shift or Day

Data QC

13s 22s R4s 41s 8X

Take Corrective Action

Report Results

No

Sigma Scale 6σ 5σ 4σ 3σ

No No No

Yes Yes Yes Yes Yes

N=2 R=1

N=2 R=1

N=4 R=1

N=2 R=2

N=2 R=4

N=4 R=2

No

Data QC

13s 2of3

2s R4s 31s 6X

Take Corrective Action

Report Results

No

Sigma Scale 6σ 5σ 4σ 3σ

No No No

Yes Yes Yes Yes Yes

N=3 R=1

N=3 R=1

N=3 R=1

N=3 R=2

N=6 R=1

No

2 Level

3 Level

WS-8. Audit/Review SQC Practices

Organization:

Laboratory:

Manager/Analyst (Date):

Test (Units)

Test System/Device


Supplier of Control Materials

Number of Control Materials Low Mid High

# Measurements

Mean

SD

%CV

Practice for Initial Calc of Mean&SD

Initial Control Rules Selected

Initial Total N Selected

Practice for Updating QC Limits

Number Patients/Run or Day

Service Priority/Turnaround Time

Mode of Operation

Location of Controls

Frequency of Controls

Response to 1st Out-of-Control

Response to 2nd Out-of-Control

Continuous Batch Unit Use Other

Events requiring SQC

Control Chart Parameters

WS-9. Review and Prioritize Available Control MechanismsOrganization:

Laboratory: Manager/Analyst (Date):

Physician test orderPatient identificationSpecimen labelingSpecimen processingSample inspection

Standard Operating ProcedureOperator trainingOperator checklistsSystem maintenanceOperator competency

Reagent storage & expirationSample acceptabilityElectronic checksFunction testsProcess testsCalibration checksStatistical QCTrueness controlsProficiency testing

Limit checksImplausible valuesDelta checksCorrelation algorithmsPatient population algorithms

Review test resultsConfirm/call critical valuesInterpretive and safety info.Turnaround TimeCustomer feedback

Analytic test system controls

Analytic test review controls

Post-analytic controls

Control Mechanisms Existing Policies & Procedures

Source Document Priority for Use

Pre-Analytic controls

Analytic operator controls

WS-10. Selection of Control Mechanisms for Total QC PlanOrganization

Laboratory

Manager/Analyst (Date)

Test (Units)

Test System/Device

Factors to consider Control Mechanism

Specimen requirements

Operator performance

Sample acceptability

Analytic variability Statistical QC

System & environment conds.

Turnaround Time

Pre-analytic controls



Possible Failure

System changes/instability



WS-11. Formulation/Assembly of a Total QC PlanOrganization:

Laboratory:


Test (Units)

Test System/Device

Control Mechanisms Frequency Recovery Disclosure


Pre-analytic controls




WS-12. Quality Assessment Plan for Monitoring Total QCPOrganization:

Laboratory/Location:Manager/Project Group (Date):

Test/Test System:Quality Indicators Number

Lost/misplaced/redrawnIncorrect ID/LabelsIncorrect specimen typeInadequate samples

Test system warningsRuns rejected/repeatedReagent failuresHardware failuresOperator failures

Control Days N Mean SD %CVLow levelMid levelHigh level

ActionSample Result Target Diff Bias %Bias

12345

Action

TAT Goal N Average Low High 95%

Customer complaints

Recommended ActionsPre-analytic: Specimens and Samples

Analytic: Test System Failures

Bias from Proficiency Testing, External Qualilty Assessment, or Peer

Comparison data

Analytic: Performance

Post-analytic: Test reports

Precision from SQC data

External sources

Turn-Around-Time (TAT)

D-IQCPDirections for Developing an Individualized QC Plan

1Organize project team to develop Individualized QC Plan. This team should include members from the pre-analytic, analytic, and post-analytic parts of the Total Testing Process. See WS-13.

2 Review the manufacturer's QC instructions to identify recommended controls, confirm there is no requirement for at least two levels of external controls per day, and confirm your decision to develop an IQCP. Review WS-2.

3Determine Sigma-metric as the primary indicator of risk based on in-house data. Use WS-4 or 5.

4 Assess hazards following CMS checklist to identify failure-modes that may be important. Use WS-14.

5 Evaluate risk of targeted failure-modes using WS-15. Rate risk factors for severity (SEV), occurance (OCC), and detection (DET) on a scale of 0, 1, or 2, where 0 represent low, 1 moderate, and 2 high. Calculate risk from the equation Risk = SEV*OCC*(2-DET). Evaluate risks of 2 and higher as needing mitigation. See WS 15. You may want to set up this form as an electronic worksheet.

6Identify controls to mitigate risks of 2 and higher. Prioritize the failure-modes on basis of highest risk. Use WS-16 to show priorities and identify mitigation procedures.

7 Assess the acceptability of residual risks by re-evaluating risk with identified mitigation procedures. Use WS-17 to recalculate risk. This WS requires signatures of responsible parties to document acceptability of risks.

8 Assemble all the risk controls and formulate an Individualized QC Plan. WS-18

9 If Repeat Patient Test (RPT) is included as a control, it is important to perform an initial study to characterize the variability that is expected and to construct a control chart for displaying the RPT results. See WS-19 and WS-20.

10 Prepare a QA plan to monitor your IQCP. See WS-21.

11 Document your IQCP in a format that is useful and practical for your laboratory.

Organize Project Team to Develop Individualized QC Plan (WS-13)

Determine Sigma Quality of Test/Test System (WS-3,4, or 5)

Assess Hazards to Identify Failure Modes (WS-14)

Evaluate Risk of Targeted Failure Modes (WS-15)

Identify Controls to Mitigate Risk (WS-16)

Formulate an Individualized QC Plan (WS-18)

Study use of Repeat Patient Tests (WS-19,20)

Assess Acceptability of Residual Risk (WS-17)

Review Manufacturer’s QC Instructions (WS-2)

Document Individualized QC Plan

Prepare QA Plan to Monitor IQCP (WS-21)

Flowchart for Developing An Individual QC Plan

WS-13. Project Plan/Activities for Developing Individualized QC Plan (IQCP)Organization:

Laboratory/Location: Manager:

Test/Test System:

Project Team Leader:

Team Members:

Schedule Activity for Developing Total QC Plan WS Assignments Done

Organizational meeting 1, 13

Review manufacturer's instructions for QC 2

Determine sigma-quality 3,4,5

Confirm decision to implement IQCP

Identify failure modes from Hazards Checklist 14

Assess risk for targeted failure modes 15

Identify risk mitigation procedures 16

Assess residual risks 17

Formulate Individualized QC Plan 18

Assess variability for Repeat Patient Tests 19

Develop control chart for RPTs 20Formulate Quality Assessment Plan 21

WS-14. Hazards Checklist for Identification of Failure Modes that Need ControlOrganization/Laboratory:

Manager/Project Group (Date): Test/Test System:

Patient preparation

Collection

Labeling

Storage, preservation, stability

Transportation

Acceptability and rejection

Referral (to other labs)

Temperature

Airflow/ventilation

Light intensity

Noise and vibration

Humidity

Altitude

Dust

Water

Utilities (elect stability)

Adequate space

Reagents

Quality control materials

Calibrators

Shipping/receiving

Storage condition requirements

Expiration data

Preparation

Intended clinical use

Serious? (Y/N/Maybe)

Step/Procedure/Activity in the Total Testing Process

What might cause a problem? (Failure Mode)

Frequent? (Y/N/Maybe)

Need Control? (Y/N/Maybe)

Pre-analytic: Specimen

Analytic: Environment

Analytic: Reagent

Analytic: Test System








Laboratory test volume

Instrument function tests

Maintenance checks

Inadequate sampling

Clot detection

Interference detection

Hemolysis

Lipemia

Icterus

Turbidity

Calibration

Mechanical/electronic failures

Optics

Pipettes, pipettors

Barcode readers

System controls & function checks

Procedural, electronic controls

Liquid controls

Temperature controls

Software/hardware

Data transmission to LIS

Result reporting

Training

Competency

Education, experience qualification

Adequate staffing

Report delivery

Post-analytic

Analytic: Testing Personnel








Turnaround Time

Critical values

WS-15. Risk Assessment of Targeted Failure ModesOrganization/Laboratory:


Testing Personnel

DET 0-2

RISK SEV*OCC*

(2-DET)

Specimen

Environment

Reagent

Test System


Targeted Failure Mode

EffectSEV 0-2

CauseOCC 0-2

Existing Controls

WS-16. Risk Mitigation Procedures

Organization/Laboratory:

Manager/Project Group (Date):

Test/Test System:

Prioritized Failure Modes Risk Mitigation Procedures

WS-17. Assessment of Residual Risks

Organization/Laboratory:


Test/Test System:

Prioritized Failure Mode: Risk Control Procedure SEV 0-2 OCC 0-2 DET 0-2Residual Risk

SEV*OCC*(2-DET)

IQCP Recommendations

Manager Review

Laboratory Director Review

WS-18. Formulation of an Individualized QC PlanOrganization/Laboratory:



Control Mechanisms Frequency/Events Recovery Disclosure

Testing Personnel

Reagent

Test System

Specimen

Environment

WS-19. Assessment of Variability from Repeat Patient Test (RPT) ControlsOrganization/Laboratory:

Manager/Supervisor:

Test/Test System/Device ID:

Patient/Sample ID Analyst/Operator Date & Time 1st Result 2nd Result Av. Result Difference

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

Averages

WS-20. Chart for Plotting Results from Repeat Patient Test (RPT) Controls

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25Day or Run Number

Operator Initials

Organization: Laboratory/Location:

Test/Test System:

Diffe

renc

e

WS-21. Quality Assessment Plan for Monitoring Individualized QCPOrganization:

Laboratory/Location:


Quality Indicators Number Recommended Actions

Lost/misplaced/redrawn

Incorrect ID/Labels

Inadequate samples

Test system failures

Runs rejected/repeated

Reagent failures

Operator failures

Bias from Proficiency Testing data

Precision from Repeat Patient Tests

Average TAT

95% limit TAT

Customer complaints

Analytic: Performance

Post-analytic: Test reports

Pre-analytic: Specimens and Samples

Analytic: Test System Failures

Appendix 1. Characteristics of QC Tools

Recommended QC Tool Control Objective Coverage Detection

Standard Operating Procedure Process for safe use Multiple runs UnknownOperator training Correct performance Total testing process UnknownOperator checklists Proper operation Single runs UnknownSystem maintenance Proper operation Multiple runs UnknownOperator competency Correct performance Total testing process Unknown

Electronic checks Analyzer components Single runs UnknownFunction tests Analyzer components Single runs UnknownProcess tests Process steps Single samples UnknownCalibration checks Analyzer stability Single runs UnknownIntegrated controls Analyzer stability Single runs Unknown

Statistical QC Method stability Single runs KnownSQC with peer comparison Method stability Multiple runs UnknownPeriodic EQA, PT Method accuracy Multiple runs UnknownTrueness controls Method accuracy Multiple runs Unknown

Implausible values Random errors Single patients UnknownDelta checks Random errors Single patients KnowableCorrelation algorithms Random errors Single patients KnowableRepeat patient testing Short-term stability Single runs KnowablePopulation statistics Long-term stability Single runs Knowable

Analyst/operator controls

Built-in analyzer controls

Stable control materials

Patient data analysis

Appendix 2. Application and Feasibility of QC Tools

Recommended QC Tool Practicality Reliability Effort

Standard Operating Procedure Man/computer Low ModerateOperator training Manual Low ModerateOperator checklists Man/computer Moderate LowSystem maintenance Manual Low LowOperator competency Man/computer Low Moderate

Electronic checks Man/computer High LowFunction tests Man/computer High LowProcess tests Computer High LowCalibration checks Man/computer High LowIntegrated controls Computer High Low

Statistical QC Man/computer High ModerateSQC with peer comparison Computer Moderate ModeratePeriodic EQA, PT Manual Moderate LowTrueness controls Man/computer Low Low

Implausible values Man/computer Moderate High/LowDelta checks Computer Low ModerateCorrelation algorithms Computer Low ModerateRepeat patient testing Man/computer Low ModeratePopulation statistics Computer Moderate High

Analyst/operator controls

Built-in analyzer controls

Stable control materials

Patient data analysis

qcp start-up kit worksheets for manual...

Documents