Phil Smith

Sales & Marketing Director

Quality Management Ltd.

Quality Management Ltd.

Manchester – North of EnglandManchester – North of England Established 1991Established 1991 30 Employees30 Employees Nearly 20,000 samples per annumNearly 20,000 samples per annum 2400 Participating Laboratories2400 Participating Laboratories 53 Countries53 Countries 7 Separate PT Schemes7 Separate PT Schemes

SUBJECTSUBJECT

Proficiency Testing Proficiency Testing (External Quality Control)(External Quality Control)

a Vital Part of a Vital Part of

Laboratory Quality SystemsLaboratory Quality Systems

OVERVIEW of PRESENTATIONOVERVIEW of PRESENTATION

Why a laboratory should participateWhy a laboratory should participate

What is Proficiency testingWhat is Proficiency testing

The essentials of a PT SchemeThe essentials of a PT Scheme

How does it workHow does it work

Accreditation of a PT SchemeAccreditation of a PT Scheme

Relationship with AccreditationRelationship with Accreditation

WHAT IS PROFICIENCY TESTING?WHAT IS PROFICIENCY TESTING?

Proficiency Testing (PT) involves the regular distribution Proficiency Testing (PT) involves the regular distribution of stable homogenous test materials to participants for of stable homogenous test materials to participants for analysis for defined determinands. Participants results of analysis for defined determinands. Participants results of analysis are then compared with those of the other analysis are then compared with those of the other participants.participants.

Proficiency Testing provides an independent means of Proficiency Testing provides an independent means of testing and comparing individual laboratory/technician testing and comparing individual laboratory/technician results of analysis with a peer group.results of analysis with a peer group.

USES OF PROFICIENCY TESTING USES OF PROFICIENCY TESTING

Monitoring PerformanceMonitoring Performance Provides Peer ComparisonProvides Peer Comparison Identifies ProblemsIdentifies Problems Encourages ImprovementEncourages Improvement VerificationVerification

TrainingTraining AccreditationAccreditation Customer DemandsCustomer Demands

HOW DOES IT WORK

TYPICAL ROUND CYCLETYPICAL ROUND CYCLE

Assigned Assigned ValuesValues

FRIFRI

Results Results deadlinedeadline

FRIFRI

Report Report issuedissued

Final test Final test date*date*

Despatch Despatch datedate

FRIFRIWEDSWEDSTUESTUES

Week 5Week 5Week 4Week 4Week 3Week 3Week 2Week 2Week 1Week 1

* Recommended* Recommended

FaxFax

Post Post

E-mailE-mail

Paper reportPaper report

WebsiteWebsite

QMS Reports

QMS Reports

QMS Reports

ILAC* G13: 2000 ILAC* G13: 2000

Guidelines for the Requirements for the Competence of Providers of Guidelines for the Requirements for the Competence of Providers of Proficiency SchemesProficiency Schemes*ILAC - International Laboratory Accreditation Co-operation*ILAC - International Laboratory Accreditation Co-operation

THE ESSENTIALS OF A PT SCHEMETHE ESSENTIALS OF A PT SCHEME

ISO IEC Guide 43 – 1ISO IEC Guide 43 – 1

Proficiency Testing by Interlaboratory ComparisonsProficiency Testing by Interlaboratory Comparisons

Part 1: Development and Operation of Proficiency Testing SchemesPart 1: Development and Operation of Proficiency Testing Schemes

ACCREDITATION OF PT PROVIDERSACCREDITATION OF PT PROVIDERS

ILAC G13 Guidelines

ISO 9001 (1994)

ISO 43-1 (1997)

ISO 17025 (2000)

Management system requirements Quality Management System Organisation & management Document & information control Contract & tender review Use of collaborators (sub-contractors) Procurement of services & supplies Client feedback Control of non-conforming material Corrective action Preventive action Records Internal audits Management reviews Technical requirements Management, staffing & training Collaborators (sub-contractors) Organisation & scheme design Choice of method or procedure Conduct of proficiency testing scheme Data analysis & interpretation Communication with participants Confidentiality Collusion & falsification of results Others Measurement, analysis & improvement Statistical design Homogeneity & stability of samples Technical competence

KEY CRITERIA KEY CRITERIA

Test Materials must:-Test Materials must:-

Contain the intended micro-organisms at the Contain the intended micro-organisms at the expected levelsexpected levels

Be sufficiently Homogenous for purposeBe sufficiently Homogenous for purpose

Be stable for at least the period of the PT roundBe stable for at least the period of the PT round

Statistics used to assess performance must:-Statistics used to assess performance must:-

Be easy to understand and interpretBe easy to understand and interpret

Enable direct comparison with other participantsEnable direct comparison with other participants

STRAIN CHARACTERISTICSSTRAIN CHARACTERISTICS

Source from recognised culture collectionsSource from recognised culture collections

Method of Preservation must maintain characteristics Method of Preservation must maintain characteristics

Verification after preservation Verification after preservation

Appropriate to target sector e.g. analysis of food Appropriate to target sector e.g. analysis of food

STRAIN CHARACTERISTICSSTRAIN CHARACTERISTICS

ATCC – ATCC – American Type Culture CollectionAmerican Type Culture Collection

NCTC – NCTC – National Collection of Type CulturesNational Collection of Type Cultures

NCIMB – NCIMB – National Collection of Industrial & Marine BacteriaNational Collection of Industrial & Marine Bacteria

Use of Wild Strains - verification Use of Wild Strains - verification

How QM assures the Authenticity of the strains usedHow QM assures the Authenticity of the strains used

Lyophilised (Freeze dried) to maintain characteristicsLyophilised (Freeze dried) to maintain characteristics

Morphological & biochemical tests after preservation Morphological & biochemical tests after preservation

STABILITYSTABILITY

ILAC G13 – states that samples must be stable for at least the ILAC G13 – states that samples must be stable for at least the conduct of the round. conduct of the round.

e.g. Culture profile

Staphylococcus aureus Gram +ve cocciCatalase and coagulase +ve

Stability tests Mixed with Skimmed milk

powder/Oatmeal and stored at

5, 25, 30,370C

Enumerate Check Purity

Freeze-dry to produce initial stock

(high number of micro-organisms in a powder format)

Resuscitation on agar medium

Storage on cryogenic beads

Micro-organisms obtained from recognised culture collection

Development tests before inclusion in QM Culture collection Development tests before inclusion in QM Culture collection

STABILITYSTABILITY

Stability of organism in skimmed milk powder

3

3.5

4

4.5

5

5.5

Initialcount

Week 1 Week 2 Week 12

Refrigerator

30°C

37°C

Room temperature

HOMOGENEITYHOMOGENEITY

Recognises that complete homogeneity cannot be achieved Recognises that complete homogeneity cannot be achieved even in a liquid sample.even in a liquid sample.

Samples taken from batch at random and tested in duplicate Samples taken from batch at random and tested in duplicate for all target organisms.for all target organisms.

International Harmonised Protocol for the Proficiency Testing of International Harmonised Protocol for the Proficiency Testing of (chemical) Analytical laboratories states that the standard (chemical) Analytical laboratories states that the standard deviation (SD) for homogeneity results must be deviation (SD) for homogeneity results must be 0.3 times the 0.3 times the acceptable range for participants.acceptable range for participants.

e.g. If Target SD is 0.3 then homogeneity SD e.g. If Target SD is 0.3 then homogeneity SD <0.09<0.09

ILAC G13 – states that the Provider shall be able to demonstrate ILAC G13 – states that the Provider shall be able to demonstrate that the test material is sufficiently homogenous for that the test material is sufficiently homogenous for the particular proficiency testing scheme. the particular proficiency testing scheme.

PRODUCTION OF TEST MATERIALS - SAMPLESPRODUCTION OF TEST MATERIALS - SAMPLES

Samples returned to QM and Stability tested

End of Life Stability tests performed after

3 months storage

Samples tested by participants

Despatch Samples to participantsOne Set posted to return to QM

Pre-despatch Quality control tests

Homogeneity, Weight

ProductionSamples are weighed (10g or 25g)

Mix working stock with SMP or OM to produce bulk stock

Check working stock

Enumerate, Profile, Growth and biochemical reactions

Dilute initial stock with irradiated SMP to produce working stock

ASSESSING PERFORMANCEASSESSING PERFORMANCE

Where: - Where: - XX = mean = mean

= standard deviation or variation around the mean = standard deviation or variation around the mean

Fre

qu

ency

99.7599.75%%

33 22 33 22 XX

68%68%

95%95%

Diagram of a Normal DistributionDiagram of a Normal Distribution

ASSESSING PERFORMANCE – Setting the Assigned ValueASSESSING PERFORMANCE – Setting the Assigned Value

Determining the Assigned ValueDetermining the Assigned Value

By FormulationBy Formulation – the assigned value is determined from knowledge of the test item – the assigned value is determined from knowledge of the test item formulation e.g preparing a solution of 1M hydrochloric acid.formulation e.g preparing a solution of 1M hydrochloric acid.

Use of certified reference material (CRM)Use of certified reference material (CRM) – the certified reference value will be used as the – the certified reference value will be used as the assigned value.assigned value.

Use of reference materialUse of reference material – a cheaper alternative to the use of certified reference materials, – a cheaper alternative to the use of certified reference materials, where a test-material is used that has been calibrated against a certified reference material. where a test-material is used that has been calibrated against a certified reference material.

Consensus values from expert laboratoriesConsensus values from expert laboratories – the expert laboratories should have – the expert laboratories should have demonstrable competence in the determination of the analyte(s) under tests, using validated demonstrable competence in the determination of the analyte(s) under tests, using validated methods.methods.

Consensus values from participantsConsensus values from participants – participant results are used to determine – participant results are used to determine the assigned value, with consideration of the possible effects of extreme values.the assigned value, with consideration of the possible effects of extreme values.

AA

BB

CC

DD

EE

Assigned Assigned value value

Sub-samples containing the target micro-organism areSub-samples containing the target micro-organism are

distributed to at least 20 laboratories with instructions for testingdistributed to at least 20 laboratories with instructions for testing

ASSESSING PERFORMANCE – Setting the Assigned ValueASSESSING PERFORMANCE – Setting the Assigned Value

Easy to understand and interpretEasy to understand and interpret

Easily summarised in tabular or graphical formEasily summarised in tabular or graphical form

Widely used and acceptedWidely used and accepted

Enable results to be compared over timeEnable results to be compared over time

Enable direct comparison with other participants Enable direct comparison with other participants

ASSESSING PERFORMANCE – Performance ScoringASSESSING PERFORMANCE – Performance Scoring

QM applies the Z-Score system QM applies the Z-Score system Z = Z = (x – X(x – Xaa))

tt

Where: - x = the result reported by a participating Where: - x = the result reported by a participating laboratorylaboratory

Xa = the assigned valueXa = the assigned value

tt = a target standard deviationa target standard deviation When a data set conforms to a standardised normal distribution When a data set conforms to a standardised normal distribution

we can expect 95% of its values to lie within 2 SDs 99.7% of its we can expect 95% of its values to lie within 2 SDs 99.7% of its values to lie within 3 SDs from the mean. values to lie within 3 SDs from the mean.

Results with a Z-Score over 2 are therefore considered questionable Results with a Z-Score over 2 are therefore considered questionable because we would expect only 5% of correct measurements to be because we would expect only 5% of correct measurements to be so different to the assigned value.so different to the assigned value.

Results with a Z-Score over 3 are considered unsatisfactory Results with a Z-Score over 3 are considered unsatisfactory because we would expect only 0.3% of correct measurements to be because we would expect only 0.3% of correct measurements to be so different to the assigned value.so different to the assigned value.

ASSESSING PERFORMANCE – Performance ScoringASSESSING PERFORMANCE – Performance Scoring

OTHER IMPORTANT CRITERIAOTHER IMPORTANT CRITERIA

How many laboratories are participatingHow many laboratories are participating

Does the Scheme allow the user to choose their test regimeDoes the Scheme allow the user to choose their test regime

Does the Provider have the required resources and financeDoes the Provider have the required resources and finance

How important is PT to the ProviderHow important is PT to the Provider

Is the Scheme AccreditedIs the Scheme Accredited

QM’s ACCREDITATION STATUSQM’s ACCREDITATION STATUS

• Based on ISO Guide 43-1:1997 and on the relevant elements of ISO 17025: 2000, applicable to the characterisation, homogeneity and stability testing of proficiency testing test materials.

• Additionally, relevant elements of ISO 9000:1994 are included to eliminate the need for separate recognition of compliance with ISO 9000.

Accredited by UKAS Accredited by UKAS (United Kingdom Accreditation Service)(United Kingdom Accreditation Service)

QM is accredited as a Proficiency Testing ProviderQM is accredited as a Proficiency Testing Provider

Certificated to ISO 9001 - 2000Certificated to ISO 9001 - 2000

WHY A LABORATORY SHOULD PARTICIPATEWHY A LABORATORY SHOULD PARTICIPATE

Because you want to – pursuit of excellenceBecause you want to – pursuit of excellence

Provides independent assessment of the quality of a Provides independent assessment of the quality of a laboratory’s performance in its routine work laboratory’s performance in its routine work

Helps identifies potential problemsHelps identifies potential problems

Identifies competent laboratories for commercial purposes Identifies competent laboratories for commercial purposes

Mandatory – Directives: - Mandatory – Directives: - 89/397/EEC Official Food Control 89/397/EEC Official Food Control

LaboratoriesLaboratories 93/99/EEC Additional Measures for Food 93/99/EEC Additional Measures for Food

ControlControl

Supports accreditation Supports accreditation

Training, validating methods, internal Q C assessments Training, validating methods, internal Q C assessments

RELATIONSHIP WITH ACCREDITATIONRELATIONSHIP WITH ACCREDITATION

The laboratory shall have quality control procedures for monitoring the validity The laboratory shall have quality control procedures for monitoring the validity of tests and calibrations undertaken. The resulting data shall be recorded in of tests and calibrations undertaken. The resulting data shall be recorded in such a way that trends are detectable and, where practicable, statistical such a way that trends are detectable and, where practicable, statistical techniques shall be applied to the reviewing of the results. This monitoring techniques shall be applied to the reviewing of the results. This monitoring shall be planned and reviewed and may include, but not be limited to, the shall be planned and reviewed and may include, but not be limited to, the following.following.

(a) (a) Regular use of certified reference materials and/or internal quality Regular use of certified reference materials and/or internal quality control using secondary reference materials;control using secondary reference materials;

(b)(b) Participation in interlaboratory comparisons or proficiency testing Participation in interlaboratory comparisons or proficiency testing programmes; programmes;

EN ISO/IEC 17025 EN ISO/IEC 17025 5.9 Assuring the quality of tests and calibration results5.9 Assuring the quality of tests and calibration results

It contains all of the requirements that testing and calibration It contains all of the requirements that testing and calibration laboratory have to meet if they wish to demonstrate that they laboratory have to meet if they wish to demonstrate that they operate a quality system, are technically competent, and able operate a quality system, are technically competent, and able to generate technically valid results. to generate technically valid results.

RELATIONSHIP WITH ACCREDITATIONRELATIONSHIP WITH ACCREDITATION

Introduction to EN ISO/IEC 17025 says: -Introduction to EN ISO/IEC 17025 says: -

Compliance with EN ISO/IEC 17025 sets the conditions that should Compliance with EN ISO/IEC 17025 sets the conditions that should

prepare the laboratory to generate technically valid results prepare the laboratory to generate technically valid results

Compliance with EN ISO/IEC 17025 does not measure the accuracyCompliance with EN ISO/IEC 17025 does not measure the accuracy

of the results produced by a laboratory of the results produced by a laboratory

Participation in a well run Proficiency Testing Scheme meets this need Participation in a well run Proficiency Testing Scheme meets this need

QM CHEMISTRY PT SCHEMES

Quality in Dairy ChemistryQuality in Dairy Chemistry

Quality in Meat ChemistryQuality in Meat Chemistry

OTHER QM MICRO PT SCHEMES

Quality in ChocolateQuality in Chocolate

Quality in BeveragesQuality in Beverages

QWAS - Quality in Water

QWAS growth

0

200

400

600

800

1000

1200

1400

1600

1800

2000

2000 2001 2002 2003

Year

Sal

es

2004!!

CS – Calibration Standards

Calibration Standards Calibration Standards

Analytical Quality

Assurance Controls

Clean Water - Organics - Inorganics

Waste Water - Organics

- Inorganics

Soils and Sediments

Sludge and Sewage

AQUACHECK

Electronic results submission and reporting

PT for Water and Environmental ChemistryPT for Water and Environmental Chemistry