Decision Rules and Risk ISO/IEC 17025:2017

Workshop

T&M 2019 Conference and Workshop

16th September 2019

Steve Sidney and John Wilson

1

Introduction

This presentation is presented in conjunction with the presentation by John Wilson who will cover the Decision Rule and Statements of Conformity now incorporated in the ISO/IEC 17025:2017, as well as a summary of what is in the New ILAC G8:2019 guidance document.

This first section will deal with Risk from a generic point of view, explain how to go about assessing Risk and then more specifically Risk in the lab context as well as the requirements of ISO/IEC 17025:2017. 2

What is risk ?

Uncertainty

what makes achieving an objective uncertain

Level of Risk

takes into account consequences and likelihood of situations

3

What is risk ?

ISO 17666:2003 – risk

“undesirable situation or circumstance that has both a likelihood of occurring and a potentially negative consequence (impact) ……..”

4

5

Likelihood of being bitten

Impact of being bitten

Another view

Outcome (Negative) – uncertain – risk

Outcome (Positive) – uncertain – opportunity

6

Four phases of risk

7

Risk Analysis

* Intended user identification* Area wise risk identification* Risk estimation

Risk Evaluation

* Risk acceptability decisions

Risk Monitoring and Control

* Option analysis* Implementation of measures* RESIDUAL RISK evaluation* Overall RISK acceptance

Post test operation information

* Post-production experience* Review of Risk management experrience – customer view* Take appropriate action

Four Phases of Risk

Risk and Opportunity

Risk identification

What can happen and why ?

What are the consequences ?

What is the probability of it occurring ?

8

Rating the evaluation – Impact

Low (1) – easy to correct

Moderate (2) – errors occurring again but clear

High (3) – serious errors with possible irreparable consequences

9

Rating the evaluation – Probability

Low – very rare (1)

Medium – rare (2)

High - frequently (3)

10

Effect of uncertainty (risk)

Risk = Impact * Probability

or

Risk = Consequences * Likelihood

11

Scaling – Version 1

12

Evaluation – 1

Lowest (green) – acceptable

Highest (red) – requires action

Medium (yellow) – decide if still acceptable or what action/s to take

13

Scaling – Version – 2

14

Mitigation

Factors that mitigate the consequence of the risk

or

Reduce the probability of the risk

15

In ISO/IEC 17025 is the concept new ?

…NOTE 2: Apart from the review of the operational procedures, the preventive action might involve analysis of data, including trend and risk analyses and proficiency-testing results………….

The procedure for corrective action shall start with an investigation to determine the root cause(s) ……….. Corrective actions shall be to a degree appropriate to the magnitude and the risk of the problem.

16

ISO/IEC 17025:2005Clauses: 4.11, 4.12

ISO/IEC 17025:2005 vs 2017

1999/2005: Managed Risk2017: Risk and

Opportunity Managed

Quality Manual Documented Information

Policies Processes

Procedures Decision Rules

Job Descriptions

Top Management

QM & TM17

2017 vs 2005

17025:2005 17025:2017

Lab shall have policies and procedures to ensure protection of confidential information…incl electronic storage and transmission of results

The Lab shall ensure the protection of confidential information.. including electronic storage and transmission of results

18

19

Foreword

the risk-based thinking applied in this edition has enabled some reduction in prescriptive requirements and their replacement by performance-based requirements

20

Introduction

… document requires the laboratory to plan and implement actions to address risks and opportunities.Addressing both risks and opportunities establishes a basis for increasing the effectiveness of themanagement system, achieving improved results and preventing negative effects. The laboratory isresponsible for deciding which risks and opportunities need to be addressed.

21

Dealing with risk

Relaxation of prescription makes it essential for each lab to consider the risk for each clause

• Discuss and agree what measures are required

• Implement in a known and controlled way for consistency

• Review22

The general case

Risk based approach is where

Breadth and depth of implementation of clause is varied to suit perceived risk for the particular laboratory

23

Risk in the context of measurement

uncertainty on meeting the objectives of the measurement

a) deviation from the expected measurement result

b) outside the calibration and measurement capability (CMC), testing capabilities or the stated uncertainties

24

Laboratory activities

25

Technical

Safety

Financial

Supply chain

Impartiality

SecurityCustomer

Management

External

Internal

Reliability

Environmental

4.1.4 & 4.1.5 (Impartiality)

Evaluate an on-going basis… include those risks that arise from its activities, ..its relationships, or relationships of its personnel.

If a risk to impartiality is identified, the laboratory shall be able to demonstrate how it eliminates or minimizes such risk.

26

4.1.4 & 4.1.5 (Impartiality)

NOTE: A relationship that threatens the impartiality of the laboratory can be based on ownership, governance, management, personnel, shared resources, finances, contracts, marketing (including branding), and payment of a sales commission or other inducement for the referral of new customers, etc

27

Risks re impartiality

Presence of objectivity

freedom from conflict of interest

freedom from bias

lack of prejudice neutrality

fairness open-mindedness

even-handedness detachment

balance

28

7.8.6.1 (Decision rules)

When a statement of conformity to a specification or standard is provided, the laboratory shall document the decision rule employed, taking into account the level of risk (such as false accept and false reject and statistical assumptions) associated with the decision rule employed, and apply the decision rule.NOTE Where the decision rule is prescribed by the customer, regulations or normative documents, a further consideration of the level of risk is not necessary.

29

Decision rule – definition

Rule that describes how measurement uncertainty is accounted for when stating conformity with a specified requirement

Calling a test/calibration result a “pass” when it is really a “fail” (false accept), or calling something a “fail” when it is really a “pass” (false reject)

30

7.10.1 (Non-conforming work)

b) actions (including halting or repeating of work and withholding of reports, as necessary) are based upon the risk levels established by the laboratory;

31

Risks – NC work and Corrective Actions

Reference standard out of tolerance (CRM, Ref Standard)

Wrong procedure or out of range

Reagents expired

Likelihood of recurrence

Impact

32

8. Management System Requirements

8.5 (Actions address risks and opportunities)

8.6 (Improvement)

8.7 (Corrective action)

8.9 (Management review)

33

Clause 8.5.1 – Purpose

The laboratory shall consider the risks and opportunitiesassociated with the laboratory activities in order to:

a) give assurance that the management system can achieve its intended results;

b) enhance opportunities to achieve the purpose and objectives of the laboratory;

c) prevent, or reduce, undesired impacts and potential failures in the laboratory activities; and

d) achieve improvement.34

8.5.2 & 8.5.3 – Lab shall….

The laboratory shall plan: a) actions to address these risks and opportunities; b) how to: integrate and implement the actions into its management system; evaluate the effectiveness of these actions.

Actions taken to address risks and opportunities shall be proportionate to the potential impact on the validity of laboratory results. 35

Risk Assessment Example

Evaluate Infrastructure

System or Instruments

ConsiderSupportability

Capacity

Efficiency

36

Risk Cube and Risk Scoring

37

High 5 5 10 15 20 25

Med/High 4 4 8 12 16 20

Medium 3 3 6 9 12 15

Med/Low 2 2 4 6 8 10

Low 1 1 2 3 5 5

1 2 3 4 5

Low Med/Low Medium Med/High High

Co

nse

qu

en

ce

Likelihood

Supportability

The condition of the various components including frequency and cost of repairs, as well as obsolescence issues.

38

Ranking Consequence Likelihood

LowSystem in good working condition, no obsolescence issues <10 %

Medium-Low

System exhibits problems, but can be maintainedSome components at or near end of support 10 to 30 %

MediumSystem exhibits problems that cannot be mitigatedMore than half components no longer supported

30 to 60 %

Medium-High

System routinely unavailableMajority of components lack support 60 to 85 %

HighSystem non-functionalSystem completely obsolete >85 % 39

Supporta

bility

Evaluation

System 10 years old

Failed and been unavailable 3 times in past 2 years

Each time down unavailable for 4 weeks

Some components still available

Repair service still available

System unavailable for 12 weeks in 2 years = 12 %

Consequence(Medium) criticality raises this to Med/High

Likelihood (Med/Low)40

Supportability

41

High 5

Med/High 4 X

Medium 3

Med/Low 2

Low 1

1 2 3 4 5

Low Med/Low Medium Med/High High

Co

nse

qu

en

ce

Likelihood

Capacity

The ability of the various components to produce the required amount of workload in order to meet customer requirements.

42

Ranking Consequence Likelihood

LowSystem capacity exceeds demand requirementNo impact to availability <10 %

Medium-Low

System capacity meets current demandAny increase may stress system 10 to 30 %

Medium

System capacity occasionally fails to keep up with demandSome impact on availability 30 to 60 %

Medium-High

System demand exceeds capacitySome equipment unavailable resulting in complaints

60 to 85 %

High Demand exceeds capacity continually >85 % 43

Capacity

Evaluation

Capable of 4 tests/calibrations per week

Adequate for current workload

One/two months/year demand increases 6 per week

Expansion likely in next 3 years to 10 months of year

Current failure to meet demand 17 %

Future failure to meet demand 83 %

44

Consequence – Med HighLikelihood – Med/Low High

Capacity

45

High 5 X

Med/High 4

Medium 3 X

Med/Low 2

Low 1

1 2 3 4 5

Low Med/Low Medium Med/High High

Co

nse

qu

en

ce

Likelihood

Efficiency

The amount of effort required to conduct the measurements and the ease of use of the components.

46

Ranking Consequence Likelihood

Low System requires minimal effort to operate <10 %

Medium-Low

System requires some effort, but effort isn’t taxing 10 to 30 %

MediumSystem requires constant attention and interaction with analyst/technician 30 to 60 %

Medium-High

System is difficult to operate Some measurements have to be repeated to validate results

60 to 85 %

HighSystem tedious to operateAll measurements made on system impacted >85 % 47

Efficiency

Evaluation

System partly automated

Analyst/technician required to monitor computer to perform manual functions

– 65 % of their time spent waiting, can’t perform other work

Consequence – Med

Likelihood – High

48

Efficiency

49

High 5

Med/High 4

Medium 3 X

Med/Low 2

Low 1

1 2 3 4 5

Low Med/Low Medium Med/High High

Co

nse

qu

en

ce

Likelihood

Lab System Health Assessment

50

System Date Supportability Capacity Efficiency Overall Risk Score

A xx/xx/yyyy High Medium Medium Medium 39

B xx/xx/yyyy Low Low Low Low 10

C xx/xx/yyyy Medium Low Medium Medium 28

D xx/xx/yyyy High High Medium High 57

Summary

What is risk

How to evaluate

Risk based approach in measurement

Risk in ISO/IEC 17025:2017

An example

51

52

Thank you !