safety management systems a review of a maturity index … · safety management systems – a...

Safety Management Systems –

a review of a maturity index

study

Chris Drew

9th April 2013

© 2013 Baines Simmons Limited

The session objectives are

to:

Provide an overview of

the 2012 Baines Simmons

SMS-MI (Maturity Index)

results and to provide our

interpretation of this

Pick out some key points

to discuss

Session Objectives

SCAA AM/QM day – 9th April 2013


What is the Safety Management

Maturity Index ?

A tool designed to offer a high level

measure and calibration of the maturity

level of an SMS

Focuses on the outcomes of a successful

SMS

Enables organisations to understand

where effort should be applied to achieve

further SMS success

Focuses progressively towards outcomes

and added value that effective SMS can

bring your business

Assumes compliance with EASA/ICAO

SMS requirementsSCAA AM/QM day – 9th April 2013

SMS-MI v 7.xls

© 2013 Baines Simmons LimitedSCAA AM/QM day – 9th April 2013

Safety Management System

Maturity Index sample view


The main dichotomy of 2012

results...

The level of top

management support

mainly stagnant vs. 2010

results

The level of return on

investment consistently

low when compared with

2010 results



So what?


Proving it is worth investing in „safety‟ is hard without

evidence that it‟s worthwhile and adds value to a

business

Safety seen as:

a cost?

as compliance issue?

a budget spend – not seen as investment?

we don‟t have accidents therefore we are safe?

our #1 priority?


Safetyleadership

Culture

KnowledgeProactive

Risk (picture)

Decision making

The essence of

SMS...

Wrong culture = wrong decisions = increased safety risk



What is it you don‟t want

to happen?

I understand safety... don‟t I?

So – what is „it‟, what is safety?


Safetyleadership

Return on Investment


What does ROI look like?


Safety leadership


Safety

leadership RAF Leuchars „switched on‟ its SMS February

2012

“In order to preserve and continuously improve

our standards and working practices we must

adopt an open and honest reporting culture:

We need to know what‟s wrong so that we can fix

it. That means that we must admit and report our

mistakes.

In a Just Culture honest and well-intentioned

mistakes are not punished, they are learned from.

If you have the courage to admit your mistakes,

submit an SMS report, and so play your part in

preventing the same mistake being made again,

you will have nothing but my thanks and

admiration”

11Effective error management


Not always make for comfortable reading

Plug assembly became impossible to

separate from the probe

The plug assembly was cut to facilitate

removal

No replacement part was available

Non-standard repair carried out

Prior to SMS would have remained hidden

Raising of the report is now welcomed and

celebrated

The

result...



Latent errors with significant safety

implications identified by reporting and

investigation process

Revealed that the labels were transposed

Had been happening for years

…known by many, but not all

The result...



Culture




The result.

Senior manager comments:

There had been a cynicism about the SMS reporting system... admitting an

aircraft engineer had made a mistake somehow de-valued the aircraft

engineer role

Since we have rolled out SMS training across the whole facility this

'credibility' issue seems to have gone away... all of a sudden the people feel

able to admit to errors, without fear of losing credibility.

We have removed the perception that admitting to making an error makes

you a less capable engineer

I firmly believe that there is now a genuine acceptance of SMS as a vehicle

to make things better and because it is across the facility, my people are

more willing to use it

This may well be an intangible aspect of what we sought to achieve, but it

is a far better indication of the effectiveness of the system than just the

number of reports raised.

15SCAA AM/QM day – 9th April 2013


The result.



Knowledge




Predictive?

Top 4 by ATA?

ATA 25

Equipment/Furnishings

ATA 71-80 Powerplant

ATA 32 Landing Gear

ATA 27 Flight Controls


Source: CHIRP MEMS


20 20 20 20

21

2121

21

25

2525

25

25 25 25

25 25

26

26

27

2727 27 27

27 27

28

28

28

31

31

31

31

32

32

32

3232

32

33

33

33

33

35

35

3535

52

52

52

52

52

52

52 52

53

53

53

53

53

53

53 53

53

54

54

54

54

57

57

57

5757

57

71

71

72

72

72 72

72

72

75

75

79

79

7979

79

80

80

80

0

5

10

15

20

25

30

35

40

45

B73

7

B74

7

B76

7

B75

7

A31

9

A32

0

B77

7

A32

1

EMB14

5

BAE14

6

Maintenance error by A/C

type & ATA


Source:

CHIRP MEMS


Error event type 2007


Source: CHIRP MEMS


Costs?



Proactive Risk

(picture)




0

20

40

60

80

100

120

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23

Aircraft Audit findings

Aircraft Audit findings Linjär (Aircraft Audit findings)

The

result...



0

1000

2000

3000

4000

5000

6000

7000

AQD Raised Technical Occurrences Sep 06 - Aug 07 Form 500 Raised Quality Occurrences Sep 07 - Aug 08

Low Risk Medium and Above

The

result...


© 2013 Baines Simmons LimitedSCAA AM/QM day – 9th April 2013

Calculating ROI

The basic equation for ROI is simple: merely divide benefit by cost.

Estimate the annual cost of a particular type of maintenance error. Call

this “Cost.”

Determine the contributing factors to the event, and estimate the cost to

mitigate these factors. Keep it simple and call this “Cost to Fix.”

Estimate a reasonable “Probability of Success” that the interventions

will be successful. Say, for example, that you estimate an 80% success

rate.

Multiply “Cost” by “Probability of Success.” Call this “Return.”

Divide (“Return” – “Cost to Fix”) by “Cost to Fix.” Call this “Return on

Investment” (ROI).

A positive ROI (> 1.0) may not be achieved in one year

Source Bill Johnson FAA


In helicopter MRO (170 people)

First year the most common event „rework‟

E.g. hydraulic motor damage cost £15,167.04 mat/labour

SMS analysis highlighted

Occurred (at least) three previous occurrences

£45,501.12 (avoidable harm).

Low cost interventions - total cost of £153.50

Subsequently no further occurrences

Extrapolating frequency of occurrence delivered demonstrable saving to Exec. Board - in excess of £45K

SMS business ROI – Medium sized org.

© Baines Simmons 2010SCAA AM/QM day – 9th April 2013


Units „dead on arrival‟ (DOA) at customer‟s aircraft

Accounted for 80% of the organisations output

failures

Investigation - distraction during the return to

service

The conditions leading to distractions - introduced

during LEAN event which saw introduction of

entirely open plan workshop

Warranty claims/disruption = £45,000 pa losses

Board concern - associated reputational losses

The costs of intervention and investigation

approximately £850

HF intervention reduced DOAs by 76% over a year

Direct saving - £34,200 year one

SMS business ROI – Medium sized org.



Darren Cook – Manager Engineering Safety

“Our journey to the Bottom of the Error Iceberg between

2005 – 2008 delivered 6,200 error reports recorded in the

system”.

. . . 50% reduction in lost work days, and AUS$500 million

in cost savings over 5 years

air safeair safe

SMS business ROI – Large org.



The

result...

Safetyleadership

Culture

KnowledgeProactive

Risk (picture)

Decision making Return on

Investment



Ultimately

The primary return on investment (ROI) in civil aviation is

to open up our culture in support of low-iceberg reporting

What we absolutely know is that we have began to undo

the blame/lack of trust cultures in aviation maintenance

The missing ROI picture at this moment in time is what is

our industry doing with this new (proactive) data source

– are we using it to prove the business case for our

continuing investment in safety?

How do you measure up? Take the SMS-MI self-survey

and find out!


safety management systems a review of a maturity index … · safety management systems – a...

Documents