organizational safety effectiveness survey (oses) human ...€¦ · leadership and) supervision --...

HFA Human Factors Associates

Copyright © Human Factors Associates, Inc.

Organizational Safety Effectiveness Survey (OSES)

HUMAN FACTORS ASSOCIATES

www.hfa-oses.com

Anthony Ciavarelli Ed.D.

[email protected]

August 2018



Agenda •  ExamplesofOrganiza2onalAccidents

•  Organiza2onalAccidentsandHumanFailures•  Organiza2onalSafetyEffec2venessSurvey–OSES

•  OSESforCivilianAvia2onapplica2ons•  GeAngStartedwithOSES


ExamplesofOrganiza2onalAccidents

•  ExamplesfromAerospace–  Challenger–  Columbia

•  ExamplesfromAvia2on–  ValuJet–  ColganAir–  France447

•  ExamplesfromHealthcare(Hospitals)– NIHReportOverview–  LaunchingofPa2entSafetymovement


WhatAreCommonRootCausesofHumanError?(Organiza2onalandCultural)

•  Poor risk perception and threat recognition •  Lack of leadership commitment to safeguards •  Inadequate management oversight and control •  Inadequate or unclear risk-decision criteria •  Too difficult to report at-risk decisions/behavior •  Policy and incentives reward excess risk taking •  Culture does not support desired attitudes/behavior •  Unhealthy attitudes about safety, risk tolerance and

mission performance expectations •  Inadequate performance standards or poor training


RCSRiskControlSystemCCRM(2012)DeepWaterHorizonStudyGroupUCBerkeley.


What is a high-reliability organization (HRO)?

An organization that:

–  Conducts nearly error free operations, with very few accidents, low incidents, and no disasters

–  Demonstrates superior performance over a long period of time –  Makes consistently good risk decisions resulting in high quality, safe

and reliable operations OSES – is a survey measurement instrument that assesses an organization from the standpoint of key HRO attributes – the survey goes to the front line employees to determine whether or not they perceive their organization as a successful HRO – one that has a strong safety culture and a successful safety management system in place and working.


Safety Culture

•  Shared values about what is safe and unsafe •  Common beliefs about how to conduct safe

operations •  Open Reporting – “just culture” advocacy •  Behavioral norms that govern risk-taking,

everyday procedures and precautions •  Transmission of values, beliefs and accepted

practices to others

JamesReason1997–“Thewaywedothingsaroundhere.”


Comparison of HRO Theoretical Models

Roberts and Libuser HRO Model (Adapted) 1.  Safety Process auditing

2.  Safety Culture & Reward system

3.  Quality assurance (QA) - best work practices

4. Risk management 5. Leadership and) Supervision

-- Command Control

Weick and Sutcliffe Model Sensitivity to operations Deference to expertise Reluctance to simplify Sensitivity to Operation and Resilience Preoccupation with failure


CivilianApplica2ons-OSES

•  Organiza2onalSafetyEffec2venessSurvey–OSES

–  EmbryRiddleFlightSchool(firstcivilianapplica2on)–  ItalianAirForce– OffshoreOilandGasTransport(Phi,Bristow)– AirMedicalTransport(Phi,HealthNet,AlleghenyHospitallifeFlight)

–  ElectricU2lity(For2sAlberta-CanadianCompany)– Hospitals(Stanford–HarvardVAhospitalstudies)–  CommercialAvia2on(Alaska,Qantas,Airnorth)

HFA Human Factors Associates CSA Survey Ratings and

Naval Accidents

•  A study conducted by the US Navy in 2006, found that the survey item average for Risk Management, a category of the High Reliability Organization (HRO) and survey-climate scale for CSA, was a good indicator of accident risk.

•  Those squadrons that took the survey were divided into four quartiles based upon their average ratings for the survey’s Risk Management items. The figure on the right shows that squadrons with the lowest HRO-climate ratings had significantly more accidents.

0

2

4

6

8

10

12

Accidents

Climate Verses Accidents

Lowest Climate Ratings

Highest Climate Ratings

Middle Climate Ratings

Schimpf and Figlock 2006


NORMATIVEANALYSIS:AEROSPACENORM

2.08% 5.89%

12.26%

47.98%

31.79%

0%

10%

20%

30%

40%

50%

60%

1 2 3 4 5

Perc

enta

ge o

f Res

pons

es

Rating Scale (1 - 5)

Percentage of Responses Over 1 - 5 Scale

Problematic

Favorable

7.97% Problematic

79.77% Favorable


NORMATIVE ANALYSIS: OFF SHORE OIL AND GAS TRANSPORT

5.30%

13.92% 16.53%

49.30%

14.94%

0%

10%

20%

30%

40%

50%

60%

1 2 3 4 5

Perc

enta

ge o

f Res

pons

es

Rating Scale (1 - 5)

Percentage of Responses Over 1 - 5 Scale

Problematic

19.22% Problematic

64.24% Favorable

Favorable


WhatthePeopleonthegroundhavetosay

FlightOpera2on

–  Schedule pressure –  Pilot shortage –  Lack of trust –  Low morale –  Reluctance to report –  Low time pilots –  Resource cutbacks

impact safety

Maintenance

–  Schedule pressure –  Inadequate staffing –  Lack of trust –  Low morale –  Reluctance to report –  Hi workload – fatigue –  Lack of qualified service

personnel –  Resource cutbacks

impact safety

Human Factors Associates, Inc. Improving performance of high risk organizations

US Navy vs. Hospitals as HROs % Problematic

Safety climate survey comparison shows much fewer “problematic” responses for US Navy than Hospitals.

0

5

10

15

20

25

30

Climate Comm.

mgmt flex

safety issues

Navy

Hospitals

(Gaba, Singer, Sinaiko, Bowen, and Ciavarelli, 2003)


US Navy and Hospitals 2010 Study Follow up

Singer, Rosen, Zhao, Ciavarelli, and Gaba 2010

•  In general, the 2010 follow on study showed that safety

climate ratings were three times better on the average for US Navy based on 16 common survey items drawn from Ciavarelli (2004) Command Safety Assessment Survey.

•  Areas of concern among hospital workers were primarily:

•  Risk decisions not made by right people •  Faulty risk perception by senior executives •  Inadequate peer pressure to ensure patient safety •  Insufficient time to perform tasks assigned •  No clear management support for patient safety

HFA Human Factors Associates OSES FOR NASA AIVIATION


NASA OSES IMPROVEMENT STUDY

•  Performed Literature Review and comprehensive statistical analysis of OSES for NASA Aviation.

•  Statistical results showed NASA OSES survey items to be highly reliable, using survey data from two separate applications (2009 – 2011).

•  NASA OSES Factor Analysis testing yielded six

critical Factors or Assessment Areas.

HFA Human Factors Associates NASA AVIATION OSES

Organizational Safety Effectiveness Survey

AssessmentAreas

1.   SafetyClimate–Culture

2.   SafetyManagementSupervision

3.   Organiza2onalEffec2veness

4.   SafetyInforma2onManagement

5.   WorkloadandFa2gue

6.   MaintenanceSpecific

SampleSurveyItem•  Allemployeesfeelfreetoreporterrorswithoutfearofreprisal.

•  Mysupervisorsetstheexampleforcompliancewithstandardprocedures.

•  IbelievethatmoraleishighatmyCenter’sAircraaOpera2onsOrganiza2on.

•  Igetalltheinforma2onthatIneedtoperformmyjobsafely.

•  Iseldomfeeloverburdenedbymydailyworkassignments

•  MyCenter’sAircraaOpera2onsOrganiza2onhasanamplesupply

ofqualifiedaircraamechanics.


Example of OSES Rating Item

Copyright2001-2007HFA,Inc.


Example of OSES Open – Ended item


Copyright©HFA

Placement of a specific rating on Normal Distribution Curve

Ra2ngis1standarddevia2onbelowmean

Ra2ngisin14thpercen2le86%ofra2ngsareabovethisone


Example of Online Statistical Summaries Fictional Data – Demonstration Web Site

(Fictitious Data)


Sample OSES Overall Results Chart (Fictitious Data)

0

10

20

30

40

50

60

70

80

90

SafetyClimtate-Culture(SCC)

SafetySupervision-Mngmt(SSM)

Organizai2onalEffec2venses(ORG)

SafetyInforma2onManagement(SIM)

Workload&Fa2gueManagement

(WFM)

MaintenanceSpecific(MAINT)

%Unfavorable

%Netutral

%Favorable


SampleOSESOverallResultsChart(Fic22ousData)

0

0.5

1

1.5

2

2.5

3

3.5

4

SafetyClimtate-Culture(SCC)

SafetySupervision-Mngmt(SSM)

Organizai2onalEffec2venses(ORG)

SafetyInforma2onManagement(SIM)

Workload&Fa2gueManagement(WFM)

MaintenanceSpecific(MAINT)

Mean

StdDev


HOW TO BUILD AN HRO Culture

1. Leadershipeduca2onandunderstandingandac2veinvolvementinHROimplementa2on.

2. Employeeempowermentinspeakingup,withnon-puni2verepor2ngandopencommunica2ons

3. Con2nuoustraining(leadership,technical.,andhumanfactors)4. Exposeallsupervisorandexecu2veranksto“grounded”experienceswithfrontlineworkers

5. Provideperiodicselfassessmentstoes2matethelevelofriskduetoprocessesthatmaynotbeworking,employeeworkload,traininglevel,adequacyofresourcesandotherfactorsthatcontributetosafetyclimateandac2onsneededtobuildandmaintainastrongsafetyculture.

Copyright © Human Factors Associates, Inc

HFA Human Factors Associates Survey Application Process

1.  HFA representatives gather safety documentation and conduct interviews and focus group discussions with a cross section of company/agency employees.

2.  HFA constructs tailored survey based on the use of previously validated survey items and inputs from document review and personnel interviews.

3.  A draft survey is presented to safety personnel for review and revision, then finalized for survey application.

4.  The final survey and demographic information are uploaded to the web or a scanable paper version is created, .

5.  The survey is administered in accordance with preplanned schedule of events, including announcements/survey instructions, survey distribution, collection and analysis of survey data, and results reporting.

6.  Results, including Normative Benchmarks, are presented along with recommendations for safety improvement interventions where warranted.

7.  A time frame is established for a follow on second survey round that will provide results to be compared to the original “benchmark” Survey.

8.  Go to link below to see online web survey demonstration.

Copyright Human Factors Associates, Inc.

hjps://www.hfa-clients.com/hfademo/login.html

HFA Human Factors Associates SUMMARY - CONCLUSIONS

HRO oriented survey (OSES) method was originally developed and successfully applied in Naval Aviation, and then migrated to Civilian Aviation, Aerospace, Healthcare and Electric Power utility domains. Measures used have high – reliability and have been partially validated. Approach incorporates a complete measurement, analysis and diagnostic display “dashboard” system hat provides diagnostic feedback and normative “benchmark” comparisons for a given organization or domain. Survey process has been used by major airlines - Alaska ,Qantas, Airnorth, helicopter transport – PHI, Bristow, HealthNet, Allegheny Hospital Life Flight – Aerospace – Lockheed Martin. NASA, Electric power – Alberta, and a medical care facility Pediatric Care– and items from the survey have been used in a number of hospital culture studies. The survey methods also proved useful in an Managed Culture Change Process in an International study of Safety Management System - SMS implementation (Ciavarelli 2010, Asia Pacific Economic Cooperation; Ciavarelli 2016, Integrating Human Factors into SMS. OTC/SPE)

hjps://www.hfa-clients.com/hfademo/login.html


END

Anthony Ciavarelli Ed.D. [email protected]


REFERENCES and READINGS

AHRQ (2010). Becoming a high reliability organization: Operational Advice for Hospital leaders. Prepared by the Lewin Group, Falls Church, VA. CCRM (2012). Final Report on the Investigation of the Macondo Well Blowout. Deep Water Horizon Study Group UC Berkeley .p.75. Ciavarelli, A.P. (2016). Integration of Human Factors into Safety and Environmental Management Systems (OTC-27015) Offshore Technology Conference, Houston. Ciavarelli, A.P. (2012, September). Survey Improvement Study for the NASA Flight Center Organizational Safety Effectiveness Survey (OSES). Lake Oswego, Oregon: Human Factors Associates. Ciavarelli, A.P. (2008, February). Culture Counts: How does your organization measure up? Aerospace Safety Magazine. Washington DC: Flight Safety Foundation. http://www.nat.tur.br/docs/aerosafety/2008/asw_0802.pdf Ciavarelli, A.P. (2007). Assessing safety climate and organizational risk. In, the 51st Annual Meeting of the Human Factors and Ergonomics Society. Baltimore, Md. Ciavarelli, A.P. (2005, September). Assessing Safety Climate and Culture: From Aviation to Medicine. Second Conference on Safety in High Consequence Industries. St. Louis. Ciavarelli, A.P. & Crowson, J. (2004, March). Organizational Safety Effectiveness Assessment. In, St. Louis University Conference Proceedings on Safety in High-Consequence Industries. St. Louis, Missouri. Ciavarelli, A.P, Figlock, R., Sengupta, K., and Roberts, K. (2001) Assessing organizational accident risk through survey questionnaire methods. In, Proceedings of the 11th Annual International Aviation Psychology Conference. Columbus.



Ciavarelli, A.P, Figlock, R., Sengupta, K., and Roberts, K. (2001) Assessing organizational accident risk through survey questionnaire methods. In, Proceedings of the 11th Annual International Aviation Psychology Conference. Columbus. Ciavarelli, A.P., (2001, February). Human Factors Checklist: A tool for Aircraft Accident and Incident Investigation. Flight Safety Digest. Washington DC: Fight Safety Foundation. Ciavarelli, A.P., (1997). Cockpit design factors: What we learn from aircraft accidents. Paper presented at the SAE/AII World Conference. Los Angeles, CA. Ciavarelli, A.P. and Figlock, R. (1997). Organizational factors in Naval Aviation accidents. Proceedings of the International Aviation Psychologists Columbus, OH. Flin, R., Mearns, K., O’Connor, P., and Bryden, R., 2000. Measuring safety climate: Identifying the common features. Safety Science, 34, 177-92. Desai, V.M., Roberts, K.R., and Ciavarelli, A.P. (2006, winter). Defensive attributions in the formation of perceived safety climate. Human Factors and Ergonomics, 48. (4) 639-650. Gaba, D., Singer, S., Sinaiko, A., & Ciavarelli, A.P. (2003, Summer). Safety Culture differences between hospital personnel and naval aviators. Human Factors and Ergonomics.



ICAO. (2013). Safety management manual (3rd ed.) (ICAO Order No. 9859). Quebec, Montréal: Author. Libuser, C.B. (1994). Organizational structure and risk mitigation (Ph.D. Dissertation. UCLA). Los Angeles CA. Lekka, C. and Sugden, C. (2012). Working towards high reliability: A qualitative evaluation (Symposium Series No.158). HMSO: UK. Lekka, C.(2011) High reliability organizations: A review of the Literature (RR899).HSE, UK. (p.6) Maurino, D. (2017, March). An introduction and overview of Safety Management Systems. Prepared for Round Table on Safety Management Systems. OECD – ITF, Paris, France. Ostrowski, K.E. (2016). USAF Aviation Safety Program Gap Analysis using ICAO Safety Management System Guidance. American Society of Safety Engineers. Perrow, C. (1984). Normal accidents: Living with high-risk technologies. NY: Basic Books. Reason, J. (1997). Managing the risks of organizational accidents. Brookfield: Ashgate Roberts, K.H. and Bea, R. (2001). Must accidents happen? Lessons from high-reliability organizations. Academy of Management Executives. Vol. 15, No.3, 70-79. Roberts, K.H. (1990, summer). Managing high-reliability organizations. California Management Review. 32, (4), 101- 113.



Roberts, K., Rousseau, D., and La Porte, T. (1994). The culture of high reliability: Quantitative and qualitative assessment aboard nuclear-powered aircraft carriers. Journal of High Technology Management Research, 5, 141-161. Royal Aeronautical Society (2017). Achieving peak safety performance: Listening and learning. https://www.aerosociety.com/news/achieving-peak-safety-performance-listing-and-learning/ Roberts, K. H. (1993). Culture characteristics of reliability enhancing organizations. Journal of Managerial Issues, 5, 165-181. Schein, E.H. (1996). Culture: The missing concept in organizational studies. Administrative Science Quarterly, 41, 229-240. Schein, E.H. (1990). Organizational culture, American Psychologist, 45, (2), 109-119. Singer, Sara, J, Rosen, A., Zhao, S. Ciavarelli, A.P. Gaba, D.M. (2010). Comparing safety climate in naval aviation and hospitals: Implications for improving patient safety. Health Care Management Review. Apr-Jun; 35(2): 134-46. Stolzer, A.J., Halford, C.D., Goglia, J.J. (eds.) 2011. Implementing safety management systems in aviation. Ashgate. Burlington, Vermont.



Waller, M.J., and Roberts, K.H. (2003). High reliability and organizational behavior: Finally, the twain must meet. Journal of Organizational Behavior, 24, 813-814. Weick, K.E. (1999). Organizing for high reliability: Processes of collective mindfulness. Research in Organizational Behavior, 21, 81-123. Weick, K.E. and Sutcliffe, K.M. (2007). Managing the unexpected. John Wiley & Sons. Westrum. R. and Adamski, A.JH. (1999). Organizational factors with safety and mission success in aviation environments In, D.J. Garland, J. Wise, and V.D. Hopkins (Eds).Handbook of Aviation Human Factors. Lawrence Erlbaum, NJ. Woods, D., Dekker, S., Cook, R., and Johannesen, L. (2010) Behind Human Error. 2nd Edition, Ashgate. Zohar, D. (2010) Thirty years of safety climate research: Reflections and future directions. Accident Analysis and Prevention, Vol. 42, pp. 1517-1522. Zohar, D (2002). Safety climate: Conceptual and measurement issues. In J. Quick & L. Tetrick (Eds.), Handbook of occupational psychology (pp. 123-142). Washington DC: American Psychological Assoc. Zohar, D. (1980). Safety climate in industrial organizations: Theoretical and implications. Journal of Applied Psychology, (1), 96-102.

organizational safety effectiveness survey (oses) human ...€¦ · leadership and) supervision --...

Documents