do we really know how well our occupational accident prevention programs work?

Safety Science, 16 (1993) 445-463 Elsevier

445

Do we really know how well our occupational accident prevention programs work?

Stephen J. Guastello Department of Psychology, Marquette University, Milwaukee, WI 53233, USA

Abstract

This article summarizes evaluation data for ten types of accident prevention program drawn

from 53 program evaluations: personnel selection variables, technological interventions, behavior

modification programs, poster campaigns, quality circles, exercise and stress management, near-

miss accident reporting, comprehensive ergonomics, the International Safety Rating System

(ISRS), and the Finnish national control program. Standard measures of effect size were com-

puted for source studies which described the effectiveness of programs in terms of the percentage

reduction in accidents attributable to the program. By this standard, personnel selection tech-

niques were the least effective (3.8% ), and comprehensive ergonomics programs (49.5% ) were

the most effective. Some methodological issues in safety program evaluation are discussed in con-

junction with some further thoughts on a previously published review of ISRS evaluation results

(in 1991).

R&urn6

Cet article donne un resume de l’kvaluation de 10 differents types de programmes de prevention

d’accidents selection&s parmi 53 evaluations de programme; les variables en ce qui concerne la

selection du personnel, les interventions techniques, les programmes de modification du comporte-

ment, les campagnes d’affichage, les cercles de qualite, les exercices et la gestion du stress, les

comptes rendus sur des accidents qui ont failli arriver et l’ergonomie, le Systkme International

d’Evaluation de la Securite’ (ISRS) et le programme de controle national finlandais. Des mesures

standard ont 6tC concues pour des etudes h la source qui permettaient de decrire l’efficacitk des

programmes en termes de reduction du pourcentage d’accidents attribuables au programme en

question. Selon cette mdthode, les techniques de selection du personnel dtaient les moins efficaces

(3.8% ) et les programmes sur l’ergonomie approfondie dtaient les plus efficaces (49.5% ). Quelques possibilites methodologiques concernant l’evaluation dun programme de skcurite sont Qtudiees en

correlation avec le document publik prkcedemment sur les rdsultats dune evaluation ISRS (en 1991).

Zusammenfassung

Dieser Artikel ist eine Zusammenfassung der Daten von zehn Arten von Unfallverhiitungspro-

446

grammen, die aus 53 Programmauswertungen ermittelt wurden. Dabei handelte es sich urn Va- riablen fur die Personalauswahl, technologische Eingriffe, Programme zur Veranderung des Ver- haltens, Aufklebercampagnen, Qualitltskreise, Gymnastik und Strefibeherrschung, Beinaheunfallberichte, ausfuhrliche Ergonomie, das Internationale Sicherheitseinstufungssys- tern (ISRS) und das finnische nationale Programm zur Einschrtikung von Unfallen. Fur Quel- lenuntersuchungen, in denen die Wirksamkeit der Programme entsprechend der prozentualen Senkung der Unfalle durch diese Programme beschrieben wird, wurden effektive Mafinahmen nach einer bestimmten Norm ermittelt. GemaR dieser Norm waren die Methoden der Personal- auswahl am wenigsten effektiv (3,s Prozent), wahrend die ausftihrlichen Ergonomieprogramme (49,5 Prozent) am effektivsten waren. Im Zusammenhang mit einigen weiteren Gedanken tiber eine friiher veriiffentlichte Besprechung der Auswertungsergebnissen des ISRS (im 1991) werden einige methodologische Fragen bei der Auswertung von Sicherheitsprogrammen diskutiert.

1. Introduction

While much well-intentioned money is spent each year to improve workplace safety, there is comparatively little available information as to which programs produce the best results. Decision-makers need to adopt policies and programs, and they need reliable information on which to base their choices. The informational needs are thus (a) to identify the salient characteristics of those programs, (b) to draw conclusions about how well they worked, (c) to prepare such information in a form that allows for comparable evaluations, and (d) to assess the generalizability of the findings.

The need for safety program evaluations appears to have been growing. In the USA, the need for evaluation has centered on the demands placed on corporate safety officers to justify the economic effectiveness of their work (Veltri, 1992). Some nations have laws requiring the certification of persons or organizations providing safety services (Oortman Gerlings and Hale, 1991). The scientific imperative, however, is to assess the intervention techniques them- selves in terms of how well they prevent accidents. Economic criteria were not considered in this review because, more often than not, they are imprecisely associated with accident outcomes due to chance effects associated with the body part that might be injured and national differences in the costs, delivery, and payment for health care services.

The primary purpose of this article is, therefore, to provide some quantita- tive comparisons of the accident prevention techniques that have been evaluated empirically in professional journals. In the spirit of this special issue I summarize briefly a recent review I completed (1991a) of program evaluations for the International Safety Rating System (ISRS ). My commentary pertains largely to how the need to review the program arose, and a summary of the key findings of a broader research effort I undertook to evaluate safety programs. The new data offer some useful context within which to frame the results of the ISRS program review.

441

2. Review strategy

The sources of review material were scientific journal articles, computerized literature scan, and evaluations that were discovered through leads from the primary sources. Studies selected for inclusion here were those that used accidents as their criterion, rather than unsafe behaviors. Non-occupational ve- hicular accident studies were omitted. Because theories and technologies evolve and improve over time, this review is limited to techniques that have been tested since 1977; earlier studies are included to the extent that they pertained to techniques still under investigation.

The measure of effect size, D, for the comparative analysis of the programs was: D = [ (Rpost - Rpre ) lRpre I - [ (Spost - Spre) /Spre I. Rpost and Rpre are the raw numbers of accidents before and after the intervention, respectively; Spost and Spre are the raw numbers of accidents before and after an intervention observed for the control group. In some studies the dependent measure was expressed as a rate. Since an accident frequency count is equal to the accident rate times the number of exposure units, a comparable D can be obtained by inserting a rate for Rpost or Rpre.

Unlike the evaluations of other types of safety programs the personnel selection studies were based on correlation coefficients. In this review, correlations were transformed to a percentage increase in relative efficiency by using utility expectancy tables (Guion, 1965; Lawshe, Bolda, Brune and Auclair (1958). The table required two additional parameters: the selection ratio and the base rate of success, which were set at 0.20 and 0.70, respectively. A base rate of 0.70 is equivalent to defining “success on the job” as an individual not incurring an accident in four years of exposure, based on the average accident rate of 8.30 for all private sector U.S. industries as reported by the National Safety Council, 1989). For personnel selection studies that were based on t tests between two groups, t was first converted to the omega-squared (w2) statistic for proportion of variance accounted for by the independent variable (Hayes, 1983 ) .

3. Accident prevention programs

Ten types of program were encountered: personnel selection, technological intervention, behavior modification, poster campaigns, quality circles, exercise and stress management, near-miss accident reporting, Finnish national program, ISRS, and comprehensive ergonomics programs. Personnel selection studies focused on characteristics of the individual worker that might explain accident involvement; six different types of selection variable were encountered. Some of the other types of programs define specific actions that management might take to solve safety problems. Other programs reflect a coor-

448

dinated series of management efforts that are sometimes accompanied by an organized diagnostic effort.

3.1. Personnel selection

The personnel selection method of accident control is based on the concept of accident proneness. The organization identifies aspects of proneness among job applicants and screens out people who have those characteristics. Mintz and Blum (1949) traced the accident proneness concept back to the 1920s when risk analysts had noticed that a large percentage of a company’s accidents were incurred by a relatively small percentage of people and therefore concluded that some people were more accident prone than others. Mintz and Blum reasoned that such irregular odds were entirely possible by chance if one assumes a Poisson distribution of accident risk. Differential accident risk can only be said to occur if the distribution of accidents deviates substantially from the Poisson expectancies. They found, after analyzing several data sets, that differential accident risk occurs inconsistently. Furthermore, differential risk does not mean that a personal characteristic is necessarily causing the out- come; environmental variables are at least equally viable.

Research on variables that could be used in a screening process continued nonetheless. Personal maladjustment, social maladjustment, impulsivity, and cognitive deficits, alcohol and drug use are regularly implicated; cognitive abil- ities have been represented as well. For this summary of effects, if a study contained r coefficients for more than one variable from one of the six categories of selection variable, the average r was used to calculate a single measure of effect size. Some of the studies of selection variables were not designed with a personnel selection objective in mind; reported correlations between a selection variable and accidents were often incidental to another objective. Most studies were postdictive in design, and many used extreme groups comparisons. It is probably also true that the categories personal maladjustment, social maladjustment and impulsivity are fuzzy rather than distinct. Effect sizes for all program types are summarized in Table 1.

Personal maladjustment variables include stressful life events anxiety (D. Guastello and S. Guastello, 1987a; S. Guastello, 1991b; S. Guastello and D. Guastello, 1988)) and a measure of distractibility composed of items from the Minnesota Multiphasic Personality Inventory (MMPI; Hansen, 1988)) and trust, tension, and insecurity scales from the Sixteen Personality Factor Ques- tionnaire (16PF; Hilakavi et al., 1989). The average effect size for this group of variables was 8.2%, with a range from 5% to 11%.

Social maladjustment variables include the Rotter Locus of Control scale (Sims, Graves and Simpson, 1984) and derivative scales such as Beliefs About Accident Control (D. Guastello and S. Guastello, 1987a; S. Guastello, 1991b; S. Guastello and D. Guastello, 1988) or Safety Locus of Control (Wuebker,

449

TABLE 1

Summary of effect sizes for ten types of accident prevention program

Type of program Number N Effect

of studies (%l

1. Personnel selection 26 19,177 3.7

Personal maladjustment 5 1,985 8.2

Social maladjustment 7 1,813 7.3

Impulsivity 3 2,007 0.9

Alcohol use 8 8,615 0.8

Drug use 2 4,686 0.0

Auditory selective attention 1 71 5.0

2. Technological interventions 4 NR 29.0

3. Behavior modification 7 2,444 59.6

Exceptional case deleted 6 1,516 38.6

4. Poster campaign 2 6,100 14.0

5. Quality Circles 1 76 20.0

6. Exercise and stress management 2 1,300 15.0

7. Near-miss accident reporting 2 NR 0.0

a. International Safety Rating 4 NR 17.0

South African mines 1 NR 1.5

Australian mines 1 NR - 10.2

North American applications 1 NR 20.6

New Zealand application 1 NR 56.0

9. Comprehensive ergonomics 3 NR 51.6

New Zealand case deleted 2 NR 49.5

10. Finnish national program 2 NR 18.3

Note: Number of studies, sample sizes, and effect sizes are nested by specific type within broader

type. NR = not reported for all studies or not relevant.

1986); a social adjustment measure composed of items from the MMPI (Han- sen, 1988); and a measure of employee reliability which was derived from the sociopathic personality (Hogan and Hogan, 1989). The average effect size for social maladjustment variables was 7.3%, with a range from 1% to 10%.

Impulsivity was measured 16PF scales for conscientiousness and self-disci- pline (Hilakivi et al., 1989), 16PF boldness (Hilakivi et al., 1989), sensation seeking personality and boredom susceptibility (Landeweerd, Urlings, de Jong, Nijhuis and Bouter 1990; Rundmo, 1992). The average effect size for these variables was 0.9%, with a range from 0% to 3%.

Alcohol use has been measured by self-report inventories (Guastello, 1991b; Webb et al., 1991) or assays of blood samples (Buchanan, 1988; Webb et al., 1991). If such measures are to be used to select personnel, it must be assumed that alcohol use off the job affects accident rates at work. In regard to alcohol, Webb et al. (1991) found that heavy drinking off the job was not associated with occupational accident involvement; problem drinking as measured by the

450

Mortimer-Filkins Test did predict concurrent accident involvement, however. Guastello (1991b) found no effect for alcohol with accidents for transit oper- ators once the effect of hazards was removed through semipartial correlation. Buchanan (1988) found a reverse effect for alcohol, and only two out of four of the other studies cited in Webb et al. (1991) showed a significant effect for alcohol. The average effect size for alcohol use was 0.8%, with a range from -6% to +ll%.

Drug use has been measured by self-report inventories (Guastello, 1991b) or urine tests (Normand, Salyards and Mahoney, 1990). Once again, if such measures are to be used to select personnel, it must be assumed that drug use off the job affects accident rates at work. Neither study obtained a significant effect for drug use (marijuana or cocaine). The effect sizes for both drug use studies were 0.0%.

In one study, auditory selective attention test scores were correlated with occupational accidents producing an effect size of 5.0% (Arthur, Barret and Doverspike, 1991). The average unweighted effect size for the six types of personnel selection schemes is 3.7%.

3.2. Technological interventions

Technology interventions fell into two categories: robotics (Karwowski, Rahmi and Mihaly, 1988) and comprehensive facility redesign (Harms-Ring- dahl, 1987; Kjellen, 1990; Mohr and Clemmer, 1989). Two of the effect sizes were based on simulation analyses (Harms-Ringdahl, 1987; Kjellen, 1990) and may be attenuated by new types of human error that could arise from intro- ducing the new system. The average (unweighted) effect size from the four applications was 29.0%, with a range from 6% to 59%.

Robotics interventions have the potential to reduce accidents, but they can introduce new types of accidents (Chignell, Lowenthal and Caspi, 1986). In accidents where deaths occurred, the operator was caught in the robot’s work envelope and impaled against a structure (Linger, 1987; Sanderson, Collins and McGlothlin, 1986). New control techniques include an emergency kill switch that is operated by the human (Sjostrom, 1989, 1990), a radar-type sensor that detects the presence of a human in the work envelope (Malm and Suominen, 1990; Sugimoto and Ikeda, 1990)) and re-engineering the programmer’s work station so that the programmer need not enter the robot’s work envelope (Linger, 1987 ) .

3.3. Behavior modification

Behavior modification programs are based on operant learning theory. Mul- tiple base line experimental designs are the most common form of experiment. When a study contained more than one experimental group, the effect sizes

451

were averaged into one effect size. Seven studies used accidents as criteria (Calkins, 1971; Fellner and Sulzer-Azaroff, 1984; Fox, Hopkins and Anger, 1987; Haynes, Pine and Fitch, 1982; Larson et al., 1980; Smith, Anger and Uslan, 1978; Sulzer-Azaroff, Loafman, Merante and Hlavacek, 1990).

The typical program consists of basic safety information training regarding proper safe behavior, followed by a period of observation and feedback. Feed- back may be provided by the supervisors or by graphs displayed in the workplace. Some program variations include goal setting, incentives, and mechan- ical aids for behavior monitoring. The substance of the behavior modification programs was reviewed in greater detail in McAfee and Winn ( 1989) and Cohen, Smith and Anger (1979).

The weighted average effect size for the seven behavior change studies was 59.6%. One of the seven studies (Fox et al., 1987) reported an effect size of 94% over a 12-year period. This exposure period was much longer than any of the other behavior modification or other studies. A possible explanation for the unusual effect size might be, according to Fox et al., that the workforce in the two uranium mines was fairly stable; a maturational effect may have been operating in addition to the intended program effect. Also, it is possible that new innovations in the handling of radioactive materials may have occurred and contributed to the effect size. If this unusual case is excluded, the weighted average effect size for the six behavior modification studies is 38.6%, with a range from 12% to 57%.

3.4. Poster campaigns

There were two reports evaluating poster campaigns against accidents as the criterion. The effect sizes were - 5% (Saarela et al., 1989) and 33% (Saa- rela, 1989). In the negative case, an experimental poster group of shipyard workers improved lo%, but the rest of the company improved 15%. The unweighted average of the two effect sizes is 14.0%.

3.5. Quality circles

A quality circle is a committee of employees who perform similar types of work who meet voluntarily to solve product quality, productivity, and cost reduction problems. Past research has shown that the quality circle technique has been successful in those objectives in addition to improving quality of work life and attendance (Marks, Mirvis, Hackett and Grady, 1986). One study has also reported success with quality circles for preventing accidents (Saarela, 1990); the effect size in that study was 20.0%.

452

3.6. Exercise and stress management

Stress management programs appear successful for reducing stress and improving some job attitudes (Ivancevich, Matteson, Freedman and Phillips, 1990; Murphy, 1984). Noteworthy, however, was Murphy’s observation that stress management programs are not designed to eliminate the sources of stress, but only to teach workers how to cope. Murphy, DuBois and Hurrell (1986) as- sessed the impact of a stress management program on accidents. Although the program appeared to have effected an 80% reduction in accident rate, a rean- alysis of the published datapoints showed that the effect was substantially autoregressive’. The net effect size was 14%.

Exercise programs have proven effective for reducing stress injuries in phys- ically demanding jobs (Gebhardt and Crump, 1990). One evaluation study addressed the accident criterion specifically. Cady, Thomas and Karwasky (1985) determined that an exercise program for firefighters produced a 16% reduction in work-related injuries and illnesses. Because both stress management programs and exercise program focus on eliminating stress as a possible cause of accidents, the two types of program are considered together here as having an unweighted average effect size of 15.0.

3.7. Near-miss accident reporting

The near-miss reporting program is based on the principle that for every real accident that occurs, approximately ten near-misses have also occurred. By investigating eleven times as many incidents, a greater number of ideas for preventative measures can be generated. One version of the technique (Carter and Menckel, 1985) resulted in an increase in the number of corrective sug- gestions generated, but no reduction in the accident rate. In another version, participants received training in how to recognize and interpret critical incidents (Menckel and Carter, 1985 ). A 56% reduction in injury severity resulted, although it was not accompanied by a reduction in the accident frequency rate.

‘Data were presented in the form of a time series analysis that showed a decrease in accidents per

month over 33 months. The monthly accident rates before and after the intervention were 3.08

and 0.63 respectively. The difference was reported as significant at p < 0.05, but the statistic used

was not reported. For this report, the data in their graph was digitized and subjected to a regression

analysis to tease out the autoregression effect. A 2-month lag length produced the best fit. Ordi-

nary least squares analysis was used because the test for serial dependency among residuals was not significant. Multiple R2 was calculated where the accident rate at time t was predicted from

accident rate at time t- 1 and the onset of the program which was entered as a dichotomous variable. The majority of the observed effect was autoregressive in nature, although a significant

semipartial r2 was obtained (p < 0.05) for the program. New raw effect size was then calculated

by reducing the observed effect size by the ratio of sr* for the program to total variance accounted

for [ (3.08-0.64)/3.08] (0.094/0.526) =0.143.

453

3.8. International Safety Rating System (ISRS)

The ISRS is a safety audit program designed to be conducted by managers of industrial sites. It was developed and first introduced in 1978 by the South African Chamber of Mines. Twenty topics are addressed by the audit in the current version of ISRS that is available to general industry: leadership and administration, management training, planned inspections, task and proce- dures analysis, accident investigation, task observation, emergency prepared- ness, organizational rules, accident analysis, employee training, personal protection equipment, health control, program evaluation system, engineering controls, personal communication, group meetings, general promotion, hiring and placement, purchasing controls, and off-the-job safety (Pringle and Brown, 1989 ). Managers are trained as auditors who assess the work environment on a yearly basis. In principle, the audit should produce recommendations for action in the year ahead. Specially certified ISRS personnel can be called upon to visit the sites; sites can be awarded up to five “stars” for safety performance.

Five forms of program evaluation data were available from two papers (Eis- ner and Leger, 1988; Pringle and Brown, 1989,1990), four of which underwent secondary analysis (Guastello, 1991a). Among South African gold mines there was no significant association between the number of stars awarded to a mine and its rate of deaths or injuries, nor was there any discernible impact on accident rates across all 33 gold mines attributable to the installation of the ISRS program in 1978. The average correlation between “stars” awarded to mines and accident outcomes reported by Eisner and Leger (1988) converted to an effect size of 3.0%; the time series trend that I analyzed (1991a) from their published data resulted in an effect size of 0.0. The average of the two effect sizes was 1.5%.

An Editor’s note published in the Journal of Occupational Accidents (Eisner, 1989) indicated that the current version of ISRS differs substantially from the South African version. Therefore it is valuable to consider other evaluation efforts independently. Pringle and Brown (1989) reported a 12.0% drop in accident rates among 2395 North American companies for the period 1978- 1979. This value compared favorably with a 7.2% increase recorded for all North America during the same year (Guastello, 1991a), resulting in an effect size of 19.2%‘. Similar levels of improvement were recorded for the ISRS companies for the 1979-1982 period (22.0%). The average effect size for North America was 20.6%. I further noted that it was hazardous to mix USA and Canadian data for most purposes, and showed some diverging trends for the two nations.

Pringle and Brown (1989) reported accident indices for six Australian coal mine sites that used ISRS during the 1982-1985 period. Each site was paired

‘The method for calculating effect size here was slightly different from the original article. The impact of this difference on the final estimate of program effect size was trivial.

454

with a control site. Split plot analysis of variance showed that while there was a significant decrease in accidents over time for all 12 twelve sites, as measured by insurance premium percentages, there was no differential improvement between ISRS and control sites (Guastello, 1991a)3. The average effect size for the six Australian ISRS coal mines where each was compared against the six that control sites was - 10.0%.

3.9. Comprehensive ergonomics

There were three evaluations of comprehensive ergonomics programs. One involved the use of ISRS with other techniques. One involved a different diagnostic survey or audit system with eight small organizations. One was a large- scale corporate program that did not utilize a diagnostic instrument.

Pringle and Brown (1989) reported pretest and post-test accident rates for a New Zealand company that used the ISRS program plus a general ergonomics program and a control company that the reader was left to assume used neither technique. The substance of the general ergonomics program was not described, but it could have been similar to the corporate comprehensive program described below. The test company experienced a 79% reduction in accidents in a one-year period, while the control company experienced a 23% decline during the same period. The effect size for ISRS plus general ergonomics was, therefore, 56.0%.

A major manufacturing conglomerate provided data (under conditions of anonymity) for this review pertaining to their comprehensive ergonomics program. Their program appears to be representative of strategies adopted by other large firms in the USA and emphasized the concept of a safety climate or safety culture and embodied seven groups of more specific programs: (a) safety performance and monthly statistical tracking (b) safety engineering services in- cluding workplace ergonomics (c ) safety seminars conducted by inside and outside consultants, (d) recordkeeping assistance (e) a videotape library, (f) specific hazard control programs (e.g. ammonia, back injury, motor vehicles), and (g) substance abuse programs (EAP and urine testing). The work group supervisors were considered the persons ultimately responsible for safety.

Over a two-year period 1988-1990, the corporation experienced a 32% reduction in its accident frequency, and a 49% reduction in its lost workday incidence rate. The average of the two effect sizes is 40.5%. The corporation’s 1988 incidence rate was 80% of the national level rate for its industry. By the the end of two years its incidence rate was 45% of the national rate. Thus it would appear that the program’s success could not be attributed to regression to the mean effect.

“Erratum: The F test for the between sites effect was incorrectly reported as 2.00. The correct value is 1.88 (~~0.20); this discrepancy has no effect on the conclusions from the analysis.

455

The Occupational Hazards Survey (OHS ) is a tool for diagnosing and con- trolling occupational accident risks from an organizational development point of view (D. Guastello and S. Guastello, 1987a, 1987b; Guastello, 1989, 1991b, 1992; S. Guastello and D. Guastello, 1988). OHS is based on a model of safety climate that expands upon the earlier work of Zohar (1980). It is also based on nonlinear system dynamics. It is treated here as a comprehensive ergonomics program because of the survey coverage and types of recommendations possible. The diagnostic surveys are completed by employees rather than management auditors, but supervisors do participate. Variables measured are: safety management ratings, danger severity, age and experience, beliefs about accident control, physical stressors, social and job stressors, environmental hazards, tools, and hazards specific to a particular industry.

Survey recommendations are based on the group level of analysis rather than the individual level. A list of recommended actions is formulated specifically for the host organization; some recommendations are geared toward specific subunits within the organization. In an evaluation of the accident strategies recommended to eight organizations in the secondary metal industry (Guastello, 1989)) accident rates were collected both before and after the interventions. Although all eight organizations received the treatment, they var- ied in the amount of time they were allotted to work with their recommendations before post-test data were collected. Results showed that reductions in accident rates were related to the amount of time the organization had to work on its recommendations. Accident rates for the 68 work groups declined 30% at the end of six months, relative to baseline accident rates. Accident rates were increasing, however, until the organization received its OHS report. The net effect size was 58.4%4 for data pooled over eight organizations.

The average effect size for the latter two programs is 49.5%. If the New Zealand company results are included, the average of three effects is 51.5%.

4The effect size was calculated from the raw data rather than from the material published in the

article. The regression equation (r= -0.27, ~~0.05) obtained for the change in accident rate

predicted from the number of months an organization had held its OHS report (M) was:

du = [ log,,( Y2 + 1) -log,, ( Y, + 1) ] = 0.534 - 0.095M. Values of A4 ranged from 1 to 9, with a mean

of 6. The Y2 accident rate was a year of exposure composed of A4 months with the report plus

12-M months without the report prior to surveying and feedback. For M= 1, Du=O.439; for

M=9, Du= -0.321. Simultaneous equations are then set up to solve for the monthly change in

logarithm of the accident rate post intervention, w, and without intervention, U: 1 lu + 1~; = 0.439,

3u + 9u, = - 0.321, w = - 0.023, u = 0.038. To obtain the changes in accident rate per month, antilog

w and u and subtract 1: w’ = -0.052,~ =0.109. To obtain the mean values of w’ and u’, multiply

each value by 6; 6w’ = -0.312, and 6~’ ~0.654. The mean effect size for this intervention can be ascertained by taking the difference between 6w’ and 6~’ and dividing by the total “damage” rate

incurred by the control condition relative to its starting point: [0.654- (-0.312)]/

[1+0.654] =0.584.

3.10. Finnish national program

Two Finnish government agencies responsible for labor protection took con- certed action to target the most problematic occupational groups and to de- velop effective control strategies. Action strategies were: safety promotion in target occupations, making safety information and research reports regarding accident causes and control methods available to work supervisors, setting standards for machine safety and repair, and worksite inspections (Bjurstrom, 1989).

Three partial effect sizes were calculated, and the more dramatic decreases in accident rate were recorded for two specially targeted occupations: internal transportation and longshoring. The 9.4% reduction at the national level was thought to be a combined result of the program plus national economic growth, where the latter had the effect of increasing casualties (Bjurstrom, 1989). It was calculated on the moving average of accident rates for 1977-1979 compared against the moving average for 19851987. Because most program evaluations considered here emanated from nations that had some sort of govern- mental agency taking some responsibility for occupational accident control, the national trend was used as a control effect size and subtracted from the effect sizes obtained for the two targeted industries. The average of the two net effect sizes was 18.3%.

4. Discussion

Evaluation of programs are not easy to perform, and they may be hampered by bankruptcies and plant closing, reassignment of key people responsible for program implementation, conflicting demands of other organizational priori- ties, concurrent uses of multiple programs, and the reluctance of corporate officials to release numerical information. In one case, recommendations that I made to an organization in 1988 were not adopted until 1991, and it would take at least another year to build, to test, and to implement the special equipment. These barriers to evaluation present some real challenges. The conclud- ing sections of this article summarize the highlighted findings of the empirical review and combine those findings with observations from industrialists who have used some of the programs. Some directions for future safety program evaluation research become evident.

4.1. Personnel selection

The largest number of available evaluation studies pertinent to the objectives of this article pertained to characteristics of individual. When transformed into the effects of personnel selection with those variables, selection

techniques emerged as the least effective method of those studied. There was some variability in effect size among individual variables, with personal and social maladjustment obtaining the largest effect sizes. The problem with those two groups of variables is, however, that the correlational link between maladjustment and accidents may be one of reverse causality. In other words, maladjustment could be the result of working too long (or not long enough) in a dangerous and poorly controlled environment.

According to Rundmo ( 1992), individual characteristics such as sensation seeking and boredom susceptibility do not have a direct effect on accidents. Individual characteristics affected strain (i.e., adjustment or other problems caused by job stress) instead. The path coefficient between individual characteristics and strain was negative, meaning that sensation seekers were less affected by job stress than others. That finding was opposite to the hypothesis advanced by Landeweerd et al. (1990) that sensation seeking personalities would take greater risks and be more likely to be involved in accidents.

For the individual characteristics that are correlated in the intended direc- tion, the estimate of population effect size (3.7%) may be different for some organizations. Effect sizes for personnel selection were based on the national (USA) average accident rate, and a four-year exposure period, and selection from the top 20% of job applicants. If an organization has a higher initial accident rate and can afford a smaller selection ratio, the impact of selection on accident rates would be larger than the values reported in Table 1. If, on the other hand, an organization has a lower than average accident rate and views an exposure period of less than four years as its criterion of success, effect sizes becomes smaller.

4.2. Technological interventions

Technological interventions can offer powerful opportunities for accident control. Large scale interventions such as automation of a plant are typically introduced to enhance output and would only by chosen by organizations ex- periencing a sharp increase in demand for its products. Re-engineering of a manufacturing facility could be grossly expensive depending on the extensive- ness of the intended changes. New human errors could arise from the change in technology; hopefully the unintended consequences would be temporary.

4.3. Behavior modification

Behavior modification techniques are potentially useful in many industries. From the existing reports, however, it appears that those programs thrive in situations where the opportunities for accidents take on a limited number of forms. If an organization has a variety of technical settings, several versions of the program would be needed.

458

Organizations that use behavior modification programs, particularly those with games and incentives, note that the novelty wears off eventually. New promotions must be generated from time to time to liven up the program. On the other hand, one foundry manager noted that his employees were so involved in their game that injuries were underreported and unattended. The pain of injury was outweighed by the ostricization from coworkers for spoiling their perfect bingo record.

4.4. Near-miss accident reporting

Only two evaluations are known, and the outcomes vary. The technique was originally devised as a method of generating new accident control ideas; the limiting link in the logical chain in the implementation of such ideas. The technique is worth pursuing, nonetheless, as a means of increasing closeness of safety supervision and increasing the amount of feedback to workers regarding the safety of operations. One organization (personal contact, 1991) embellished the idea into an index of subcritical incidents which they report to workers and supervisors alongside the traditional accident indices.

4.5. Comprehensive ergonomics

The strength of these techniques lies in their comprehensive treatment of virtually all conditions that could affect accident rates from the viewpoint of person-situation interactions. At the present time they appear to work best when they are coupled with an efficient diagnosis, with the result that time and effort can be targeted to the largest problems in specific areas.

Unlike some diagnostic strategies and audit programs, employees actively participate in the OHS audit. This technique allows for a psychologically based diagnosis from the employee’s point of view, creates a ZO-minute training op- portunity, and builds commitment on the part of the workforce to programs based on their responses. The idea of building commitment through enpow- erment is common to many organizational theories. Once again, the management must act upon the survey recommendations.

4.6. ISRS

The ISRS is a commercially offered program that is widely used throughout the world. As such, it would be logical to include it in a collection of safety program evaluations. Although there is testimonial support for the program, there was one formal publication on the ISRS (Editor’s note: In a version used in South Africa, which differs from the system in North America and Europe in a number of aspects, among others the independence of the auditors making

459

its rating.) (Eisner and Leger, 1988) which was thorough but unfavorable. I was, therefore, interested in Pringle and Brown’s (1990) abstract to their conference paper (1989) presented to the International Conference on Strategies for Occupational Accident Prevention. The abstract was published along with many other abstracts and articles from the conference in a special issue of Journal of Occupational Accidents. I wrote to New Zealand for the ISRS paper and to Europe for a few others. Everyone responded promptly with pertinent material they had available on their work.

Pringle and Brown (1989) contained tables of accident data without statistical analysis from Australia, some underreported descriptions of what appeared to have been a large effort in North America, and data from a New Zealand plant that mixed ISRS with another type of accident control effort. I wrote to New Zealand to request elaborations of the material in the conference paper, but received no reply. Therefore, I performed some secondary analysis on the available data in order to gain estimates of effect size and to test some comparisons for statistical significance.

There are now three new ideas that have emerged since my original article on my findings. The first is the shift in focus from statistical significance to effect size. The objective of a meta-analysis such as this one is to obtain a population estimate of the size of the impact a program can make. That goal is accomplished by pooling measures of effect size over many samples.

This shift in focus did not alter my interpretation of the South African data or the Australian data, but it did cast a different light on the North American data. Some reasonably good effect sizes were reported here and in my original article. The test of significance for the difference between accident rate reduction for ISRS companies and the North American population did not register a significant effect; I noted the situation as I have now unraveled the matter, which is the second new idea.

The z test that I used to compare two rates was, according to Eisner and Leger (1988) the method recommended by ISRS to calculate the difference in accident rate for two periods of time at one site. (A site needed to show a significant improvement in its accident rate to qualify for certain rating points within the ISRS system.) It appeared to be derivable from McNemar’s t for correlated proportions, although it does not appear in most commonly available statistics books. I have since discovered by working with the formula that it can be derived from McNemar’s t, and that it is very sensitive to variable scaling effects5. The new result is that the significance tests that I reported (1991a, p. 258) should be ignored.

’ The z for two proportions assumes that the rates being compared are units out of 100. Thus,

comparisons involving the USA accident rate of government-reportable cases per 100 person-

years of exposure would work as intended. But, if one were to convert the USA rate to accidents

per million person-hours by multiplying the rate by 5, a different result for z would be obtained.

The correct value of z would be restored by dividing the obtained value by ,/%. As a further result,

the statistic is inappropriate for comparing two changes in rates.

460

The third new idea is the interpretation of the results from the New Zealand company that used the ISRS in combination with a “general ergonomics” program. The program description remains vague, but the application now has its own effect size and a place in Table 1.

4.7. Future directions

The results of this review should be interpreted as a starting point rather than as a final word on the status of any of the programs evaluated herein. Personnel selection techniques have been given the most attention, perhaps because the study of individual characteristics performs a double duty of ex- plaining a more complex system when they are considered in conjunction with other variables and of contributing to our knowledge of system diagnostic techniques. Behavior modification has been well-represented. The next step in studying this technique may be to compare results from variations of the program such as the relative influence of supervisor feedback, job complexity, graphs, and external incentives and games.

Other techniques have surfaced in the literature only once or twice, and their results have been variable. Further concentration on those techniques would be desirable to obtain a clearer estimate of their true effectiveness and reasons for the variability in their effectiveness. In the broader picture, it appears that programs take on an idiosyncratic form when they are applied from one site to another (Turpin and Sinacore, 1991). Program differences are sometimes traceable to social or cultural differences among organizations (S. Guastello and D. Guastello, 1991). Such variations in accident prevention programs are already known to exist, and have not been fully explored or expressed.

What has been accomplished is that we now have a standardized indicator of effect size that can be used with data that researchers have been reporting all along. There are now some generalized estimates available for interesting programs with which to continue to work.

Another set of issues addressed here centered around how to combine results from different studies into generalized estimates. For personnel selection and behavior modification, weighting averages by sample size appeared appropri- ate. There is a precedent for doing so, and it is based on the principle that estimates based on large samples have smaller standard errors than those based on small samples. For technological interventions it was not clear how many people were affected by the program. In those cases, unweighted averages were used.

Finally, one further avenue of research would be to explore other criteria of safety program success and how they interrelate. Veltri (1992) and Oortman Gerlings and Hale ( 1991) have suggested several already.

461

Author’s note

The author would greatly appreciate reprints from any published or unpub- lished evaluation studies of accident prevention programs that have not been included in this review.

References

Arthur, W. Jr., Barret, G.V. and Doverspike, D., 1990. Validation of an information-processing-

based test battery for the prediction of handling accidents among petroleum-product transport

drivers. J. Appl. Psychol., 75: 621-628.

Bjurstrom, L.M., 1989. Priority to key areas and management by results in the national accident

prevention policy. In A. Mital (Ed. ), Advances in Industrial Ergonomics and Safety I, Taylor

and Francis, London, pp. 801-808.

Buchanan, D.J., 1988. Studies on blood alcohol in the workers of a Zambian copper mine. Alcohol

Alcoholism, 23: 239-242.

Cady, L.D., Thomas, P.C. and Karwasky, R.J., 1985. Program for increasing health and physical

fitness of fire fighters. J. Occup. Medicine, 27: 110-114.

Calkins, M.D., 1971, August. A municipal safety program that works. The American City: 67-68.

Carter, N. and Menckel, E., 1985. Near-accident reporting: A review of Swedish research. J. Oc-

cup. Accid., ‘7: 61-64.

Chignell, M.H., Lowenthal, A. and Caspi, A., 1986. The principles of caveat vendor, caveat emptor,

and caveat operator in robotic safety. J. Occup. Accid., 8: 13-23.

Cohen, A., Smith, M.J. and Anger, W.K., 1979. Self-protective measures against workplace haz-

ards. J. Safety Res., 11: 121-130.

Eisner, H.S., 1989. Editor’s note. J. Occup. Accid., 11: 69.

Eisner, H.S. and Leger, J.P., 1988. The International Safety Rating System in South African

mining. J. Occup. Accid., 10: 141-160.

Fellner, D.J. and Sulzer-Azaroff, B., 1984. Increasing industrial safety practices and conditions

through posted feedback. J. Safety Res., 15: 7-21.

Fox, D.K., Hopkins, B.L. and Anger, W.K., 1987. The long-term effects of a token economy on

safety performance in open-pit mining. J. Appl. Behavior Anal., 20: 215-224.

Gebhardt, D.L. and Crump, C.E., 1990. Employee fitness and wellness programs in the workplace. Am. Psychologist, 45: 262-272.

Guastello, D.D. and Guastello, S.J., 1987a. A climate for safety in hazardous environments: A

psychosocial approach. Report No. 2839. Social and Behavioral Sciences Documents, 17: 67.

Guastello, D.D. and Guastello, S.J., 1987b. The relationship between work group size and occu-

pational accidents. J. Occup. Accid., 9: l-9.

Guastello, S.J., 1989. Catastrophe modeling of the accident process: Evaluation of an accident

reduction program using the Occupational Hazards Survey. Accid. Anal. Prevention, 21: 61- 77.

Guastello, S.J., 1991a. Some further evaluations ofthe International Safety Rating System. Safety Sci.,, 14: 253-259.

Guastello, S.J., 1991b. Psychosocial variables related to transit accidents: A catastrophe model. Work and Stress, 5: 17-28.

Guastello, S.J., 1992. Accidents and stress-related health disorders: Forecasting with catastrophe theory. In J.C. Quick, J.J. Hurrell and L.M. Murphy (Eds.), Work and Well-being: Assess-

462

ments and Interventions for Occupational Mental Health. American Psychological Associa-

tion, Washington, D.C.

Guastello, S.J. and Guastello, D.D., 1988. The Occupational Hazards Survey: Second Edition

Manual and Case Report. Milwaukee: Author.

Guastello, S.J. and Guastello, D.D., 1991. How organizations differ: Implications for multisite

evaluation. In: R.S. Turpin and J.M. Sinacore (Eds.), New Directions for ProgramEvaluation,

Vol. 50: Multisite Evaluations. Jossey-Bass. San Francisco.

Guion, R.L., 1965. Personnel Testing. McGraw-Hill, New York.

Hamilton, J.E. and Hancock, P.A., 1986. Robotic safety: exclusion guarding for industrial opera-

tors. J. Occup. Accid., 8: 69-78.

Hansen, C.P., 1988, August. Development and validation of two industrial accident liability meas-

ures. Paper presented to the American Psychological Association, Atlanta, GA.

Harms-Ringdahl, L., 1987. Safety analysis in design - Evaluation of a case study. Accid. Anal.

Prevention, 19: 305-317.

Hayes, W., 1983. Statistics for the Social Sciences (3rd Ed). Holt, Rinehart, Winston, New York.

Hilakivi, I., Veilahti, J., Asplund, S., Sinivuo, J., Laitinen, L. and Koskenvuo, K., 1989. A sixteen-

factor personality test for predicting automobile driving accidents of young drivers. Accid.

Anal. Prevention, 21: 413-418.

Hogan, J. and Hogan, R., 1989. How to measure employee reliability. J. Appl. Psychol., 74: 273-

279.

Ivancevich, J.M., Matteson, M.T., Freedman, S.M. and Phillips, J.S., 1990. Worksite stress man-

agement interventions. Am. Psychologist, 45: 252-261.

Karwowski, W., Rahmi, M., Mihaly, T., 1988. Effects of computerized automation and robotic

safety performance of a manufacturing plant. J. Occup. Accid., 10: 217-233.

Kjellen, U., 1990. Safety control in design. Experiences of an offshore project. J. Occup. Accid.,

12: 49-61.

Landeweerd, J.A., Urlings, I.J.M., de Jong, A.H.J., Nijhuis, F.J.N. and Bouter, L.M., 1990. Risk

raking tendency among construction workers. J. Occup. Accid., 11: 183-196.

Larson, L.D., Schnelle, J.F., Kirchner Jr., R. Carr, A.F., Domash, M. and Risley, T.R., 1980.

Reduction of police vehicle accidents through mechanically aided supervision. J. Appl. Behav-

ior Anal., 13: 571-581.

Lawshe, C.H., Bolda, R.A., Brune, R.L. and Auclair, G., 1958. Expectancy charts II. Their theo-

retical development. Personnel Psychol., 11: 545-559.

Linger, M., 1987. Robot safety. The Swedish Institute of Production Engineering Research,

Stockholm.

Marks, M.L., Mirvis, P.H., Hackett, E.J. and Grady Jr., J.F., 1986. Employee participation in a

quality circle program: Impact on quality of work life, productivity, and absenteeism. J. Appl.

Psychol., 71: 61-69.

Malm, T. and Suominen, J., 1990. Intelligent safety systems provide production adapted safety.

J. Occup. Accid., 12: 150.

McAfee, R.B. and Winn, A.R., 1989. The use of incentives/feedback to enhance work place safety: A critique of the literature. J. Safety Res., 20: 7-19.

Menckel, E. and Carter, N., 1985. The development and evaluation of accident prevention rou-

tines: A case study. J. Safety Res., 16: 73-82. Mintz, A. and Blum, M.L., 1949. A re-examination of the accident proneness concept. J. Appl.

Psychol., 33: 195-21. Mohr, D.L. and Clemmer, D.I., 1989. Evaluation of an occupational injury intervention in the

petroleum drilling industry. Accid. Anal. Prevention, 21: 263-271. Montag, I. and Comrey, A.L., 1987. Internality and externality as correlates of involvement in

fatal driving accidents. J. Appl. Psychol., 72: 339-343.

Murphy, L.R., 1984. Occupational stress management : A review andappraisal. J. Occup. Psychol.,

57: 1-16.

Murphy, L.R., Dubois, D. and Hurrell, J.J., 1986. Accident reduction through stress management.

J. Business Psychol., 1: 5-17.

Normand, J., Salyards, SD. and Mahoney, J.J., 1990. An evaluation of pre-employment drug

testing. J. Appl. Psychol., 75: 629-639.

Oortman Gerlings, P.D. and Hale, A.R., 1991. Certification of safety services in large Dutch in-

dustrial companies. Safety Sci., 14: 43-59.

Pirani, M. and Reynolds, J., 1976, February. Gearing up for safety . or these boots were made for

wearing. Personnel Management: 25-29.

Pringle, D.R.S. and Brown, A.E., 1989, September. International Safety Rating System: New

Zealand’s experience with a successful strategy. Paper presented to the International Confer-

ence on Strategies for Occupational Accident Prevention, Stockholm.

Pringle, D.R.S. and Brown, A.E., 1990. International Safety Rating System: New Zealand’s ex-

perience with a successful strategy. J. Occup. Accid., 12: 41.

Rundmo, T., 1992. Risk perception and safety on offshore petroleum platforms - Part II: Perceived

risk, job stress and accidents. Safety Sci., 15: 53-68.

Saarela, K.L., 1989. A poster campaign for improving safety on shipyard scaffolds. J. Safety Res.,

20: 177-185.

Saarela, K.L., 1990. An intervention program utilizing small groups: A comparative study. J. Safety

Res., 21: 149-156.

Saarela, K.L., Saari, J. and Aaltonen, M., 1989. The effects of an informational safety campaign

in the ship building industry. J. Occup. Accid., 10: 255-266.

Saari, J., 1990. On strategies and methods in company safety work: From informational to moti-

vational strategies. J. Occup. Accid., 12: 107-118.

Sanderson, L.M., Collins, J.W. and McGlothlin, J.D., 1986. Robot-related fatality invo1vingaU.S.

manufacturing plant employee: Case report and recommendations. J. Occup. Accid., 8: 13-23. Sims, M.T., Graves, R.J. and Simpson, G.C., 1984. Mineworkers’ scores for the Rotter Internal-

External Locus of Control scale. J. Occup. Psychol., 57: 327-329.

Sjostrom, H., 1989, September. Production adapted safety systems. Paper presented at the Inter-

national Conference on “Strategies for Occupational Accident Prevention”, Stockholm.

Sjostrom, H., 1990. Production adapted safety systems. J. Occup. Accid., 12: 149.

Smith, M.J., Anger, W.K. and Uslan, S.S., 1978. Behavior modification applied to occupational

safety. J. Safety Res., 10: 87-88.

Sulzer-Azaroff, B., Loafman, B., Merante, R.J. and Hlavacek, A.C., 1990. Improving occupational

safety in a large industrial plant: A systematic replication. J. Organizational Behavior Man-

agement, 11: 99-120.

Sugimoto, N. and Ikeda, H., 1990. The construction of a radar-type safety device to complement

human vision safety production. Ergonomics, 33: 735-762.

Turpin, R.S. and Sinacore, J.M. (Eds.), 1991. New Directions for Program Evaluation, Vol. 50:

Multisite Evaluations. Jossey-Bass, San Francisco.

Veltri, A., 1992. Evaluating the safety functions: A conceptual model. J. Safety Res., 23: 27-38.

Webb, G.R., Redman, S., Henrikus, D.J., Kelman, G.R., Gibberd, R.W. and Sanson-Fisher, R.W.,

1991. The relationships between high-risk and problem drinking and the occurrence of work

injuries and related absences. Technical Report. University of Newcastle, Faculty of Medicine,

Newcastle, New South Wales, Australia. Wuebker, L., 1986. Safety locus of control as a predictor of industrial accidents and injuries. J.

Business Psychol., 1: 19-30.

Zohar, D., 1980. Safety climate in industrial organizations: Theoretical and applied implications.

J. Appl. Psychol., 65: 96-102.