electrical disaster recovery rev 3

13
Revision 3, 2016 1 Utility Electric Power Challenges Related to a Level 3 Trauma Hospital’s Electrical Vault Failure, its Recovery Operations, Regulatory Challenges and the Evolving Challenges of Temporary Generation OR The Hospital, the Utility, the Explosion and the Aftermath Mike Moore Walker Engineering 1505 W. Walnut Hill Lane Irving, Texas 75038 USA [email protected] Abstract - This case study will be a chronicle of events that were encountered through on site interactions with qualified electrical workers, executives, corporate managers, and safety professionals while interacting with a large nationally recognized medical management corporation and one of its hospital locations in the Dallas, Texas Area. The medical facility experienced a complete electrical failure promulgated by the actions of an unqualified electrical worker that resulted in an arc flash/blast incident in one of four hospital electrical utility vaults. The event severely burned an experienced electrical worker, an experienced utility worker, a hospital maintenance director as well as a general maintenance worker. The goal of this presentation is to communicate the specific details, accounts and challenges that allowed the incident to occur and the human failures, poor safety related work practices and cultures that were exhibited during this project. These poor safety related work practices and cultures seem to plague many industrial facilities, utilities and large commercial enterprises even to this very day even though the ability to calculate and identify electrical hazards, communicate them, mitigate them and protect the electrical worker are easier to manage today than ever before. There were several key strategies employed by the disaster recovery personnel to counter the site hazards that included dealing with exposed energized conductors, poor electrical equipment grounding, inclement weather, human limitations, limited qualified recovery personnel, site inner and inter-company politics, as well as the disaster sites management’s limited knowledge about electrical safety, their electrical system, disaster preparedness and limitations of liability when it comes to forcing contractors to work in a hazardous condition. I. WEEKEND FIRE INJURES THREE WORKERS AT A LOCAL HOSPITAL From: The News Headlines on May 4 th , 2010 An accidental weekend explosion at a local hospital burned three hospital workers and caused $300,000 in damage to an outbuilding. The local firefighters arrived at the hospital shortly after 2 p.m. Sunday and quickly extinguished the fire in a one-story structure near the hospital's loading docks, the Fire-Rescue spokesman said in a news release today. Witnesses said the explosion was sparked by an electrical short created when a contractor attempted to transfer power, the spokesman said. Three men were injured in the explosion, he said. Two victims were taken to a burn unit at a local hospital by air ambulance, and the third was treated at the hospital where the event occurred. The extent of their injuries is unknown. II. THE BLAST A. Events Leading Up to the blast On May 2nd, 2010 just after midnight an electrical testing contractor had completed the replacement of a 4000 Amp fused low-voltage bolted pressure switch on side Aof the hospitals switchgear being fed from the West Vault of the local utility. Once completed the electrical testing contractors technicians communicated to their customer (the hospital) that the site was clear of their personnel, that all safety grounds and locks had been removed from the equipment in the facility and that the hospital was clear to discuss energizing the site with the utility. The electrical testing contractor’s personnel withdrew from the power building to the parking lot adjacent to the main utility vault and switchgear room. The utility worker entered the vault to begin the process of energizing the incoming 15 kV switch while close behind him were three other workers or hospital staff

Upload: mike-moore

Post on 13-Feb-2017

35 views

Category:

Documents


5 download

TRANSCRIPT

Revision 3, 2016 1

Utility Electric Power Challenges Related to a Level 3 Trauma

Hospital’s Electrical Vault Failure, its Recovery Operations, Regulatory

Challenges and the Evolving Challenges of Temporary Generation

OR

The Hospital, the Utility, the Explosion and the Aftermath

Mike Moore

Walker Engineering

1505 W. Walnut Hill Lane

Irving, Texas 75038

USA

[email protected]

Abstract - This case study will be a chronicle of events

that were encountered through on site interactions with

qualified electrical workers, executives, corporate managers,

and safety professionals while interacting with a large

nationally recognized medical management corporation and

one of its hospital locations in the Dallas, Texas Area. The

medical facility experienced a complete electrical failure

promulgated by the actions of an unqualified electrical worker

that resulted in an arc flash/blast incident in one of four

hospital electrical utility vaults. The event severely burned an

experienced electrical worker, an experienced utility worker,

a hospital maintenance director as well as a general

maintenance worker. The goal of this presentation is to

communicate the specific details, accounts and challenges

that allowed the incident to occur and the human failures,

poor safety related work practices and cultures that were

exhibited during this project. These poor safety related work

practices and cultures seem to plague many industrial

facilities, utilities and large commercial enterprises even to

this very day even though the ability to calculate and identify

electrical hazards, communicate them, mitigate them and

protect the electrical worker are easier to manage today than

ever before. There were several key strategies employed by

the disaster recovery personnel to counter the site hazards that

included dealing with exposed energized conductors, poor

electrical equipment grounding, inclement weather, human

limitations, limited qualified recovery personnel, site inner

and inter-company politics, as well as the disaster sites

management’s limited knowledge about electrical safety, their

electrical system, disaster preparedness and limitations of

liability when it comes to forcing contractors to work in a

hazardous condition.

I. WEEKEND FIRE INJURES THREE WORKERS AT A

LOCAL HOSPITAL

From: The News Headlines on May 4th, 2010

An accidental weekend explosion at a local hospital burned

three hospital workers and caused $300,000 in damage to an

outbuilding. The local firefighters arrived at the hospital

shortly after 2 p.m. Sunday and quickly extinguished the fire

in a one-story structure near the hospital's loading docks, the

Fire-Rescue spokesman said in a news release today.

Witnesses said the explosion was sparked by an electrical

short created when a contractor attempted to transfer power,

the spokesman said. Three men were injured in the explosion,

he said. Two victims were taken to a burn unit at a local

hospital by air ambulance, and the third was treated at the

hospital where the event occurred. The extent of their injuries

is unknown.

II. THE BLAST

A. Events Leading Up to the blast

On May 2nd, 2010 just after midnight an electrical testing

contractor had completed the replacement of a 4000 Amp fused

low-voltage bolted pressure switch on side “A” of the hospital’s

switchgear being fed from the West Vault of the local utility.

Once completed the electrical testing contractor’s technicians

communicated to their customer (the hospital) that the site was

clear of their personnel, that all safety grounds and locks had

been removed from the equipment in the facility and that the

hospital was clear to discuss energizing the site with the utility.

The electrical testing contractor’s personnel withdrew from the

power building to the parking lot adjacent to the main utility

vault and switchgear room. The utility worker entered the vault to

begin the process of energizing the incoming 15 kV switch while

close behind him were three other workers or hospital staff

Revision 3, 2016 2

members observing the utility worker. Two of these hospital

workers entered the vault while the third waited just outside the

door. Moments later a loud explosion occurred and the entire site

went dark.

B. The Rest of the Story

The utility worker involved tried to energize the vaults

main 15 kV oil filled circuit breaker several times. This is

typically performed remotely through a network of

communications and controls. The circuit breaker would “trip

free” and reset to the charged position. After several failed

attempts to close the circuit breaker the utility worker noticed

that an over-current relay was flagged and what was blocking

the circuit breaker from closing. The utility worker removed

the sliding disconnect paddle from beneath the protective

relay, isolating it from the trip circuit. The utility worker tried

to operate the circuit breaker one more time manually. The

results were catastrophic. All 4 workers in the vault were

medically evacuated via helicopter to another local trauma

center that was equipped to deal with electrical burn injuries.

The hospital’s emergency generators did not all come on line

due to some transfer switching issues, but eventually with the

testing contractor’s support the “Life and Safety” power was

restored to a limited capacity as well as some power was

restored to the critical facilities on the hospital campus. One

of the workers was treated and released quickly after the

events. This worker became the contact for all disaster

operations at the hospital.

After the smoke cleared the carnage was indescribable. The

oil filled circuit breaker had exploded covering the room in

burning mineral oil from the 15kV switch. The overhead 480

VAC distribution bus and thermal plastic insulators were

covered in burned mineral oil, soot and soda. The fire

extinguishers used to put out the fire had left a nasty mess that

would later be a challenge to contend with. The utility worker

suffered at least a 60% body burns of 2nd and 3rd degree. Both

of the unauthorized and unqualified hospital workers who just

happened to be in the utility vault at the same time suffered

2nd and 3rd degree burns as well. The two hospital workers

were administrators and maintenance management personnel

nowhere even qualified to be where they were. The third

worker who was standing in the doorway to the utility vault

was the campus maintenance superintendent and only

received small areas of first and second degree burns. None

of the utility or hospital workers were wearing any arc-rated

PPE. The utility worker was wearing a standard issue FR only

rated clothing, a hardhat and safety glasses. The hospital

workers were wearing poly-blend business casual clothing, no

safety glasses or hard hats. It has never been communicated to

the hospital or to the electrical testing contractor why the oil

circuit breaker failed. Even to this day the discussion of the

events is evasive and a point to “not to be discussed”.

The next several days had to be extremely challenging for

the utility company. One of their own long time workers as

well as two hospital workers were now in a regional burn

center with little or no way of communicating what had

happened and what factors lead up to the incident. The utility

was on an information lock down about the events of the

incident and were not delivering any commitments, at all to

anyone, even to the hospital about recovering their electrical

system. The entire site was at a virtual standstill and nobody

seemed to have a direction of what needed to happen next or

maybe they did!

As the next day progressed a frenzy of utility construction

personnel mobilized to the site to repair the damage to the

utility vault and its equipment. The utility’s electrical testing

personnel were testing the transformers, medium voltage

power cables, the low voltage riser cables and overhead bus

work to determine their serviceability. The utility construction

personnel replaced the charred remains of the oil circuit

breaker with a new vacuum circuit breaker. All control

indication and station metering wiring harnesses between the

West & East vaults was replaced, as well as heavy cleaning

and painting of the balance of the electrical equipment. The

damaged utility equipment that left the hospital site did so

covered with tarps cloaked with a thick veil of secrecy. See

figure 1 - 3

Figure 2 – New utility vacuum switch, control transformer,

control cabling and riser cables in the vault

Revision 3, 2016 3

Figure 2 – Outside view of the melted utility vault

ventilation system that was damaged from the arc blast.

Figure 3 – New fan installed on the utility vault ventilation

system & new medium voltage conductors that feed from

“Vault B”.

By the end of the first day the utility had completely

replaced all of the damaged electrical equipment, tested the

balance of the questionable electrical equipment and made

ready the site to be energized, as long as you didn’t need in

the vault. A simple piece of caution or barrier tape that

crossed what was left of the two doors leading into the vault.

The communication to the hospital was limited to phone calls

only to communicate that they “could repair their bus

whenever they wanted to”. In short; the utility poorly

communicated to the hospital what they were allowed to do or

how to interact or tie into the utilities equipment and chose

never to come to the site and work with the hospital staff.

The testing contractor took the initiative and acted as the

catalyst to begin the communication process between the

temporary power contractor, the hospital and the utility.

Repair, restoration and communication were finally moving

forward.

III. The Hospital Electrical Recovery Project

The electrical testing contractor that had been on site since

the initial disaster event immediately started assisting the

hospital to place the site in an “electrically safe condition”.

That meant locking out all of the four (4) the incoming 4000

Amp bolted pressure switches fed from the “A Vault” that

were associated with the failure, and limiting access into the

hospitals power house and the electric utilities vault notifying

local authorities that the site was on temporary power and that

no utility service existed.

Media arrived on the site looking for comment. They

were lost on who to talk too or where to go. The testing

contractor asked that they come at another day in time when

the proper hospital and utility staff were present to address

their needs and questions.

As the first day drew to a close, temporary power assets

arrived on the site and safety controls were set into place

many local electrical contractors came to the site for a piece

of the action. Some of the electrical contractors wanted the

opportunity to make a quick buck and could care less about

the sensitive nature of the situation. The hospital had already

chosen its support team immediately after the event. That

team consisted of an electrical testing contractor and an

electrical construction contractor. The utility had yet to

include itself into restoring power to the hospital.

A. Temporary Power

The hospital allowed the testing contractor top start

working with their nationally contracted emergency rental

generation equipment provider. The rental electric equipment

supplier and the testing contractor worked nearly 30 hours

straight to get the hospitals electrical system up to some

reasonable level of reliability, but the hospital wanted more

assurances that there would be no more unplanned outages.

The rental equipment supplier had originally positioned 3

MW of generation in the first days after the disaster, but as

the reliability requirements of the hospital grew as did the

capacity of temporary generation which eventually to 6 MW

on-line with an additional reserve of capacity of 3 MW of

generation fueled, cabled in and ready to go when needed.

This configuration offered the hospital redundancy and back-

up if one of the other generators failed. See figures 4 - 6

Revision 3, 2016 4

Figure 4 – “Switch B” that was fed from the utility “Vault A”

via bus duct, this switch was back-fed from a rental generator

Figure 5 – Day 1 generator set-up with 3 MW of generation.

Figure 6 – Day two with 3 MW of generation capacity on line

and an additional 2 MW of stand-by generation.

If needed, the hospital now had the ability to transition to

the spare generators, but it would take some time and

additional outages to tie in the complex network of generator

cables into the main wiring harness. Additionally all these

generators had maintenance requirements that required

downtime and short start time windows that would cause

small outages. The thought of these potentially outages and

delays were unacceptable to the hospital. Several forced

outages occurred on the second and third days of the project

due to switching, routing fueling and filter changes, but the

hospital refused to allow or accept these challenges and

constantly complained that we were impacting business as we

added reliability; meaning that surgeries and procedures were

delayed and interrupted during the generation events. No

matter how hard we tried to convince them how vulnerable

they were and the challenges we had; it was business as usual

for the hospital.

Additional switching equipment and generators were

located, installed at the site to limit outages related to

maintenance routines and enhance future reliability related tie

in’s, but two more outages would be required for a final tie in

to limit any additional outages See figure 7.

Figure 7 - Day 1 cable harness that fed switchgear. This

system grew more complex as the project went on.

The operation and maintenance of these generators were

extremely critical to the reliability of the site. The back-up

generator’s ability to start and energize the emergency circuits

was even more critical. The failure of any one of the three

generator systems placed in service would leave portions of

the hospital in the dark or could possibly create a life

threatening scenario in the critical care unit of the hospital.

The generators had two critical maintenance routines

that could not be missed or deferred. The routines involved

fuel and air filtration and if any of the two routines were

missed, it would actually force the generators into a shutdown

Revision 3, 2016 5

mode. The need to perform these maintenance routines were

discussed with the hospital on several occasions. The hospital

administration refused to entertain the idea of taking short

outages to perform these services. Even with a back-up plan

to shed load from the generator to be maintained and to

change its status through an open transition scheme to an

additional stand-by generator, the hospital still refused to

allow the mandatory maintenance process stating “that it was

too risky” and that they “cannot take another outage no matter

how small or short it was”. Therefore the only way these

filters could be changed was during unplanned outages

basically allowing the generator to enter a self-shutdown

mode. These unplanned and unscripted outages happened

routinely every day and created chaos for the hospital and a

few yelling incidents by the staff of the hospital to the

contractors recovering the site. Other than filtration outages,

these generators ran 100% event free for nearly 3000 hours

and consumed close to 35,000 gallons of diesel fuel.

B. Electrically safe

The first order of business with any electrical project is to

secure a copy of an engineered one line diagram of the

existing electrical system. This is even more important when

dealing with a failed electrical distribution system that has

injured workers, has created the possibility of litigation and

will mostly likely create a reason for an OSHA compliance

officer to investigate the site. There was a need for a complete

site one line drawing to accurately size loads and calculate the

power requirements for each building In this instance, an

accurate one line of the site was never found, only small

pieces of various expansions that had happened over the

years. The lack of the one line drawing created a need for a

detailed site temporary power plan and a method to

communicate the changes to the system as they were made.

This temporary power plan would let the hospital and other

contractors know exactly where the temporary power

resources were located and tied in. As the project progressed,

miles and miles and miles of temporary power cables were

strewn across multiple buildings, across parking lots, through

tunnels, through hallways and even through windows and up

the sides of several buildings on the campus. This created

some potentially hazardous situations where hospital

personnel and contractors would have had to walk on the

energized 480VAC cables. One solution was to build

temporary ramps and elevated walkways on top and across

the cables. See figure 8 & 9

Figure 8 – Ramp built over the generator cables going into

the main switchgear room.

Figure 9 – Ramps built over the generator cables to allow for

emergency vehicle access.

The electrical systems bonding and grounding was a

nightmare. Years and years of unqualified and inexperience

electricians had left panel bonding jumpers in place creating

downstream bonding. This installation issue created havoc

with the ground fault relays on the temporary breakers and

generator main circuit breakers. The introduction of the

generator grounds to the facilities grounding system created

large amounts of circulating ground currents system wide.

These circulating currents heated bonding jumpers to high

temperatures causing the connections to smoke, isolated

grounding panels to trip and to eventually fail. Several runs of

the facilities cables failed in large manholes that were filled

with 12 feet of water from recent rains. Cable failures on top

of grounding issues with the added pressure for uptime by the

hospital was becoming a challenge. After two days of

troubleshooting, intersecting cables and isolation of other

cables at various points in the campus electrical system the

grounding problems were somewhat temporarily resolved.

By the time the temporary power scheme was completed

on the third day the entire hospital was now isolated from the

electric utility provider and operating 100% on rental

temporary power generation with a double ended closed

Revision 3, 2016 6

transition system for redundancy. The hospital; still moved

forward with a business as usual mindset with no regard to the

limitations that had been placed on the electrical

infrastructure, missing support personnel and the fact that

there was no guarantee that the utility would allow anyone to

repair the service entrance anytime soon. Day three was soon

to come as nightfall went away and the sun rose in the

gorgeous Texas Spring sky.

C. The Service Entrance

Since the electrical testing contractor was already on-site

and contracted to support switchgear repair, the hospital

maintenance staff asked them to evaluate the feasibility of

rebuilding all four service entrance transition pieces from the

utility vault that went into each hospital power room. Access

to work on the customer owned equipment inside the utility

owned vault had its own challenges. Early discussions with

key management of the utility company, the hospital and the

electrical testing contractor yielded a small four-day window

for the electrical testing contractor to remove the three

customer-owned service entrance transition pieces from the

utility vault, rebuild them, set them in place and reconnect

them into the utility’s distribution system.

Day (3) three started out as a major challenge. The utility

had not communicated its plan to allow contractors into the

vault to its own workers. Several vice presidents, managers

and utility workers arrived at the site to evaluate the damage

and the repairs to the vault. These workers would “run us out”

of their vault and tell us not to re-enter. There were multiple

times where we had to demobilize due to poor utility

communications. Additional generator related outages,

rainstorms, hailstorms and hospital meetings created

leadership challenges for the very small qualified crew to

perform very niche work in a very critical and extremely

dangerous work environment. At this point at least once an

hour to every two hours a challenge occurred that impacted

and slowed the recovery processes and at times stopped all

work. The initial communication to the testing contractor was

that the transition sections were remain in place as is. The

utility company communicated that since the service was only

480 VAC equipment that “just blowing them out would

suffice”. Additional educational discussions concerning long

term reliability on behalf of the hospital persuaded the

hospital management company’s executives to pressure the

electric utility to allow the complete removal and repair of all

of the transition sections. The recovery tasks of the

customer’s cables and bus duct associated with each transition

piece was even more of a challenge than originally planned.

Two of these transition pieces were connected to 3000 A bus

duct, the third was connected to forty (40) 500MCM THHN

cables. The three service entrance transition pieces from the

utility vault were severely burned and required complete

disassembly and cleaning, re-plating and reinsulating. See

figures 10 through 12

Figure 10 – Transition section “Feeder C” and “Feeder F”

equipment after the arc blast and covered with soda from the

fire extinguishers. “Feeder C” is the transition section fed by

cables on top of “Feeder F”.

Figure 11 – Bus duct “Feeder D” transition after the arc blast

and covered with soda from the fire extinguishers

Figure 12 – Bus duct “Feeder F” transition after the removal

from the vault. Note the ingress of soot and soda.

Revision 3, 2016 7

OSHA made their appearance to the disaster site on the

third day as they would with any incident that injures three or

more workers. The OSHA compliance officer had a very

professional approach to their line of questioning. The OSHA

representative evaluated the overall site for any existing

hazards, thoroughly questioning the testing contractors about

their knowledge of the events about the prior days and the

evening of the blast, the current condition of the electrical

distribution system, the damaged electrical equipment, the

temporary power installation contractor and qualifications,

the plans for the recovery of the hospitals electrical system

and if there were any safety challenges that were being

experienced on site. The electric utility had no representation

at the disaster site during OSHA’s review, the utility vault

was open, unsecure and exposed to the public. The hospital

staff did not seem to understand the gravity of the situation

and could not properly answer the questions being asked of

them by the OSHA compliance officer. The OSHA

compliance officer left after just less than an hour on the site

and with more questions than answers. OHSA documented

the events on their enforcement site. See figure 13

Figure 13 – OSHA documented case of the injuries of 4

workers, not three as reported by the news and other sources.

In addition to the concerns about the removal of the bus

duct connectors the utility refused to allow any of the site

contractors to clean any of the 480 VAC complex weave of

overhead station bus work, stand-off insulators, bracing and

barriers that supported the hospital. To this day it has never

been cleaned or maintained. To this day the utility accepted

Insulation Resistance readings that were left in electric

service at 30K Ohms, ANSI/NETA MTS states that it should

be 100M Ohms at a minimum. See figure 15

Figure 15 – The utility vault’s 480 VAC bus work that

remains in service to this day and still charred and covered

with soda from the fire extinguishers. Bus duct “Feeder D” is

far left, “Feeder C” & “Feeder F” are located to the far right

and mounted one on top of another.

The transitions sections required 6 electricians working

about six to eight hours each to remove from the utility vault.

Once these were removed from the site they were transported

to the electrical testing contractor’s electrical repair facility

for a complete disassembly, cleaning and remanufacture to

“like new”. All copper energized parts were bead blasted to

bare copper and polished. All connections were re-plated with

6 MILS of tungsten silver plating, all insulating boards,

barriers and stand-off bushings were replaced, all exterior

metal and bracing were sand blasted and powder coated with

ANSI 61 Grey epoxy paint, and all hardware was replaced.

Mylar insulating tapes and sheets replaced the factory

fluidized epoxy bus coating. After an around the clock

process the components were re-assembled, electrically tested

and ready for delivery in two-and–a-half days. See figures 14

through 17.

Revision 3, 2016 8

Figure 14 – Completely remanufactured bus transition section

that fed bus duct “Feeder D”

Figure 16 - Completely remanufactured bus transition section

that fed bus duct Bus duct “Feeder F”

Figure 17 – Completely remanufactured “Feeder C” bus

section.

The two 3000 Amp bus ducts sustained heavy smoke and

blast damage. The soot and soda material from the fire

extinguishers used to put out the fire and the extinguishers

high pressure nozzle pushed contaminates well inside two of

the ten foot sections of the bus duct on “Feeder D” & “Feeder

F”. The conduits for “Feeder C” sustained heavy soot and

soda contamination as well. There were signs of cable

insulation failure and shorting of conductors at the entrance to

the conduits. See figure 18 & 19.

Figure 18 – Transition section “Feeder C” and “Feeder F”

equipment as seen from the hospital equipment room after the

arc blast. Evidence of the blast is visible, note soot on the

sides of the bus duct, and the arcing on the lower left corner

of the transition box for “Feeder C”

Though this was only 480VAC equipment, the dissimilar

compounds at the connections combined with moisture had

already created the beginning of an acidic process that turned

the silver plating black and started the “whiskering” process.

The copper had turned from a dark brown and was slowly

turning to a bright green as well. Soot from the blast was

found 80ft to 100ft feet away in opposing conduits not even

related to the blast.

Figure 19 – Transition section “Feeder D” equipment as seen

from the hospital equipment room after the arc blast.

Evidence of the blast is visible, note soot on the sides of the

bus duct.

Four each of the ten foot sections of the hospitals bus duct

for “Feeder D” & “Feeder F” had to be disassembled and

cleaned while on site. Soot traveled inside both sections of

Revision 3, 2016 9

bus duct on each feeder. The bus duct compression plates

were buffed to bare copper and polished with the connection

points being re-plated in the field with 2 to 3MILS of

tungsten silver. The insulating plates were cleaned with

denatured alcohol and were determined to be suitable for

continued service.

The third and fourth service entrances “Feeder C”

connected forty 500MCM conductors to an older installation

in the hospital equipment room and presented the greatest

repair challenge. How do you replace eighty conductors that

are still connected to the rear of a section of the switchgear

directly above the energized common bus when power

outages are forbidden? The hospitals answer to all involved

was; “you do it energized” or at least that was the hospital’s

original intention.

D. “Doing it Energized”

The hospital had repeatedly requested and directed that

“there are to be no more outages taken for any reason

whatsoever”. This requirement was repeated in each and

every operations and safety meeting the contractors had with

the hospital staff. The electrical testing contractor and its

electrical contractors did not relish the idea of a demolition of

conductor’s job energized. Cutting away and removing cables

directly above energized 3000 Amp and 4000 Amp common

switchgear buses. Performing this task while the bus was

energized was extremely risky and of the highest hazard, but

someone had to convince the hospital that it was not in their

best interest to risk anyone else’s life to perform an unsafe

task especially since OSHA had just left the site and three

workers were still fighting for their life in the hospital. Even

though it was communicated verbally that “You can’t ask

someone to perform an unsafe task, much less make them for

monetary reasons”, they still refused to submit.

The electrical testing contractor’s plan was to develop an

energized work strategy for two reasons;

1) To acknowledge the risk and the hazards to the

hospital, to place that burden of risk on them and to remove it

from the electrical testing contractor; it’s their electrical

hazard, they should assume the risk

2) To develop a plan to perform the task if they

were forced to execute the tasks while the equipment was

energized.

The plan had several key steps.

Step 1 was to identify the hazards.

The shock hazard was 480 VAC fed through

temporary cables directly from the output terminals of an

insulated case circuit breaker from a 2 MW generator. The

case was made to protect the workers from a possibility of

shock by isolating the energized bus by securing voltage rated

blankets in place with clips and tapes manufactured

specifically for the operation and task. The arc flash/blast

hazard was calculated at 38 cal/cm2. Though below 40

cal/cm2 it was considered high enough to bring into question

the reliability of the unmaintained generator main circuit

breaker, the maintenance cycle of the breaker and any

established performance data from the maintenance program.

No maintenance data existed and there was no knowledge of

when the last tests were performed on that breaker.

Additionally, performing the cutting and removal of cables

while wearing arc rated clothing introduced additional

concerns, especially with the wearing of Class 0 gloves and

the need for fine motor skills to turn Allen head bolts, all

while positioned above energized bus.

Step 2 was to identify the most qualified and experienced

electrical installers.

The electrical contractor had some extremely

qualified personnel, but did not have the skill sets needed to

perform the high hazard task. The site manager and safety

manger selected key experienced and trusted personnel. They

asked and documented a barrage of questions of each

electrical worker about the company safety practices,

company safety policy, electrical theory, their individual

families, arc flash knowledge as well as key points about the

task at hand as they understood it. The worker selection

process identified the most qualified workers for the task.

Step 3 was to develop a written procedure.

There had to be a written procedure for limiting non-

essential and un-qualified personnel egress into the building.

This included hospital personnel, generator support

personnel, utility personnel, and even the contractor’s support

personnel. There was also a procedure developed to install

insulating blankets over the common bus with specific

mounting locations of the hardware to be used to secure it.

The most in-depth procedure was the cutting and extraction of

the cables from the top side of the main switch located just

above the common bus. The procedure detailed what to cut,

how to cut it, the number of personnel involved in the task

and their position relative to the equipment they are working

on.

Step 4 was to notify the personnel.

The contractor’s energized safe work plan required that all

parties be notified prior to the performance of an energized

task. This included their senior management & ownership, the

electrical testing contractor’s management and the hospitals

staff and management. The most personal or key part of the

plan was to notify the family of the selected workers about the

risk they were taking on in behalf of the hospital and offer

them a voice in the matter. This required the contractor’s

manager to call and discuss the plan with the spouse or

family, as well as the worker communicating his willingness

to participate verbally to his spouse.

Revision 3, 2016 10

Step 5 was to develop an emergency response plan

The development of a disaster response plan was critical

in performing a task such as this. If medical attention was

needed, the hospital would most likely not be able to support

it, as their power would be out. The plan included emergency

first responders already on site, the identification and

notification of a trauma center, a “first call” plan, as well as a

plan on what to do to get the site restored if the worst were to

happen.

Step 6 the final step.

This highly elaborate step specifically required that the

workers, the workers supervisors and the electrical testing

contractor management all understood the task, its hazards

and the gregarious amount of risk involved. The plan

required the hospital management to sign a document

accepting these facts and acknowledging some key facts:

1. They had asked a contractor to perform a very

risky task on their electrical hazards.

2. They had a contractor that had qualified its

workers and what workers would be performing

what task.

3. What the limitations were for the workers –

PPE, training & rate of survival if an electrically

related incident was to occur.

4. Scales of responsibility, care and liability were

communicated in writing.

Basically the hospital; had to sign documentation that they

were responsible, liable and “could go to jail” if something

went wrong.

Even though there were a tremendous amount of

preparations made for the task of the energized demolition of

the existing cables, a new plan would have to be written for

the pulling and installation of the new cables as well. The

energized demolition work plan was never signed by the

hospital, as it realized it could not accept the liability

involved for such a hazardous task. At this point, the hospital

agreed to a twelve-hour outage for the end of the week.

E. The Outage

The outage was scheduled for a Friday night and was to end

on Saturday night at midnight. This was a full 24-hour outage,

just shy of a week from the initial event. The outage was

uneventful for the most part. The electrical testing contractor

and its contractors installed the remanufactured customer-

owned transition pieces into place and tied two of them into

3000 Amp bus connectors. The electrical testing contractor

performed insulation resistance testing phase-to-phase and

phase-to-ground and recorded satisfactory test results.

Contact resistance testing was performed end-to-end and

phase-to-phase from the utility side of the transition piece to

the line side of each service entrance switch and again

recorded satisfactory test results. The third customer-owned

transition piece was installed and thirty runs or about 2500 ft.

of 500 MCM cable were installed, terminated and electrically

tested with satisfactory test results. During this time the

electrical testing contractor assisted the generator support

personnel in removing all the temporary cables, while

maintaining all associated hospital-owned service entrance

switches and the placement of covers back on the switchgear.

Though the tasks went down to the wire time-wise, they were

error free, injury free and, at completion, the system was

deemed safe to energize.

F. Energizing

The electrical testing contractor worked closely with the

utility so they could verify the test results. The utility was not

satisfied with the testing contractor’s test data and chose a

different methodology of test to verify that it was safe to

energize the system. The utility placed a 6 A glass fuse across

the open bus terminals between the utility and the hospital

and energized each and every phase for a few seconds. A

blown fuse meant it is not safe to energize. This process took

about two hours and prolonged the outage beyond the

hospitals planned 24 hours. Once the equipment was deemed

safe to energize by the utility personnel, they began working

with the network management personnel to “close the system

in”. At this time the large fuses between the hospital and the

vault had not been installed. A utility worker who was

donning his arc-rated suit for the very first time to perform

this task made a comment that “he had owned this suit for a

year and never wore it”. That was evident, since it still had

the plastic extra-large sticker on the pant leg and plastic

manufacturer’s labels hanging off the jacket. These stickers

were never removed from the suit while in use by the utility

worker. After some quick discussions between the electrical

testing contractor and the utility workers management staff,

they were finally convinced by the testing contractor that

installation of 3000 Amp fuses while energized may not be

the safest method; the fuses were installed de-energized. The

vault was formally energized early Sunday morning around

6:30AM. Prior to the break of daylight and just a few hours

past the one-week anniversary of the original event all utility

power was restored to the hospital. See Figures 19 & 20.

Revision 3, 2016 11

Figure 19 - Restored Low Voltage Vault Feeders to Overhead

bus with new lighting.

Figure 20 - Repaired Low Voltage Bus ducts in the hospital

power room

G. Regulatory Enforcement (Updated 9/2016)

OSHA performed their investigation at the disaster site.

Their summary and narration of events stated that; On May 2,

2010, Employee #1 and Employee #2 of Medical Hospital and

Employee #1 of Utility, were burned due to an arc flash and fire

while Employee #1 of Utility was opening a phase potential

switch from a “Load Break Switch” of 13,200 volts phase to

phase to perform the transfer switch. All Employees were

hospitalized and treated for their burns and scalds due to the

flash. No mention of the 4th worker from the hospital was made

by OSHA since an actual hospitalization overnight did not

occur. The worker was treated for minor second degree burns

and released. OSHA ultimately issued seven (7) citations the

electric utility company. Citations issued were for

inappropriate PPE for the hazards, inappropriate electrical

work practices, no documented training for the hazards or the

task, no training and/or procedures for the task and bypassing

equipment interlocks or safeguards resulting in equipment

failure and injury to electrical workers.

a. Six (6) “Serious” citations were issued

a. Four (4) were issued at $6300 each

b. One (1) was issued at $0 amount

b. One (1) “Other” for $6,300.was issued as well for

defeating the electrical equipment’s safety interlocks and

mechanical safeguards.

Ultimately OSHA deleted all of the “Serious” citations and

“zero balanced” they as well abated the “Other” citation to

only a $3,100. “Other” than Serious.

Figure 21 – Examples of the citation schedule.

H. Today and the safety practices of those involved

Not much has changed since this event for the hospital or

the utility. The hospital has performed an electrical arc flash

hazard analysis on its distribution switchgear, labeled and

documented the equipment, but has not implemented any

standardized safety related electrical maintenance programs.

The company that performed the arc flash hazard analysis was a

low cost provider using a commercially available software. The

report that was issues made numerous assumptions about the

performance of the equipment and did not reference any

maintenance or frequency data. Much of the data gathered for

the arc flash study had to be assumed because prior electrical

performance data on the equipment was not available since it

had never been routinely performed in its 20 years of service.

The personnel have received some lecture and video electrical

safety training and provided their workers with some limited

PPE suitable for the arc flash/arc blast hazards downstream of

the “Main” low voltage switches, but the daily wear worn by

the staff is still a poly-blend professional wear. No PPE exists

for the workers to operate the main switchgear. N training on

the main switchgear has ever been performed.

The utility has seen a large amount of attrition of its

senior qualified field personnel and is working to replace

them as quickly as possible. New workers do receive

organized electrical safety and hazard awareness training with

focused on-the-job training is a. Much of the burden of

dealing with the hazardous operation, maintenance and repair

Revision 3, 2016 12

of the equipment in the utility system has been passed on to

contractors. Some good, some bad. Not one contractor is the

same. The utility typically works with the lowest cost

provider. Safety is not a focus!

IV. CONCLUSIONS

The dilemma here is that a hospital charged with saving

lives can be discussing pushing a contractor to perform a

high hazard, high risk task and have no accountability for

safety of their workers, much less their patients and worst of

all it was never recognized at any point during the disaster.

The hospital still performed surgeries daily and managed

their emergency room as if the rental generator power was

just as reliable as the utility power. The refusal to plan

around generator maintenance routines proved extremely

costly for the hospital staff, as well as placed each and every

patent in jeopardy. Though the hospital touted that “we

need to do this as safe as possible” they never once engaged

with any of the electrical contractor’s about their electrical

safety practices, procedures or electrical worker

qualifications. They never attended one safety briefing.

Even after the submission of the energized work plan for the

removal of the cables, the hospital showed no interest in

interacting with the electrical contractors about safety,

maintenance or planning challenges that arose during the

outages. Many attempts were made by the electrical

contractors on site to directly inform the hospital’s facility

management and inform them that the electrical worker’s

lives were worth as much or more than the patients in their

hospital. This did not seem to sink in, nor did it ever

resonate with the hospital management. The hospital staff

was more worried about another inconvenient outage, not

taking an outage and requiring that risky energized work be

performed.

All utility and hospital workers survived their injuries and

are out of the hospital. The utility worker and hospital

administrator will have to undergo some additional medical

procedures in the future. Additional litigation ensued

afterwards.

It’s funny that we have known and used electricity in

some form since 1746. We didn’t understand the hazards of

shock and didn’t do something about it until 1972 with the

OSHA Act. We didn’t understand nor could we document

the hazards associated with electrical flash and blast until

1996 with the issuance of the IEEE 1584-2002. We clearly

didn’t understand how to safely work near, on, around or

interact with electricity until 2004 with the issuance of the

NFPA 70E. It took OSHA until 2007 to determine that the

NFPA 70E was a “necessary tool for the prevention of

electrically related death and injuries”. Its 2016 and the

NFPA 70E is the gold standard for protecting electrical

workers, establishing safety related maintenance routines

and dealing with PPE and task related tooling.

Regardless of the regulatory environment and limitations

to finding qualified workers you would think that protecting

our narrowing field of qualified electrical workers would be

the highest priority. Sadly the mindset of many contractors,

utilities, end users and electrical workers still has not

changed about how they train and protect their employees.

Mitigating shock and flash hazards does not seem to have

the appeal that it should.

V. ACKNOWLEDGEMENTS

Update (09/26/2016)

This is a modified IEEE Industry Applications Society

White Paper originally written and presented in 2012. This

updated paper and presentation includes the public details of

regulatory enforcement actions, general electric worker

recovery updates and additional details not made public at the

time of the initial publication of this paper. As a note, this paper

and the presentation were awarded “Best Case Study” for the

IEEE/IAS Electrical Safety Workshop in 2013.

VI. REFERENCES

VII.

1584-2002 - IEEE Guide for Performing Arc Flash Hazard

Calculations

NFPA 70E: Standard for Electrical Safety in the Workplace,

2015 Edition

VIII. ANSI/NETA: The 2015 Edition ANSI/NETA Standard for

IX. Maintenance Testing Specifications for Electrical Power

X. Equipment and Systems

XI.

XII. VITA

Mike Moore has been with Walker Engineering since early

2016. Prior Mike was the Vice President of Sales, Marketing

and Business Development. Mike Led Shermco through its

largest period of organic growth in its 40 year history. Mike

Left Shermco in 2013 after its sale to GFI Energy. Mike’s

experience includes project management on multiple large

and long term maintenance outages, disaster recovery projects

and startup services for Shermco Industries, Emerson Process

Management (eti/Electro-Test), National Switchgear Systems,

and Roundhouse Electric & Engineering Company. Mike is

qualified as an electrical safety & skills trainer training

commercial, industrial and government electricians,

technicians and engineers for the last 10 years. Mike is

currently a member of the International Association of

Revision 3, 2016 13

Electrical Inspectors (IAEI), the International Electrical

Testing Association (NETA), the Institute of Electrical and

Electronic Engineers (IEEE), Dallas Chapter of the

Independent Electrical Contractors (IEC), Past Marketing

Director (2008) and Membership Director (2006), as well as

current member of The Oklahoma Predictive Maintenance

Users Group (OPMUG), and Past member of the US Army

Infantry and US Army Persian Gulf Veteran.