l15. case study: the therac-25eliza.newhaven.edu/ethics/attach/l15_therac.pdf · 2018-08-23 ·...

Incidents Denials and Coverups Investigation and Action Technical Causes Organizational Causes Stakeholder Responsibilities

L15. Case Study: The Therac-25

Alice E. Fischer

October, 2018

The Therac-25. . . 1/35


I. Background and Incidents

The Therac-25. . . 2/35


Background

The Therac 25 was a Canadian product ...

I A medical linear accelerator used to treat cancer

I Put on the market in 1983

I Preceded by Therac-6 and Therac-20 (pre-1981)

I Designed by AECL (Atomic Energy Commission Limited)

I Bimodal use: Electron beam and X-ray modes. Earlier modelswere not bimodal.

I Completely dependent on software for operation.

I Safety analysis excluded an analysis of software.

Eleven machines were installed, 6 in Canada and 5 in the U.S.

The Therac-25. . . 3/35


The Cast of Characters (Stakeholders)

I AECL, the manufacturer and distributor

I The programmer who wrote the code

I Eleven medical centers that purchased Therac-25

I Several Therac-25 operators

I 11 Radiation MDs in charge of the treatments

I 11 Hospitals, medical physicists and technicians

I Many patients receiving treatments

I The U.S. Food and Drug Administration

I Canadian Radiation Protection Bureau (CRPB), later mergedinto the Bureau of Radiation and Medical Devices

The Therac-25. . . 4/35


The Six Accidents

I June 1985, Georgia, severe radiation burns (front and back),swelling, great pain, skin sloughing off, shoulder paralyzed.

I July 1985, Ontario, patient lost hip, then died of cancer.Would have needed hip replacement.

I December 1985, Yakima WA, burns in a grid pattern. Chronicskin ulcer, pain, necrosis under skin. Damage was repairedsurgically, has a minor disability and scarring.

I March 1986, Texas, patient died from complications after 5months of pain

I April 1986, Texas, patient died from overdose 3 weeks later

I January 1987, Yakima, patient died in April fromcomplications related to overdose.

The Therac-25. . . 5/35


II. Denials and Coverups

The Therac-25. . . 6/35


Marietta Georgia: July, 1985

Kennestone Regional Oncology CenterThe patient was severely burned and ended up with a paralyzedarm and shoulder.

I Hospital and technician denied any wrongdoing.

I They continued to send the patient for more treatments.

I The patient filed a lawsuit.

I AECL denied knowledge of incident and said it was impossiblefor the equipment to operate in the wrong mode.

I Nobody notified the FDA.

I Other Therac users were not notified.

The Therac-25. . . 7/35


Hamilton, Ontario: July, 1985Ontario Cancer FoundationThe patient lost a hip, then died soon from an aggressive form ofcancer.I The hospital contacted AECL.I AECL sent a service engineer to investigate.I The FDA and Canadian Radiation Protection Bureau (CRPB)

were informed.I Hospitals were told to visually confirm the treatment

parameters until further notice.I AECL could not replicate the malfunction. Suspected a

microswitch. Looked for problems in the turntable positioning.I AECL announced they had improved safety by 5 orders of

magnitude.I The hospital hired an independent investigator.

The Therac-25. . . 8/35


Canadian Investigation: 1985

As a result of the Hamilton accident, the head of advanced X-raysystems in the CRPB, Gordon Symonds, wrote a report thatanalyzed the design and performance characteristics of theTherac-25 with respect to radiation safety.

I Besides citing the flawed microswitch,

I the report faulted both hardware and software components ofthe Therac’s design.

I It concluded with a list of four modifications to the Therac-25necessary for minimum compliance with Canada’s RadiationEmitting Devices (RED) Act (which gave government officialspower to ensure the safety of radiation-emitting devices).

The Therac-25. . . 9/35


Yakima, Washington: December, 1985

Yakima Valley Memorial HospitalTreatment left burns in a grid pattern, leading to a chronic skinulcer, pain, necrosis under skin. Damage was repaired surgically,has a minor disability and scarring.

I Hospital contacted AECL, spoke to a technical supportsupervisor.

I AECL replied that the damage could not have been caused bythe machine or by an operator.

I Its claim was supported by 2 pages of details why such anaccident was impossible.

I AECL claimed that there were no other similar incidents.

The Therac-25. . . 10/35


Tyler, Texas: March, 1986East Texas Cancer CenterThe patient died from complications after five months of pain.

I The video monitor was unplugged and the audio unit wasbroken. The Therac commonly had glitches and delays.

I After the first treatment, the machine displayed an error codeand showed a severe under-dose.

I The operator (as usual) hit the “proceed” button.

I The patient went to the door and pounded on it.

I He was immediately examined by a physician.

I The machine was examined next day by AECL, ETCC

I AECL again claimed there were no other incidents. Theythought an electrical glitch had caused the problem.

The Therac-25. . . 11/35


Tyler, Texas: April 1986

East Texas Cancer Center, one month after the previous incident.The patient died 3 weeks later from the radiation overdose.

I The hospital physicist recognized the seriousness of theproblem immediately and gathered information.

I ETCC took the machine out of service. Called AECL.

I The physicist and technician documented the procedure.

I ETCC Informed AECL that speed was necessary to cause thetrouble.

I AECL admitted to previously discovering a cursor-up problembut thought they had fixed it.

I The FDA was notified, and notified other users.

The Therac-25. . . 12/35


Yakima, Washington: January, 1987Yakima Valley Memorial Hospital9 months after the nature of the problem was understood!The patient died in April from complications of the overdose.

I After 5 or 6 seconds of treatment, the machine paused anddisplayed an error message that may have disappeared quickly.The operator was not sure.

I The treatment console displayed no dose.

I The operator pressed the “proceed” button.

I The machine shut down again displaying the error message“Flatness”.

I The operator went to check on the patient, who reported“burning”.

I Four days later, the skin showed burns in a striped pattern.

The Therac-25. . . 13/35


III. Investigation and Action

The Therac-25. . . 14/35


AECL report: April 1986The AECL issued a report following their investigation at ETCC

I The “frying” sound heard by patients happened when the ionchamber became saturated.

I Effective immediately, and until further notice, the key usedfor moving the cursor back through the prescription sequence(i.e. cursor UP, inscribed with an upward pointing arrow)must not be used for editing or any other purpose.

I To avoid accidental use of this key, the key cap must beremoved and the switch contacts fixed in the open positionwith electrical tape or other insulating material.

I Disabling this key means that, if any prescription entered isincorrect, then a Reset command must be used and the wholeprescription reentered.

The Therac-25. . . 15/35


FDA Response: May, 1986

We have reviewed AECL’s April 15 letter to purchasers and haveconcluded that it does not satisfy the requirements for notificationto purchasers of a defect in an electronic product. Specifically,

I It does not describe the defect nor the hazards associatedwith it.

I The letter does not provide any reason for disabling the cursorkey and

I the tone is not commensurate with the urgency for doing so.

I In fact, the letter implies the inconvenience to operatorsoutweighs the need to disable the key.

We request that you immediately renotify purchasers.

The Therac-25. . . 16/35


Resolution: July, 1987

I The software race condition was understood.

I AECL agreed to fix it and make a number of othermodifications to the machine.

I Those changes included safety interlocks.

I Operator error messages were to be written in clear Englishinstead of cryptic codes.

I Up-to-date documentation was to be provided.

The Therac-25. . . 17/35


IV. Technical Causes

The Therac-25. . . 18/35


Rapid editing of an error.Every incident involved a skilled and experienced Therac operator.

I The prescription was initially entered incorrectly, and thecursor reached the final position that signals completion of theprescription.

I This triggers the machine to start the setup process.

I Then the operator noticed an error and used the up-cursor keyto revisit the wrong field and correct it, then used cursor-downto get back to the “done” position. If the correction task wasfast enough, the control process would not be aware thatanything had changed.

I However, the concurrent setup had already begun. Theincorrect data was used to set one of the radiation parameters.Later, the corrected data was used to set the other.

The Therac-25. . . 19/35


Bad programming practice.

I A one-byte variable (0 ... 255) was used for indicatingwhether the upper collimator was in position.

I It was incremented every time its parent process wasrescheduled. It eventually overflowed and wrapped to 0.

I A non-zero value indicated setup was not complete. A 0indicated that the positioning was complete.

I Instead of incrementing the variable, it should have been setto some fixed non-zero value.

The Therac-25. . . 20/35


Poor design and construction practices.

I The system was developed over a span of several yearswithout adequate structure.

I There was no overall specification and no documented testplan.

I The model behind the software did not take concurrency andtiming issues into account.

I No redundancy was designed into the system.

I The code was written by a single developer with no significantsoftware review.

I There was no software audit trail.

I Code was written in assembly code (probably avoidable in1983).

The Therac-25. . . 21/35


Code reuse without re-analysis.

I The software for Therac-25 was ported from the Therac-20without a new analysis and without consideration of thechanges in the hardware.

I Hardware interlocks on the Therac-20 were removed andreplaced by software checks on the Therac-25.

The Therac-25. . . 22/35


Bad concurrency management.

I Extensive use was made of concurrent programming withshared variables.

I Synchronization was not implemented properly, so the systemwas prone to race conditions.

I The code relied on status flags that were set and reset bymultiple subsystems for more than one reason. It usedphantom tables for getting dosage values if too low a valuewas entered.

The Therac-25. . . 23/35


Confusing user interface, bad operating conditions.

I The user interface was bad. Error messages were cryptic andpoorly documented.

I As a consequence of common software glitches and hiccups,operators were desensitized and ignored error comments.

I Standard operating procedure was to keep pressing ”Proceed”.

I One accident happened when safety systems were disabled:broken audio and video.

The Therac-25. . . 24/35


Faulty testing and debugging.

I No unit tests were designed, documented or installed.

I Simulation and testing were inadequate.

I Errors were not easily reproduceable because half-trainedpeople did the testing.

The Therac-25. . . 25/35


V. Organizational Causes

The Therac-25. . . 26/35


Denials and sloppy process at AECL

I No software quality assurance procedures were in place.

I In a letter to the FDA, AECL erroneously or falsely claimed alawsuit notification was received in March 1986 instead ofNovember 1985, as was the case.

I After the second accident (Hamilton), overexposure wasblamed on faulty microswitch.

I AECL misrepresented facts and did not assume responsibilityfor its product after the first death.

I AECL failed to replicated the error but, after minor hardwareand software modifications, they claimed “five orders ofmagnitude”safety improvement.

The Therac-25. . . 27/35


More Problems at AECL

I AECL ignored Symonds’ and CRPB’s recommendations tomodify the data entry procedure. The number of retriesallowed changed from five to three.

I After accident #3, AECL claimed that no operator error orsystem malfunction could have caused the overexposure.

I In March, 1986, after the fourth incident, AECL’s engineercompleted inspections in one day.

I The quality assurance manager lied to the ETCC physicist,saying that AECL knew of no accidents involving radiationoverexposure by the Therac-25.

I AECL’s first recall notification letter to users was inadequateand confusing.

The Therac-25. . . 28/35


Organizational Problems Elsewhere

I One of the accidents occurred when audio and videomonitoring of patients was broken/turned off. Use ofpotentially fatal equipment should have been discontinued.

I The FDA (June 1987) gave the AECL a Class II recall,meaning the risk of serious consequences is remote, or injuriesare reversible.

I Nine months after the problem was diagnosed, anotheraccident happened.

The Therac-25. . . 29/35


VI. Stakeholder Responsibilities

The Therac-25. . . 30/35


Manufacturer of Safety-Critical EquipmentResponsibilities:

I Have working safety locks in place.

I Train programmers and engineers that safety comes first.

I Start with a correct model. if it is wrong, testing can’t detectthe problem.

I Thoroughly check / test software and hardware before release.I Use a responsible design, development, and testing process.

I One guy. No code review. Were there unit tests?I There should have been reviews at different levels, all should

have been documented.

I Keep track of code reviews and equipment design changes.

I For customers, provide high-quality documentation anddescriptions of error conditions.

The Therac-25. . . 31/35


Manufacturer Responsibilities (Continued)

I Respond in a truthful, competent, appropriate and timelymanner to problems.

I Announce recalls promptly and clearly.

I Establish good communication among departments within thecompany.

I Make sure there are proper checks and balances onresponsibilities of departments.

I Coders, legal staff, managers, complaint deskI Follow best practices for customer support and complaints

The Therac-25. . . 32/35


Programmer / Programming Department ResponsibilitiesUse a responsible design and development process including

I Do code review between versions of the machine.

I Use modular design.

I Implement a “commit” step before using a life-criticalmachine.

I Synchronize the concurrent processes properly.

I Design a user interface that is clear, especially with respect toerrors.

I Use peer review during coding.

I Create unit tests and a test suite.

I Do regression testing over a long period of time.

I Document and comment the code.

The Therac-25. . . 33/35


Hospitals

Responsibilities:

I Put patient safety and welfare at highest priority.

I Make sure that maintenance checks are timely and problemsare addressed promptly.

I Do not operate with broken safety equipment (video,intercom).

I Contact the manufacturer in case of problems ( The GAhospital didn’t think the problem was serious)

I Follow-up on treatments. (This might have prevented laterincidents.)

The Therac-25. . . 34/35


Regulatory Agencies

I Be prompt if it is a life-threatening problem.

I Do not minimize the severity of a problem that has caused adeath.

I Issue a warning even if a full investigation is not complete.

I Follow the rules about recalls.

The Therac-25. . . 35/35

l15. case study: the therac-25eliza.newhaven.edu/ethics/attach/l15_therac.pdf · 2018-08-23 ·...

Documents