Understanding the EMR Error Control Practices

among Gynecologic Physicians

Ritu Kharea, Yuan Anb, Sandra Wolfa, Paul Nyirjesy,a Longjian Liua, Edgar Choua

aDrexel University College of Medicine

bDrexel University College of Information Science and Technology

Philadelphia, PA, USA

1

iConference 2013,

February 13 2013

Motivation: EMR Errors

EMR Errors

◦ incomplete, inaccurate, or inconsistent information entered in Electronic Medical Records (Brown & Patterson, 2001; Phillips & Gong, 2009)

Occur because

◦ unusable EMR interfaces situated within demanding clinical environment

◦ clinicians inadvertently make mistakes while documenting patient visits and diagnosis information

Are expensive …

◦ poor data quality

◦ unsafe quality of care

◦ physicians liable for medical malpractice (Classen, Pestotnik, Evans, Lloyd, & Burke, 1997; Fichman, Kohli, & Krishnan, 2011).

2

Controlling EMR Errors Develop computational error control algorithms (Redwood, Rajakumar, Hodson, & Coleman,

2011).

◦ alert physicians in real time, and minimize further medical errors

An “inside-out” approach

Step1: Understand the existing error control mechanisms

Step II: Design the algorithms according to the observed limitations, and

opportunities.

Existing EMRs offer limited error control functionality

◦ Clinicians resort to MANUAL techniques to review, detect, and

resolve the errors (Phillips & Gong, 2009).

We take the first step toward algorithm development, and

investigate the manual practices.

◦ assess abilities of physicians to detect EMR errors

◦ elucidate their strategies

◦ derive implications for algorithm design

3

The Error Simulation User Study

Outpatient Clinics

◦ clinicians document the patient

visit information into the EMRs in

an on-the-spot “narrative”

manner.

◦ documentation occurs under

extreme time constraints

◦ conducive to a variety of data

errors (George & Bernstein, 2009).

Gynecologic Field of Medicine

◦ Physicians responsible for

documenting information on

yeast infections, bacterial vaginitis,

menstrual cycle issues, pre-natal

and post-natal complaints, regular

gynecologic examination, etc.

Study Workflow

1. Fabricate gynecologic visit

scenarios and develop the

corresponding EMR patient visit

notes.

2. Purposefully introduce several

EMR errors into the notes.

3. Present experienced gynecologic

physicians with the flawed notes,

and ask them to

◦ Identify any data errors

◦ Reveal error detection and

resolution strategies

4

EMR Gynecologic Visit Notes

5

Reason for Visit

Ms ABC presents for a new patient visit

Chief Complaints

Yeast Infections

HPI

After having the PE and DVT had to go off birth control which she was on for endometriosis. Few months later, started to have yeast infections.

Initially, noticed external itching, soreness, redness. Her OB/GYN told she had yeast and put her on fluconazole to take as two dose treatment.

Sometimes it would work but not reliable. Treated herself monthly in this manner. Started to have more internal symptoms, as well as little

discharge. Started nystain-triamcinolone which helped the external itching. They then tried inserts x7 days , it helped again. Now finds that her

partner is getting sore at times. Finally went on fluconazole once a week, started it a few months ago. Has not been much better on this regimen.

Allergies

Benadryl: CAPS

Lovenox: SOLN

Coumadin: TABS

Latex Exam Gloves MISC

PMH:

Bipolar disorder

Current Meds

Diflucan 150 mg oral tablet: Qty0, R0, RPT

Terconazole 0.8% Vaginal Cream; Qty0; R0; RPT

Probiotic CAPS; ; Qty0; R0; RPT

Assessment

• Bacterial vaginosis

Tests

LABS ORDERED; yeast culture

Orders

Fluocinolone Acetonide 0.025% External Ointment; APPLY SPARINGLY TO AFFECTED AREA(S) TWICE DAILY; Qty1; R1; Rx.

Plan

Reviewed at length w/pt. Discussed vulvar hygiene sheet. Discussed pathophysiology of lichen simplex. Start fluocinolone ointment. Stop

fluconazole as she has no evidence of VVC today. RTC – 1 month.

Signature

Electronically Signed by XYZ, MM-DD-YY

Type: Amb New Pt Visit

Owner: XYZ, XYZ

Status: Final

ABC, ABC, 34 yr old F, DOB: DD-MM-YY

Providers find it far more efficient to enter impromptu

narrative notes (as opposed to structured data) during

patient visits (Doğan et al., 2010).

A typical gynecoloigic

visit note is organized

into 19 Sections

(1) Reason for Visit

(2) Chief Complaint

(3) History of Present

Illness (HPI)

(4) Allergies

(5) Current Medicines,

(6) Active Problems

(7) Past Medical History

(PMH)

(8) Past Surgical History

(PSH)

(9) Family History

(10) Personal or Social

History

(11)Gynecologic History

(12)Obstetric History

(13)Review of Systems

(14)Vital Signs

(15)Physical Examination

(PE)

(16)Assessment

(17)Tests

(18)Plan

(19)Orders

The 5 Kinds of EMR Errors

Inconsistent Information ◦ Contradictory information

across sections

Incorrect Information ◦ Based on note scenario

◦ Based on clinical guidelines

Incomplete Information ◦ Essential information

omitted

Missing Section ◦ An entire section omitted

Miscellaneous Errors ◦ Inappropriate placement

◦ Use of un-established acronyms

6

User Study Design

Participants

◦ 11 females, 9 males

◦ Gynecologic physicians with

Drexel College of Medicine Using Allscripts EMRs since 2008 or later

Data

◦ 7 fabricated visit notes

different hypothetical

gynecologic patients

Purposefully introduced errors Gold standard errors (total 97)

prepared by 2 clinical investigators

with 20 years of patient visit

documentation

Introduced more incomplete and

missing section errors as they are

more frequent in real world

7

5

52

8

Number of Participants

1 yearexperience

2 yearexperience

3 yearexperience

4 yearexperience

86

2554

4

Total Number of Introduced ErrorsInconsistent

Incorrect

Incomplete

MissingSection

Miscellaneous

User Study Design

Objectives

◦ assess participants’ ability to detect/resolve errors

◦ explicate intuitive strategies

◦ infer guidelines for algorithm design.

Conducted one-to-one session with each participant.

◦ Analysis Stage

present the paper prototypes of the patient notes

participant carefully studies the note, detect any data error(s), and document/annotate them on the same sheet of paper.

◦ De-briefing Stage

the participant answers follow-up questions regarding the detected errors what makes you conclude that certain data are erroneous?

what in your medical training allowed you to detect this error?

what measures would you take to resolve a certain error?

why do you think these errors occur?

8

Results: Note Analysis Stage (avg. 32.6 min)

Re-grouped Errors:

◦ Mod-liability: missing section, missing

information, misc. errors

◦ Hi-liability: incorrect and incomplete

information

Error Precision: 100%

◦ Each detected error could be mapped to a

gold standard item

Error Recall: Recall for hi-liability (0.49)

statistically higher(p<=0.05) than recall for

mod-liability (0.36)

Correlation (Pearson's) between

◦ Task performance(recall) and task

duration

◦ Task performance and years of

experience

◦ Not significant at 0.05 level

Best Recall Performance 70%

◦ Participant P5 had 4 years experience

and spent 53 minutes reviewing the

note

Lowest Recall 17%

◦ Participant P10 had 1 year experience

and spent only 19 minutes reviewing

the note

9

Time

spent

Experience

Hi-liability 0.29 0.43

Mod-liability 0.1 0.19

Results: Debriefing Stage (avg. 13.3 min)

Participants were very confident of their performances during the

note analysis stage, and were very vocal about their experiences.

How do you gain the ability to detect errors? ◦ Field experience of writing EMR

notes in clinical settings 4 participants

◦ Training in medical school 6 participants

◦ Both (Experience + Academic training) 5 participants

Why do the errors occur? ◦ Because of poor

documentation practices 17 participants believe that

physicians Should write for others to be

able to read

Should ask more questions of the patient

Should write in a list format

◦ Because of system design 5 participants

Allscripts EMR propagates all problems through previous visits creating obsolete information

Clinicians tend to write free text note because structured interface is not friendly enough.

10

Results: Debriefing Stage What are the triggers for detecting mod-liability errors?

Detection of abnormal history events

◦ if history of abnormal pap smear in the gynecologic history section is observed, it

must always accompany more information such as the diagnosis date

◦ If history of hypertension in the family history section is specified then it should

also be specified which family member suffered hypertension.

General Information Recall

◦ If the patient is having an annual visit, then HIV screening must be

present in at least one of the sections.

◦ If the patient is over 60 years of age, then a health monitoring plan

should be created and specified in the Plan section

11

Results: Debriefing Stage What are the triggers for detecting hi-liability errors?

Observation of discrepant information between two sections

◦ The reason for visit should be consistent with the active problem list.

◦ The information on the same drugs should match across different

sections, e.g., in one of the study notes

strength of the drug “Fluconazole” is 200mg in the Plan section, and strength of the drug

“Diflucan” (market brand name for same drug) is 20mg in the Orders section

Only 5 participants detected the above error in our study

Detection of abnormal results

◦ Any abnormal body mass index should be alerted in the Plan and

Assessment sections.

Identification of broken information links

◦ Each abnormal result from Physical Examination, should be linked to a

corresponding diagnosis in the Assessment section. Each diagnosis item

should have a corresponding item in the Plan section.

12

General Implications for Error Control Algorithm Design

Despite the experience and expertise, error recall performance < 50% (3 participants had >55%, 5 participants had <30%)

It is imperative to replace existing manual strategies with effective computational algorithms

The participants delivered statistically better performance for hi-liability errors than mod-liability errors

Underlines the significance of learning from their expert abilities to minimize potential physician liability

The computed correlations between performance and experience/time were not significant

No clear conclusion regarding learning-based algorithms

The results on the provenance of abilities suggest that future algorithms should

incorporate domain knowledge from a wide range of sources

learn and infer from the contextual information in the EMR data

13

Guidelines to programmatically fire the Error Detection Triggers

14

Trigger Computational Technique Knowledge Sources

General Information Recall Basic if-then rules Extraction of key information such as age, visit type, etc.

Clinical guidelines • Centers for Disease Control and Prevention • American Congress of Obstetricians and Gynecologists

Detection of Abnormal Results

Basic if-then rules Extraction of examination results.

• Davis’s Laboratory and Diagnostic Tests • Agency for Healthcare Policy and Research • Archimedes 360 Medical Calculator

Detection of abnormal history event

Advanced if-then rules Extraction of abnormal events and their attributes Extraction of medications and their attributes.

Conceptual model for drugs, disease conditions, and habits. • UMLS RxNorm • DailyMed • MedlinePlus

Observation of discrepant information between two sections

Comparison of problems, and medications across sections Drug and Disease recognition

Controlled vocabulary for describing problems and drugs, linkages between drug ingredients, and brand names, drug-drug interactions. • DrugBank • UMLS • FDA National Drug Code Directory • Classification of Diseases, Functioning, and Disability • RxDrugs • Davis’s Drug Guide

Identification of broken links across multiple sections

Extraction of results, diagnosis, plan, order information Linking items from different sections, and discovering the missing links.

Drug indication, prescriptions, physical examination resources. • SIDER 2 • Health Assessment Through the Life Span • Outlines in Clinical Medicine • DailyMed • NDF-RT

Study Summary and Contributions As a pre-step to design EMR error control algorithms, we learn algorithm design lessons from

their abilities and behaviors of physicians on gynecologic visit notes.

Participants could detect only 49% of the inaccuracy and inconsistency

errors, and only 36% of the omission errors from the notes.

◦ Need for more effective and efficient error control solution.

An in-depth investigation of manual strategies helped develop guidelines for algorithm design.

1. 5 data triggers that naturally prompt participants to sense a potential error in gynecologic notes: detection of abnormal examination results, recall of generic clinical guidelines, detection of abnormal history events, observation of discrepant information, identification of broken

information links.

2. Participants can identify the triggers using NLP abilities, and an immense amount of intuitive domain knowledge accumulated through experience and medical school training.

◦ In addition to sophisticated NLP techniques, the algorithms should incorporate a wide range of federally established free resources for clinical guidelines, controlled vocabularies, drugs, diseases, drug indications, gynecologic best practices, etc.

3. We briefly provide the linkages among triggers, NLP techniques, and the relevant trustworthy knowledge sources.

15

Study Limitations

Demanding schedules of our participants

◦ Could devote limited time to the study

◦ Set of derived implications is by no means complete

Study had inherent bias

◦ Participants knew in advance that the note contain errors

◦ This is contrary to real-world practices

Frequency of errors introduced in each note (avg. 13) is

not based on empirical evidence (due to lack of related work)

◦ Some participants might have assumed the notes to contain

fewer errors and terminated their analysis earlier

16

Conclusions and Future Work

the first step to algorithm design by exploring an untapped knowledge resource, i.e., the physicians.

Explored their abilities to detect EMR data errors

Derived algorithm design implications from their intuitive knowledge and personal strategies.

In comparison to the manual expert strategies, the existing automated algorithms only scratch the surface of error control

We plan to design customized algorithms for gynecologic notes by building on the identified triggers

◦ To simulate the narrative information extraction existing NLP algorithms to extract drug, disease and specific clinical information from

texts (Doğan et al., 2010; Li et al., 2012; Névéol & Lu, 2010).

◦ To simulate the physicians’ knowledge in the head utilize, integrate, and organize several available trustworthy knowledge sources hosted by

the US Government.

17

In the USA, medical errors kill more people than highway accidents every year. (Kohn, Corrigan, & Donaldson, 1999).

“30 years from now the EMRs should make sense – otherwise defeats the purpose of EMRs”

– a study participant on documentation malpractices

THANK YOU

Acknowledgements - The 2 anonymous reviewers - The 20 User Study Participants - Drexel’s Jumpstart grant for health informatics - Drexel’ Institutional Review Board

18