ehr usability test report for epiccare inpatient base...the design team, comprised of software...

© 2018 Epic Systems Corporation.

1

EHR Usability Test Report

for EpicCare Inpatient

Base August 2018

Report based on NISTIR 7742 Customized Common Industry Format Template for Electronic Health Record Usability Testing

Dates of Usability Study: February, March & August 2018

Date of Report: September 7, 2018

Report Prepared by: Epic

© 2018 Epic Systems Corporation.

2

Table of Contents

1 Executive Summary .........................................................................................................................................4

2 Introduction .......................................................................................................................................................4

3 Method.................................................................................................................................................................7

3.1 Participants................................................................................................................ 7

3.2 Study Design ........................................................................................................... 18

3.3 Tasks ....................................................................................................................... 19

3.4 Procedures............................................................................................................... 28

3.5 Test Location ........................................................................................................... 28

3.6 Test Environment .................................................................................................... 28

3.7 Test Forms and Tools ............................................................................................... 29

3.8 Participant Instructions ............................................................................................ 29

3.9 Usability Metrics...................................................................................................... 29

4 Results ............................................................................................................................................................... 31

4.1 §170.315(a)(9) Clinical Decision Support ................................................................... 31

4.2 §170.315(b)(2) Clinical Information Reconciliation and Incorporation ........................ 33

4.3 §170.315(a)(1) Computerized Provider Order Entry - Medications............................. 35

4.4 §170.315(a)(2) Computerized Provider Order Entry - Laboratory .............................. 37

4.5 §170.315(a)(3) Computerized Provider Order Entry – Diagnostic Imaging ................. 39

4.6 §170.315(a)(5) Demographics.................................................................................... 41

4.7 §170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks ..................................... 43

4.8 §170.315(b)(3) Electronic Prescribing......................................................................... 45

4.9 §170.315(a)(14) Implantable Device List .................................................................... 47

4.10 §170 315(a)(8) Medication Allergy List .................................................................... 49

4.11 §170 315(a)(7) Medication List ................................................................................ 51

4.12 §170.315(a)(6) Problem List ..................................................................................... 53

5 Results Conclusion....................................................................................................................................... 55

Appendices ....................................................................................................................................................... 561

Appendix 1 ................................................................................................................... 56

Appendix 2 ................................................................................................................... 61

________________________________________________________________________________________________

© 2018 Epic Systems Corporation 3

Appendix 3 ................................................................................................................... 63

Appendix 4 ................................................................................................................... 64

Appendix 5 ................................................................................................................... 69

Appendix 6 ................................................................................................................... 70

________________________________________________________________________________________________


1 Executive Summary Epic staff conducted a usability study of the August 2018 version of EpicCare Inpatient Base1 in February,

March, and August of 2018 at multiple healthcare organizations. The purpose of this study was to evaluate the

usability of the user interface and provide quantitative analysis of the usability of EpicCare Inpatient. During

the usability test, 195 healthcare providers used EpicCare Inpatient in simulated, representative tasks. Each

task was analyzed for risk using the methods detailed in section 3.3.

This study collected performance data on various tasks typically conducted by physicians and nurses. The tasks correspond to certification criteria identified in 45 CFR Part 170 Subpart C of the Health Information

Technology: 2015 Edition Health Information Technology (Health IT) Certification Criteria, 2015 Edition Base Electronic

Health Record (EHR) Definition, and ONC Health IT Certification Program Modifications . For information about the

overall User-Centered Design (UCD) process at Epic, see the User Centered Design document as submitted

for 9.1.4.

Participants in the usability study had experience with a prior version of EpicCare Inpatient. The moderators

introduced the test and instructed participants to complete a series of tasks, given one at a time, using EpicCare

Inpatient, following the testing procedures outlined in section 3.4. After each task, the participants were asked

to complete an Ease of Task Completion rating for that task. The participants’ interactions with the screen,

facial expressions, and verbalizations were recorded electronically during the test and later analyzed to collect

time measurements and evaluate performance. The moderators did not assist the participants in completing

the tasks. All participant data was de-identified.

In accordance with the examples in the NIST 7742 Customized Common Industry Format Template for Electronic

Health Record Usability Testing, various recommended metrics were used to evaluate the usability of the

software. The following quantitative metrics were collected for each eligible participant:

Task completion

Time to complete each task

Number and type of unnecessary steps

Number and type of extra steps

Participant’s Ease of Task Completion ratings

In addition to the performance data, the following qualitative observations were made:

Post-test debrief comments

Major findings

Areas for improvement

2 Introduction

2.1 Procedure Overview

The usability study was conducted on the August 2018 version of EpicCare Inpatient Base, which facilitates

inpatient care workflows and presents healthcare providers in an inpatient setting with integrated medical

information documented in a single patient chart. This usability study included realistic scenarios typically

1 ONC Health IT Certification (for Meaningful Use) information including pricing and limitations is available here: http://www.epic.com/Docs/MUCertification.pdf.

http://www.epic.com/Docs/MUCertification.pdf

________________________________________________________________________________________________


encountered by clinicians who use EpicCare Inpatient. Scenarios were constructed in collaboration with

clinicians to ensure clinical accuracy.

The purpose of the usability study was to validate and provide quantitative evidence of the usability of

EpicCare Inpatient Base. Accordingly, the testing data measured efficiency, effectiveness, and user satisfaction

through a collection of metrics including time spent on each task, extra and unnecessary steps taken per task,

and Ease of Task Completion ratings from participants.

For the purpose of reporting findings from this usability study, commonly used terms are defined as follows:

Participant: A clinician who has experience with a prior version of EpicCare Inpatient, is eligible for

participation in the usability study as determined by the Recruiting Screener (see Appendix 1), and

has completed the usability test

Scenario: A patient synopsis, given to participants to provide clinical context for tasks

Task: A verbal and written clinical workflow that is provided to all participants in the usability study

and has a predefined desired outcome

Subtask: The portion of a task relating to a specific criterion for which data is analyzed

Test: The compilation of tasks specific to studied criteria given in a single sitting to a participant

Path: A series of actions that can be taken in EpicCare Inpatient Base to reach an outcome

The study was performed using the Epic 2018 and August 2018 versions of EpicCare Inpatient Base. For more

information on the testing method, see Section 3.1.

2.2 Certification Usability Testing in Context

The usability testing Epic publically reports for ONC 2015 Edition certification represents a small segment of

the work Epic puts toward user centered design and quality assurance. Epic’s user centered design processes

and quality management system focus on designing and developing high quality software based on a deep

understanding of the user, task, and environment.

Epic approaches usability from multiple angles to create workflows that are simple, intuitive, and attractive

while providing sophisticated functionality and promoting productivity and efficiency.

________________________________________________________________________________________________


Fig 1: User Centered Design Process

Research It’s critical that Epic software developers understand users’ requirements, needs, values, and goals. All Epic

software developers conduct extensive ethnographic research including user interviews, workflow

observation, and focus groups.

Collaborative Design and Development

Development projects are selected and prioritized based on feedback from Epic users. Following that input,

the design team, comprised of software developers, clinicians, usability professionals, users, and operational

stakeholders, detail how proposed development addresses the user’s goals.

From wireframe mockups to prototype-based formative usability testing, users are involved in all aspects of

the design process. Epic conducts group design sessions in person and via webcasts and calls with members of

the Epic community. In addition, internal user experience experts evaluate designs based on industry-standard

usability heuristics and Epic’s usability style guide.

User Testing

Epic performs both formative and summative testing. Formative testing early in the cycle informs broad

design decisions. Summative testing on pre-release and released development quantifies usability and

identifies major areas for future focus and improvement. Epic uses industry-standard usability testing

techniques, including card sorts, preference testing, and eye tracking, to collect feedback and establish design

goals.

Customer staff who come to Epic for training and conferences have the opportunity to participate in usability

labs and simulations labs to take part in formative testing and give feedback on in -process development.

________________________________________________________________________________________________


3 Method

3.1 Participants

A total of 95 inpatient physicians and 101 inpatient nurses participated in the study. Nurses were asked to

complete 10 tasks. Physician testing was divided into two sessions due to updates in the user interface in the

discharge ordering workflow for the August 2018 release. During the first session, conducted in March of 2018,

physicians were asked to complete 15 tasks in the Epic 2018 version. In the second session, conducted in

August of 2018, physicians were asked to complete three tasks that were affected by the updates in the August

2018 release. Twelve of the 15 physician tasks from the first session were not changed by the updated user

interface in the August 2018 release and were not retested. None of the nursing workflows tested in February

and March of 2018 were affected by the August 2018 release. The demographics, task, and results information

reflect all sessions conducted in February-March and August of 2018.

Participants were recruited by leaders at their organizations and Epic staff. Participants were not compensated

by Epic for their time. EpicCare Inpatient Base is designed to accommodate physician and nurse specialists,

therapists, and other specialized care providers, in addition to general medicine providers, in an inpatient

setting. The participants were actual users of a previous version of EpicCare Inpatient. Participants were not

directly connected to the development of EpicCare Inpatient, nor were they employed by Epic.

Participants completed a recruiting screener that was used to gather demographic data and to verify eligibility

for participation in the study (see Appendix 1 for the Recruiting Screener used for this study). Recruited

participants represented a mix of backgrounds and demographic characteristics. Participants were assigned

an alphanumeric participant identifier at the time of the usability test to de-identify results.

Of the 123 physicians who were given the Recruiting Screener, 95 qualified to participate in the usability study.

Of the 150 nurses who were given the Recruiting Screener, 101 qualified to participate in the usability study.

The most common reason for ineligibility was that the participant did not provide patient care (see Appendix

1 for a full list of eligibility criteria). Participants were scheduled for individual 30-minute testing sessions.

Demographic data is listed in Tables 1-2 and summarized in Appendix 3.

Table 1: Participant Demographics - Inpatient Physicians

ID Gender Role Education Leve l Epic User

Experience (Years)

Specialty (as reported by

user)

Age First

Language English?

Experience

in Fie ld (Years)

Computer

Use (Years)

1 I08 F Physician Doctorate (MD, DO,

PhD, DNP)

7 OB/GYN 30-39 Yes 10 25


PhD, DNP)

3 Pediatrics 40-49 Yes 12 20


PhD, DNP)

2 Hospitalist 40-49 Yes 11 20

4 J02 F Physician Doctorate (MD, DO,

PhD, DNP)

1.5 Internal Medicine 20-29 Yes 1.5 16

________________________________________________________________________________________________



Experience (Years)


user)

Age First

Language English?

Experience

in Fie ld (Years)

Computer

Use (Years)

5 J03 M Physician Doctorate (MD, DO,

PhD, DNP)

2 Internal Medicine 30-39 No 17 20


PhD, DNP)



PhD, DNP)

2 OB/GYN 50-59 Yes 25 20


PhD, DNP)

1.5 OB/GYN 30-39 Yes 2 15


PhD, DNP)



PhD, DNP)

2 Internal Medicine 20-29 Yes 3 20


PhD, DNP)

3 Internal Medicine 20-29 Yes 2.5 10


PhD, DNP)

2 Internal Medicine 30-39 No 2.5 10


PhD, DNP)



PhD, DNP)

3 Pediatric Hospital

Medicine

40-49 Yes 12 15


PhD, DNP)



PhD, DNP)

5 Cardiac Surgery 70-79 Yes 37 50

17 K01 M Physician Doctorate (MD, DO,

PhD, DNP)



PhD, DNP)


19 K04 F Physician Doctorate (MD, DO,

PhD, DNP)

3 Family Practice 30-39 Yes 3 20


PhD, DNP)



PhD, DNP)



PhD, DNP)

1.5 Internal Medicine 30-39 No 7 20

________________________________________________________________________________________________



Experience (Years)


user)

Age First

Language English?

Experience

in Fie ld (Years)

Computer

Use (Years)


PhD, DNP)



PhD, DNP)


25 L01 M Physician;

Director

Doctorate (MD, DO,

PhD, DNP)


26 L03 M Physician;

CMIO

Doctorate (MD, DO,

PhD, DNP)


27 L04 F Physician Doctorate (MD, DO,

PhD, DNP)



PhD, DNP)



PhD, DNP)


30 L07 M Physician Doctorate (MD, DO,

PhD, DNP)


31 L08 F Physician;

Director

Doctorate (MD, DO,

PhD, DNP)

5 Neonatology 40-49 Yes 16 20


PhD, DNP)

6 Pediatric Endocrinology 30-39 Yes 1 25


PhD, DNP)

0.5 Family Medicine,

Geriatrics

40-49 Yes 24 25


PhD, DNP)

7 Pediatric 40-49 No 7 18


PhD, DNP)

6 Family Medicine 30-39 No 6 20

36 L15 M Physician;

CMO

Doctorate (MD, DO,

PhD, DNP)

5 Pulmonology; Internal

Medicine

60-69 Yes 38 20


PhD, DNP)

5 Family Practice,

Hospitalist

30-39 No 5 20

38 L17 M Physician;

CMIO

Doctorate (MD, DO,

PhD, DNP)

7 Neurology; Pediatric

Neurology

40-49 Yes 15 10

39 M01 M Physician;

Director

Doctorate (MD, DO,

PhD, DNP)


40 M02 M Physician Doctorate (MD, DO,

PhD, DNP)


________________________________________________________________________________________________



Experience (Years)


user)

Age First

Language English?

Experience

in Fie ld (Years)

Computer

Use (Years)

41 M03 M Physician Doctorate (MD, DO,

PhD, DNP)

8 Neurology 30-39 No 5 20

42 M04 F Physician Doctorate (MD, DO,

PhD, DNP)


43 N01 M Physician Doctorate (MD, DO,

PhD, DNP)

8 Cardiac Transplant 30-39 Yes 6 10


PhD, DNP)

7 Critical Care 30-39 Yes 10 20


PhD, DNP)



PhD, DNP)

5 Palliative Medicine 30-39 Yes 5 20

47 N05 M Physician;

CQO

Doctorate (MD, DO,

PhD, DNP)



PhD, DNP)

4 Infectious Diseases,

Internal Medicine

40-49 Yes 14 30


PhD, DNP)


50 N08 M Physician;

Director

Doctorate (MD, DO,

PhD, DNP)



PhD, DNP)

5.5 Internal Medicine 40-49 Yes 2.5 30

52 N10 M Physician;

IT

Doctorate (MD, DO,

PhD, DNP)

7 Hematology 30-39 Yes 7 30

53 N11 F Physician Doctorate (MD, DO,

PhD, DNP)

6 Pediatrics, Internal

Medicine

30-39 Yes 6 20


PhD, DNP)

9 OB/GYN 50-59 Yes 28 28


PhD, DNP)

6 Anesthesiologist 30-39 Yes 6 34

56 Z03 M Physician Doctorate (MD, DO,

PhD, DNP)

5 OB/GYN 30-39 Yes 5 20

57 Z04 M Physician Doctorate (MD, DO,

PhD, DNP)


58 Z05 F Physician Doctorate (MD, DO,

PhD, DNP)


________________________________________________________________________________________________



Experience (Years)


user)

Age First

Language English?

Experience

in Fie ld (Years)

Computer

Use (Years)

59 Z06 F Physician;

CMIO

Doctorate (MD, DO,

PhD, DNP)

4 Pediatric Critical Care 40-49 Yes 19 40

60 AA01 M Physician Doctorate (MD, DO,

PhD, DNP)

0.25 Internal Medicine 30-39 Yes 3 20

61 AA04 M Physician Doctorate (MD, DO,

PhD, DNP)

3 Psychiatry 40-49 No 20 30

62 BB01 M Physician Doctorate (MD, DO,

PhD, DNP)

0.25 Internal Medicine 40-49 No 15 30

63 BB02 M Physician Doctorate (MD, DO,

PhD, DNP)


64 BB03 F Physician Doctorate (MD, DO,

PhD, DNP)


65 BB04 F Physician Doctorate (MD, DO,

PhD, DNP)

1 Pulmonology/Critical

Care

40-49 Yes 20 15

66 CC04 M Physician Doctorate (MD, DO,

PhD, DNP)


67 CC06 M Physician Doctorate (MD, DO,

PhD, DNP)

3 Psychiatry 30-39 Yes 6 18

68 DD01 F Physician Doctorate (MD, DO,

PhD, DNP)

4 GI 30-39 Yes 5 20

69 DD02 M Physician;

Director

Doctorate (MD, DO,

PhD, DNP)

6 Family Medicine 40-49 No 9 25

70 DD03 M Physician Doctorate (MD, DO,

PhD, DNP)

1.5 Infectious Disease,

Internal Medicine

40-49 No 10 31

71 DD04 F Physician;

CMIO; Trainer

Doctorate (MD, DO,

PhD, DNP)

1 Family Medicine 30-39 Yes 16 20


PhD, DNP)

0.5 Cardiology 40-49 Yes 11 30


PhD, DNP)


74 U01 M Physician;

Director

Doctorate (MD, DO,

PhD, DNP)

6 Anesthesiology, Internal

Medicine

50-59 Yes 20 20

75 U02 M Physician Doctorate (MD, DO,

PhD, DNP)

8 Pediatric Orthopedic

Surgery

50-59 Yes 20 25


PhD, DNP)

7 Pediatric Pulmonology 30-39 No 5 20

________________________________________________________________________________________________



Experience (Years)


user)

Age First

Language English?

Experience

in Fie ld (Years)

Computer

Use (Years)

77 U04 F Physician;

CMO

Doctorate (MD, DO,

PhD, DNP)

0.67 Pulmonology/Critical

Care

50-59 No 18 30


PhD, DNP)


79 U07 M Physician;

CMIO

Doctorate (MD, DO,

PhD, DNP)

8 Physical Medicine and

Rehabilitation

50-59 Yes 27 20


PhD, DNP)

13 HBS 50-59 Yes 20 20


PhD, DNP)


82 U10 M Physician;

Trainer

Doctorate (MD, DO,

PhD, DNP)

0.33 Anesthesia 50-59 Yes 16 30


PhD, DNP)


84 U12 M Physician;

Director

Doctorate (MD, DO,

PhD, DNP)



PhD, DNP)

10 Family Medicine,

Hospitalist

30-39 Yes 8 20

86 U14 F Physician;

CMIO

Doctorate (MD, DO,

PhD, DNP)


87 U15 M Physician;

Director

Doctorate (MD, DO,

PhD, DNP)

5 Hospitalist, Internal

Medicine

40-49 Yes 12 20

88 U16 M Physician;

CMIO

Doctorate (MD, DO,

PhD, DNP)



PhD, DNP)

2 Hospitalist, Internal

Medicine

40-49 Yes 16 20

90 U18 M Physician;

CMIO

Doctorate (MD, DO,

PhD, DNP)


91 U19 M Physician;

CMIO

Doctorate (MD, DO,

PhD, DNP)

13 Cardiology 70-79 Yes 40 40


PhD, DNP)


93 U22 F Physician Doctorate (MD, DO,

PhD, DNP)

3 Pediatric

Hematology/Oncology

40-49 Yes 14 25


PhD, DNP)

7 Pediatric

Hematology/Oncology

40-49 Yes 15 25

________________________________________________________________________________________________



Experience (Years)


user)

Age First

Language English?

Experience

in Fie ld (Years)

Computer

Use (Years)

95 U24 M Physician;

Informatics

Doctorate (MD, DO,

PhD, DNP)

6 Infectious Disease 50-59 Yes 20 15

Table 2: Participant Demographics - Inpatient Nurses


Experience (Years)

Specialty (as reported

by user)

Age First

Language English?

Experience

in Fie ld (Years)

Computer

Use (Years)

1

A01 F Nurse Bache lor's

degree

2 Mother-Baby 20-29 Yes 2 20

2 A09 F Nurse Master's degree

(MSN, MS)

13 Med/Surg 50-59 No 30 38

3 A11 F Nurse Bache lor's

degree

9 Med/Surg 50-59 No 16 30


(MSN, MS)

8 Med/Surg 40-49 No 18 18


degree

5 Med/Surg 30-39 Yes 5 10


degree

10 Surgical ICU 30-39 Yes 13 20


degree



degree

10 Oncology 40-49 Yes 12 22

9 A22 F Nurse Associate degree 3 Med/Surg 40-49 Yes 7 20

10 A23 F Nurse Associate degree 2 Med/Surg 50-59 No 20 5




degree

9 Transplant; Med/Surg 40-49 No 21 25


degree

3 Not reported 20-29 No 5 10


degree

2 Neurology 30-39 Yes 6 25


degree

3.5 ICU 20-29 Yes 3.5 10

________________________________________________________________________________________________



Experience

(Years)


by user)

Age First

Language

English?

Experience

in Fie ld

(Years)

Computer

Use

(Years)


degree

10 OB 40-49 Yes 17 10


degree

8 Family Centered Care 60-69 Yes 10 30

19 A228 F Nurse Associate degree 13 Neonatal ICU 50-59 Yes 30 30


(MSN, MS)

5 Med/Surg 30-39 No 5 20

21 A32 F Nurse Associate degree 1 Oncology 30-39 Yes 9 15


degree

1 Labor & Delivery 50-59 Yes 8.5 20

23 A49 M Nurse Bache lor's

degree

0.75 Med/Surg 20-29 Yes 0.75 13

24 A54 F Nurse Associate degree 2 Pre -op; Post-op 30-39 Yes 10 15


degree

3 Surgical 20-29 Yes 3 10


degree

2 Orthopedics 50-59 Yes 6 36


degree

2.5 NSICU 30-39 Yes 9 20

28 A66 M Nurse Master's degree

(MSN, MS)

3 Cardiovascular ICU 40-49 Yes 20 30


degree

10 CICU 30-39 Yes 10 30


degree

9 Mother-Baby 50-59 Yes 9 >20


degree

9 Neonatal 30-39 Yes 10 >20

32 B01 F Nurse Bache lor's

degree

2 Surgical Intermediate

Care

20-29 Yes 5 15

32 B02 M Nurse Master's degree

(MSN, MS)

2.5 BMT 40-49 Yes 5 26

34 B06 F Nurse Master's degree

(MSN, MS)

2 Intermediate

Care /Medical

30-39 Yes 4 20

35 B07 M Nurse Bache lor's

degree

2 Trauma 30-39 Yes 2 15

________________________________________________________________________________________________



Experience

(Years)


by user)

Age First

Language

English?

Experience

in Fie ld

(Years)

Computer

Use

(Years)


degree

8 Cardiovascular ICU 40-49 Yes 9 25


(MSN, MS)



degree

2 Mother-Baby 20-29 Yes 4 15


degree



degree

1 Te lemetry; Med/Surg 20-29 Yes 5 25


degree

1.5 ICU 20-29 Yes 2.5 10

42 B30 M Nurse Bache lor's

degree

2 Not reported 40-49 No 16 >2


degree

2 Med/Surg 40-49 Yes 20 20


degree



(MSN, MS)

1.5 Hospice 60-69 Yes 42 40

46 C01 M Nurse ;

Manager

Bache lor's

degree


47 C02 M Nurse ;

Manager

Bache lor's

degree


48 C03 M Nurse Master's degree

(MSN, MS)

5 ICU, CCU 30-39 Yes 8 26

49 C06 M Nurse Bache lor's

degree

5 CCU 30-39 Yes 13 25

50 C08 F Nurse Bache lor's

degree

5 Postpartum 40-49 Yes 10 >20

51 C09 F Nurse Associate degree 3 Critical Care 50-59 Yes 25 14


degree

2 Behavioral Health 20-29 Yes 3 15

53 C14 F Nurse Master's degree

(MSN, MS)


________________________________________________________________________________________________



Experience

(Years)


by user)

Age First

Language

English?

Experience

in Fie ld

(Years)

Computer

Use

(Years)


degree

1.5 NICU 60-69 No 25 5

55 C20 F Nurse Associate degree 2 Mother-Baby 40-49 Yes 16 25

56 C21 F Nurse Associate degree 2.5 NICU 60-69 Yes 20 38


degree

4 Intensive Care 60-69 Yes 25 30

58 C430 F Nurse Associate degree 7 Not reported 50-59 Yes 32 20


degree


60 C434 F Nurse Associate degree 6 Surgical 30-39 Yes 6 20

61 C435 F Nurse Associate degree 10 Labor & Delivery 30-39 Yes 2.5 >20


degree

5 ICU 40-49 Yes 7 25

63 C437 M Nurse Associate degree 10 Med/Surg 30-39 Yes 15 30

64 C438 F Nurse Associate degree 6 Med/Surg 50-59 No 28 26


degree



degree

5 Psychiatry 50-59 No 10 10


degree

0.5 NICU 30-39 Yes 2 20


degree

1.5 Psychiatry 30-39 Yes 3 15

69 C448 F Nurse Associate degree 4 Psychiatry 40-49 Yes 4 5


degree

2.5 Postpartum 30-39 Yes 2.5 20

71 D01 F Nurse Associate degree 5 Neonatal 50-59 Yes 30 15

72 D02 F Nurse Bache lor's

degree


73 D03 F Nurse Associate degree 5 Pediatrics, PICU 50-59 Yes 32 20

74 E14 M Nurse ;

Analyst

Master's degree

(MSN, MS)

4 Med/Surg 40-49 No 10 26

________________________________________________________________________________________________



Experience

(Years)


by user)

Age First

Language

English?

Experience

in Fie ld

(Years)

Computer

Use

(Years)

75 E15 F Nurse Bache lor's

degree

10 PICU 40-49 Yes 26 35

76 E409 F Nurse ;

Analyst

Bache lor's

degree

11 CICU 40-49 Yes 18 20

77 E410 F Nurse ;

Analyst

Master's degree

(MSN, MS)

10 CICU 40-49 Yes 16 20

78 H02 F Nurse Bache lor's

degree

0.5 Med/Surg 20-29 Yes 1 10

79 H04 F Nurse Associate degree 5 Oncology 40-49 Yes 15 20

80 H06 F Nurse Associate degree 5 Trauma Med/Surg 20-29 Yes 5 15


degree

2 Med/Surg 20-29 No 1 20

82 H08 F Nurse Master's degree

(MSN, MS)

2 Te lemetry; Cardiac 20-29 Yes 1 10

83 H09 F Nurse Associate degree 5 Med/Surg 30-39 Yes 5 10


degree



degree


86 I06 F Nurse Bache lor's

degree


87 I12 F Nurse ;

Information

Technology

Bache lor's

degree



degree

3 ICU 50-59 Yes 22 33

89 I18 F Nurse Associate degree 4 Med/Surg 40-49 Yes 10 27


degree

1 Rehab 40-49 Yes 20 15

91 I24 F Nurse Master's degree

(MSN, MS)

2 ICU 30-39 Yes 20 20

92 I26 M Nurse Bache lor's

degree

2 ICU 30-39 Yes 3 12

93 I27 M Nurse Associate degree 3 CRRN 50-59 Yes 36 12

________________________________________________________________________________________________



Experience

(Years)


by user)

Age First

Language

English?

Experience

in Fie ld

(Years)

Computer

Use

(Years)


degree

2 Trauma, Oncology,

Ortho

20-29 Yes 2 16


degree

1 Ortho Trauma 20-29 Yes 6 20


degree

8 Rehab 30-39 Yes 8 23

97 I34 F Nurse Master's degree

(MSN, MS)

3 Med/Surg 60-69 Yes 40 15


degree

2 Orthopedics 40-49 Yes 20 20

99 J09 F Nurse Bache lor's

degree

2 NICU 50-59 No 30 15


degree

3 Te lemetry 30-39 Yes 6 25


degree

2 Med/Surg 30-39 Yes 12 20

3.2 Study Design

The objective of this study was to demonstrate areas where the application suite performed well – that is,

effectively, efficiently, and satisfactorily – and identify areas where improvements can be made.

Participants interacted with either the Epic 2018 or August 2018 version of EpicCare Inpatient Base. Each

participant used the system in a designated location, usually a conference or training room at the site where

the participant is employed. All participants were provided with the same instructions by the test moderator.

The system was evaluated for effectiveness, efficiency, and satisfaction as defined by metrics collected and

analyzed for each participant:

Task completion

Time to complete each task

Number and type of unnecessary steps

Number and type of extra steps

Participant’s Ease of Task Completion ratings

For additional information on usability metrics, see section 3.9.

________________________________________________________________________________________________


3.3 Tasks

3.3.1 Task List

Tasks were constructed to be realistic and representative of typical activities a user would complete using

EpicCare Inpatient Base. The tasks were prioritized and selected through a combination of the risk analysis

framework outlined in section 3.3.2 and the priorities outlined in NISTIR 7804-012.

NISTIR 7804-01 is an industry standard that provides scenarios and guidelines for usability testing of

Electronic Health Records. Tasks for the usability study were constructed to emphasize priorities articulated

in NISTIR 7804-1 (e.g. identification of information, consistency of information, and integrity of information),

and incorporated the NIST test scenarios when applicable.

Tasks are split into subtasks that are measurable components related to criteria supplied by the ONC. See

Appendix 4 for full task wording.

Physician Tasks/Subtasks, Session 1

Scenario 1: Gertrude is a 55-year-old female who has been admitted to your unit from the ED after

experiencing a fall at home. She is a diabetic patient being treated for dehydration, malnutrition, abrasions,

and a possible concussion.

Task 1: Reconcile a problem from a primary care provider at an outside organization.

§170.315(b)(2) Clinical information reconciliation and incorporation

Task 2: Add a medication allergy to a patient’s allergy list and assess any interactions.

§170.315(a)(4) Drug-drug, drug-allergy interaction checks

§170.315(a)(8) Medication allergy list

Task 3: Modify an order for a diagnostic imaging procedure.

§170.315(a)(3) Computerized provider order entry – Diagnostic imaging

Task 4: Place a medication order and place the consult order suggested by the system’s clinical decision

support.

§170.315(a)(1) Computerized provider order entry – Medications

§170.315(a)(8) Clinical decision support

Scenario 2: Sheryl is a 68-year-old female who is recovering from a recent knee replacement and is currently

admitted to your unit.

Task 5: Review the patient’s problem list.

§170.315(a)(6) Problem list

Task 6: Modify the details of a patient-controlled analgesic (PCA) medication order.


2 NISTIR 7804-1 Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records: Empirically Based Use Cases for Validating Safety-Enhanced Usability Guidelines for

Standardization

________________________________________________________________________________________________


Task 7: Modify the details of an inpatient laboratory order.

§170.315(a)(2) Computerized provider order entry – Laboratory

Task 8: (Replaced by Task 16)


Scenario 3: Arthur is a 65-year-old male with a history of type 2 diabetes mellitus, hypercholesterolemia, and

bradycardia. He came to the ED complaining of weakness in his right arm and was admitted to the ICU for a

CVA last night.

Task 10: Add a problem to the problem list and place the consult order suggested by the system’s clinical

decision support.



Task 11: Modify a medication allergy on a patient’s allergy list.

§170.315(a)(8) Medication allergy list

Task 12: Place a medication order and assess any interactions.

§170.315(a)(4) Drug-drug, drug-allergy interaction checks


Task 13: Order an ultrasound to diagnose symptoms.

§170.315(a)(3) Computerized provider order entry – Diagnostic imaging

Task 14: Resolve a problem on the patient’s problem list.



Physician Tasks/Subtasks, Session 2

Scenario 1: Sheryl is a 68-year-old female who is recovering from a recent knee replacement and is currently

admitted to your unit. Sheryl has recovered enough to be discharged.

Task 16: Order an outpatient lab to monitor a discharge medication.

§170.315(a)(2) Computerized provider order entry – Laboratory

Task 17: Electronically prescribe a discharge medication.


§170.315(b)(3) Electronic prescribing

Scenario 2: Arthur is a 65-year-old male with a history of type 2 diabetes mellitus, hypercholesterolemia, and

bradycardia. He came to the ED complaining of weakness in his right arm and was admitted to the ICU for a

CVA a few nights ago. Several days have passed, and Arthur is ready to be discharged.

Task 18: Electronically prescribe a discharge medication.


§170.315(b)(3) Electronic prescribing

________________________________________________________________________________________________


Nurse Tasks/Subtasks

Scenario 1: Walter is a 79-year-old male with a complex medical history that includes CHF, osteoporosis,

dementia, hypertension, and hyperlipidemia. He is directly admitted to your hospital for a wound infection.

Task 1: Document the patient’s preferred language.

§170.315(a)(5) Demographics

Task 2: Reconcile allergy information from an outside urgent care facility.


Task 3: Review a complex medication list based on information provided by the patient .

§170.315(a)(6) Medication list

Task 4: Add an implantable device to the patient’s chart using information provided by the patient .

§170.315(a)(14) Implantable device list

Task 5: Determine if the patient has any active devices on the implantable device list that may be contributing

to symptoms.


Task 6: Indicate that the patient has declined the intervention suggested by the system’s clinical decision

support.


Scenario 2: Robin is a 50-year-old who has been admitted for diabetic ketoacidosis.

Task 7: Update gender identity as reported by the patient.

§170.315(a)(5) Demographics

Task 8: Reconcile allergy information from an outside facility.


Task 9: Add a patient-reported medication to the medication list.

§170.314(a)(7) Medication list

Task 10: Update the information in a patient’s implantable device list.


Task selection was based on criticality of function and the risk analysis described in sections 3.3.2 and 3.3.3

below.

3.3.2 Risk Analysis Framework

Risk assessment for each task involves assessing the Likelihood of Risk Occurrence and the Significance of

Risk Materialization (referred to as Likelihood and Significance, respectively). Overall risk for each task is then

categorized as High, Moderate, Low, or Negligible.

________________________________________________________________________________________________


Likelihood is determined by a combination of two factors: Frequency of Workflow and Possibility of

Alternative Outcome. Frequency of Workflow reflects the general prevalence of a specific or closely analogous

workflow. Possibility of Alternative Outcome is an assessment of the likelihood that a variation might occur

during a specific or closely analogous workflow. The way in which Frequency of Workflow and Possibility of

Alternative Outcome contribute to Likelihood is outlined in Appendix 6.

Significance is the measurement of the impact of possible outcomes that result from a variation from an

expected task workflow. Impact of possible outcomes of each task was determined in consult with clinicians

and other subject matter experts.

Figure 2 lists representative factors used to evaluate Likelihood and Significance. Human and system factors

that affect the evaluation of each component were discerned from conceptual and historical analysis and

include, but are not limited to, those that appear in Figure 2.3

Fig. 2: Factors which Influence Risk

Likelihood Significance

↑ Factors increasing Frequency of Workflow

Task involves a common sequence of events

↑ Factors increasing Possibility of Alternative Outcome

Clinician must make a decision that requires an increased cognitive load (i.e. the clinician must find information in multiple locations and must synthesize or remember high volumes of information to complete the workflow)

Clinician is limited in ability to recover from an issue in documentation or action

Workflow completes an event or otherwise closes an instance of care to any further documentation

High intrinsic complexity of information or information management

Clinician is likely to experience an interruption during the workflow

↑ Factors increasing Significance

Clinician is likely to take inappropriate action after the issue occurs

Issue affects mostly high-acuity patients, high-risk medication, or urgent/critical workflows

Affected data directly informs significant clinical decisions

3 See Beasley, J. W., Wetterneck, R. B., Temte, J., Lapin, J., Smith, P., Rivera-Rodriguez, J., & Karsh, B. (2011). Information Chaos in Primary Care: Implications for Physician Performance and Patient Safety. J Am Board Fam Med. , 24(6), 745-751. doi:10.3122/jabfm.2011.06.100255; Carayon, P. Sociotechnical systems approach to healthcare quality and patient safety. Work, 4(1). doi:10.3233/WOR-2012-0091-38Carayon; Holden, R. J., Carayon, P., Gurses, A. P., Hoonakker, P., Hundt, A. S., Ozok, A., & Rivera-Rodriquez, A. (2013). SEIPS 2.0: A human factors framework for studying and improving the work of healthcare professionals and patients. Ergonomics, 53(11). doi:10.1080/00140139.2013.838643; Meeks, D. W., Takian, A., Sittig, D. F., Singh, H., & Barber, N. (2014). Exploring the sociotechnical intersection of patient safety and electronic health record implementation. Journal of the American Medical Informatics Association, 21, 2834. doi:10.1136/amiajnl-2013-001762. Epub 2013 Sep 19; O'Hara, R.,

& Et al. (2014). A qualitative study of decision-making and safety in ambulance service transitions. Health Services and Delivery Research, 2(56). doi:10.3310/hsdr02560; Vincent, C., Taylor-Adams, S., & Stanhope, N. (1995). Framework for Analysing Risk and Safety in Clinical Medicine. BMJ: British Medical Journal, 316(7138), 1154-1157; Wogalter, M. S., & Laughery, K. R. (1996). WARNING! Sign and Label Effectiveness. Current Directions in Psychological Science. doi:10.1111_1467-8721.ep10772712

________________________________________________________________________________________________


↓ Factors decreasing Frequency of Workflow

Task involves an unusual sequence of events

↓ Factors decreasing Possibility of Alternative Outcome

Clinician does not make decisions in the course of the workflow or has robust decision-making support within the workflow

Clinician finds information in one location, or the workflow requires low degree of information synthesis or memory

Clinician has ample opportunity to recover from an issue in documentation or action

Task or workflow occurs in the midst of an event; documentation can easily be edited or added

Low intrinsic complexity of information or information management

Clinician is unlikely to experience an interruption during the workflow

↓ Factors decreasing Significance

Clinician is likely to take appropriate action regardless of the issue

Issue affects mostly low-acuity patients, low-risk medications, or non-urgent/non-critical workflows

Affected data does not inform significant clinical decisions

________________________________________________________________________________________________


Figure 2 illustrates how Likelihood and Significance contribute to overall risk. Significance is weighted more

heavily than Likelihood when completing the overall risk assessment for each task. The relationship between

Significance and Likelihood was created in consult with literature on risk evaluation.4

Fig. 2: Overall Risk Assessment

Significance

Negligible Low Moderate High

Like

lih

oo

d

High Negligible Risk Moderate Risk Moderate Risk High Risk

Moderate Negligible Risk Moderate Risk Moderate Risk High Risk

Low Negligible Risk Low Risk Moderate Risk Moderate Risk

Negligible Negligible Risk Low Risk Moderate Risk Moderate Risk

3.3.3 Risk-Based Task Selection

Tasks were selected to emphasize moderate to high-risk workflows based on the framework presented

above. Risk analysis of physician and nurse tasks is summarized in Tables 3 and 4, respectively.

Table 3: Risk Analysis of Physician Tasks

Physician Task

Criteria Likelihood Significance Risk

1 Reconcile a problem from a primary care provider at an outside organization.

Clinical information reconciliation and incorporation High Moderate Moderate

2 Add a medication allergy to a patient’s allergy list and assess any interactions.

Drug-drug, drug-allergy interaction checks; Medication allergy list

Moderate High High

3 Modify an order for an inpatient diagnostic imaging procedure.

Computerized provider order entry - Diagnostic imaging Moderate Moderate Moderate

4 Place a medication order and place the consult order suggested by the system’s clinical decision support.

Computerized provider order entry - Medications; Clinical decision support

Moderate Low Moderate

4 NASA. (1994). Systems Engineering (EIA/IS-632). Electronic Industries Association (EIA); Ben-Asher, J. Z. (2004). Systems engineering aspects in theatre missile defense? Design principles and a case study. Systems Engineering. doi:10.1002/sys.10058; Ben-Asher, J. Z. (2008). Development Program Risk Assessment Based on Utility Theory. Risk Management, 10(4), 285-299. doi:10.1057/rm.2008.9; Ben-

Asher, J. Z., Zack, J., & Prinz, M. (2000). Development Program Risk Management. AIAA Progress in Aeronautics and Astronautics, 192, 341-351.; Blanchard, B. S., & Fabrycky, W. J. (1981). Systems engineering and analysis. Englewood Cliffs, NJ: Prentice-Hall.; Robertson, T. C. (Ed.). (2000). Systems Engineering Handbook (2000 ed.). INCOSE; Tummala, V. M., & Mak, C. L. (2001). A risk management model for improving operation and maintenance activities in electricity transmission networks. Journal of The Operational Research Society, 52, 125-134. doi:10.1057/palgrave.jors.2601044.

________________________________________________________________________________________________


Physician Task


5 Review the patient’s problem list.

Problem list Moderate Moderate Moderate

6 Modify the details of a patient-controlled analgesic medication order.

Computerized provider order entry - Medications Moderate High High

7 Modify the details of an inpatient laboratory order.

Computerized provider order entry - Laboratory Moderate Low Moderate

8 Replaced by Task 16


10 Add a problem to the problem list and place the consult order suggested by the system’s clinical decision support.

Problem list; Clinical decision support Moderate Negligible Negligible

11 Modify a medication allergy on a patient’s allergy list.

Medication allergy list Moderate Moderate Moderate

12 Place a medication order and assess any interactions.

Computerized provider order entry – Medications; Drug-drug, drug-allergy interaction checks

Moderate Moderate Moderate

13 Order an ultrasound to diagnose symptoms.

Computerized provider order entry - Diagnostic imaging Moderate Moderate Moderate

14 Resolve a problem on the patient’s problem list.

Problem list Moderate Low Moderate


16 Order an outpatient lab to monitor a discharge medication.

Computerized provider order entry – Laboratory Moderate High High

17 Electronically prescribe a discharge medication.

________________________________________________________________________________________________


Physician Task


Computerized provider order entry - Medications; Electronic prescribing

Moderate High High

18 Electronically prescribe a discharge medication.

Computerized provider order entry - Medications; Electronic prescribing

Moderate High High

Table 4: Risk Analysis of Nurse Tasks

Nurse Task


1 Document the patient’s preferred language.

Demographics Moderate Low Moderate

2 Reconcile allergy information from an outside urgent care facility.

Clinical information reconciliation Moderate Moderate Moderate

3 Review a complex medication list based on information provided by the patient.

Medication list Moderate Moderate Moderate

4 Add an implantable device to the patient’s chart using information provided by the patient.

Implantable device l ist Moderate Moderate Moderate

5 Determine if the patient has any active devices on the implantable device list that may be contributing to symptoms.

Implantable device l ist Low Low Low

6 Indicate that the patient has declined the intervention suggested by the system’s clinical decision support.

Clinical decision support Moderate Low Moderate

7 Update gender identity as reported by the patient.

Demographics Moderate Moderate Moderate

8 Reconcile allergy information from an outside facility.

Clinical information reconciliation Moderate Moderate Moderate

9 Add a patient reported medication to the medication list.

________________________________________________________________________________________________


Nurse Task


Medication list Moderate Moderate Moderate

10 Update the information in a patient’s implantable device list.

Implantable device l ist Moderate Low Moderate

________________________________________________________________________________________________


3.4 Procedures

Moderators were Epic employees with experience in usability testing who underwent training specific to the

type of usability testing being conducted.

Participants arrived at the testing location where they were greeted by a moderator and oriented to the testing

computer and equipment. Participants were then assigned an alphanumeric participant identifier in order to

de-identify results. Each participant was asked for verbal consent to record the testing session.

To prepare the participant for testing, the moderator outlined the format of th e test and gave general

instructions. The moderator then began recording the session using screen capture, a microphone, and a

portable camera. Before starting the tasks, the moderator showed the participant a brief instructional recording

representative of the training typically given to users before the implementation of new functionality. Pieces

of functionality were included in the instructional recordings when research indicated that not all users were

familiar with the testing setup used, due to organizational variations in configuration. (Epic provides

organizations with a high degree of latitude in setting up the software to meet specific organizational needs.)

Physicians viewed an instructional recording on updated discharge order reconciliation and clinical

information reconciliation. Nurses viewed an instructional recording on clinical information reconciliation and

the implantable device list. At this point, the participant was given an opportunity to ask any questions or

express any concerns. The moderator continued to administer general instruction and tasks during the session.

Participants were instructed to perform tasks:

At their normal pace

Without assistance; moderators were allowed to give immaterial guidance and clarification on tasks,

but not instructions on use of the software

Withholding comments until the test was completed

Before each task, the moderator gave participants a written copy of the task and oral instruction. Task timing

began and ended when the participant reached particular predetermined points in the task.

After the participants finished the tasks, they participated in individual debrief sessions. During these sessions,

the moderators solicited feedback from participants on any areas where the participants had ex tra steps,

unnecessary steps, or incomplete tasks or subtasks. The purpose of this session was to gain additional

information about the possible causes of the unnecessary steps, extra steps, or incomplete tasks or subtasks.

3.5 Test Location

Testing was conducted at healthcare organizations where participants were employed or at the Epic campus.

Participants’ organizations provided testing rooms, typically small conference or training rooms. To ensure

that the surroundings were comfortable for participants, noise levels were kept to a minimum and the ambient

temperature kept within a normal range. See Appendix 2 for a table summarizing the testing dates and

locations.

3.6 Test Environment

EpicCare Inpatient Base is typically used in an inpatient facility. The testing was conducted at a variety of

participating organizations in rooms made available for this purpose. Testing workstations were either a

Lenovo T440P, Intel Core i5-4300M processor (2.60GHz) with 16 GB RAM or a Lenovo T450S, Intel Core i7-

5600U processor (2.60 GHz) with 12 GB RAM. Workstations of both types used either Windows 10 Enterprise

________________________________________________________________________________________________


or Windows 7 Enterprise and Microsoft Internet Explorer 11 and a 23-inch flat panel display in landscape

orientation, 1600 by 900 pixel resolution, and set to thousands of colors.

The application was locally installed and used an InterSystems Caché 2017.1 database server on a loopback

connection.

The participants used a mouse and keyboard when interacting with EpicCare Inpatient Base. The overall

system performance was comparable to what users would experience in a field implementation.

3.7 Test Forms and Tools

During the usability test, the following documents were used:

Recruiting Screener

Moderator Guide

Participant Packet

The participant’s interaction with EpicCare Inpatient Base was captured and recorded digitally with screen

capture software running on the test workstation. Each participant’s facial expressions were recorded, along

with onscreen actions and verbal comments. Recordings were saved and used for further analysis.

3.8 Participant Instructions

The moderator read general introductory statements and instructions aloud to the participant before

administering the test. See Appendix 5 for the Sample Participant Orientation script.

The participant was then asked to complete a number of tasks that were read aloud by the moderator and

provided on paper to the participant for reference.

3.9 Usability Metrics

According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records ,

EHRs should support a process that provides a high level of usability for all users. The goal is for users to

interact with the system effectively, efficiently, and with high satisfaction. As such, metrics for these measures

were captured during the usability testing.

The goals of the test were to assess:

1. Efficiency of EpicCare Inpatient Base by measuring the average task time and extra steps

2. Effectiveness of EpicCare Inpatient Base by measuring task completion rates and unnecessary steps

3. Satisfaction with EpicCare Inpatient Base by measuring Ease of Task Completion ratings

________________________________________________________________________________________________


Data Scoring

Table 5 details how metrics measuring efficiency, effectiveness, and satisfaction were scored.

Table 5: Usability Scoring Metrics

Measures Rationale and Scoring

Efficiency:

Average Task Time

Each task is timed from a predetermined starting point until the participant reaches the predetermined end point for the task or verbalizes completion.

Only task times for tasks that were successfully completed are included in the average task time analysis and standard deviation (reported in parentheses in the table below).

Efficiency:

Average Extra Steps per Task

The number of steps in a participant’s path through the application is recorded and compared to the number of steps in the closest acceptable path. An extra step is recorded if the participant performs a step that is not included in the defined path but is also not counterproductive to completing the task.

The total number of steps taken by a participant is counted and the difference between the steps in their path and the closest defined path is calculated. The average of the differences for the participants is calculated. Only extra steps for tasks that were successfully completed are included in the average extra steps per task analysis and standard deviation.

Effectiveness:

Binary Task Completion Rate

A task is considered a success if the participant achieves the defined task outcome without assistance.

The total number of successes was calculated for each task and then divided by the total number of times that the task was attempted. The results are presented as a percentage.

The task failure percentage can be calculated by subtracting the binary task completion rate from 100.

Effectiveness:

Partial Task Completion Rate

A task is considered 100 percent completed if the participant achieves the defined task outcome without assistance. For participants unable to successfully complete a task, the number of steps completed are counted and divided by the number of steps in the closest defined path to calculate the percentage of the task the participant completed. These results are, in turn, added together and divided by the number of participants who attempted the task to obtain the average partial task completion rate.

Effectiveness:

Average Unnecessary Steps per Task

Unnecessary steps are recorded each time a participant performs an action in the system that is not his intended action. Examples of unnecessary steps include typing mistakes and errant clicks that do not contribute to the completion of the task. The total number of unnecessary steps is calculated and divided by the number of participants to obtain the average number of unnecessary steps committed per participant.

Only unnecessary steps for tasks that were successfully completed are included in the average unnecessary steps per task analysis and standard deviation.

Satisfaction:

Ease of Task Completion Rating

The participant’s subjective impression of the ease of use of the application is recorded for each task. After each task was completed, the moderator asked the participant to rate the task on a 5-point Likert scale: 1 (Very Difficult), 2 (Somewhat Difficult), 3 (Neither Difficult nor Easy), 4 (Somewhat Easy), 5 (Very Easy).

These values are averaged across participants for each task with the calculated standard deviation reported in parentheses in the table in the Data Analysis and Reporting section for each criterion.

________________________________________________________________________________________________


4 Results The usability testing results for EpicCare Inpatient Base are detailed below (see Tables 6-17). Results are

organized and analyzed by criteria. The results were calculated according to the methods specified in section 3.9

Usability Metrics.

4.1 §170.315(a)(9) Clinical Decision Support

4.1.1 Data Analysis and Reporting

Table 6: Clinical Decision Support Physician and Nurse Task Results

N=Number of participants

4.1.2 Discussion of the Findings

Clinical decision support testing with inpatient physicians covered two tasks:

Placing a consult order based on a medication order (Task 4)

Placing a consult order based on problem list documentation (Task 10)

Testing with inpatient nurses covered one task:

Indicating that the patient declines intervention (Task 6)

Efficiency

Physicians completed Task 4 in an average of 8.38 seconds with a negligible number of extra steps. (Negligible

is hereafter defined as within one confidence interval of zero.) Physicians completed Task 10 in an average of

5.61 seconds. Nurses completed Task 6 in an average of 13.95 seconds.

Clinical Decision Support

Efficiency Effectiveness Satisfaction

N Average Task Time


Binary Task Completion

Rate

Partial Task Completion

Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)

Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very

easy

Physician Subtasks

Placing a consult order based on a

medication order

57 8.38 (5.65) 0.05 (0.23) 98 (2) 99 (1) 0.02 (0.13) 4.75 (0.54)

Placing a consult order based on

problem list documentation

57 5.61 (3.30) 0.07 (0.26) 100 (0) 100 (0) 0 (0) 4.77 (0.46)

Nurse Subtasks

Indicating that the patient declines

intervention

96 13.95

(11.13)

0.05 (0.23) 97 (2) 98 (2) 0.01 (0.10) 4.46 (0.79)

________________________________________________________________________________________________


Effectiveness

The binary task completion rates for Task 4 and 10 are 98 percent and 100 percent respectively, and physicians

performed a negligible number of unnecessary steps for Task 4 and no unnecessary steps for Task 10. The binary

task completion rate for Task 6 is 97 percent, and nurses performed a negligible number of unnecessary steps.

Satisfaction

The majority of physicians rated both tasks as Very Easy with scores of 4.75 and 4.77 for Task 4 and Task 10,

respectively. The average satisfaction rating for nurse Task 6 is between Somewhat Easy and Very Easy with a

score of 4.46.

Major Findings

Physicians effectively performed tasks with few or no unnecessary steps and high satisfaction rates. Nurses also

completed the clinical decision support task with negligible unnecessary steps, though physicians' satisfaction

ratings were slightly higher than nurses' ratings overall. Physicians tend to interact with decision support

functionality more frequently, so this familiarity could explain the variation.

Extra steps were observed involving acknowledgement reasons, which are used to provide a reason why a

clinician is dismissing decision support suggestions. Two participants viewed the acknowledgement reasons list

without selecting a reason, and one participant entered an acknowledgement reason but also accepted the

suggested decision support action.

Areas for Improvement

Overall, participants' effectiveness and high satisfaction ratings indicate decision support is easy to use and

understand. Additional study of the wording and display of acknowledgement reasons could potentially lead

to improvements in efficiency by clarifying the intended purpose of acknowledgement reasons.

________________________________________________________________________________________________


4.2 §170.315(b)(2) Clinical Information Reconciliation and Incorporation


Table 7: Clinical Information Reconciliation and Incorporation Physician and Nurse Task Results


Clinical information reconciliation and incorporation testing with inpatient physicians covered one task:

Reconciling a problem from a primary care provider at an outside organization (Task 1)

Testing with inpatient nurses covered two tasks:

Reconciling allergy information from an outside urgent care facility (Task 2)

Reconciling allergy information from a primary care provider at an outside facility (Task 8)

Efficiency

Physicians completed Task 1 in an average of 18.06 seconds, with 52 out of 55 participants completing the task

with no extra steps. Nurses completed Task 2 and Task 8 in an average of 13.77 seconds and 9.02 seconds,

respectively.

Effectiveness

Physicians performed Task 1 with a binary task completion rate of 98 percent. Nurses performed Task 2 and

Task 8 with a binary task completion rate of 95 percent and 97 percent, respectively, with a negligible number

of unnecessary steps.

Clinical Information Reconciliation and

Incorporation


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)


easy

Physician Subtasks

Reconciling a problem from an outside organization

55 18.06 (19.24)

0.19 (0.43) 98 (2) 99 (2) 0.06 (0.24) 4.45 (0.90)

Nurse Subtasks

Reconciling an allergy from an outside urgent care facility

100 13.77 (15.34)

0.15 (0.38) 95 (2) 97 (2) 0.01 (0.15) 4.55 (0.90)

Reconciling an allergy from an outside facility

100 9.02 (7.60)

0.04 (0.20) 97 (2) 99 (1) 0.01 (0.10) 4.70 (0.78)

________________________________________________________________________________________________


Satisfaction

The average satisfaction rating for physician Task 1 is between Somewhat Easy and Very Easy with a score of

4.45. The majority of nurses rated both tasks as Very Easy with scores of 4.55 and 4.70 for Task 2 and Task 8,

respectively.

Major Findings

Physicians and nurses completed the tasks with few or negligible unnecessary steps and rated tasks as

Somewhat Easy to Very Easy on average. Extra steps were observed when participants navigated to other

areas of the chart to view outside information before pursuing the task workflow. This could be explained by

variations in workflow configuration for how users access outside information at participants' organizations.


Epic 2018 includes visual updates to the clinical information reconciliation activity that participants were using

for the first time during the study. Participants completed all tasks with average ease of task completion ratings

of Somewhat Easy to Very Easy, though nurses rated the tasks somewhat easier than physicians. Results

suggest that improvements could be made to efficiency by addressing variations in workflow configuration.

Further study could explore ways to bolster the consistency of user experience among various organizations.

________________________________________________________________________________________________


4.3 §170.315(a)(1) Computerized Provider Order Entry – Medications


Table 8: Computerized Provider Order Entry – Medications Physician Task Results

Computerized Provider Order Entry - Medications


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean

(seconds) (SD)

Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD)

5 = very easy

Physician Subtasks

Placing a medication order for nicotine replacement therapy

58 22.14 (10.85)

0 (0) 100 (0) 100 (0) 0.09 (0.29) 4.72 (0.59)

Modifying the details of a PCA medication order

58 21.93 (17.79)

0.09 (0.30) 98 (2) 99 (1) 0.05 (0.23) 4.67 (0.63)

Placing a medication order and assessing any interactions

55 12.51 (20.38)

0.18 (0.43) 100 (0) 100 (0) 0.09 (0.30) 4.62 (0.76)

Ordering a PPI for discharge 33 11.12 (12.48)

0.00 (0) 100 (0) 100 (0) 0.06 (0.25) 4.88 (0.33)

Ordering an electrolyte for discharge 35 15.94 (20.41)

0.03 (0.17) 100 (0) 100 (0) 0.00 (0) 4.86 (0.36)


Computerized provider order entry of medications testing with inpatient physicians covered five tasks:

Placing a medication order (Task 4: Subtask A)

Modifying the details of a PCA medication order (Task 6)

Placing a medication order and assessing any interactions (Task 12: Subtask A)

Ordering a discharge medication (Task 17: Subtask A)

Ordering a discharge medication (Task 18: Subtask A)

Efficiency

Medication ordering tasks have an average task time of 16.73 seconds. Tasks 4 and 6 have above average task

times, which is expected due to the additional complexity inherent in those tasks. Task 4 includes decision

support to select additional orders, and Task 6 includes a PCA order, which has multiple dose components.

Effectiveness

Tasks 4, 12, 17, and 18 all have perfect binary task completion rates. Task 6 has a binary task completion rate of

98 percent and negligible unnecessary steps.

________________________________________________________________________________________________


Satisfaction

Each task was rated Very Easy by a majority of participants, indicating a high degree of satisfaction with

medication ordering. The average task ease rating ranges from 4.62 to 4.88.

Major Findings

Four of the five tasks were completed with 100 percent task completion. The relatively short task time of Task

12 demonstrates the benefits of ordering medications from an order set, especially when drug-allergy

interactions are triggered, because they provide inline warnings and quick selection of substitute medications.


Overall, physicians were very successful ordering medications. The effectiveness metrics for Task 6 highlight

PCA medications as a possible opportunity for improvement. PCA orders are a complex area of patient care that

are vital for pain management and require providers to specify multiple interrelated parameters, such as patient

bolus dose and basal rate. The system assists providers with discrete dosing parameter fields and dose warnings.

Additional study could explore ways to clarify the interactions between the various parameters in a dynamic,

intuitive display.

________________________________________________________________________________________________


4.4 §170.315(a)(2) Computerized Provider Order Entry – Laboratory


Table 9: Computerized Provider Order Entry – Laboratory Physician Task Results

Computerized Provider Order Entry - Laboratory


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean

(seconds) (SD)

Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD)

5 = very easy

Physician Subtasks

Modifying the details of an inpatient lab order

58 9.84 (4.98) 0.03 (0.19) 100 (0) 100 (0) 0.02 (0.13) 4.86 (0.40)

Ordering an outpatient lab to monitor a discharge medication

33 12.48 (16.15)

0.00 (0) 100 (0) 100 (0) 0.06 (0.25) 4.76 (0.61)


Computerized provider order entry of laboratory tests with inpatient physicians covered two tasks:

Modifying the details of an inpatient lab order (Task 7)

Ordering an outpatient lab to monitor a discharge medication (Task 16)

Efficiency

Task 7 has an average task time of 9.84 seconds and negligible extra steps. All participants completed Task 16

without any extra steps. The average task time for Task 16 is 12.48 seconds.

Effectiveness

Both Tasks 7 and 16 have negligible unnecessary steps and perfect task completion .

Satisfaction

Each task was rated Very Easy by a majority of participants, indicating a high degree of satisfaction.

Major Findings

Task 16 was participants' first time interacting with an updated user interface for discharge ordering as part of

the August 2018 release. All participants were able to complete the task successfully, indicating that the new

user interface is highly intuitive.

________________________________________________________________________________________________



Results indicate that users are easily able to enter laboratory orders, which may be due to the consistency with

other ordering workflows. As healthcare researchers continue to develop new best practices, we will monitor

this area for further opportunities for improvement with a particular focus on the following areas:

Incorporating decision support more seamlessly at the point of ordering

Preventing duplicate laboratory orders by displaying results from outside organizations

________________________________________________________________________________________________


4.5 §170.315(a)(3) Computerized Provider Order Entry – Diagnostic Imaging


Table 10: Computerized Provider Order Entry – Diagnostic Imaging Physician Task Results

Computerized Provider Order Entry – Diagnostic Imaging


N Average

Task Time

Average Extra

Steps per Task

Binary Task

Completion Rate

Partial Task

Completion Rate

Average

Unnecessary Steps per Task

Task Ease

Rating

# Mean (seconds)

(SD)


easy

Physician Subtasks

Modifying a diagnostic imaging

procedure

57 20.56

(9.37)

0.02 (0.13) 100 (0) 100 (0) 0.05 (0.23) 4.79 (0.45)

Ordering a diagnostic imaging

procedure

58 14.74

(7.99)

0.02 (0.13) 100 (0) 100 (0) 0.09 (0.29) 4.83 (0.53)


Computerized provider order entry testing for diagnostic imaging tests with inpatient physicians covered two

tasks:

Modifying a diagnostic imaging procedure to include a comment (Task 3)

Ordering a diagnostic imaging procedure (Task 13)

Effectiveness

Tasks 3 and 13 both have negligible extra steps. Task 3 has a slightly longer task time, which is expected

because the task required the participant to type a free-text comment instead of completing discrete fields.

Effectiveness

Both tasks have a perfect task completion rate. Task 3 has a negligible number of extra steps. Most of the extra

steps performed during Task 13 were typing errors while searching for the procedure that were quickly

corrected.

Satisfaction

Both tasks were rated Very Easy by a majority of participants.

Major Findings

All of the imaging order tasks were completed with high levels of effectiveness, efficiency, and satisfaction,

suggesting that physicians are comfortable using this functionality.

________________________________________________________________________________________________



Physicians successfully completed all imaging orders tasks. Diagnostic image ordering continues to be an

important topic with implications for patient care and the financial health of organizations. Although the test

results did not indicate specific areas for improvement, as new industry practices arise, future testing should

focus on the following areas:

Decision support for Appropriate Use Criteria

Display of radiation exposure information

________________________________________________________________________________________________


4.6 §170.315(a)(5) Demographics


Table 11: Demographics Nurse Task Results

Demographics


N Average

Task Time

Average Extra

Steps per Task

Binary Task

Completion Rate

Partial Task

Completion Rate

Average

Unnecessary Steps per Task

Task Ease

Rating

# Mean (seconds)

(SD)

Mean (SD) Mean % (SD) Mean % (SD)

Mean (SD) Mean (SD) 5 = very

easy

Nurse Subtasks

Documenting preferred language 98 17.95 (21.21)

0.24 (0.49) 89 (4) 89 (3) 0.01 (0.11) 4.30 (1.09)

Updating sexual orientation and gender identity

99 9.44 (19.58)

0.10 (0.32) 99 (1) 99 (1) 0.07 (0.27) 4.56 (0.81)


Demographics testing with inpatient nurses covered two tasks:

Documenting the patient’s preferred language (Task 1)

Updating sexual orientation and gender identity as reported by the patient (Task 7)

Efficiency

Participants who completed Tasks 1 and 7 did so with an average time between 9.44 and 17.95 seconds.

Effectiveness

Participants completed Task 1 with negligible number of unnecessary steps and a binary completion rate of 89

percent. Participants completed Task 7 with a 99 percent binary completion rate and 94 out of 99 participants

completed the task without unnecessary steps.

Satisfaction

The majority of participants rated Task 1 Very Easy. Task 7, which a majority of participants also rated Very

Easy, was rated slightly easier overall.

Major Findings

Task 7 received generally high usability metrics, with a satisfaction rating of Very Easy and a 99 percent binary

task completion rate. This indicates a strong user experience, even with the relatively recent addition of gender

identity documentation at most organizations.

The form used to document information for Task 1 has been updated for users to collect more data about social

determinants of health to create a more comprehensive health record. Some participants noted the additional

________________________________________________________________________________________________


fields and visual differences. Comments from participants also indicate that many organizations document

preferred language in a different area of the patient’s chart, which could explain the extra steps taken by some

participants, the slightly lower success rate, and the amount of time taken to complete Task 1. A majority of

participants rated the task as Very Easy, which indicates general satisfaction with the task.


As we continue to update the demographics documentation tool to include more detailed information about

social determinants of health, just-in-time training could help orient users to the updated functionality. Further

study to determine the optimal arrangement of language documentation within other demographics sections

could be beneficial.

________________________________________________________________________________________________


4.7 §170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks


Table 12: Drug-Drug and Drug-Allergy Interaction Checks Physician Task Results

Drug-Drug, Drug-Allergy Interaction Checks


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)


easy

Physician Subtasks

Adding a medication allergy and assessing any interactions

57 13.16 (11.37)

0.16 (0.40) 100 (0) 100 (0) 0 (0) 4.40 (0.73)

Ordering an inpatient medication and assessing any interactions

55 12.57 (20.44)

0.07 (0.27) 98 (2) 99 (2) 0.02 (0.14) 4.62 (0.76)


Drug-drug and drug-allergy interaction checks testing with inpatient physicians covered two tasks:

Adding a medication allergy and assessing any interactions (Task 2: Subtask B)

Ordering an inpatient medication and assessing any interactions (Task 12: Subtask B)

Efficiency

On average, participants completed each of the three tasks within 12 or 13 seconds. Four out of 57 participants

took one extra step for Task 2, and two participants took extra steps while completing Task 12. The majority of

participants who took an extra step spent time viewing and selecting interaction override reasons.

Effectiveness

Participants had a 100 percent binary task completion rate for Task 2 and a 98 percent binary task completion

rate for Task 12. Participants took no unnecessary steps while completing Task 2, and some participants took a

negligible number of unnecessary steps while completing Task 12.

Satisfaction

The majority of participants rated Task 2 as Somewhat Easy or Very Easy. Part icipants rated Task 12 between

Somewhat Easy and Very Easy, with the majority of participants rating each task as Very Easy.

Major Findings

Physicians completed Tasks 2 and 12 with high effectiveness, and spent a consistent amount of time on each

task. The satisfaction score for Task 2 was slightly lower than the score for Task 12. However, all participants

________________________________________________________________________________________________


completed Task 2 successfully. Participants took no or a negligible number of unnecessary steps for each task.

Of the participants who took extra steps, most were looking for override reasons specific to their organization.


The efficiency scores indicate that approximately 5 percent of physicians who completed the tasks spent extra

time reviewing and documenting override reasons. Simplifying the selection and documentation screen for

overriding drug-drug and drug-allergy interactions could reduce the number of extra steps taken by physicians

and improve physician satisfaction.

________________________________________________________________________________________________


4.8 §170.315(b)(3) Electronic Prescribing


Table 13: Electronic Prescribing Physician Task Results


Electronic prescribing (e-prescribing) testing with inpatient physicians covered two tasks:

E-prescribing a PPI for discharge (Task 17: Subtask B)

E-prescribing an electrolyte for discharge (Task 18: Subtask B)

Efficiency

Participants took an average of 23.80 seconds to complete Task 17 and an average of 30.58 seconds to complete

Task 18. No participants took extra steps while completing Task 17, and a negligible number of participants took

extra steps during Task 18.

Effectiveness

Thirty out of 32 participants completed Task 17 successfully, with a negligible amount of unnecessary steps.

Thirty-three out of 35 participants successfully completed Task 18, and the majority of unnecessary steps taken

during the task were typing errors that were quickly corrected.

Satisfaction

The majority of participants rated both tasks as Very Easy, indicating a high level of satisfaction with electronic

prescribing.

Major Findings

The partial task completion rate for both Tasks 17 and 18 is 98 percent compared to the 94 percent binary task

completion rate. The two participants who did not complete the tasks were able to queue the expected order and

Electronic Prescribing


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)


easy

Physician Subtasks

E-prescribing a PPI for discharge 32 23.80 (13.02)

0.00 (0) 94 (0.05) 98 (0.03) 0.07 (0.26) 4.88 (0.34)

E-prescribing an electrolyte for discharge

35 30.58 (22.51)

0.03 (0.17) 94 (0.04) 98 (0.02) 0.00 (0) 4.86 (0.36)

________________________________________________________________________________________________


complete the tasks up to the point of signing the order. In an actual clinical setting, clinical decision support

would prompt the user to sign orders before the discharge ordering process is complete.


Further study is needed that more accurately simulates the clinical setting to ensure that the full e-prescribing

workflow can easily be complete in a real-world context.

________________________________________________________________________________________________


4.9 §170.315(a)(14) Implantable Device List


Table 14: Implantable Device List Nurse Task Results

Implantable Device List


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)


easy

Nurse Subtasks

Adding a historically implanted device 99 51.67 (20.98)

0.07 (0.27) 99 (1) 99 (1) 0.19 (0.44) 4.65 (0.58)

Verifying the information for a historically implanted device

99 16.86 (19.40)

0.04 (0.21) 95 (2) 97 (2) 0.01 (0.10) 4.61 (0.92)

Updating the information for a historically implanted device

99 7.21 (4.02)

0.02 (0.14) 100 (0) 100 (0) 0.01 (0.10) 4.88 (0.46)


Implantable device list testing with inpatient nurses covered three tasks:

Adding a historically implanted device based on information provided by the patient (Task 4)

Verifying the information for a historical entry on the implantable device list (Task 5)

Updating the information for a historical entry on the implantable device list based on information

provided by the patient (Task 10)

Efficiency

Tasks 5 and 10 have average task times of 16.86 seconds and 7.21 seconds, respectively. As anticipated, Task 4,

which involved adding a new implant to the chart, has a longer task time than the other two tasks.

Effectiveness

The binary task completion rate for Tasks 4, 5, and 10 is 99 percent, 95 percent, and 100 percent respectively.

Satisfaction

The majority of participants rated each task as Very Easy. This indicates a high degree of satisfaction with the

implantable device list functionality.

Major Findings

Epic 2018 introduced visual updates to the implantable device list functionality. Ninety-eight out of 99 users

effectively completed Task 4 by manually inputting multiple pieces of data to add an implant. All participants

________________________________________________________________________________________________


successfully updated information for an existing implant. In addition, many participants indicated that they

were not familiar with this workflow prior to the testing session. This suggests that the functionality is highly

learnable.


The task completion rates for these tasks indicates that documenting within the implantable device list is highly

usable. The slightly lower task completion rate for Task 5 suggests that reviewing existing data might be an area

for improvement. Further study could explore ways to optimize review activities and improve the findability of

less frequently used fields.

________________________________________________________________________________________________


4.10 §170.315(a)(8) Medication Allergy List


Table 15: Medication Allergy List Physician Task Results


Medication allergy list testing with inpatient physicians covered two tasks:

Adding a medication allergy and assessing any interactions (Task 2: Subtask A)

Entering new patient-reported information for an existing allergy (Task 11)

Efficiency

Physicians took an average amount of time of 24.69 seconds to add a new allergy to the patient's chart and

complete Task 2. They took an average amount of time of 9.02 seconds to update an existing allergy and complete

Task 11. Five out of 57 participants completed Task 2 with only one extra step, and participants completed Task

11 with a negligible amount of extra steps.

Effectiveness

Task 2 had a binary task completion rate of 96 percent and a partial task completion rate of 99 percent. Task 11

had a binary task completion rate of 100 percent, and 54 out of 58 participants completed the task with no

unnecessary steps. The majority of unnecessary steps taken during Task 2 were minor typing errors that were

quickly corrected.

Satisfaction

The majority of participants rated Task 2 as Somewhat Easy or Very Easy and Task 11 as Very Easy. The lower

satisfaction rating for Task 2 as compared to Task 11 could be attributed to the additional complexity of either

adding a new allergy or completing the drug-allergy interaction check.

Medication Allergy List


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)

Mean (SD) Mean % (SD) Mean % (SD) Mean (SD) Mean (SD) 5 = very easy

Physician Subtasks

Adding a medication allergy and assessing any interactions

57 24.69 (10.83)

0.09 (0.30) 96 (3) 99 (1) 0.11 (0.33) 4.35 (0.79)

Entering new information for an existing allergy

58 9.02 (6.53) 0.03 (0.19) 100 (0) 100 (0) 0.07 (0.26) 4.86 (0.40)

________________________________________________________________________________________________


Major Findings

All participants successfully completed Task 11. In Task 2, all participants were able to add the new allergy to

the chart; however, two out of 57 participants did not document that the patient’s reaction was a rash. Task 11

had a high satisfaction rating of 4.86, and Task 2, which also included a drug-allergy interaction check, was rated

4.35 on the ease of task completion scale.


The ease of task completion ratings for Task 2 indicate that drug-allergy interaction checking is an area in which

we could improve physician satisfaction. Improvements to drug-allergy interaction checks are discussed in

section 4.7 §170.315(a)(4)Drug-Drug, Drug-Allergy Interaction Checks.

________________________________________________________________________________________________


4.11 §170.315(a)(7) Medication List


Table 16: Medication List Nurse Task Results


Medication list testing with inpatient nurses covered two tasks:

Reviewing a complex medication list based on information provided by the patient (Task 3)

Adding a patient-reported medication (Task 9)

Efficiency

Participants completed Tasks 3 and 9 with high levels of efficiency. A negligible number of extra steps were

taken during Task 9, and three out of 99 participants took only one extra step during Task 3.

Effectiveness

When reviewing a complex medication list, the majority of participants completed the task with no unnecessary

steps, and 6 percent of participants took only one unnecessary step. When adding a patient-reported medication,

two out of 100 participants took one unnecessary step.

Satisfaction

Tasks 3 and 9 were given task ease ratings of 4.52 and 4.63 respectively, indicating that the majority of

participants found both of the tasks very easy.

Major Findings

For Tasks 3 and 9, the majority of participants reviewed and added a medication to the medication list efficiently

and effectively. All participants added the correct medication to the patient's chart in Task 9. The few participants

who did not complete Tasks 3 or 9 did not document the last taken date for prior-to-admission medications.

Medication List


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)


easy

Nurse Subtasks

Reviewing a complex medication list 99 36.60 (25.75)

0.03 (0.18) 94 (3) 94 (2) 0.06 (0.25) 4.52 (0.75)

Adding a patient-reported medication 100 36.96

(16.99)

0.01 (0.10) 95 (2) 99 (1) 0.02 (0.15) 4.63 (0.60)

________________________________________________________________________________________________



Tasks 3 and 9 received high efficiency and satisfaction scores, with low percentages of participants taking an

extra or unnecessary step. The majority of participants rated Tasks 3 and 9 as Very Easy, demonstrating high

satisfaction. The main area for improvement is to better prompt users to document the last taken date for prior-

to-admission medications. The effectiveness of this workflow could be improved by enhancing visual indicators

to highlight this outstanding documentation.

________________________________________________________________________________________________


4.12 §170.315(a)(6) Problem List


Table 17: Problem List Physician Task Results

Problem List


N Average Task Time



Rate


Rate

Average Unnecessary

Steps per Task

Task Ease Rating

# Mean (seconds)

(SD)


easy

Physician Subtasks

Reviewing a patient’s problem list 58 3.05 (2.89) 0.02 (0.13) 100 (0) 100 (0) 0 (0) 4.84 (0.59)

Adding a problem to a patient’s problem list

54 15.72 (7.95)

0 (0) 100 (0) 100 (0) 0.02 (0.14) 4.78 (0.46)

Resolving a problem on a patient’s problem list

58 2.38 (1.93) 0.03 (0.19) 100 (0) 100 (0) 0 (0) 4.91 (0.28)


Problem list testing with inpatient physicians covered three tasks:

Reviewing a patient’s problem list (Task 5)

Adding a problem to a patient’s problem list (Task 10: Subtask A)

Resolving a problem on a patient’s problem list (Task 14)

Efficiency

Physicians completed Tasks 5, 10, and 14 with average task times of 3.05 seconds, 15.72 seconds, and 2.38

seconds, respectively. All tasks were completed with either zero or negligible extra steps.

Effectiveness

The binary task completion rate for all tasks is 100 percent. Participants completed the tasks with zero or

negligible extra steps.

Satisfaction

The majority of participants rated the tasks as Very Easy with average ease ratings of 4.84, 4.74, and 4.91 for

Tasks 5, 10, and 14, respectively.

Major Findings

Task 5 and Task 14, which involved reviewing and resolving problems from the problem list, have high ease of

completion ratings, indicating that participants could efficiently and effectively complete these tasks. The

average task time for these tasks is less than the average task time for adding a problem to the problem list in

________________________________________________________________________________________________


Task 10. This is likely due to the additional complexity and the necessary clinical documentation involved in

adding a new problem compared to resolving or reviewing an existing problem. The overall completion rate of

100 percent indicates that participants were highly effective at completing these tasks.


Overall, participants reported high satisfaction with problem list tasks, and they performed the tasks effectively.

Though the results indicate positive experiences with problem list documentation, potential areas for future

exploration include:

Integrated use of the problem list across care settings

Development and progression of problems over time

Cooperative documentation by all members of a patient care team

________________________________________________________________________________________________


5 Results Conclusion Physicians and nurses performed tasks that were selected to emphasize areas of Moderate to High risk (see

section 3.3.3 for details). Common areas for improvement include:

Clarifying expected fields to complete

For various criteria, efficiency was impacted by the participant's uncertainty about which fields

needed to be updated to complete the workflow. Although indicators for required and

recommended fields are already used in many places throughout the system, future research

could explore additional ways to clarify whether a response is expected in a given field in a

specific situation. Improvements in this area could support the usability heuristic of "Recognition,

not Recall" and reduce users’ cognitive load.

Bolstering onboarding techniques

EpicCare Inpatient is continually being improved and enhanced to assist users in providing the

highest level of patient care. Incorporating training into the user interface is an important way to

disseminate tips and information so that users can quickly adjust to and make use of new

features. This study highlighted potential areas to further improve onboarding techniques to

make the transition to new and updated activities smoother for users, as observed in sections 4.2,

4.3, 4.6, and 4.9.

The overall average number of extra steps is 0.06 per task, the average binary task completion rate is 98 percent,

and the average ease of task completion rating is 4.69, indicating that tasks were generally completed with a

high degree of efficiency, effectiveness, and satisfaction.

________________________________________________________________________________________________


Appendices

Appendix 1

Recruiting Screener

Note: Italicized text indicates information used for internal determination of eligibility and was not included

on the copy given to the participant.

________________________________________________________________________________________________


Recruiting Screener Demographic Information

1. Name:

2. Credentials:

3. Highest Level of Education:

a. High school graduate/GED

b. Some college

c. College graduate

d. Postgraduate

e. Other

4. Organization:

5. Primary Work Location:

6. Contact method (please provide one of the following):

a. Work phone:

b. Cell phone:

c. Email:

7. What is your gender?

a. Male

b. Female

c. Other (please specify):__________________

8. Which of these best describes your current age?

a. <20

b. 20-29

c. 30-39

d. 40-49

e. 50-59

f. 60-69

g. 70-79

h. ≥80

________________________________________________________________________________________________


Additional Information

9. Is English your first language?

a. Yes

b. No

10. Are you a fluent English speaker? [if No, disqualify]

a. Yes

b. No

11. Due to logistical restraints and the parameters of this study, we cannot provide assistive technologies

during the testing session. Do you require any assistive technologies to use a computer? [if Yes,

disqualify]

a. Yes

b. No

12. Do you, or does anyone in your household, have a commercial interest in an electronic health record

software or consulting company? [if Yes, disqualify]

a. Yes

b. No

13. How many years of experience do you have using computers for personal and professional activities

(such as email, shopping, record keeping, etc.)?

a. <5 years

b. 5-10 years

c. 10-20 years

d. >20 years

________________________________________________________________________________________________


14. What is your current role? [if not Nurse or Physician, disqualify]

a. Analyst

b. Application Coordinator

c. Certified Nursing Assistant (CNA)

d. CEO, CMIO, CIO, etc.

e. Consultant

f. Director

g. Information Technology

h. Licensed Practical Nurse (LPN)

i. Marketing/Communications

j. Medical Assistant (MA)

k. Nurse

l. Nurse Practitioner (NP)

m. Office Manager

n. Pharmacist

o. Physician

p. Physician Assistant (PA)

q. Project Manager

r. SVP, AVP, VP, etc.

s. Trainer

t. Other (please specify)

15. Do you currently provide direct patient care? [if No, disqualify]

a. Yes

b. No

16. In which setting do you primarily work? [if Ambulatory or Emergency Department, disqualify]

a. Inpatient

b. Emergency Department

c. Ambulatory

17. What is your specialty? [if role is Physician and specialty is Radiology, Ophthalmology or Pathology,

disqualify]

18. How many years have you been working in your field?

a. <5 years

b. 5-10 years

c. 10-20 years

d. >20 years

19. Have you participated in Epic usability testing previously?

a. Yes

b. No

If yes, please describe.

________________________________________________________________________________________________


20. How do you capture patient data in your organization? [if Primarily on paper, disqualify]

a. Primarily on paper

b. Primarily electronically

21. Is Epic the EHR you use most often in your organization? [if No, disqualify]

a. Yes

b. No

22. How long have you been using Epic? [if <3 months, disqualify]

23. How frequently do you use Epic? (daily, weekly, monthly)

________________________________________________________________________________________________


Appendix 2

Testing Dates and Locations

Test Dates Locations

1 February 20, 2018 Dallas, TX


3 February 21, 2018 Baltimore, MD

4 February 21, 2018 Walnut Creek, CA

5 February 21, 2018 Concord, CA

6 February 21, 2018 Mountain View, CA


8 February 22, 2018 Fort Worth, TX

9 February 22, 2018 Baltimore, MD

10 February 22, 2018 Concord, CA

11 February 22, 2018 San Pablo, CA

12 February 22, 2018 Martinez, CA

13 February 22, 2018 Rockford, IL


15 February 28, 2018 Skokie, IL

16 March 1, 2018 Chicago, IL

17 March 1, 2018 New Orleans, LA

18 March 1, 2018 Lancaster, PA

19 March 6, 2018 Denver, CO

20 March 7, 2018 Centennial, CO

21 March 7, 2018 New York, NY



24 March 21, 2018 Verona, WI

25 March 27, 2018 Wauwatosa, WI

________________________________________________________________________________________________


26 March 27, 2018 Springfield, MO

27 March 27, 2018 Kalamazoo, MI

28 March 28, 2018 Toledo, OH

29 March 28, 2018 Columbus, OH

30 August 6, 2018 Verona, WI







________________________________________________________________________________________________


Appendix 3

Participant Demographics

Following is a high-level summary of participants in this study.

Gender Men 86 Women 110 Total (participants) 196

Occupation/Role RN/BSN 101 Physician 95 Total (participants) 196

Years of Experience Years of experience with Epic (average)

4.66

________________________________________________________________________________________________


Appendix 4

IP Physician Session 1

IP Physician Scenario 1

Your first patient is Gertrude. Gertrude is a 55-year-old female who has been admitted to your unit from the

ED after experiencing a fall at home. She is a diabetic patient being treated for dehydration, malnutrition,

abrasions, and a possible concussion.

Task 1:

Gertrude mentions that she saw her PCP at an outside organization, River Hills Health. She remembers that

he diagnosed her with something new. Reconcile Gertrude’s problem list by adding and accepting the

information from River Hills Health.

Task 2:

Gertrude mentions that sulfa antibiotics give her a rash. Add sulfa antibiotics to her allergy list with a

reaction of rash. You know that any drug-allergy interaction due to her furosemide is unlikely. Use your

clinical judgment to respond to any warnings you receive.

Task 3:

Gertrude complains of pain in her right wrist and you see that it is bruised and swollen. You suspect that

she may have fractured it when she fell. The ED physician already ordered a right wrist x-ray. Update the

order with a comment to the radiologist to look for a possible scaphoid fracture. Sign the order when

complete.

Task 4:

Gertrude is struggling to quit smoking, and she says that she is experiencing symptoms of nicotine

withdrawal since being admitted. Place an order for a nicotine 21 mg patch, and based on the information

provided by the system, place and sign any recommended follow-up orders.


Sheryl is a 68-year-old female who is recovering from a recent knee replacement and is currently admitted to

your unit.

Task 5:

In addition to osteoarthritis of the left knee, Sheryl mentions that her current conditions include high

cholesterol, type 2 diabetes mellitus, and atrial fibrillation . Document that you have reviewed her problem

list.

Task 6:

As you speak with Sheryl, you learn that she is still experiencing significant pain, despite constant use of her

PCA. Increase the dose of her existing HYDROmorphone (Dilaudid) PCA order slightly by decreasing the

lockout interval to 5 minutes and increasing the basal rate to 0.2 mg/hour. After modifying, sign the order.

Task 7:

________________________________________________________________________________________________


There were concerns with how much blood Sheryl lost in surgery, so a CBC was ordered for every 12 hours.

Her hemoglobin levels are looking good, so the frequency of the CBC order can be decreased. Modify the

existing CBC order to have a frequency of daily. After modifying, sign the order.

Task 8:

Replaced by Task 16

Task 9:

Replaced by Task 17


Arthur is a 65-year-old male with a history of type 2 diabetes mellitus, hypercholesterolemia, and bradycardia.

He came to the ED complaining of weakness in his right arm and was admitted to the ICU for a CVA last

night.

Task 10:

While rounding, you notice that although Arthur is stabilized, he is still exhibiting weakness of his right side.

Add right-sided weakness to his problem list. Based on the information provided by the system, place and

sign any recommended follow-up orders.

Task 11:

While reviewing Arthur’s chart, you notice that there is no reaction for his aspirin allergy. He tells you that

aspirin gave him hives. Update Arthur’s aspirin allergy with a reaction of hives.

Task 12:

Your organization has an initiative to increase antiplatelet utilization for stroke and CVA patients. You’ve

already opened the order set made to support this initiative. Using the order set and your clinical judgment,

select and sign an order for an appropriate antiplatelet therapy.

Task 13:

You decide to order a carotid ultrasound to further investigate the underlying cause of Arthur’s CVA. Place

and sign an order for a bilateral carotid ultrasound.

Task 14:

Several days have passed and Arthur is ready to be discharged. Shortly after admission Arthur was

diagnosed with hypokalemia (low potassium). He responded well to the prescribed potassium drip and

now his levels are much better. Resolve hypokalemia in the problem list.

Task 15:

Replaced by Task 18

________________________________________________________________________________________________


IP Physician Session 2


Sheryl is a 68-year-old female who is recovering from a recent knee replacement and is currently admitted to

your unit. Sheryl has recovered enough to be discharged.

Task 16:

You’ve already completed most of Sheryl’s discharge order reconciliation. Since she will continue her current

warfarin dose, you would like to continue monitoring Sheryl’s INR post -discharge. Enter an order for

Protime-INR. You’ll also be prescribing a medication, so wait to sign this order.

Task 17:

Sheryl mentions that she has been experiencing heartburn, and you want to prescribe her pantoprazole. Your

system automatically sends prescriptions electronically to the patient’s preferred pharmacy, which has

already been specified as Epic Apothecary Pharmacy. Place an order for 40 mg pantoprazole (Protonix). Sign

all orders when complete.


Arthur is a 65-year-old male with a history of type 2 diabetes mellitus, hypercholesterolemia, and bradycardia.

He came to the ED complaining of weakness in his right arm and was admitted to the ICU for a CVA a few

nights ago. Several days have passed, and Arthur is ready to be discharged.

Task 18:

You’ve already completed most of Arthur’s discharge order reconciliation. His potassium levels are still a

little low, and you would like him to continue taking potassium chloride. Your system automatically sends

prescriptions electronically to the patient’s preferred pharmacy, which has already been specified as Epic

Apothecary Pharmacy. Place and sign an order for potassium chloride (K-dur).

IP Nurse Scenario 1

Your first patient is Walter. Walter is a 79-year-old male with a complex medical history that includes CHF,

osteoporosis, dementia, hypertension, and hyperlipidemia. He is directly admitted to your hospital for a

wound infection.

Task 1:

Walter’s preferred language is Spanish and his PCP speaks Spanish, so he has never needed an interpreter

until now. You have already documented that Walter needs interpreter services. Add Spanish as Walter’s

primary language.

Task 2:

Walter tells you that he went to an outside organization’s urgent care clinic and they determined he has a

new allergy, but he can't remember what the allergy is. You discuss the allergy with Walter and his wife and

________________________________________________________________________________________________


confirm what is displayed as accurate. Reconcile Walter’s allergy list by adding and accepting the

information from River Hills.

Task 3:

Walter’s wife gives you a list of Walter’s at-home medications and states that he took them all yesterday.

Review his medications, adding or changing as needed. Indicate that Walter last took doses for all his

medications yesterday. Document that you have reviewed his medication list.

Task 4:

You’re taking Walter’s vitals when he mentions he has a pacemaker. You see that the pacemaker is not on

his Implants List. Walter does not have his pacemaker card with him but his wife gives you some details.

Add Walter’s pacemaker to the Implants List.

Task 5:

While discussing implants, Walter also mentions he had his hip replaced 6 years ago. You remember that

Depuy Orthopaedics recalled several of their hip replacement systems due to the implants shedding metal

shards. You want to determine if Walter could be affected. Review Walter’s hip implant and state verbally if

the manufacturer is Depuy Orthopaedics.

Task 6:

When giving Walter a snack, you notice he has difficulty chewing. After documenting this, you see that a

nutritional consult is recommended by the system. You discuss this with Walter, but he refuses to see a

nutritionist. Document that Walter has difficulty chewing and move to the next section. Review the

advisory given by the system and indicate that Walter refused the nutritional consult.

IP Nurse Scenario 2

Your next patient is Robin. Robin is a 50-year-old who prefers male gender pronouns. He has been admitted

for diabetic ketoacidosis.

Task 7:

Your organization has an initiative to more accurately document gender identity. There is already

documentation present for Robin and you discuss this with him. Robin informs you that while his sex

assigned at birth was female, he identifies as a man. Update this information in the chart.

Task 8:

Robin recently saw his PCP at an outside organization, River Hills Medical System, for an allergic reaction.

His PCP documented a new allergy which you discuss with Robin and confirm is accurate. Reconcile Robin’s

allergy list by adding and accepting the information from River Hills.

Task 9:

Robin mentions that he is taking Lasix (furosemide) 20 mg at home and last took it yesterday. You also

confirm he is still taking his other medications. Add Lasix to the medication list and indicate that he also

took his other medications yesterday.

Task 10:

________________________________________________________________________________________________


You notice Robin touching his right knee as he mentions some tenderness. He confirms that he had a right

knee replacement last year. You review the implant and notice there is no laterality listed for it. Add the

appropriate laterality to Robin’s chart.

________________________________________________________________________________________________


Appendix 5

Sample Participant Orientation

1) We are not testing you or your ability to use the system. We are measuring the usability of the system

itself.

2) You will be taken to the appropriate starting point for each task.

3) You will have a written copy of the task to read.

4) Work at your normal speed and only do what you are specifically asked to do in the system.

5) Because we are testing specific pieces of functionality, you may not complete the entirety of your

normal clinical workflow with a patient.

6) There may be multiple ways to complete a task. You can complete the task in whichever way is

apparent to you or easiest for you.

7) Verbalize that you are done upon completion of each task.

8) Fill out the Ease of Task Completion rating after each task.

9) You will complete a survey about your experience after all tasks are complete.

10) At the end of the test, we may discuss your thought process during specific tasks.

11) Save your comments until all tasks are completed. The facilitator will not offer help or answer any

questions during the test.

12) All of the information you provide will be kept confidential and your name will not be associated with

the results of this session.

________________________________________________________________________________________________


Appendix 6

Frequency of Workflow and Possibility of Alternative Outcome

Possibility of Alternative Outcome

Negligible Low Moderate High

Fre

qu

en

cy o

f W

ork

flo

w

High Negligible Likelihood Moderate Likelihood Moderate Likelihood High Likelihood

Moderate Negligible Likelihood Moderate Likelihood Moderate Likelihood High Likelihood

Low Negligible Likelihood Low Likelihood Moderate Likelihood Moderate Likelihood

Negligible Negligible Likelihood Negligible Likelihood Low Likelihood Low Likelihood

ehr usability test report for epiccare inpatient base...the design team, comprised of software...

Documents