TURF – A Unified Framework

For EHR Usability

Jiajie Zhang & Muhammad Walji

National Center for Cognitive Informatics & Decision Making

in Healthcare

Research Goals for TURF

• Language for

– defining usability

• Theory for

– describing, explaining, and predicting usability differences

• Methods for

– evaluating

– measuring

– designing usability

• Software tools for

– partially automating the methods

2

What is Usability?

• Usability refers to how

–useful,

–usable,

–satisfying

• a system is for the intended users to accomplish goals in the work domain by performing certain sequences of tasks

3

Zhang J, Walji MF. TURF: A Unified Framework of EHR Usability. Journal of Biomedical Informatics. 2011, In Press.

Electronic Health Record

System Usability

TURF Framework for EHR Usability

4

Functions

Users

Representations

Tasks

Useful

Usable

Satisfying

Intrinsic Complexity

Extrinsic Difficulty

Rapid Usability Assessment

Comprehensive Usability Assessment

Laboratory based Focus on usableness Usability Problem Identification Collaborate with GCREC & Vendors

Also includes: Usefulness Clinical settings Design and redesign

Laboratory based Focus on usableness Usability Problem Identification Collaborate with GCREC & Vendors

Rapid Usability Assessment

5

Stage 1 Meaningful Use Objectives

6

Core Set Objectives (selected from 15)

1. CPOE 2. E-prescribing 3. Demographics 4. Problem list 5. Medication list 6. Medication allergy list 7. Vital signs, BMI, growth chart 8. Smoking status 9. E-Copy of health information 10. Clinical summary

Usability Violations

Consistency & Visibility 1. Window shown is

called the Patient Select popup but the Alerts being shown are for the physician and not specific to a patient and there is no indication to the user that these are for the user logged on to the system.

Memory & Match 2. The user has to

remember that the Alerts are for all patients. The user has to know where to look for the Alerts

3. The icons do not match the expected use of these symbols.

Representation Analysis – Expert Review

7

1. Consistency 2. Visibility 3. Match 4. Memory 5. Feedback + Error 6. Undo 7. Document

Record CBC for PATIENT,CLINICAL F

Operator Sequence:

KLM

Code Time (s)

… … …

Find the "Orders" tab M 1.2

Point to "Orders" tab P 1.1

Click "Orders" tab B 0.1

Find the "Common Orders" M 1.2

Point to "Common Orders" P 1.1

Click "Common Orders" BB 0.2

Wait for the system to show "Common

Orders" pop-up window W 0.1

Find the "LAB ORDERS" M 1.2

Find the "Search All Labs" M 1.2

Click "Search All Labs" B 0.1

Wait for the system to show "Order a Lab

Test" pop-up window W 0.2

Find "Available Lab Tests" M 1.2

Point to "Available Lab Tests" field P 1.1

Click "Available Lab Tests" field B 0.1

Type "CBC" K(3) 0.84

Point to "CBC" on the lab test list P 1.1

Click "CBC" on the lab test list B 0.1

Point to "Order" button P 1.1

Click "Order" button B 0.1

Point to "Close" button P 1.1

Click "Close" to close the "Order a Lab

Test" pop-up window B 0.1

… … …

Total Time 39 35.06

Task Analysis - Keystroke Level Model

Findings from Expert Reviews

9

23 28 19 38 48

15 13 15

124

46 72

125

35 44

21

29

93

30 22 45

165

44

88

154

2.5 2.5

2.3 2.2

2.6 2.6 2.7 2.7

2.4 2.4 2.5 2.5

1.0

1.5

2.0

2.5

3.0

3.5

4.0

0

50

100

150

200

250

300

350

Nu

mb

er o

f U

sab

ility

Pro

ble

ms

Minor Issues Major Issues Avg Severity

Findings from Expert Reviews

10

508

426

362

355

343

90

0 100 200 300 400 500 600

Match

Feedback + Error

Memory

Visibility

Document

Undo

Findings from Task Analysis - Time on Task (seconds)

11

27

44

28

59

129

208

127

22

546

118

127

349

0 100 200 300 400 500 600

Demographics

Vital Signs

Body Mass Index

Growth Chart

Problem List

Medication List

Medication Allergies

Smoking Status

Clinical Summary

e-Copy of Health Info

e-Prescribing

CPOE

Optimal Time after Redesign

Average User Time

Optimal Time

194 seconds

97 seconds

129 seconds

Comprehensive Usability Assessment

Moving from the lab to the field

12

Assess Usefulness and

Usableness Redesign Re-evaluate

13

TURF in Action: Redesigning EHR

Harrington, Wood, Breuer, Pinzon, Howell, Pednekar, Zhu, & Zhang (AMIA 2011)

– AMIA Session S85 @ 8:30am on Wed 10/26

75 50

81

41

91

14

97

10

21

15

21

31

187

79

199

82

0

50

100

150

200

250

OpenVista: Steps Prototype: Steps OpenVista: Timing Prototype: Timing

Record

Modify

Retrieve

Total

Time (seconds) # Steps

99

28 19

53

0

20

40

60

80

100

120

Overhead Functions Essential Functions

OpenVista Prototype

TURF Architecture

Data Layer

Mapping editor

Data Collection

Presentation layer

Business layer

Data access components

Usability Evaluator System administrator Developer Provider

Data Capturing

Representation data: Screenshot, video, widget

Interaction data: Keystroke & mouse movement

User data: Profiles

Function Data: Work domain ontology

Model-Driven Analysis

Usability Metrics

Usability Benchmarks

Usability & safety patterns

EHR domain ontology

Entity-relational database Ontology database

Modeling

Modeling engine

Data integration

Repository management

Populate TURF

models

Data utilities Service agents

Other usability test/analysis services (e.g., Cogtool, Ulog, Noldus, etc.)

Modeling Analysis & Report

Users Other Users

TURF – Video / Screen Shots here

This project was supported by :

1) ONC SHARP Grant for Patient-Centered Cognitive Support

2) ONC Texas Gulf Coast Regional Extension Center for Health IT

Acknowledgements

SHARPC Project 1 Personnel

16

Project Leaders:

Jiajie Zhang Keith Butler Muhammad Walji

Project Advisor:

Mark Haselkorn

Project Co-Is:

Amy Franklin Debora Simmons Brent King Yan Xiao Ali Mokdad Emeka Okafor Axel Roessler

Consultants:

Ali Bahrami Ellen Bass Chris Esposito David Kieras Mark Musen David Woods

Postdocs, GRAs, Programmers:

Min Zhu Thomas Kannampallil Louis Lee Yingliu (Meg) Gu

REC Personnel:

Zhen Zhang Krisanne Graves Craig Harrington Yuanyuan Li