computerized provider order entry: initial analysis of current and … · 2010. 12. 10. · errors,...

Computerized Provider Order Entry: Initial Analysis of

Current and Predicted Provider Ordering Workflow

by

Niiokai Alcide

A thesis submitted in conformity with the requirements

for the degree of Master of Health Science in Clinical Biomedical Engineering,

Institute of Biomaterials and Biomedical Engineering, University of Toronto

Supervised by

Dr. Edward Etchells

Department of Medicine, Centre for Health Services Sciences, and Information Services,

Sunnybrook Health Sciences Center

© Copyright by Niiokai Alcide 2009

ii

Niiokai Alcide

Computerized Provider Order Entry: Initial Analysis of Current and Predicted Provider Ordering Workflow

November 2009

Master of Health Science in Clinical Biomedical Engineering Institute of Biomaterials and Biomedical Engineering

University of Toronto

Abstract

Background: Computerized Provider Order Entry (CPOE) allows providers to enter medication

and service orders electronically. Workflow analysis is a critical component of CPOE

implementation.

Objectives

1. To develop a nosology for provider ordering workflow.

2. To describe actual provider ordering workflow focusing on chart and computer usage

3. To model the impact of computerized ordering on provider workflow in three future state

scenarios

Method: 20 hours of participant observation was performed for nosology development, time

motion studies totaling 47 hours and predictive modeling to project effects of possible

implementation scenarios

Results/Conclusions: Unique nosology was developed. Clinicians spent 27% of their time with

the patient, 2.2% writing and 1.1% locating patient charts. Our study predicted that the E-All

scenario (computerization of all orders) would be the best implementation choice.

Limitations: Small sample size (25 clinicians), participant frame of reference and other

assumptions may have affected the results of this study.

iii

Acknowledgements

I would like to express my most profound gratitude to the entire CPOE team at the Sunnybrook

Health Science Center, my dynamic supervisor Dr Edward Etchells, my thesis supervisory

committee: Dr. Kaveh Shojania and Paul Milgram, my colleague Julie Chan for her help with my

observational studies. My mother Catherine, father Anthony, my fiancée Tiffany and my sister

Kimmel who all pushed me to make the necessary sacrifices to achieve my goals. Without your

support my efforts would have been futile. Thank you

iv

Table of Contents

Chapter 1: Background……………….…………………….………….……………….….……1

1.1 Benefits of CPOE…………………………………………...………………...…..…...1

1.2 Importance of pre-implementation workflow analysis……………………..…..……..3

1.3 Fragmentation of clinician workflow………………………………………...………..6

1.4 Current state ordering process at Sunnybrook……………...………...………….……9

1.5 Methods for evaluating clinician workflow…………………………...……………..12

1.6 Predictive modeling………………………………………...………………...………15

Chapter 2: Objectives……………..……………………….……….…………………………..18

Chapter 3: Methods.................................................................................................................... 19

3.1 Study environment………….………...………………...………………………..…..19

3.2 Participant observation…………….……….…………….…………………………..19

3.3 Time motion study…………………………………….……...………………………20

3.4 Measures. ………………..…………………………………………………….……..23

3.5 Other data sources………….……………………………………………...…………23

3.6 Data analysis…………………………………….…………………………...………23

3.7 Predictive modeling……………….……………...………………………….………24

Chapter 4: Results………………………………………………………………………..….…28

4.1 Results: Objective 1: Nosology development …………...………..……..…………..28

4.2 Results: Objective 2: Time motion study ………………...……………...……..……30

4.3 Predictive modeling decision tree ………………..………….…………...…...……..34

4.4 Base case analysis.………….…………………………………………..……………37

4.5 One way sensitivity analysis …………………………...……………………………37

4.6 Two way sensitivity analysis………………..………..………………………………43

Chapter 5: Discussion……………………………………………………………..……………49

5.1 Development of a Nosology …………..……………...………..……………...……..49

5.2 Description of provider workflow…………………………….……….………..……50

5.3 Effects of order entry on provider workflow…………………….…..…………...…..52

Chapter 6: Limitations……………………………………………….…….…………………..56

Chapter 7: Recommendations…………………….……………...……………………………61

Chapter 8: Conclusion…………………………….……………………………………………63

References…………………………..…...………………………………………………………64

Appendix…………………………..…...……………………………………………………..…67

v

LIST OF FIGURES

Figure 1: Data collection tool ....................................................................................................... 20

Figure 2: Screen shots from time-motion study software ............................................................. 21

Figure 3: Decision Tree ................................................................................................................ 34

Figure 4: One way sensitivity graph for time to find a free PC workstation. ............................... 39

Figure 5: One way sensitivity graph for time to log on to EPR system........................................ 40

Figure 6: One way sensitivity graph for time to enter order into EPR system ............................. 41

Figure 7: One way sensitivity graph for time to write order in chart. .......................................... 42

Figure 8: Two way sensitivity graphs for time to find free pc and logon to EPR system ............ 44

Figure 9: Two way sensitivity graphs for time to find free pc and enter order into EPR system . 46

Figure 10: Two way sensitivity graphs for time logon to EPR system and enter order ............... 47

vi

LIST OF TABLES

Table 1: Summary of reviewed literature on impact of CPOE on physician workflow ................. 6

Table 2: Types of order clusters, and clinician workflow for each type of order cluster in various

CPOE implementation scenarios .................................................................................................. 11

Table 3: Avantages and Disadvantages of workflow evaluation methods ................................... 11

Table 4: Ordering activity nosology ............................................................................................. 28

Table 5: Characteristics of study and participants ........................................................................ 30

Table 6: (a)Time captured by activity (b) Usage of patient chart and EPR system ...................... 30

Table 7: Charateristics of order clusters for time-motion study ................................................... 31

Table 8: Interobserver reliability results ....................................................................................... 32

Table 9: Predictive modeling variables......................................................................................... 35

Table 10: Probabilities used for predictive modeling ................................................................... 35

Table 11: Averaged variables used in decision tree analysis ........................................................ 36

Table 12: One way senstivity analysis of probability variables .................................................... 38

Table 13: One way sensitivity analysis of averaged variables……………….......……………...38

Table 14: Two way sensitivity analysis results……………………..………………………..…..44

Table 15: Comparison of time motion study results with prior studies………………..…..…….51

vii

LIST OF ABBREVIATIO:S

ADE: Adverse Drug Event

CDS: Clinician Decision Support

CPOE: Computerized Provider Order Entry

EPR: Electronic Patient Record

E-MED: electronic ordering of medication orders only

E-MOST: electronic ordering of medication, radiology, laboratory and echo-cardiology orders

only

E-ALL: electronic ordering of all orders

OE: Order Entry

PC: Personal Computer

PDA: Personal Data Assistant

PO: Participant Observation

POE: Provider Order Entry

SPSS: Statistical Package for the Social Sciences

TM: Time Motion

WS: Workstation

1

Chapter 1: BACKGROU:D

Computerized Provider Order Entry (CPOE) is a computer application that accepts the

provider’s (physician, pharmacist or nurse) orders for medications, laboratory tests, diagnostic

radiology, and other diagnostic tests. CPOE replaces the manual process whereby the providers

record orders using paper based methods. According to Drazen (2000), the immediate interest in

CPOE is focused on medication order entry and its potential to reduce medication prescribing

errors. The application can compare the orders against standards for dosing, check for allergies

or interactions with other medications and warn the physician about potential problems. CPOE

systems may also reduce costs through avoided adverse events, reduced utilization and shorter

lengths of stay, and reduce unnecessary variations in care by encouraging recommended care

practices (Drazen 2000). However, unintended consequences and CPOE induced errors can

stem from improper design and implementation. There are many reports of failed CPOE

implementation, and a low rate of the adoption of this technology. Goddard et al in 2000

reported the reasons for the failure of an implementation undertaken in three acute care

hospitals. They highlighted financial pressures, personal unwillingness to change and integrator

inexperience with health care. In addition to those reasons, other studies have reported lack of

medical professional (clinician) involvement, inadequate capture and analysis of pre and post

implementation workflows, and lack of post implementation clinical and technical support as

other frequently reported reasons for failure. Keel et al (2005)

1.1 The benefits of CPOE

A major potential benefit of CPOE is the reduction of medication errors. Bates et al (1999)

evaluated the impact of computerized physician order entry on medication errors. Non-

2

intercepted medication errors, defined as medication errors that reached the patient, were reduced

by 86%. Non-missed dose medication errors (i.e. medication errors resulting from route errors,

frequency errors, substitutions, drug-drug interactions, inappropriate drugs, illegible orders,

known allergies to drugs, drugs not being available, avoidable delays in treatment, and

preparation errors) fell by 81%.

CPOE may also reduce adverse drug events. The effect of CPOE with clinical decision

support (CDS) on potential adverse drug events (ADEs) was explored by Wolfstadt et al in 2008.

The authors reviewed 10 CPOE studies and found that 5 reported statistically significant

reductions in ADEs. One of the studies reported a significant reduction in the rate of total ADEs

per 100 drug orders from 1.0 in a paper-based unit to 0.15 in a computer-based unit consisting of

CPOE with CDS (P<.01) (Colpaert et al 2006).

The second benefit of CPOE is that it serves as a powerful tool for the reduction of

unnecessary variation in care by encouraging recommended practices and increasing

responsiveness to new information (Drazen et al, 2000). CPOE facilitates the standardization of

care through decision support and standardized order sets. In one study there was a 94%

compliance with the new recommendations after 4 weeks of using computerized decision

support, compared to 16% before the implementation of the system. (Teich et al 1996).

A final potential benefit of CPOE is cost reduction. Cost reductions may be realized by avoiding

errors, improving efficiency, optimizing drug utilization, and avoiding the costs of adverse drug

events. Shojania et al (1998) demonstrated reduction of use of an expensive antibiotic with a

CPOE platform and clinical decision support. Clinicians wrote 32% fewer orders for the drug

and the overall duration of therapy with the drug was 36% lower than that of the study’s control

3

group. Another study showed that CPOE reduced utilization of services, wastage of

medications, decreased length of stay and overall costs (Classen et al 1997).

CPOE implementation and maintenance also carries costs however, CPOE requires an

analysis and redesign of important clinical processes, as well as an implementation of

technology. According to Kuperman (2003), technical costs, costs of process redesign, and cost

of implementation and support often deter the adoption of this technology. However, cost

savings from high costs of ADEs both medical and legal are solid incentives to put CPOE into

widespread use. A study at Brigham Women’s Hospital revealed an implementation cost of $1.9

million, with a yearly maintenance of $500,000. The institution now reports a return on initial

investment of $5 to $10 million in annual savings (Leapfrog Group 2008).

1.2 Importance of Pre implementation Workflow Analysis

The effect of CPOE on provider workflow is one of the major pitfalls in the adoption of the

technology. Successful implementation requires adequate knowledge of local workflow patterns.

Niazkhani et al. (2008), emphasize the important differences between conceptual (what is

supposed to happen) and actual workflow patterns. Successful CPOE projects require an

understanding of these differences, so that implementation can maximize efficiency to ensure

that the clinical information system is in tune with the healthcare professionals’ existing

processes and workflows. (Shabot 2004)

Workflow routines vary tremendously amongst various groups of healthcare providers.

Callen et al (2008) described the dissimilar ways doctors and other health professionals carry out

4

their routines at the point of care. Physicians and residents review test results, provide or seek

consults, and order drugs, tests and other services. Nursing staff on the other hand execute orders

and are involved in direct care for patients. Nurses would therefore spend less time interacting

with a computer while carrying out their daily routine. In contrast, physicians are involved in

indirect care, as they spend more time with the paper charts and workstations. The article also

reveals that there is tremendous diversity in the work patterns of clinicians and that careful

analysis must be in place before they can adapt to any workflow changes. For example,

Niazkhani et al. (2009) reported that providers from one institution’s CPOE implementation

experienced greater workflow support after the implementation and felt that the system was in

sync with their habits.

The changes in provider workflow brought about by CPOE may increase the amount of

time spent on the medication ordering process and for clinical documentation. As a result, many

physicians express concern that ordering with CPOE takes longer than ordering with paper.

Some studies have confirmed this aspect of CPOE. Possiant et al undertook a review of 23

studies centered on electronic health records and clinical documentation times, from the

perspective of physicians and nurses. The general trend of the studies was an increase in the time

spent documenting on paper versus the computer. Only 10 of the studies focused on physicians

and 3 of the 10 studies included order entry (CPOE) systems. The three studies (Bates 1994, Shu

2001 and Tierney 1993) reported similar results. Bates et al. revealed that in a post CPOE

implementation inpatient setting, 15 clinicians spent on average 44 more minutes daily (5.3% to

10.3%) writing orders. On a pre and post implementation study conducted on 43 clinicians for

1554 hours, Shu et al (2001) found an increase from 2.1 % to 9.0% (approximately 35 minutes

5

more per clinician per day) for time spent order writing. Similarly in 1993, Tierney et al

observed 24 clinicians in a pre and post implementation setting over 957 hours and reported an

average increase of 33 minutes for writing orders (2.5% to 9.3% minutes) after CPOE

implementation during day shifts ( 10am to 8pm).

Other current studies have shown that electronic ordering can be time-neutral compared

with a paper process. A study conducted by Overhage et al 2001 in an inpatient setting on pre

and post implementation ordering times revealed that physicians and residents spent 2.12

minutes longer per patient (6.2% to 6.9% of total time over a typical 30 minute clinic visit)

writing orders using CPOE than control physicians using paper-based methods with some

computer usage for displaying lab results. Similarly, Lo et al revealed an increase in physician

order writing time by 2 minutes (11% to 15% of total time) per patient, for a 30 minute average

patient care period using an integrated EPR (Electronic Patient Record) and CPOE system. They

also reported that clinicians spent on average 5.3 minutes for ordering medication and services

and 3.5 minutes reading various lab results on the EPR-CPOE system per every 30 minutes of

patient care. In contrast, physicians using paper-based ordering spent on average 3.3 minutes per

30 minutes of patient care writing orders and 1.7 minutes using the computer to review lab

results. Given the above mentioned time changes, the researchers concluded that clinicians do

not spend more time overall on patient care after the implementation of CPOE. The results of

these studies may however only pertain to the particular study environment, as all orders and

charting activities were hand written and lab results were reviewed electronically. The following

table summarizes the results of the reviewed literature pertaining to time spent ordering.

6

Table 1: Summary of reviewed literature on impact of CPOE on physician workflow

1.3 Fragmentation of clinician workflow

Over the past 15 years, there has been gradual computerization of various aspects of the

ordering process. For example at Sunnybrook today, physicians must write, then electronically

enter, radiology and echocardiography orders. By contrast, medication orders and all other

orders (such as nursing orders) are written on paper order sheets. Clinical documentation is

done almost entirely on the paper chart, but discharge summaries and certain clinical notes are

available on the computer. Clinical data review is done both on the computer, for radiology and

laboratory results, and the paper chart (such as pulmonary function tests, electroencephalography

reports).This process fragmentation can create problems for clinicians’ workflow, because many

Study Study Setting and Design Participants Sample

Size

Workflow

Classification

Methods

Main measures Results/ comments

Bates et al

1994

Inpatient Pre and post

implementation

Medical

Interns

22 Time Motion Time spent on order entry. Time

for other activities after order

entry implementation

Interventional participants 44 minutes

longer per day than control (5.3% to 10.5%

of total time spent on order writing.

Lo et al

2007


implementation

Physicians 17 Time Motion Measure of clinician time spent on

85 activities

Interventional participants 2 minutes longer

per patient than control on time spent on

order writing.

Overhage et

al 2000

Internal medicine practice


implementation

Physicians 34 Time Motion Time spent on various activities

per patient

Interventional participants 2.12 minutes

longer per patient than control (6.2% to 6.9)

% of total time spent on order writing.

Teirney et al

1996


implementation

Interns

Medical

students,

24 Time motion Time motion study of selected

interns on system’s time

consumption


longer per day than control (2.5% to 9.3%

of total time spent on order writing.

Shu et al

2001


implementation

Interns

Medical

students,

43 Random sampling Impact of CPOE system on

clinician time


longer per day than control (2.1% to 9% of

total time spent on order writing.

7

tasks now require both the paper chart and the computer workstation for successful completion.

For example, a physician may need to view laboratory results on the computer while writing

progress notes in the paper chart. Clinicians may also be performing multiple tasks

simultaneously, or sequentially, with the paper chart or computer. For example, a clinician could

be using the computer to review results so that orders can be written in the paper chart. The

critical workflow consideration is that the clinician is performing the tasks with both the paper

chart and the computer. Process fragmentation, and the related concept of task sequencing, may

also represent an opportunity, because clinicians may now be more willing to increase the

amount of time spent at the computer in order to reduce process fragmentation.

Previous clinician workflow studies have not thoroughly explored the amount of time

clinicians spend multitasking while performing tasks centered on ordering medication and

services in a pre CPOE implementation setting. Studies by Overage and Lo, where the setting

involved the use of paper charts and computer for results display, revealed a less drastic impact

of CPOE on physician time and workflow as clinicians were already spending time on the

computer.

Other workflow studies have not thoroughly considered the concepts of process

fragmentation or task sequencing. Studies by Bates (1994), Shu (2001), and Overhage et al.

(2001) group clinician tasks as isolated activities. For example they generated categories by

media (computer or chart) and included write, read and other activities as sub categories. During

the observation periods, observers used their judgment to determine what the major activity

would be logged as, and possibly disregarding the fact that the clinician may have been

8

multitasking at the time. This may have impacted the results of the studies as one of the activities

would be chosen more frequently than another, without accounting for performing simultaneous

tasks. The end result would therefore be less time being recorded for particular tasks.

A study by Asaro took a different approach and captured to some degree the multitasking

nature of clinician workflow. They captured the various “overlapping” (simultaneously

performed) tasks of clinicians in an emergency department. For example tasks were grouped as

using EPR system while on the phone, writing orders while talking to nurse, and using the EPR

system while working with the patient chart. The latter category thus included all the possible

uses of the chart, namely writing order, reviewing or documentation. The study was not aimed at

documenting the particular activities being performed or whether they included the EPR system

and chart, but focused on how much time was spent doing any type of multitasking.

The present study will explore the current fragmented nature of physicians’ workflow,

focusing on whether the clinician is using a computer workstation, a paper chart, or both, during

ordering, documentation, or clinical data review. These data will allow a more accurate

prediction of the impact of changes to the ordering process on clinician workflow.

1.4 Current State Medication Ordering Process at Sunnybrook

Currently at Sunnybrook clinicians are required to write all medical orders in the

patient’s chart. In addition echocardiography and diagnostic imaging are required to be entered

directly into the EPR system by the ordering provider. Laboratory orders are then entered into

the electronic system by nurses or ward clerks. For a typical medication order, a clinician would

9

write the order on an order sheet found in the patient’s chart. The chart is then flagged (red knob)

to alert nurses that an order has been written. After acknowledging the order, the nurse removes

the yellow carbon copy of the order sheet and places it in the pharmacy tray for pickup by

pharmacy technicians. Pharmacy technicians then collect order sheets on a unit at 30 to 45

minute intervals. The forms are taken to the pharmacy department for pharmacist review. The

pharmacist will enter the order into the computerized dispensing system. The first doses are

dispensed in the pharmacy department, then brought to the unit by pharmacy technicians and

placed in the patient’s tray in the medication cart for administration by nurses.

The Sunnybrook CPOE project is currently choosing between three future state

implementation scenarios

(i) “Electronic Med” (E-Med): Medication orders are only entered electronically by the

ordering provider. There are no other changes to current state. All other orders are written on

paper chart. The physician must then also electronically enter radiology and echocardiography

orders.

(ii) “Electronic Med-Lab-Radiology-Echo” (E-Most): Medication, laboratory, radiology

and echocardiography orders entered only electronically by the ordering provider. There are no

paper chart orders for medication, laboratory, radiology and echocardiography. All remaining

orders are written on paper chart.

(iii) “All Orders Electronic” (E-All): All orders only entered electronically by the

ordering provider; no orders written on paper chart.

10

Clinicians do not necessarily write orders solely for medications at a single time. A

clinician may also write a lab order and a general order (for example, activity as tolerated) at the

same time. We define the term “order cluster” as a group of orders written by a single clinician

at a single time for a single patient. Prior studies have examined “order sets”, which are

predefined standard groups of orders, but there has been little consideration of the concept of an

order cluster in studies of clinician workflow. Clusters could include any combination of various

types of orders. For example, a doctor may write an order for Heparin (medication), a Complete

blood count (Lab), and nothing by mouth for 24 hours (general order). The analysis of a cluster

of orders is highly relevant when process fragmentation occurs. Depending on the characteristic

of the orders in the order cluster, and the workflow for each type of order in the order cluster, the

clinician will have a very different workflow. For example, if an order cluster has three orders

(see example in bottom row of table 2, below), the clinician may need to interact with a paper

chart only, both a chart and computer, or a computer only, depending on the implementation plan

chosen. No prior time motion studies or predictive models of CPOE implementation have

explicitly accounted for the concept of order clusters and process fragmentation when evaluating

the potential effects of CPOE on clinician workflow.

Our predictive model will explore three implementation scenarios: E-Meds: all computerized

entry of medication orders, no chart for medications; E-Most: computerized entry of

medications, laboratory, and echocardiography orders (no paper use); and E-All: all orders are

entered electronically, no use of paper chart. The three proposed implementation scenarios will

thus incorporate combinations of the order clusters highlighted in table 2 below.

11

Table 2: Types of order clusters, and clinician workflow for each type of order cluster in various

CPOE implementation scenarios

Media Required to create Order

Order Type Frequency*

%

Typical Example Current

State

E-Meds E-Most E-All

Medication Only 38% Lasix 40 mg po once daily Paper

chart

Computer Computer Computer

Laboratory/radiology or

echocardiography only

8% Electrolytes tomorrow

morning

Paper

chart**

Paper

chart**

Computer Computer

General order only 13% Daily weights Paper

chart

Paper

chart

Paper

chart

Computer

Medication plus

laboratory/radiology or

echocardiography

13% 1) Lasix 40 mg po once daily

2) Electrolytes tomorrow

morning

Paper

chart

Both Computer Computer

Medication plus general order 11% 1) Lasix 40 mg po once daily

2) Daily weights

Paper

chart

Both Both Computer

Laboratory/radiology or

echocardiography and General

Order

4% 1) Electrolytes tomorrow AM

2) Daily weights

Paper

chart

Paper

Chart

Both Computer

All types (medication,

laboratory/radiology/echocardio

graphy, and general order)

13% 1) Lasix 40 mg po once daily

2) Electrolytes tomorrow AM

3) Daily weights

Paper

chart*

Both Both Computer

*Data obtained from CPOE project team based on 300 order clusters written on D2 at Sunnybrook, October 2008.

**In some cases the clinician would use BOTH the chart and the computer, but not in the example on this table

E-Meds : computerized entry of medication orders no chart for medications

E-Most: computerized entry of medications, laboratory, and echocardiography orders (no paper use)E-All: all orders are entered

electronically no paper use

12

1.5 Methods for Evaluating Clinician Workflow

There are numerous methods for the evaluation of clinician workflow in a hospital

setting. Some of the more commonly used methodologies are highlighted in the table below.

Method of Workflow

Evaluation

Strength Weakness

Participant observation: whereby an

observer is immersed in daily

activities of the observed and makes

note of activities of interest

Observer sees all activities and can

concentrate only on the relevant

ones

The observer effect- Participants

may change behavior due to the

presence of the observer. Difficulty

in noting activities.

Work sampling: this technique is

used to measure work activity at set

intervals of time using an observer

or a self logged method

General understanding of activities

observed as observer observes

participants in groups

Easier for observer to blend in, as

many are being observed

simultaneously.

The observer effect- Participants

may change behavior due to the

presence of the observer. One

observer to multiple participants

thus some aspects of work habits

may be overlooked.

Self Reported Survey- Participants

respond directed questions.

Directed to large number of

participants and they can remain

anonymous

Responses are not always accurate

and or honest

Time Motion Studies: Method which

allows time stamps to be attached to

performing various activities.

Gives more detailed information

Observer is in close proximity to

participant and can log exact times

for activities.

The incidence of the observer effect

is greater.

Labor intensive as it requires one on

one observations

Random sampling: Participants are

given pagers which alarm to remind

them to record activities

Less disruptive from the observer’s

perspective and can be used on a 24

hour basis. No human observer

necessary

Individuals may still forget to record

activities and they may become

bothered by the alarms

Table 3: Advantages and disadvantages of workflow evaluation methods

Participant Observation and Time-motion studies were chosen for this study as they are

complementary methods for workflow evaluation. The participant observation formulates the

categories from the observed activities. The time motion studies then allow the association of a

time factor to the generated categories. Participant observation is a qualitative research method

involving direct participation of the researcher in the events being studied. In participant

13

observation the researcher is immersed in the day-to-day activities of the population being

studied. The objectives are to record conduct and reveal patterns for various tasks under the

widest range of possible settings. Data obtained through participant observation serve as a check

against participants’ subjective reporting of what they believe should be done and what they do.

Through this method, researchers can also uncover factors important for a thorough

understanding of the research problem but that were unknown when the study was designed.

(Pina 2006) Participant observation studies are an important tool in understanding how to

redesign CPOE systems to avoid harm and achieve the full potential of benefit for improved

patient safety. For example a participant observation study by Jamie Pina in 2006 on an EPR

system revealed that issues such as duplicate patient records arose due to understaffing in the

clinic’s data entry department and variations in the current staff’s levels of training on the

system. The information can be used to suggest and project work patterns when the change is

implemented.

Participant observations can be passive, whereby the observer does not take part in the

activities of the observed, or active where the observer participates and is involved in the daily

activities of the observed. Active participant observation is known to foster increased observer

acceptance as he/she appears to have a more significant role. Passive participant observation

however, is a less intrusive method than active participant observation as there is no observer

involvement in the observed activities. Thus active participant observation may in some ways

increase the occurrence of the “observer effect” which is behavioral change in the observed

brought about by the observer’s presence. .

14

From reviewed literature, time-motion studies have been noted to be the most accurate

method for capturing task duration in the evaluation of workflow compared to simple participant

observation random sampling and self reported surveys (Lo et al. 2007). Time motion studies are

used to measure time required for the completion of a particular task. The fragmented nature of

clinicians work and the relatively short duration of specific tasks makes a time-motion study a

critical research method choice in workflow analysis (Overhage et al. 2001) The time motion

study can highlight the time spent during each area of a clinician’s daily patient care workflow,

despite the direct (patient interaction) or indirect (no patient interaction) nature of the activities.

The time motion studies and participant observation can be performed simultaneously. Time

motion study results may identify aspects of order entry which are more time consuming and

allow CPOE project teams to predict the various ways in which a fully integrated EPR and

CPOE system may impact on task times.

The development of time-motion software that runs on Personal Data Assistants (PDAs)

has greatly increased the resolution and speed of data recording, which was previously done with

a stopwatch and paper, limiting the scope of such studies. (Khan 2006). Using stop watches to

record durations cannot provide information about the interactions of activities across care

providers. Capturing starting and ending times of activities along with a description of the

activity is difficult, particularly in an emergency department or a general medicine unit, due to

frequently changing and overlapping tasks. In this setting handheld computer applications have

been found useful in capturing activities at high granularity. The applications are capable of

highlighting the most general to the more specific activities performed by clinicians.(Asaro

2004) Previous studies dating as far back as 1993 have utilized time motion studies for time

15

capture using laptop computers or stopwatches. More recent studies, e.g. VanDenKerkhof et al in

2003, have incorporated handheld devices (PDAs) for increased accuracy and mobility.

1.6 Predictive Modeling

Predictive modeling involves the creation of a statistical model of future behavior. A

predictive model is composed of predictor variables that are likely to influence future behavior or

results. A good understanding of current state and future state workflow are also essential to

building a meaningful predictive model. To create a predictive model, data are collected from

workflow observation and time motion studies and then a predictive model is formulated.

Predictions are made and the model is validated (or revised) as additional data become available.

Predictive modeling will aid in the projection of the changes to clinician ordering

workflows which may be experienced after CPOE implementation. The predicted model may

serve as a guide to decisions about hardware and software revisions which are essential for the

success of POE systems.

Predictive modeling techniques include neural networks, Markov analysis and decision

trees. Before any analysis can take place, the modeling techniques must match the requirements

of the clinical problem, but should not be too complex for the available data (Chapman et al

2003). In addition, the issues of accuracy, simplicity, timeframe and availability of data are key

determinants of which decision model is best suited.

A neural network model is generated through a set of computer algorithms which mimic

human brain function. The purpose of a neural network is to learn to recognize patterns in data

16

by making various associations. The neural network model can be trained from data samples,

allowing it to make predictions by detecting similar patterns in future data. The major advantage

of neural networks lies in their ability to represent both linear and non-linear relationships and in

their ability to learn these relationships directly from the data being modeled. (Traditional linear

models are simply inadequate when it comes to modeling data that contains non-linear

characteristics.) The disadvantage of neural networks is that there is tremendous difficulty in

interpreting results, as the model structure does not relate directly to the features of the

underlying problem. Neural networks were deemed too complex for this preliminary analysis of

ordering workflow, given the simple nature of the data captured.

A Markov analysis uses a sequence of events, and analyzes the tendency of one event to

be followed by another. Using this analysis, it is possible to generate a new sequence of random

but related events, which will look similar to the original. A Markov process is useful for

analyzing dependent random events, i.e. events whose likelihood depends on what happened last.

Markov analyses allow modeling of complex systems over time and they rely on sequenced

events as in the case of the intention to order medication. The major limitation of this decision

analysis methodology is that in some cases Markov analyses are too complex for the situation

being analyzed and, similar to the neural networks, results are not easily interpreted.

A decision tree is a branching structure in which various node symbols are used to

represent different kinds of events, including decisions, probabilities and uncertainties, and a

node’s branches represent the outcomes or alternatives associated with that event. Every series of

actions and outcomes is clearly represented with a distinct path. This decision analysis model

offers easy construction and interpretable results, making it highly amenable to explanation and

17

human inspection. However, a decision tree represents a situation in its simplest form and as a

result assumptions must be clearly defined and accounted for.

Given the preliminary nature of our study and the nature of the captured data, decision

tree analysis was chosen to predict values for average ordering times given the various CPOE

implementation scenarios. We reviewed sample studies reported by Detsky et al (1997) to ensure

that this method would be sufficient to capture the key aspects of our study given the many

assumptions generated. Our relatively small data set and small number of probability variables

made using a decision tree a better choice.

18

Chapter 2: Objectives

1. To develop a nosology for provider workflow

2. To describe actual provider workflow, with a focus on (i) time spent with the paper chart (ii)

time spent with the computer workstation, and (iii) time spent with both chart and computer

workstation simultaneously

3. To model the predicted impact of computerized ordering on time to complete an order in three

future state CPOE implementation scenarios




orders.





(iii) “All Orders Electronic” (E-All): All orders only entered electronically by the

ordering provider; no orders written on paper chart.

19

Chapter 3: Materials and Methods

3.1 Study Environment

We enrolled a convenience sample of staff physicians and junior and senior residents working on

D2, a general medicine and nephrology ward at Sunnybrook Health Science Center. We chose

D2 because it is the ward where the CPOE pilot implementation at Sunnybrook is being

conducted. Participants were recruited on a voluntary basis via direct approach on the study unit.

Before each observation period, the observer informed participants about the purpose of the

study, after which the individual gave verbal consent to be observed. Each participant was given

a unique identifier to enable the observer to keep track of how many times the participant was

observed.

3.2 Participant Observation

Observations took place only on the patient care unit in the vicinity of the nursing station and the

hallways. One observer shadowed 5 clinicians on the study unit for 20 hours over 10 days (2

hours/day). The observer noted all activities and processes carried out daily for 20 pilot

observation periods highlighting usage of the patient charts and the EPR system. After the 13th

observation period it was noted that no new categories emerged, as the observer was able to

group the activities using previous information.

20

3.3 Time Motion Study

To capture the time spent on various clinical activities we utilized time motion study software

installed on Personal data assistants. (Figure 1 below)

Figure 1: HP IPAQ Personal Data Assistant.

Device: Hewlett Packard IPAQ, Personal Data Assistant Model: Rx3115 (above)

Operating system: Windows Mobile 2003

Time Motion study Software: UMTPlus® by Laubrass Inc installed on the 2 PDAs

The nosology developed during the participant observation phase of the study was

programmed into the time motion study software (UMTPlus). The configuration was then

uploaded into the PDA. The device utilized color touch screen technology which made it easy

for choosing the correct color coded keys and entering auxiliary study notes/ occurrences.

21

(a) (b)

(c) (d)

Figure 2 PDA screen for time motion study.(a) Main Study screen ,(b) Sub screen for activities such as

Reviewing Chart, Writing order and progress notes, phone orders and verbal orders (c) Sub screen for

Walking , (d) Sub screen for Other/ Miscellaneous activities.

Time-Motion Observations

One of the main focuses of the study was to determine the amount of time a clinician

spends interacting with the patient’s chart, computer or both. As a result we used the clinician,

rather than the chart or the patient, as the unit of analysis.

Each observation period lasted for 25-30 minutes. First, the level of training of

participant and the time of day were recorded. Using the time motion study software (UMTPlus)

22

installed on the PDA, the observer keyed in the activity and location of the participant and

focused on medication and services ordering tasks and usage of workstations (which are located

in the nursing stations and in a conference room) and 2 workstations on wheels (WOWs) located

in the hallway near the nursing station. The observer remained 8 feet from the participant for the

duration of observation, but did not enter patient rooms. The observer did not witness any

instances where the clinician took a chart into a patient’s room, and there are no workstations in

patient rooms.

After each session, participants were asked whether the presence of the observer made

them uneasy and ultimately affected their behavior. Finally, the observer retrieved the patient’s

paper chart(s) used by the clinician and recorded the characteristics of the newly written orders.

The observation data were uploaded to a desktop computer via a cradle connection and

imported into Microsoft Excel. This procedure was repeated for 105 observation periods. Our

target was 50 hours of observation, but the observations totaled 47 hours, as a few sessions were

cut short because participants were leaving the hospital or called upon for an emergency.

To evaluate inter-observer reliability of time motion study software, a second observer

was trained during a 30 minute session. The use of the tool was demonstrated and the structure of

the task categorization scheme was reviewed. The second observer then practiced on the

handhelds for 30 minutes for one observation period. The second observer had already conducted

workflow observations on D2 for other aspects of the CPOE project, and was very familiar with

the electronic ordering process at Sunnybrook. When the second observer was comfortable with

23

the software and categorization of activities, both observers simultaneously but independently

coded 5 observation periods (150 minutes of data).

3.4 Measures

Major categories of activity were: Reviewing chart, writing notes in chart, writing order,

walking, talking, in patient room and miscellaneous tasks. Sub-categories were based on

ordering media type (chart, computer or both) (See Table 2).

3.5 Other Data Sources

To supplement our analysis of the types of orders written, we obtained data from the

Sunnybrook CPOE team. These data are from 300 order clusters written for D2 patients in

October 2008 and sent to the D2 satellite pharmacy for verification. In general, orders that

contain at least one medication are sent to the pharmacy, so these data tend to over-represent

medication orders. Orders that contain no medication orders will tend not to be sent to the

pharmacy. Also, the CPOE project team had done studies measuring the time required to enter

10 medication orders into the test electronic system.

3.6 Data Analysis

UMTPlus included a statistical suite which was used to tabulate and perform the descriptive

statistics on the data captured during each observation period. The data was exported to a MS

Excel format. Microsoft Excel 2007 statistical analysis suite was then used to perform more in-

depth statistics and generate graphical results.

24

Interobserver reliability was evaluated using intraclass correlation coefficients. The data for the

two observers was entered into SPSS v15 (statistical program) and the intraclass coefficients

were generated along with the 95% confidence interval.

3.7 Predictive Modeling

The main output variable of the model was the average time required to write/enter a typical

order. The other aspects of CPOE, such as order acknowledgement by nurses, verification of

orders by pharmacists, and medication administration by nurses were not explored in this

predictive model.

We modeled current state ordering workflow, as well as future workflow changes after

CPOE implementation, based on several potential future state scenarios

Current state: Medication and general orders written on chart, Echo and diagnostic imaging

written on both chart and then entered into EPR/CPOE system

Future state scenarios:




orders.





25

(iii) “All Orders Electronic” (E-All): All orders only entered electronically by the ordering

provider; no orders written on paper chart.

The participant observation and time motion data are a central part of the predictive model.

The following variables were used in the predictive modeling/decision tree analysis. We used

data from our time motion study plus the data obtained from the Sunnybrook CPOE project team

to estimate the following variables:

• Proportion of Orders entered electronically without any written order

• Proportion of Orders written on paper chart AND entered electronically

• Proportion of orders written on the paper chart only.

We used data from the time motion study to estimate:

• The proportion of clinician time spent at a computer (either logged into EPR, or using the

computer for other reasons)

• The proportion of time spent with the paper chart

• The proportion of time with the paper chart AND at a computer.

We used data from the time motion study and as well as the data obtained from the CPOE project

team to estimate the following time variables:

• Time to Locate chart ( T_locate_chart )

• Time to Locate free workstation (T_findPC)

• Time to Log into EPR (Tlogon.EPR)

26

• Time to Log out of EPR (Tlogoff.EPR)

• Time to Enter order into EPR (Tenter.EPR)

• Time to Write order on chart (T write-order)

TreeAge ProR 2009 was used to model the decision trees for the above scenarios and to

perform decision analyses. The output of the decision tree was the average time to enter a typical

order in each of the four scenarios.

We performed one-way and two-way sensitivity analyses on the decision trees using various

combinations of the study variables.

3.8 Sample Size

A sample of 25 clinicians participated in this study. During the observation period there were

approximately 12 staff physicians or clinical associates and 20 residents on the five general

medicine teams. This represents the teams on service on the D2 general medicine unit for one

month. Each participant was observed between 1 to 5 periods for 25 – 30 minutes each. The time

period was derived during participant observation/nosology development periods, when it was

noted that 30 minutes was sufficient to capture the nature of clinician activities without

encountering unnecessary repetition. In addition we chose the time interval based on the fact that

participants may feel burdened by our presence if they were being observed for a prolonged

period of time.

27

3.9 Ethics

Prior to any observations and data collection, we obtained ethics approval from the Sunnybrook

Health Sciences Center’s and University of Toronto’s research ethics review board

28

Chapter 4: Results

4.1 Results: Objective 1: :osology Development (Table 4)

We successfully grouped the clinical activities centered on patient care and medication ordering

using paper charts and the available EPR system. From the table, activities were categorized into

major groups, e.g. review chart, writing order, talking, walking and miscellaneous. Further

groupings were constructed for activities which included the use of computers (for both EPR and

non EPR usage) and the patient charts simultaneously.

Table 4: Ordering activity 1osology. Generated from participant observation periods and review

of prior literature

Major Activity Categories Specific Activity

At WorkStation

PC Use and EPR Use

Pc Use and no EPR Use

No PC Use

Review Patient Chart

Not at Workstation

At Workstation no PC use

At Workstation PC use for Non EPR Activity

At Workstation PC use for EPR

Writing Med/Service Order

Not at Workstation




Writing Progress Notes in

Chart

Not at Workstation




Telephone Med/Service Order

Not at Workstation




29

Verbal Med/Service Order

Not at Workstation




Walking

On Unit

To Nurse Station

To other Unit

Look for Nurse

To Patient Room

Talking

To Clinician

To Patient

To Family

Miscellaneous Activities

Telephone Non- Ordering

Answering Page

Reading Reference Text

Locate Progress notes sheet

Locating Order sheet

Locate Auxiliary Forms

In Patient's Room

Checking MAR

Locating Patient's Chart

Waiting for Chart

Wait for Free workstation

Logging into EPR

Logging out of EPR

Reviewing EPR only

Ordering Services in EPR only

30

4.2 Results: Objective 2 (Tables 5,6a, 6b)

Table (5) depicts the characteristics of our study and the participants.

We observed 25 participants during 105 observation periods for a total of 47 hours of

observation. The majority of participants (84 %) were residents, as Sunnybrook is a teaching

hospital. Staff physicians accounted for the remaining 16%. Although the observer was

available at all times on weekdays, most of the observations (73%) were performed between 9am

– 12pm Monday to Friday, and the remaining 27% during the afternoon hours (12pm – 4pm) due

to the availability of participants.

Clinician Activity (Table 6a)

Major Activity No of

Occurrences

Percentage of Observation

time Total Time Spent (h:mm:ss)

Locate Patient Chart 79 1.1% 0:32:23

Write Med/Service Order 40 2.2% 1:06:52

Review Paper Chart 206 13.8% 6:39:11

Write Progress Notes 153 14.2% 6:40:16

In Patient Room 120 26.6% 12:31:18

At Workstation (No Chart) 101 11.4% 4:45:46

Logging into EPR system 51 0.7% 0:19:04

Ordering Services in EPR only 10 0.20% 0:14:42

Miscellaneous Activities 119 7.3% 3:34:50

Talking 231 19.6% 9:48:13

Walking 228 1.8% 0:50:12

Participant* (n=25) % of Observations Periods (n=105)

Junior Resident 52% (13)* 53%(56)

Senior Resident 28% (7)* 31%(32)

Staff Physician 20% (5)* 16%(17)

Time of Day

9am-12pm 73%(77)

12pm -4pm 27%(28)

* There were 25 different physicians observed during the study including 13 different junior

residents, 7 different senior residents and 5 different staff physicians.

31

Total -- 100.00% 47:02:47

Table 6a: Time Captured by activity

Table 6b: Usage of Patient chart and EPR system

We captured 47 hours of clinician activity during 105 observation periods. Table 6(above)

highlights the total time spent per major activity and the percentages of total observation time.

“In patient room” was the most time consuming category with 27%, while 20% of time was

spent talking to clinicians or family members. Also, 14% of the time was spent writing progress

notes and filling out other forms using the patient chart. For the duration of the observations, 1hr

06 minutes (2% of total time) was spent writing 40 order clusters in the patient chart (or about 95

seconds per order cluster), as well as 32 minutes (1.1% of time) locating 79 patient charts (or

about 25 seconds per chart).

From table 6b, clinicians spent 16% (7.5 hours) using the chart and PC simultaneously, while

11% was allotted to PC usage without the chart and 12 % to using the patient’s chart only.

Time Motion Study Data

Study Data from CPOE team

(October 2008)

Order Cluster Type

Frequency % (n=40 order

clusters)

Frequency % (n=300) order

clusters)

Medication orders only (M) 18%(7) 38% (115)

General Orders only (G) 25%(10) 13% (41)

EPR orderable services only (E) 25%(10) 8% (23)

Medication & EPR orderable services (ME)

5%(2) 13% (39)

Activity Percentage of Observation time Total Time Spent (h:mm:ss)

Chart Only 12% 5:42:27

PC/EPR system Only 11% 5:18:16

Chart and PC/EPR 16% 7:30:02

Other Clinical Activity 61% 28:32:02

Total 100% 47:02:47

32

Medication & General order & EPR (MEG)

3%(1) 13% (38)

Medication & General order (MG) 10%(4) 11% (32)

General Orders & EPR orderable services (GE)

15%(6) 4% (12)

Table 7: Characteristics of Order Clusters from Time Motion Study

(As described on page 11, EPR orderable services include diagnostic imaging, laboratory and

echocardiography orders. General orders include orders such as vital signs, activity,

consultations, and other tests that cannot currently be ordered in EPR.)

The above table (7) depicts the order cluster types of 40 orders written by clinicians during the

study. As previously mentioned we also obtained data and compared with 300 orders written by

clinicians in another study performed by the CPOE project team which was used in the

determination of the frequency of order types for our predictive model (See table 9). The CPOE

team data reflect only orders that were received by the inpatient pharmacy, so that orders that

include medications are overrepresented. From the current study, the majority of orders were

general (G) and EPR orderable (E) at 25% and medication orders followed with 18%. Order

clusters with all three types of orders (MEG) were the least common at 3%. These order cluster

data were used for the predictive modeling so that future state workflow could be predicted for

different future state electronic order entry scenarios. (Table 7)

Interobserver reliability

Time Spent per Activity (mm:ss)

Activity Observer 1 Observer 2

Review Chart 06:17 06:19

33

Write Progress Note 34:41 34:46

Write Med/Service Order 03:46 03:42

At Work Station 02:38 02:38

Locate Patient's Chart 02:43 02:43

Talking 15:33 15:09

Walking 01:06 01:12

In Patient's Room 25:39 25:29

Logging into EPR 02:35 02:40

Table 8: Results from 5 observation periods conducted by 2 observers

The table shows that the times captured by each observer differed by only a few seconds in every

case.

We performed intraclass correlation statistics for the observations from the 2 observers and the

analysis generated an intraclass correlation coefficient of +1.This indicates that each logged

activity time was identical for both observers.

34

4.3 Predictive Modeling (Decision Analysis)

a. Decision Tree

Figure 3: Decision tree

Figure 3 illustrates the decision tree created for data analysis. It begins with the decision to write

an order and the 3 options are using chart only, EPR only or a both chart and EPR. The branches

then separate into the availability of the chart and PC and whether or not the clinician is logged

on to the EPR System. The unbolded text represents the probability at each point on the tree. The

end terminals give the total value of each branch depicted by the addition of the variables defined

in table (11).

35

Variables used in decision tree (Tables 9 10, 11)

Percentage of types orders for each Scenario

Characteristics of Ordering Activity Current state E-Meds E-Most E-All

Orders entered into EPR Only 25% 38% 59% 100%

Orders entered into chart & EPR 23% 49% 28% 0%

Orders entered on chart only 52% 13% 13% 0%

Total 100% 100 100 100


E-Most: computerized entry of medications, laboratory, and echocardiography orders (no paper use)

E-All: all orders are entered electronically no paper use

Table 9: Predictive modeling variables

The values for each scenario were derived by simple addition of the order types reported in table

7 (pg 31-32) based on the scenario being implemented (Also See Appendix)

The table (9) shows the frequency of order types given current state workflows which require the

chart only, EPR only or a combination of the chart and EPR. Clinicians used the chart 52% of the

orders placed during the study. The remaining 48% (23%+25%) was almost evenly distributed

for orders placed using EPR system only and both the chart and the EPR system simultaneously.

Each future state ordering scenario has a different proportion of order clusters requiring

the chart, the computer or both. . As expected, table 9 shows a decrease in the frequency of use

of the paper chart across the 3 scenarios. It is important to note that for the future state scenario E

–Meds (medication orders are entered into EPR; all other orders are managed as per current

state), there is an increase in the proportion of order clusters that require both EPR and the paper

chart. This occurs because 18% of order clusters include a medication plus another type of

order. (See Table 7 and Table 2 in background)

Variable Activity Probability Range for sensitivity analyses

AT_PC At PC 0.18 0.02 – 0.8

Logged_On Logged On 0.56 0.02 – 0.9

Has_Chart Has Chart 0.34 0.02 – 0.9

Table 10: Probabilities

36

Table (10) highlights study generated probabilities for being at a PC, having a chart and being

logged on to the EPR system. For example from the study, 34% of the clinician’s time included

having the chart in their possession, giving the 0.34 value. For sensitivity analyses, we used wide

ranges of probabilities (0.02-0.9) to account for the limited number of observations and the

limited scope of observation.

Variable Name Activity Average(seconds)**

Range for sensitivity

analyses (seconds)

T_locate_chart Time to Locate Chart 25 5 - 60

T_findPC Time to a Free PC* 0* 0 -240

T_Epr_Logon Time to log onto EPR 22 5 -60

T_write_order Time to Write Order on Chart 95 10 -285

T_Epr_enter Time to enter order in EPR 35 10 – 105**

*there were no observations where clinicians waited for a pc

** range from finding of study by Ogura et al. 1985

Table 11: Averaged Variables for decision tree analysis

The variables used in the decision tree analysis were generated by averaging (total time spent

divided by the total number of occurrences) the time spent on each task from the time motion

study. For sensitivity analyses, we used wide ranges of values to account for the limited number

of observations and the limited scope of observations. The ranges for sensitivity analysis were

arbitrarily set as one third (lower limit) and three fold (upper limit) the base case value. For time

to enter order in EPR we used a similar range to findings by Ogura et al. 1985

37

4.4 Results: Base Case Analysis

The base case analysis predicted the time to order an order cluster of 110 seconds in current

state. This compares to our actual observation of 100 seconds from our 40 observed ordering

sessions. This result suggests that our model is reasonably well calibrated. For the E-med

scenario, the time required to order a cluster was 117 seconds (a 6% increase over current state).

By contrast, the time required to order for the E-Most scenario and the E-All scenarios were

lower than for current state (93 seconds and 55 seconds respectively). When interpreting this

base case result, it is important to note that the time required to find a workstation in the base

case analysis was 0 seconds, because we did not observe any clinicians waiting to find a

workstation.

4.5 One Way Sensitivity Analyses (Table 12-13)

We conducted one way sensitivity analyses over wide ranges for all variables. The time required

to order a cluster, and the preferred ordering strategy, was insensitive to the following variables:

probability that clinician has the chart, probability that the clinician is at a workstation, and

probability that the clinician is logged on to EPR when at a workstation. For all of these

variables, the absolute time required to order changed less than 20% and the ranking of each

scenario from shortest to longest time was unchanged.

38

Sensitivity Analyses (One Way)

Time Per Order Cluster (seconds)

Sensitivity Parameters Current E-Med E-Most E-All

Base Case 110 117 93 55

Probability that clinician has chart at time of ordering =90% (base

case =34%) 99 108 88 55

Probability that clinician has chart at time of ordering = 2% (base

case =34%) 113 119 95 55

Probability that clinician is at computer workstation at time of

ordering =80% (base case = 18%) 106 110 87 47

Probability that clinician is at computer workstation at time of

ordering = 2% (base case = 18%) 111 119 95 57

Probability that clinician is logged onto EPR system at time of

ordering = 90% (base case =56%) 109 116 92 53

Probability that clinician is logged on to EPR system at time of

ordering = 2% (base case =56%) 111 119 95 57




Table 12: One way Sensitivity analysis result for probability variables. For each row, the

preferred strategy is shown in BOLD.

Time Per Order Cluster (seconds)

Sensitivity Parameters Current E-Med E-Most E-All

Base Case 110 117 93 55

Time to log on to EPR = 60 secs ( base case - 22 secs) 127 147 123 89

Time to find computer workstation= 240 secs (base case - 0 secs) 204 288 265 252

Time to enter order into EPR system = 105 secs (base case - 35 secs) 143 178 154 125

Time to locate chart = 75 secs (base case - 25 secs) 134 137 107 55

Time to write order in chart = 285 secs (base case - 95 secs) 252 235 171 55




Table 13: One way Sensitivity analysis result for averaged time variables for each row, the

preferred strategy is shown in BOLD.

By contrast, the absolute values for time required to order, and the preferred ordering strategy,

were sensitive to the following variables: time to find a computer workstation, time to log on to

EPR, time to enter an order into EPR, and time to write an order in the chart. The average time

39

to order was significantly affected by the availability of PC, as the time taken to write and order

increased significantly. The time required to enter an order into EPR in each case also increased

the time it would take to write an order. As expected, time spent locating a chart had less effect

on the more EPR intensive scenarios (E-Most and E-All).

The following figures (4, 5, 6 &7) depict the changes to the preferred implementation scenario

based on one way sensitivity analyses on our study variables. The preferred scenario in all cases

is the line which suggests the time to order is minimized.

Figure 4: One way sensitivity analysis for time to find free PC.

The preferred ordering strategy was sensitive to the time to find a free PC workstation. At base

case (time to find free workstation = 0 seconds), the preferred strategy was E-all (all orders

40

electronic). When the time to find a free pc exceeds 132 seconds, then current state becomes the

preferred strategy over e-all. When the time to find a free pc exceeds 48 seconds, then current

state is preferred over e-most. Current state is always preferred over e-med (dominant strategy)

for all values of time to find a free pc.

Figure 5: One way sensitivity analysis for time to log onto EPR system

At base case (time to find log on to EPR system = 22 seconds), the preferred strategy was e-all

(all orders electronic). As the time required to logon to the EPR system increases E-All remains

the preferred strategy. When the time to logon to EPR is less than 5 seconds then E-Meds

becomes the preferred strategy over current state.

41

Figure 6: One way sensitivity analysis for time to enter order into EPR system

At base case (time to enter order into EPR = 35 seconds), the preferred strategy was E-all (all

orders electronic). Current state is always preferred over E-med if the time to enter an order

exceeds 17seconds and also favored over E-Most if time to enter an EPR order exceeds 75

seconds

42

Figure 7: One way sensitivity analysis for time to write order in chart

At base case (time to enter order in chart = 95 seconds), the preferred strategy was E-all (all

orders electronic). If the time to enter an order in the chart exceeds 50 seconds then the Emost

strategy is preferred.Also, if the time to write an order in the chart exceeds 160 seconds then the

E-meds scenario is preferred over the current state.

43

4.6 Two Way Sensitivity Analysis

In two-way sensitivity analyses, two of the variables in the decision tree are varied

simultaneously.

Time Required to Order (seconds)

Conditions Current E-Meds E-Most E-All

Base Case 110 117 93 55

Time to find computer workstation LONG = 240 secs (base case = 0

secs)

Time to enter order into EPR system LONG= 105 secs (base case = 35

secs)

238 349 326 322

Time to find computer workstation LONG= 240 secs (base case = 0

secs)

Time to enter order into EPR system SHORT= 10 secs (base case = 35

secs)

192 266 243 227


secs)

Time to log on to EPR LONG = 60 secs ( base case - 22 secs)

197 318 294 286


secs)

Time to log on to EPR SHORT = 5 secs ( base case = 22 secs)

221 275 251 236




Table 14: Two way Sensitivity analysis results For each row, the preferred strategy is shown in

BOLD.

We conducted two-way sensitivity analyses for variables that affected the model in our one-way

sensitivity analyses: “time to find computer workstation”, “time to enter order in EPR system”

and “time to logon to EPR system”. In our two way analyses, the dominant variable was time to

find a computer workstation. At an upper extreme value of 240 seconds to find a workstation,

the impact of other variables was minimal (table 14). The current state scenario was preferred

across all values of the other variables, and the relative ranking of the scenarios was consistent

across the tested ranges for the other variables. (Table 14)

44

Each line on the following two-way analysis graphs represents the “line of equality”. At any

point on this line both scenarios will be equally preferred given the combination of the variables

being explored. For combinations of the variables below the line and to the left the highlighted

scenario is preferred. Also, for combinations of the variables above the line and to the right, the

other highlighted scenario would be preferred.

Figure 8: Two way analyses for Time to Find PC and Time to log onto EPR (a)Current state &

E-Meds, (b)Current state & E-Most and (c) Current state and E-All

(a)

(c)

(b)

45

For figure 8a, if the time to log on to EPR is 5 seconds, then the time to find a pc must be 0

seconds for E-meds to be the preferred strategy. If the time to find a pc is 100 seconds then time

to log on must be 0 seconds for E-Meds to be preferred. Similarly for figure 8b, if the time to log

on to EPR is 70 seconds, then the time to find a pc must be 0 seconds for E-Most to be the

preferred strategy. If the time to find a pc is 80 seconds then time to log on must be 0 seconds for

E-Most to be preferred. Also for figure 8c, if the time to log on to EPR is 140 seconds, then the

time to find a pc must be 0 seconds for E-All to be the preferred strategy. If the time to find a pc

is 165 seconds then time to log on must be 0 seconds for E-All to be preferred.

Our two way analyses can define situations where certain scenarios will always (or

never) be preferred (Figures 8a-c). For example, figure 8a demonstrates that if the time to log on

to EPR is more than 5 seconds, then current state will always be preferred to the E-meds

strategy. Similarly, if the time to find a pc is more than 100 seconds, the current state will

always be the preferred strategy (regardless of time to log on). As shown in figure 8c, for E-all

to be preferred over current state, the time to find a pc must always be less than 165 seconds,

regardless of time to log on to EPR.

46

Figure 9: Two way analyses for Time to Find PC and Time to enter order into EPR system

(a)Current state & E-Meds, (b)Current state & E-Most and (c) Current state and E-All

For figure 9a, if the time to enter order into EPR system is 20 seconds, then the time to find a pc

must be 0 seconds for E-meds to be the preferred strategy. If the time to find a pc is 20 seconds

then time to enter order into EPR system must be 0 seconds for E-Meds to be preferred.

Similarly for figure 9b, if the time to enter order into EPR system is 80 seconds, then the time to

find a pc must be 0 seconds for E-Most to be the preferred strategy. If the time to find a pc is 100

seconds then time to enter order into EPR system must be 0 seconds for E-Most to be preferred.

(c)

(b) (a)

47

Also for figure 9c, if the time to enter order into EPR system is 140 seconds, then the time to find

a pc must be 0 seconds for E-All to be the preferred strategy. If the time to find a pc is 180

seconds then time to enter an order into EPR system must be 0 seconds for E-All to be preferred.

It is worth mentioning again that it currently takes about 25-35 seconds to enter a mediction

order in the current test system, so figure 9a suggests that E-meds strategy will never be

preferred over current state, no matter how many pcs we make available.

Figure 10: Two way analyses for Time to log on to EPR System and Time to Enter EPR Order

(a)Current state & E-Meds, (b)Current state & E-Most

For figure 10a, if the time to enter order into EPR system is 40 seconds, then the time to log on

to EPR system must be 0 seconds for E-meds to be the preferred strategy. If the time to log on to

EPR system is 40 seconds then time to enter order into EPR system must be 0 seconds for E-

Meds to be preferred. Similarly for figure 10b, if the time to enter order into EPR system is 100

seconds, then the time to log on to EPR system must be 0 seconds for E-Most to be the preferred

(a) (b)

48

strategy. If the time to log on to EPR System is 108 seconds then time to enter order into EPR

system must be 0 seconds for E-Most to be preferred

In the final two way analysis ( Current state Vs E-All) at all values of the variables :“time to find

a PC”, “time to log on to the EPR system” and “time to enter and order into EPR system”, E-all

was preferred over current state.

49

Chapter 6: Discussion

Our objectives were to develop a nosology for clinician workflow, use time motion studies to

describe activities focused on time spent with paper charts and at computer workstations, and to

predict the workflow changes causes by various phases of CPOE implementation. We developed

a nosology which successfully captured the activities of clinicians on the general medicine unit.

Through the time-motion study we described the workflow activities with a particular focus on

time spent with the paper chart and/or the computer workstation. We constructed a simple

decision tree that was reasonably well calibrated, based on comparison to results from our time

motion studies. Our predictive model revealed that the preferred future state ordering strategy is

highly sensitive to the time required to find a free workstation, enter order into EPR system and

to log onto the EPR system

6.1 Development of a :osology

Our nosology categorizing clinician activities differs from prior physician ordering workflow

studies. Prior studies have not sufficiently explored the mixed mode (paper and computer

workstation) analysis of incomplete CPOE environments. Generating a nosology focusing on

chart and PC based activities allows the capturing of single or simultaneous media usage in a

fragmented environment where both paper charts and computer workstation are used in the

ordering process. Lo et al. (2007) used media (PC and paper) as major categories and set the

action (e.g. “reading”, “writing” and “looking for”) as sub-categories. Overhage (2001) also

used a similar nosology using “computer”, “examine”, “looking for” and “phone” as major

categories and generalized the minor ones without considering the use of both media

50

simultaneously. The only common categories reported from the reviewed studies and the current

one were walking and talking. Our study differed by designating activities as major categories,

e.g. review chart and writing order in chart and at workstation and using sub categories for usage

of media (chart or PC). The nosology was similar to that of Bates, Shu et al. (2001) where they

categorized activities such as writing notes, writing orders, and reading the computer. They did

not, however explore, the clinician’s multitasking using the various media types.

For this study it was necessary to separate activities such as review chart, write notes and

write order to facilitate the simple capture of detailed time motion information while performing

these tasks and using a combination of the chart and EPR system. The nosologic system will be

useful for future workflow studies focused on other aspect of clinical care, such as clinical

documentation.

6.2 Description of Provider workflow

Our results are show some similarities and differences to prior studies of clinician ordering

workflow. (Table 15 below)

51

% of Physician Time Spent in Various Activities

Activity Current

Study

Bates et al

1998

Shu et al

2001

Tierney et al

1994

Bates et al

1994

Asaro et al

2004

Lo et al

2007

Locating Patient Chart 1.1 % 1.1% -- -- -- -- 6.7**

Write Order in Chart 2.2 % 2.1% 2.1% 2.5% 5.3% 3.6% 6.3*

Talking 19 % 38% 50% -- -- 18% 7

Walking

1.8% 2.7% -- -- -- -- --

Writing Progress Notes 14.2% 18% 14% -- -- -- --

Reviewing Chart 13.8% -- -- -- -- -- --

In Patient Room 27% -- -- -- -- -- --

Write Order in Chart while using

EPR for review 0.8% -- -- -- -- 2.6 % --

Write Progress notes in Chart

while using EPR for review 7.9% -- -- -- -- -- --

Review Patient Chart While

using EPR 6.5% -- -- -- -- -- --

Logging into EPR 0.7% -- -- -- -- --

*Included progress notes, orders and other forms.

** Included looking for chart, electronic results and clinician (no distinction was made between electronic and paper chart activities)

***Blank fields represent activities not captured by study

52

Table 15: Comparison of time-motion study results with previous studies

Our results are similar to Bates (1998) pre-CPOE workflow study, where clinicians spent 1.1%

of time locating the patient chart and 2.1% of time writing orders. Similarly Shu (2001) and

Tierney (1993) found 2.1% and 2.5% of time spent writing orders respectively. By contrast, Lo

found that clinicians spent 6.7% of time looking for charts and 6.3% of time writing orders. The

reason for this high value was because Lo used different methods for classifying the workflow

data. The locating chart option was grouped with locating other clinicians and electronic results.

Similarly, the writing category from Lo’s nosology included writing letters, scripts and other

miscellaneous documents. The above mentioned studies did not report actual times (hrs) for the

various activities. This made it impossible to generate ranges for our study using their data (e.g.

table 9).

Our study adds to existing knowledge by describing clinician activity using paper and computer

media. We found that clinicians spent 0.8% of their time writing orders while reviewing EPR.

The only similar data was reported by Asaro in 2004, who found 2.6 %. This value may be as a

result of the study location, as the emergency department may experience more ordering activity

than a general medicine unit. No other CPOE workflow study made similar groupings for the

usage of the paper chart and EPR system. Our study also highlighted many other important

workflow analysis categories, such as logging into the EPR system, writing progress notes while

reviewing the EPR system, and reviewing the chart while using the EPR system, which were not

captured in any of the prior workflow studies. Our preliminary reliability evaluation revealed

high levels agreement between observers, suggesting that our nosology is ready for formal

reliability testing.

53

6.3 Effects of Order Entry on Clinician Workflow

Our base case result differs from observations from other studies. Poissant et al. (2003)

reviewed 23 studies of the impact of electronic health records on efficiency, including 10 studies

on physicians, and 3 studies on computerized physician order entry. The studies were all

complete computerization of all order types (E-All). Based on the 3 CPOE studies from the

Possiant view, all three reported increased time spent ordering after implementation of CPOE (E-

All). Bates et al. (1994) reported a change from 5.3% to 10.3% on total time spent writing orders.

Shu et al (2001) found an increase from 2.1 % to 9.0%, and Tierney et al (1994) a change from

2.5% to 9.3% minutes) after CPOE implementation. These results are in sync with the findings

of this study on the current state workflow, where clinicians spent 2.1% of their time writing

orders. The predictive modeling results, however, were opposite to the findings of the studies for

post implementation order times. They revealed decreased order times as the shift was made

from paper to all electronic. These contrasting results may be due to assumptions that were made

in generating the decision trees. In our base case we assumed that time would be spent finding a

workstation, and that the clinician would not need find or refer to the paper chart (e.g. for clinical

information) while entering electronic orders.

The contrasting result with the other prior workflow studies may also be due to values for

certain variables used in our base case analysis. Our estimate for time to enter an order into EPR

was based on ten medication orders submitted by an experience user in the test EPR system. We

could expect the time required for an average user on the real EPR system would be longer.

Also the assumption for the base case scenario that clinicians spent 0 seconds on the study unit

finding a computer is unique to our study, as prior literature has not explored this aspect during

54

predictive modeling analyses. D2 had recently installed several new workstations, which may

explain why we did not observe any physician waiting for a free workstation during our time

motion studies. Finally, we assumed that clinicians would not need to locate and review the

paper chart (e.g. for clinical information) prior to entering electronic orders, so some of the

estimates of workflow in future state may be unduly optimistic. A more complex model could

address this limitation.

Overall sensitivity analyses suggested that the time required to find a computer

workstation, log onto EPR and enter an order electronically are the more important factors to be

considered when going forward with the various phases of implementation. In other words,

technology and software must be readily available and easily accessed for minimizing time and

maximizing the efficiency of the ordering processes. The increased time spent interacting with a

PC has been viewed as the major pitfall to CPOE implementation. The results of the study

indicate that clinicians on general medicine services already spend 18% of their time using a PC

for both EPR and Non EPR activities and 16% of their time using both the patient’s chart and PC

simultaneously, even though only 2% of their time is spent on ordering.

From the sensitivity analysis and also suggested in previous literature there must be

sufficient access to workstations and other devices ( laptops or PDAs) which are to be used for

the ordering of medication and services. The speed of logging into the EPR/CPOE application

plays a less important yet significant role in the time required for ordering, but is necessary for

the satisfaction of the user. Thus, single, quicker methods, such as access cards may be sought to

gain access to the ordering software.

55

During our time motion studies, we found that 25% of orders were entered into EPR without

a written paper chart order. This observation is noteworthy because in current state, clinicians

are never supposed to enter orders into EPR only; orders are supposed to be written in the paper

chart, then selected orders (radiology, echocardiography) are supposed to be entered into EPR.

This suggests that they are either violating current state policies, or that they were writing the

orders in the chart prior to, or after, the observation period. It also possible that some physicians

were entering laboratory orders directly into EPR without a written order for efficiency purposes,

even though current state policy is that physicians write laboratory orders, and nurses enter the

order into EPR.

Our predictive model was not sensitive to several variables, including time to locate chart,

and probability of being logged on to EPR. This means our analysis is likely applicable to other

order writing workflows on other hospital units. For example on surgical services, it is common

to do morning rounds with all paper charts on a wheeled chart rack, so that time to locate chart is

very short. By contrast, on D2, it is unusual for clinicians to assemble all paper charts on a

wheeled rack.

56

Chapter 7: Limitations

This preliminary study has several important limitations. First, our nosology was

developed and applied on only one inpatient ward at a single hospital. We recommend that the

classification be used for future workflow studies in different settings at Sunnybrook, to ensure

that the system is robust in many clinical settings.

Our nosology did not allow us to capture one of the major benefits of CPOE (remote

access ordering), as all observations took place in and around the nursing station. Observing

doctors ordering in all environments would further validate the value of implementing

computerized systems. It is therefore necessary that future studies explore this aspect of CPOE.

A second limitation is that our time motion observations were on physicians working in a

single hospital ward for patients who have already been admitted. The workflow for these

physicians could be very different in other hospital wards (e.g. the emergency room), and may be

different for other types of physicians such as surgeons, intensivists or psychiatrists. Another

limitation of this study may be attributed to its location. The general medicine unit is more

likely to see a slightly different population of clinicians and consequently different workflows

than that of a surgical ward. In addition, the captured workflows are only representative of a

small sample of clinicians at Sunnybrook Health Sciences Center, and may not represent

workflows of clinicians in other healthcare facilities, as every institution has varying protocols

57

and infrastructural constraints. However, our method for describing workflow, and predicting

future state workflows, should be applicable to other settings.

A third limitation is that we focused on the clinical workflow of junior, senior residents

and staff physicians on D2. Other D2 providers, such as nurses, pharmacists, dietitians and allied

health workers also write medication orders; however, they were not included in the study. In

addition, the workflow related to order acknowledgement, order verification, dispensing,

medication administration, and execution of other (non medication) orders were not considered.

In addition, all observations were performed during the day shift on the general medicine unit.

The majority of the observations (78%) were carried out during 9am to 12pm. These conditions

however may not be representative of the workflow and ordering activities on the unit on

evenings and weekends.

A fourth limitation of the time motion study was the choice of using the healthcare

provider as the frame of reference. Many previous studies have used the patient, paper chart or

chart rack as the frame of reference for observations. Overhage (2001) and Lo (2007) used the

patient as the frame of reference. However, in this study our focus was on the proportion of time

the clinician spent with different media (computer, paper chart, both or neither). Therefore, we

chose the clinician as the frame of reference. The 30 minute observation period may not have

been sufficient to capture the true workflow of clinicians. It was however chosen due to the

fragmented nature of the activities on the study unit and the difficulty in observing the same

individual for longer periods without affecting their workflow.

58

A fundamental limitation of any participant observation study is the possibility of the

observer effect. This is the condition whereby the presence of the observer alters the behavior

and routines of the observed, decreasing the significance of the results. The observer attempted

to curb its incidence by remaining at least 8 feet away from the participant at all times. In

addition, no communication was made with the observed until the observation periods were

concluded when participants were asked if they felt bothered by the observer. The general

responses were however that they had forgotten that they were being observed.

The small number of orders (n=40) placed during the study may have been due to the fact

that we only captured ward based ordering, and because we believed that the majority of orders

were written in the morning, which may have not been correct. Admission orders also increase

the ordering activities of clinicians. Further study in other settings where admission orders are

written, such as the emergency department, and at other times of day are needed to ensure

robustness of our time motion data.

We incorporated data from the CPOE project teams’ Test System were two expert users

estimated the time to enter orders into EPR. This value may not have been robust as actual times

would be significantly longer based on the number of users logged on and system processing

speed. However, the results from our sensitivity analysis show that this is an important variable

and therefore requires further study in future projects.

Our predictive model is a preliminary and imperfect attempt to model ordering workflow.

Detsky et al (1997), emphasize that the tree must adhere to the following guidelines to facilitate

59

sound results. The tree must be balanced, symmetrical and the branches must be linked. The

issue of balance refers to the probabilities that one branch clearly dominates the other. The

symmetry aspect of the tree relates to the repeated usage of probabilities for each strategic

branch. Symmetrical trees allow the rapid generation of sound sensitivity analyses, which reduce

the risk of programming errors. The tree generated for our study was not perfectly symmetrical,

because of the inclusion of combinations of EPR+ Chart branch. The sensitivity analyses were,

however, carefully performed and revealed logical results. The concept of branch linking also

suggests the same expressions should be used for similar variables and probabilities throughout

the structure to generate logical results. This guideline was also carefully followed in the

construction of our decision tree.

Our tree carried some fundamental assumptions about clinician workflow. We assumed

that the proportion of time observed with the chart or at a computer translated into a probability

of being with the chart or at a PC for our predictive model. The translated probabilities then

caused the predicted values to deviate from that of actual captured times slightly (E.g. predicted

current state 110 seconds per order cluster, actual value 100 seconds). This assumption was made

as we were only able to capture what the participant did and not what they wanted to do. A more

interactive time-motion study and participant observation would remedy this issue as observers

would always be aware of the intentions of the participants. Also, the EPR only branch assumed

that clinicians would not need to refer to the chart to order electronically (E.g. the use of the

chart to review results before ordering). We assumed that the order entry application would

always be fast and there would be no downtimes or system hang-ups. This assumption this

however is not representative of the actual system functioning. Increased numbers of non expert

60

users would significantly reduce entry and processing speed. A downtime scenario would also

mean major disruptions in workflow and general patient care. Our model assumed that the

ordering process is linear to some extent however; studies by Niazkhani et al. (2009) show that

ordering is a complex nonlinear task involving more than one provider (consults) and activity.

Despite these assumptions, we are encouraged that the predicted time to order of 110

seconds is similar to the actual time of order we observed in our study, and that the incremental

time to convert to medication orders (an increase of 6%) was similar to the increaser actually

observed by Overhage (2001).

61

Chapter 8: Recommendations

1. We recommend that Sunnybrook CPOE projects use our nosology for

capturing current state workflows. Our nosology adds to current knowledge

by capturing paper based and workstation activities, allowing for a more

precise prediction of future state impact of CPOE. It may also be adapted for

usage on other units, or for other providers, where workflows may differ.

2. We recommend that CPOE project teams continue to use our time-motion

methods to capture current and quantify current state workflow. The collection

of time-motion data by activity type rather than media provides a better view

of the fragmented, multitasking nature of clinician workflow. Further studies

should include a broader group of clinicians (e.g. pharmacists and surgeons)

on various units. There also needs to be more data captured for the following

variables: “time to find PC”, “time to log on” and “enter order into EPR

system” before using the variables in the predictive modeling analysis.

Observation periods should also be longer and should cover both day and night

shifts to ensure complete capturing of workflow. Prior to the use of our

nosology in time-motion studies, formal reliability evaluations are required for

further validation.

62

3. The assumptions made in our decision tree must be minimized for proceeding

versions. The data from the time motion studies used to construct the tree

must sufficiently capture both intended and actual activities performed by the

observed providers.

4. We recommend that the Sunnybrook CPOE project team focus on ensuring (in

order)

a. Adequate access to order entry workstations

b. Speed of the order entry application is minimized

c. Speed of log onto the EPR system is minimized

63

Chapter 9: Conclusion

Our developed nosology successfully captured the activities of clinicians on the general medicine

unit although it differed from that of prior studies. The time motion study allowed us to describe

the workflow activities focusing on time spent with the paper chart and/or the computer

workstation. They revealed that clinicians currently spend a significant amount of time using

workstations and very little time ordering medications and services. Our decision tree provided a

simplistic projection into the future for possible implementation scenarios. The predictive model

revealed that the preferred future state ordering strategy is highly sensitive to the time required to

find a free workstation, enter order into EPR system and to log onto the EPR system.

64

References

1. Aarts J, Doorewaard H, Berg M. Understanding implementation: the case of a

computerized physician order entry system in a large Dutch university medical center. J

Am Med Inform Assoc. 2004; 11:207–16.

2. Asaro Phillip V. Synchronized Time motion study in the Emergency department using a

Handheld Computer Application. International Medical Informatics Association,

Medinfo 2004:701-705

3. Bates, D.M., J.M. Teich, J. Lee, et al., The Impact of Computerized Physician Order

Entry on Medication Error Prevention. Journal of the American Medical Informatics

Association 6(4):313-321, (1999).

4. Bates David W., MD, MSc, Shu Kirstin, Naraimhan Deepa, Horsky Jan, Comparing

Time Spent Writing Orders on Paper and Physician Computer Order Entry. American

Medical Informatics Association Symposium Proceedings 1998,2000; 965

5. Callen Joanne L. MPH, PhD, Brathwaite Jeffrey, MIR, MBA, PhD., WestBrook J.,

MHA, PhD. Contextual Implementation Model: A Framework for assisting Clinical

Information System Implementations Journal of the American Medical Informatics

Association 2008;15:255-262.

6. Chapman Gretchen B. PhD, Decision making in healthcare. Cambridge series on

judgment and decision making, 2003; 5:100

7. Detsky Allan S.,Nagile Gary, Krahn Murray D., Naimark David, Redelmeier Donald,

Primer on Medical Decision Analysis: Part 2 – Building a Tree, Medical Decision

Making 1997;17 :126

8. Drazen Erica, Kilbridge Peter, Metzger Jane, and Turisco Fran, A primer on Physician

order entry, Califonia Healthcare Consulting Group September 2000;3-44

Goddard Bryan L., MD, Termination of a Contract to Implement an Enterprise Electronic

Medical Record System. Journal of the American Medical Informatics Association

2000:7:564-568

65

9. Khan Sharib A., MBBS MA, Payne Phillip R.O., MPhil, Johnson Stephen B., PhD,

Bigger Thomas, MD, Kukafka Rita, DrPH, MA. Modeling Clinical Trials Workflow in

Community Practice Settings. American Medical Informatics Association Symposium

Proceedings 2006; 419 -423

10. Kuperman Gilad J. MD, PhD, Gibson Richard F., MD, PhD, Computer Physician Order

Entry: Benefits, Costs, and Issues. Journal of the American Medical Informatics

Association 2003;139:31-39

11. Lee Fiona, PhD, Teich Jonathan M., PhD, Spurr Cynthia D., MBA, RN, Bates David W.,

MD, MSc., Implementation of Physician Order Entry: User Satisfaction and Self-reported

Usage Patterns, Journal of the American Medical Informatics Association Volume 3

Number 1, 1996 42-55

12. Lindo John. Corrigan Janet M., and Donaldson M., To Err Is Human: Building a Safer

Health System. Institute of Medicine, 1ational Academy Press (1999).

13. Lo Helen G., Newmark Lisa P., Yoon Catherine ,MS, Volk Lynn A., MHS, Carlson

Virginia L., Kittler Anne F., Lipincott Margaret, Wang Tiffany, Bates David

W.,MD,MSC, Electronic Health Records in Specialty Care: A Time-Motion Study,

Journal of the American Medical Informatics Association Volume 14 Number

5,2007:609 - 615

14. Niazkhani Zahra, Evaluating the Impact of CPOE Systems on Medical Workflow: a

Mixed Method Study, Institute of Health Policy and Management (iBMG), Erasmus

University Medical Center, Rotterdam, the 1etherlands, IOS Press 2008;881-882

15. Niazkhani Zahra MD, Pirnejad H. MD, PhD, Berg M.,MD,MA, PhD, The Impact of

Computerized Provider Order Entry Systems on InPatient Clinical Workflow: A literature

Review, Journal of the American Medical Informatics 2009;16:539-549

16. Niazkhani Z, van der Sijs H, Pirnejad H, Redekop WK, Aarts J. Same system, different

outcomes: Comparing the transitions from two paper-based systems to the same

computerized physician order entry system. International Journal of Medical Informatics

2009 Mar;78(3):170 – 81

66

17. Ogura H, Sagara E, Yamamoto K, Furutani H, Kitazoe Y,Takeda Y. Analysis of the

online order entry process in an integrated hospital information system. Comput Biol

Med.1985;15:381–95.

18. Overhage Marc J., MD, PHD, Perkins Susa, PHD, Tierney William M., MD, Mc Donald

Clement J., MD, Controlled Trial of Direct Physician Order Entry: Effects on Physicians’

Time Utilization in Ambulatory Primary Care Internal Medicine Practices. Journal of the

American Medical Informatics Association Volume 8 Number 4,2001:361- 371

19. Pina Jamie, M.S.P.H., Using Participant Observation for Organizational Discovery and Systems Analysis, American Medical Informatics Association Symposium Proceedings 2006;1064

20. Poissant Lise, PhD, Pereira Jennifer, MSc, Tamblyn Robyn, PhD, Kawasumi Yuko, MSc.

The Impact of Electronic Health Records on Time Efficiency of Physicians and Nurses:

A Systematic Review. Journal of the American Medical Informatics Association Volume

12 Number 5,2005:361- 371

21. Poon Eric G., Blumenthal D., Jaggi T.,. Honour Melissa M., Bates David W. and Kaushal

Rainu, Overcoming Barriers To Adopting And Implementing Computerized Physician

Order Entry Systems In U.S. Hospitals. Health Affairs, (2004) 184-190

22. Shojania Kaveh G. MD., Yokoe D, Fiskio J., Ma’luf N., Bates D. MD., Reducing

Vancomycin use utilizing a computer guideline. Journal of the American Medical

Informatics Association. 1998 Nov-Dec; 5(6):585-6.

23. Tierney WM, Miller ME, Overhage JM, McDonald CJ. Physician inpatient order writing

on microcomputer workstations. Effects on resource utilization. JAMIA 1993; 269:379-

383

67

Appendix

Raw Data for Two way sensitivity Analysis graphs (figures 8,9,10 pg 44,46 & 47)

Two way sensitivity data (Time to find Pc and Time to log on to EPR system)

Current State Time to Log on to EPR system (Seconds)

Time to Find PC ( Seconds) 5 10 15 20 25 30 35 40 45 50 55 60

0 103 105 107 109 111 113 116 118 120 122 124 126

20 111 113 115 117 119 121 123 126 128 130 132 134

40 118 121 123 125 127 129 131 133 136 138 140 142

60 126 128 131 133 135 137 139 141 144 146 148 150

80 134 136 138 141 143 145 147 149 151 154 156 158

100 142 144 146 148 151 153 155 157 159 161 164 166

120 150 152 154 156 158 161 163 165 167 169 171 174

140 158 160 162 164 166 169 171 173 175 177 179 181

160 166 168 170 172 174 176 179 181 183 185 187 189

180 173 176 178 180 182 184 186 189 191 193 195 197

200 181 184 186 188 190 192 194 196 199 201 203 205

220 189 191 194 196 198 200 202 204 206 209 211 213

240 197 199 201 204 206 208 210 212 214 217 219 221

E-Meds Time to Log on to EPR system (Seconds)


0 103 107 111 115 119 123 127 131 135 139 143 147

20 118 122 126 129 133 137 141 145 149 153 157 161

40 132 136 140 144 148 152 155 159 163 167 171 175

60 146 150 154 158 162 166 170 174 178 181 185 189

80 161 164 168 172 176 180 184 188 192 196 200 204

100 175 179 183 187 190 194 198 202 206 210 214 218

120 189 193 197 201 205 209 213 216 220 224 228 232

140 203 207 211 215 219 223 227 231 235 239 242 246

160 218 222 225 229 233 237 241 245 249 253 257 261

180 232 236 240 244 248 251 255 259 263 267 271 275

200 246 250 254 258 262 266 270 274 277 281 285 289

220 260 264 268 272 276 280 284 288 292 296 300 303

240 275 279 283 286 290 294 298 302 306 310 314 318

E-Most Time to Log on to EPR system (Seconds)


68

Two way sensitivity data (Time to find PC and Time to order into EPR system)

Current State Time to enter order into EPR system (seconds)


0 93 98 103 108 112 117 122 127 132 136 141 146

20 101 106 111 115 120 125 130 135 139 144 149 154

40 109 114 118 123 128 133 138 142 147 152 157 162

60 117 122 126 131 136 141 146 150 155 160 165 170

0 80 84 88 92 96 100 104 107 111 115 119 123

20 94 98 102 106 110 114 118 122 126 130 133 137

40 109 113 116 120 124 128 132 136 140 144 148 152

60 123 127 131 135 139 142 146 150 154 158 162 166

80 137 141 145 149 153 157 161 165 168 172 176 180

100 151 155 159 163 167 171 175 179 183 187 191 194

120 166 170 174 177 181 185 189 193 197 201 205 209

140 180 184 188 192 196 200 203 207 211 215 219 223

160 194 198 202 206 210 214 218 222 226 229 233 237

180 208 212 216 220 224 228 232 236 240 244 248 252

200 223 227 231 234 238 242 246 250 254 258 262 266

220 237 241 245 249 253 257 260 264 268 272 276 280

240 251 255 259 263 267 271 275 279 283 286 290 294

E-All Time to Log on to EPR system (Seconds)


0 39 44 48 53 57 62 66 71 75 80 84 89

20 56 60 65 69 74 78 83 87 92 96 101 105

40 72 77 81 86 90 95 99 104 108 113 117 122

60 89 93 98 102 107 111 116 120 125 129 134 138

80 105 110 114 119 123 128 132 137 141 146 150 155

100 121 126 130 135 139 144 148 153 157 162 166 171

120 138 142 147 151 156 160 165 169 174 178 183 187

140 154 159 163 168 172 177 181 186 190 195 199 204

160 171 175 180 184 189 193 198 202 207 211 216 220

180 187 192 196 201 205 210 214 219 223 228 232 237

200 203 208 212 217 221 226 230 235 239 244 248 253

220 220 224 229 233 238 242 247 251 256 260 265 269

240 236 241 245 250 254 259 263 268 272 277 281 286

69

80 125 129 134 139 144 149 153 158 163 168 173 177

100 132 137 142 147 152 156 161 166 171 176 180 185

120 140 145 150 155 160 164 169 174 179 184 188 193

140 148 153 158 163 167 172 177 182 187 191 196 201

160 156 161 166 170 175 180 185 190 194 199 204 209

180 164 169 174 178 183 188 193 198 202 207 212 217

200 172 177 181 186 191 196 201 205 210 215 220 225

220 180 185 189 194 199 204 209 213 218 223 228 233

240 188 192 197 202 207 212 216 221 226 231 236 240

E-meds Time to enter order into EPR system (seconds)


0 86 95 104 112 121 130 139 147 156 165 173 182

20 101 109 118 127 135 144 153 162 170 179 188 196

40 115 124 132 141 150 158 167 176 184 193 202 211

60 129 138 147 155 164 173 181 190 199 207 216 225

80 143 152 161 170 178 187 196 204 213 222 230 239

100 158 166 175 184 192 201 210 219 227 236 245 253

120 172 181 189 198 207 215 224 233 242 250 259 268

140 186 195 204 212 221 230 238 247 256 265 273 282

160 200 209 218 227 235 244 253 261 270 279 287 296

180 215 223 232 241 250 258 267 276 284 293 302 310

200 229 238 246 255 264 273 281 290 299 307 316 325

220 243 252 261 269 278 287 295 304 313 322 330 339

240 258 266 275 284 292 301 310 318 327 336 345 353

E-Most Time to enter order into EPR system (seconds)


0 63 72 80 89 98 106 115 124 133 141 150 159

20 77 86 95 103 112 121 129 138 147 155 164 173

40 91 100 109 118 126 135 144 152 161 170 178 187

60 106 114 123 132 141 149 158 167 175 184 193 201

80 120 129 137 146 155 163 172 181 190 198 207 216

100 134 143 152 160 169 178 186 195 204 213 221 230

120 149 157 166 175 183 192 201 209 218 227 236 244

140 163 172 180 189 198 206 215 224 232 241 250 259

160 177 186 194 203 212 221 229 238 247 255 264 273

180 191 200 209 217 226 235 244 252 261 270 278 287

200 206 214 223 232 240 249 258 267 275 284 293 301

220 220 229 237 246 255 263 272 281 289 298 307 316

240 234 243 252 260 269 278 286 295 304 312 321 330

70

E-All Time to enter order into EPR system (seconds)


0 20 30 40 50 60 70 80 90 100 110 120 130

20 36 46 56 66 76 86 96 106 116 126 136 146

40 53 63 73 83 93 103 113 123 133 143 153 163

60 69 79 89 99 109 119 129 139 149 159 169 179

80 85 95 105 115 125 135 145 155 165 175 185 195

100 102 112 122 132 142 152 162 172 182 192 202 212

120 118 128 138 148 158 168 178 188 198 208 218 228

140 135 145 155 165 175 185 195 205 215 225 235 245

160 151 161 171 181 191 201 211 221 231 241 251 261

180 167 177 187 197 207 217 227 237 247 257 267 277

200 184 194 204 214 224 234 244 254 264 274 284 294

220 200 210 220 230 240 250 260 270 280 290 300 310

240 217 227 237 247 257 267 277 287 297 307 317 327

Two way sensitivity data ( Time to log on to EPR system and Time to order into EPR system)

Current State Time to enter order into EPR system (seconds)

Log on to EEPR system

(seconds) 0 10 20 30 40 50 60 70 80 90 100 110

0 84 88 93 98 103 108 112 117 122 127 132 136

5 86 91 95 100 105 110 115 119 124 129 134 139

10 88 93 98 102 107 112 117 122 126 131 136 141

15 90 95 100 104 109 114 119 124 128 133 138 143

20 92 97 102 107 111 116 121 126 131 135 140 145

25 94 99 104 109 114 118 123 128 133 138 142 147

30 97 101 106 111 116 121 125 130 135 140 145 149

35 99 104 108 113 118 123 128 132 137 142 147 152

40 101 106 110 115 120 125 130 134 139 144 149 154

45 103 108 113 117 122 127 132 137 141 146 151 156

50 105 110 115 120 124 129 134 139 144 148 153 158

55 107 112 117 122 127 131 136 141 146 151 155 160

60 110 114 119 124 129 134 138 143 148 153 158 162

E-Meds Time to enter order into EPR system (seconds)

71


(seconds) 0 10 20 30 40 50 60 70 80 90 100 110

0 69 78 87 95 104 113 121 130 139 147 156 165

5 73 82 90 99 108 117 125 134 143 151 160 169

10 77 86 94 103 112 120 129 138 147 155 164 173

15 81 90 98 107 116 124 133 142 150 159 168 177

20 85 93 102 111 120 128 137 146 154 163 172 180

25 89 97 106 115 123 132 141 150 158 167 176 184

30 93 101 110 119 127 136 145 153 162 171 180 188

35 97 105 114 123 131 140 149 157 166 175 184 192

40 100 109 118 127 135 144 153 161 170 179 187 196

45 104 113 122 130 139 148 157 165 174 183 191 200

50 108 117 126 134 143 152 160 169 178 187 195 204

55 112 121 130 138 147 156 164 173 182 190 199 208

60 116 125 133 142 151 160 168 177 186 194 203 212

E-Most Time to enter order into EPR system (seconds)


(seconds) 0 10 20 30 40 50 60 70 80 90 100 110

0 46 54 63 72 81 89 98 107 115 124 133 141

5 50 58 67 76 84 93 102 111 119 128 137 145

10 54 62 71 80 88 97 106 114 123 132 141 149

15 57 66 75 84 92 101 110 118 127 136 144 153

20 61 70 79 87 96 105 114 122 131 140 148 157

25 65 74 83 91 100 109 117 126 135 144 152 161

30 69 78 87 95 104 113 121 130 139 147 156 165

35 73 82 90 99 108 117 125 134 143 151 160 169

40 77 86 94 103 112 121 129 138 147 155 164 173

45 81 90 98 107 116 124 133 142 151 159 168 177

50 85 94 102 111 120 128 137 146 154 163 172 181

55 89 97 106 115 124 132 141 150 158 167 176 184

60 93 101 110 119 127 136 145 154 162 171 180 188

E-All Time to enter order into EPR system (seconds)


(seconds) 0 10 20 30 40 50 60 70 80 90 100 110

0 0 10 20 30 40 50 60 70 80 90 100 110

5 4 14 24 34 44 54 64 74 84 94 104 114

10 9 19 29 39 49 59 69 79 89 99 109 119

15 13 23 33 43 53 63 73 83 93 103 113 123

20 18 28 38 48 58 68 78 88 98 108 118 128

25 22 32 42 52 62 72 82 92 102 112 122 132

72

30 27 37 47 57 67 77 87 97 107 117 127 137

35 31 41 51 61 71 81 91 101 111 121 131 141

40 36 46 56 66 76 86 96 106 116 126 136 146

45 40 50 60 70 80 90 100 110 120 130 140 150

50 45 55 65 75 85 95 105 115 125 135 145 155

55 49 59 69 79 89 99 109 119 129 139 149 159

60 54 64 74 84 94 104 114 124 134 144 154 164

Responses for Post observation Questions: Did the presence of the observer change the way you

performed your duties?

Observation Number Participant Responses to the presence of the Observer on their Behavior

1 Forgot you were there.

2 Not at all




6 Was weird knowing you were being watched but it was fine

7 No communication

8 No communication

9 No communication

10 no


12 Not at all

13 Not at all

14 Not at all

15 Not at all

16 No

17 No

18 No

19 No

20 No

21 No

22 No

23 No

24 No

73

25 No

26 No

27 No

28 No

29 No

30 Not at all

31 Not at all

32 Not at all

33 Not at all

34 Nothing communicated

35 no

36 no




40

41

42 no

43 Not at all

44 no

45 Not at all

46 Not at all

47 Not at all

48 Not at all

49 Not at all

50 You are like staff so it didn’t bother me


52 Not at all

53 Not at all


55 no

56 no

57 no

58 no

59 no








74

67 didn’t bother me



70 No






76 no

77 no

78 no

79 no

80 no






86 NO

87 NO

88 NO

89 NO

90 NO

91 NO

92 NO

93 NO

94 NO

95 NO






75

Percentages of Order types ( for tables 7 and 9)

Time Motion Study Data

Data from CPOE team

(October 2008)

Order Cluster Type

Frequency % (n=40 order

clusters)

Frequency % (n=300) order

clusters)

Medication orders only (M) 18%(7) 38% (115)

General Orders only 25%(10) 13% (41)

EPR orderable services only (E) 25%(10) 8% (23)

Medication & EPR orderable services (ME)

5%(2) 13% (39)

Medication & General order & EPR (MEG)

3%(1) 13% (38)

Medication & General order (MG) 10%(4) 11% (32)

General Orders & EPR orderable services (GE)

15%(6) 4% (12)

E-Most – medication, radiology, echo-cardiology and laboratory orders are entered electronically.

General orders and others written in patient chart.

Therefore, using the CPOE project team data,

Percentage of orders entered into EPR system only = 38+8+13= 59%

Percentage of orders entered into chart and EPR system= 13+11+4 =28%

Percentage of orders entered into chart only = 13%

Total =100%

computerized provider order entry: initial analysis of current and … · 2010. 12. 10. · errors,...

Documents