onc hit program test results summary for 2014 …...test results summary for 2014 edition ehr...

Test Results Summary for 2014 Edition EHR Certification

4788782557R1PRA‐1.0, December 17, 2018

Part 1: Product and Developer Information1.1 Certified Product Information

1.2 Developer/Vendor Information

Part 2: ONC‐Authorized Certification Body Information2.1 ONC‐Authorized Certification Body Information

12/17/18

ThinkHealth ‐ Practice Management Software (EHR)

ONC HIT Certification Program


Product Name:

Product Version: 2.1.0.0

Domain: Ambulatory

Developer/Vendor Contact: Clyde Wafford

Oklahoma City, OK 73114

Website: www.iorion.com

Test Type: Modular EHR

Developer/Vendor Name: OrionNet Systems, LLC

Address: 510 E. Memorial Rd., Suite 2

ONC‐ACB Name: InfoGard Laboratories

Address: 709 Fiero Lane Suite 25

Email: [email protected]

Phone: (405) 286‐1674

ONC‐ACB Contact: Milton Padilla

This test results summary is approved for public release by the following ONC‐Authorized Certification Body

Representative:

Milton Padilla EHR Certification Body Manager

ONC‐ACB Authorized Representative Function/Title

San Luis Obispo, CA 93401

Website:


Phone: (805) 783‐0810

https://industries.ul.com/medical‐and‐

laboratory/healthcare‐it‐security

Signature and Date

©2018 InfoGard. May be reproduced only in its original entirety, without revision

Template 24 V3.7 1


4788782557R1PRA‐1.0, December 17, 2018

2.2 Gap Certification

(a)(1) (a)(19) (d)(6) (h)(1)

(a)(6) (a)(20) (d)(8) (h)(2)

(a)(7) (b)(5)* (d)(9) (h)(3)

(a)(17) (d)(1) (f)(1)

(a)(18) (d)(5) (f)(7)**

*Gap certification allowed for Inpatient setting only

**Gap certification allowed for Ambulatory setting only

2.3 Inherited CertificationThe following identifies criterion or criteria certified via inherited certification

(a)(14)

(c)(3)

(f)(1)

(a)(2)

(a)(15)

(d)(1)

(f)(2)

The following identifies criterion or criteria certified via gap certification

§170.314

No gap certification

§170.314

(a)(3)

(a)(16) Inpt. only

(d)(2)

(f)(3)

(a)(4)

(a)(17) Inpt. only

(d)(3)

(f)(4) Inpt. only

(a)(1)

(a)(7)

(b)(3) (d)(6) (a)(8)

(b)(4) (d)(7)

(a)(5)

(b)(1) (d)(4) (a)(6)

(b)(2) (d)(5)

(e)(2) Amb. only

(g)(4)

(a)(9)

(b)(5) (d)(8)

(g)(1)

(a)(10)

(b)(6) Inpt. only (d)(9) Optional

(g)(2)

(b)(8) (a)(13)

(c)(2)

(e)(3) Amb. only

No inherited certification

(f)(5) Amb. only

(f)(6) Amb. only

(f)(7) Amb. Only

(h)(1)

(h)(2)

(h)(3)

(a)(11)

(b)(7) (e)(1)

(g)(3)

(a)(12)

(c)(1)

(b)(9)

(a)(18)

(a)(19)

(a)(20)


Template 24 V3.7 2


4788782557R1PRA‐1.0, December 17, 2018

Part 3: NVLAP‐Accredited Testing Laboratory Information

5/11/2015, 5/12/2015, 5/15/2015

3.1 NVLAP‐Accredited Testing Laboratory Information

3.2 Test Information

3.2.1 Additional Software Relied Upon for Certification

2014‐EHRA552649‐2015‐0515‐00

Test Date(s):

ATL Name: ICSA Labs, an independent division of

Report Number:


Phone: 717.790.8100

ATL Contact: Michelle Knighton

Accreditation Number: NVLAP Lab Code 200697‐0

Address: 1000 Bent Creek Boulevard, Suite 200

Mechanicsburg, PA 17050

Website:https://www.icsalabs.com/technology‐program/onc‐

ehr

Signature and Date

Additional Software Applicable CriteriaFunctionality provided by

Additional Software

For more information on scope of accreditation, please reference

http://ts.nist.gov/Standards/scopes/1004320.htm

Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory

Representative:

Michelle Knighton Health IT Test Lab Manager

ATL Authorized Representative Function

Michelle Knighton 5/15/15


Template 24 V3.7 3


4788782557R1PRA‐1.0, December 17, 2018

No additional software required

3.2.2 Test Tools

Version

2.6.0

1.0.5

1.7.2

180

3.0.3

No test tools required

3.2.3 Test Data

3.2.4 Standards

3.2.4.1 Multiple Standards Permitted

EMR DirectONC 314b1, ONC

314b2, ONC 314e1The HISP we used for Direct

Microsoft SQL Server

ONC 314a1 through

ONC 314a8, ONC

314a11, ONC 314a14,

ONC 314a15, ONC

314b1 through ONC

314b7, ONC 314c1

through ONC 314c3,

ONC 314d1 through

ONC 314d8, ONC

314e1 through ONC

314e3, ONC 314g2,

ONC 314g3, ONC

314g4

All data accessed, saved or

updated is stored in the SQL

database. The system uses SQL

as the backend database.

HL7 v2 Laboratory Restults Intervace (LRI) Validation Tool

HL7 v2 Syndromic Surveillance Reporting Validation Tool

Transport Testing Tool

Direct Certificate Discovery Tool

Alteration (customization) to the test data was necessary and is

described in Appendix [insert appendix letter ]

No alteration (customization) to the test data was necessary

Test Tool

Cypress

ePrescribing Validation Tool

HL7 CDA Cancer Registry Reporting Validation Tool

HL7 v2 Electronic Laboratory Reporting (ELR) Validation ToolHL7 v2 Immunization Information System (IIS) Reporting

Valdiation Tool

Edge Testing Tool


Template 24 V3.7 4


4788782557R1PRA‐1.0, December 17, 2018

(a)(13)

§170.207(a)(3)

IHTSDO SNOMED CT®

International Release July 2012

and US Extension to SNOMED

CT® March 2012 Release

§170.207(j)

HL7 Version 3 Standard: Clinical

Genomics; Pedigree

(a)(15)(i)

§170.204(b)(1)

HL7 Version 3 Implementation

Guide: URL‐Based

Implementations of the Context‐

Aware Information Retrieval

(Infobutton) Domain

§170.204(b)(2)


Guide: Context‐Aware

Knowledge Retrieval

(Infobutton) Service‐Oriented

Architecture Implementation

Guide

The following identifies the standard(s) that has been successfully

tested where more than one standard is permitted

Criterion # Standard Successfully Tested

(a)(8)(ii)(A)(2)

§170.204(b)(1)


Guide: URL‐Based

Implementations of the Context‐

Aware Information Retrieval

(Infobutton) Domain

§170.204(b)(2)


Guide: Context‐Aware

Knowledge Retrieval

(Infobutton) Service‐Oriented

Architecture Implementation

Guide

(b)(7)(i)

§170.207(i)

The code set specified at 45 CFR

162.1002(c)(2) (ICD‐10‐CM) for

the indicated conditions

§170.207(a)(3)

IHTSDO SNOMED CT®




(e)(1)(i) Annex A of the FIPS Publication 140‐2

AES‐128, SHA‐1

(a)(16)(ii)

§170.210(g)

Network Time Protocol Version 3

(RFC 1305)

§170. 210(g)

Network Time Protocol Version

4 (RFC 5905)

(b)(2)(i)(A)

§170.207(i)


162.1002(c)(2) (ICD‐10‐CM) for


§170.207(a)(3)

IHTSDO SNOMED CT®




(b)(8)(i)

§170.207(i)


162.1002(c)(2) (ICD‐10‐CM) for


§170.207(a)(3)

IHTSDO SNOMED CT®




(e)(1)(ii)(A)(2)

§170.210(g)

Network Time Protocol Version 3

(RFC 1305)

§170. 210(g)

Network Time Protocol Version

4 (RFC 5905)

(e)(3)(ii) Annex A of the FIPS Publication 140‐2


Template 24 V3.7 5


4788782557R1PRA‐1.0, December 17, 2018

None of the criteria and corresponding standards listed above are applicable

3.2.4.2 Newer Versions of Standards

No newer version of a minimum standard was tested

3.2.5 Optional Functionality

Common MU Data

Set (15)

§170.207(a)(3)

IHTSDO SNOMED CT®




§170.207(b)(2)


162.1002(a)(5) (HCPCS and CPT‐

4)

The following identifies the newer version of a minimum standard(s) that has

been successfully tested

Newer Version Applicable Criteria

AES‐128, SHA‐1

(b)(1)(i)(B)

Receive summary care record using the standards

specified at §170.202(a) and (b) (Direct and XDM

Validation)

(b)(1)(i)(C) Receive summary care record using the standards

specified at §170.202(b) and (c) (SOAP Protocols)

(b)(2)(ii)(B)

Transmit health information to a Third Party using

the standards specified at §170.202(a) and (b) (Direct

and XDM Validation)

Criterion # Optional Functionality Successfully Tested

(a)(4)(iii) Plot and display growth charts

(b)(2)(ii)(C)

Transmit health information to a Third Party using

the standards specified at §170.202(b) and (c) (SOAP

Protocols)

(f)(3)

Ambulatory setting only – Create syndrome‐based

public health surveillance information for transmission

using the standard specified at §170.205(d)(3) (urgent

care visit scenario)

Common MU Data

Set (15)

Express Procedures according to the standard

specified at §170.207(b)(3) (45 CFR162.1002(a)(4):

Code on Dental Procedures and Nomenclature)

(e)(1)

View, download and transmit data to a third party

using the standard specified at §170.202(d) (Edge

Protocol IG version 1.1)

(f)(7) Ambulatory setting only – transmission to public

health agencies – syndromic surveillance ‐ Create Data

Elements


Template 24 V3.7 6


4788782557R1PRA‐1.0, December 17, 2018

No optional functionality tested

3.2.6 2014 Edition Certification Criteria* Successfully Tested

TP** TD*** TP** TD***

1.3 1.5 1.12

1.2 1.2

1.2 1.4 1.6

1.4 1.3 1.3

1.4 1.3 1.2

1.3 1.4 1.2

1.3 1.3 1.2

1.3 1.2

1.3 1.3 1.2

1.2 1.4 1.2

1.3 1.11 1.5

1.3 1.2 1.6

1.2 1.3

1.2 1.2 1.2

1.5 1.3 1.3.0

1.4 1.2 1.3 1.3.0

1.2 1.3 1.3.0

1.1 1.5

1.1 1.5

1.1 1.5

1.7 1.4

1.4 1.6 1.1

1.4 1.0.4 2.0 2.0

1.3 1.4 2.0 2.0

1.4 1.7.0 1.4

1.3 1.3.0 1.2

1.5 1.7 1.1

1.2 1.1

1.1 1.4 1.1

1.12

1.12

Common MU Data

Set (15)

Express Procedures according to the standard

specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD‐

10‐PCS)

Criteria #Version

Criteria #Version

(a)(4) (d)(3)

(a)(5) (d)(4)

(a)(6) (d)(5)

(a)(1) (c)(3)

(a)(2) (d)(1)

(a)(3) (d)(2)

(a)(10) (d)(9) Optional

(a)(11) (e)(1)

(a)(12) (e)(2) Amb. only

(a)(7) (d)(6)

(a)(8) (d)(7)

(a)(9) (d)(8)

(a)(16) Inpt. only (f)(3)

(a)(17) Inpt. only (f)(4) Inpt. only

(b)(1)

(f)(5) Optional & Amb. only

(b)(2)

(a)(13) (e)(3) Amb. only

(a)(14) (f)(1)

(a)(15) (f)(2)

(a)(18)

(a)(20)

(a)(19)

(b)(7)

(g)(3)

(c)(1)

(g)(4)

(c)(2)

(b)(3)

(f)(6) Optional & Amb. only

(b)(4)

(b)(5)

(g)(1)

(b)(6) Inpt. only

(g)(2)

(b)(8)

(b)(9)

(f)(7) Amb. only

(h)(1)

(h)(2)

(h)(3)

1.2 1.2

1.4 4.0

*For a list of the 2014 Edition Certification Criteria, please reference

http://www.healthit.gov/certification (navigation: 2014 Edition Test Method)

**Indicates the version number for the Test Procedure (TP)

***Indicates the version number for the Test Data (TD)


Template 24 V3.7 7


4788782557R1PRA‐1.0, December 17, 2018

3.2.7 2014 Clinical Quality Measures*Type of Clinical Quality Measures Successfully Tested:

CMS ID Version CMS ID Version CMS ID Version CMS ID Version

2 90 136 155 v3

22 117 137 v3 156 v3

50 122 138 v3 157

52 123 139 158

56 124 140 159

61 125 141 160

62 126 v3 142 161

64 127 143 163

65 128 v3 144 164

66 129 145 165 v3

68 v4 130 146 166

69 131 147 167

74 132 148 169

75 133 149 v3 177

77 134 153 179

82 135 154 182

CMS ID Version CMS ID Version CMS ID Version CMS ID Version

9 71 107 172

26 72 108 178

30 73 109 185

31 91 110 188

32 100 111 190

53 102 113

55 104 114

60 105 171

*For a list of the 2014 Clinical Quality Measures, please reference

http://www.cms.gov (navigation: 2014 Clinical Quality Measures)

Ambulatory CQMs

Inpatient CQMs

Ambulatory

Inpatient

No CQMs tested


Template 24 V3.7 8


4788782557R1PRA‐1.0, December 17, 2018

3.2.8 Automated Numerator Recording and Measure Calculation

3.2.8.1 Automated Numerator Recording

Automated Numerator Recording was not tested

3.2.8.2 Automated Measure Calculation

Automated Measure Calculation was not tested

3.2.9 Attestation

(e)(1)

(a)(6) (a)(15) (b)(3)

(a)(3) (a)(12) (a)(19) (b)(8)

(a)(4) (a)(13) (a)(20) (b)(9)

(e)(2)

Quality Management System** B

Privacy and Security C

*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16),

(a)(18), (a)(19), (a)(20), (b)(3), (b)(4), (b)(9)

**Required for every EHR product

(a)(7) (a)(16) (b)(4)

(a)(3) (a)(12) (a)(19) (b)(8)

(a)(4) (a)(13) (a)(20) (b)(9)

(a)(7) (a)(16) (b)(4)

Automated Numerator Recording Successfully Tested

(a)(1) (b)(6)

(a)(5) (a)(14) (b)(2)

Safety‐Enhanced Design* A

(a)(5) (a)(14) (b)(2)

Attestation Forms (as applicable) Appendix

(a)(11) (a)(18)

Automated Numerator Recording Successfully Tested

(a)(1) (a)(11) (a)(18) (b)(6)

(e)(1)

(e)(3)

(a)(9) (a)(17) (b)(5)

(a)(9) (a)(17) (b)(5)

(e)(2)

(e)(3)

(a)(6) (a)(15) (b)(3)


Template 24 V3.7 9


4788782557R1PRA‐1.0, December 17, 2018

Appendix A: Safety Enhanced Design


Template 24 V3.7 10

OrionNet System UDC Design Process

The design process that we use is based off of the Participatory Design process.

Participatory Design

• An approach to design attempting to actively involve all stakeholders (e.g. employees, partners, customers, citizens, end users) in the design process to help ensure the result meets their needs and is usable. The term is used in a variety of fields as a way of creating environments that are more responsive and appropriate to their inhabitants' and users' cultural, emotional, spiritual and practical needs. Participatory design is an approach which is focused on processes and procedures of design and is not a design style.

While we use much of what this design process there are things we do differently. In our design process participants are invited to help with designing new parts of the system. The users can work with our team to hammer out details, discuss rules, design flow, etc. They work with us during several stages but we begin the initial exploration and design and bring to the table at the start a rough process to begin the conversation. Dealing with many agencies mean we have to kick off the process then everyone can be part of the process by starting with the same base. The user helps us with problems, building on the initial design, ideas to that add to the solution, etc. The users do not help with the development but they continue to be part of the process as we roll out prototypes, screen shots of design, and initial testers to evaluate the solution.

Outline of how we use our design:

• OrionNet System team gathers details on a new function o We lay out the details of the new function o Describe a general flow of the function as it would fit into our system

• Details we create are sent to our full user community o Users come together with OrionNet System team to discuss the function o User work with our team on the initial details to make sure we know all the

Rules Processes Needed data elements

o Discuss the propose flow o Discuss how screens can be used to make the user job easier o Discuss other areas in the system that the new function would be tied to

• OrionNet System takes details from these meeting and layout new screen design and program flow

• OrionNet Systems brings back to the user group the propose changes o Mockups are shown to the users

o Prototypes are created so users can try the flow o Users comments and concerns are taken and evaluated

• OrionNet System then build into our system the new function o Users can sign up for beta testers of the new function o User receive the new function before others to use and try out

• We roll the new function to all users once the design looks good.

ThinkHealth – Practice Management Software (EHR) EHR Usability Test Report Test Dates: March 16, 2015 Report Date: April 14, 2015 Prepared By: OrionNet Systems, LLC. 510 E. Memorial Road Oklahoma City, OK 73013

Executive Summary A usability test of ThinkHealth – Practice Management Software (EHR) Version 2.1.0.0 was conducted on March 16, 2015 in Oklahoma City, OK by OrionNet Systems, LLC. The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of the usability of the EHR Under Test (EHRUT). During the usability test, 3 users matching target demographics served as participants and used the EHRUT in simulated, but representative tasks. This study collected performance data on 29 tasks typically conducted on an EHR: 170.314(a)(1) Computerized provider order entry Select a Patient Record and Record Medication Order Select a Patient Record and Change Medication Order Select a Patient Record and Access Medication Order Select a Patient Record and Record Laboratory Order Select a Patient Record and Change Laboratory Order Select a Patient Record and Access Laboratory Order Select a Patient Record and Radiology/Imaging Order Select a Patient Record and Radiology/Imaging Order Select a Patient Record and Radiology/Imaging Order 170.314(a)(2) Drug-drug, drug-allergy interactions checks Create drug-drug and drug-allergy interventions prior to CPOE completion Adjustment of severity level of drug-drug interventions 170.314(a)(6) Medication list Record Medication List Change Medication List Access Medication List 170.314(a)(7) Medication allergy list Record Medication Allergy List Change Medication Allergy List Access Medication Allergy List 170.314(a)(8) Clinical decision support Problem List Interventions Medication List Interventions Medication Allergy List Interventions Demographics Interventions Lab Tests and Results Interventions Vital Signs Interventions Identify User Diagnostic and Therapeutic Reference Information Configuration of CDS interventions by user 170.314(b)(3) Electronic prescribing Create prescriptions 170.314(b)(4) Clinical information reconciliation

Reconcile patient’s active medication list with another source Reconcile patient’s active problem list with another source Reconcile patient’s active medication allergy list with another source The test administrator introduced the test, and instructed participants to complete a series of tasks (given one at a time) using the EHR. During the testing, the administrator timed the tests and recorded user performance data on paper. The test administrator did not provide the participants with assistance on how to complete the tasks, other than to setup each task exercise with a brief definition of the task objectives. Participants had all previously received some basic end-user system instruction. User tasks were prioritized in accordance with the risk associated with user errors and common workflow. The following types of data were collected for each participant: • Number of tasks successfully completed within the allotted time without assistance • Time to complete the tasks • Number and types of errors • Participant’s satisfaction ratings of the system All participant data was de-identified – no correspondence could be made from the identity of the participant to the data collected. Following the conclusion of the testing, participants were asked to complete a post-test questionnaire. Various recommended metrics, in accordance with the examples set forth in the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance and rating data collected on the EHRUT.

Measure Task Success Task Time Task Ratings 5 = Easy

Task Mean (SD) Mean (SD)

Deviations (Observed/Optimal)

Mean (SD)

Record Medication Order 3 – 100% 150s 150/130 = 1.15

5.0

Change Medication 3 – 100% 75s 75/72 = 1.04

5.0

Access Medication Order 3 – 100% 13s 13/15 = .87 5.0 Record Lab Order 3 – 100% 135s 135/152 =

.89 1.6

Change Lab Order 3 – 100% 126s 126/135 = .93

2.3

Access Labe Order 3 – 100% 27s 27/31 = .87 3.0 Record Radiology Order 3 – 100% 88s 88/95 = .92 1.6 Change Radiology/Imaging Order 3 – 100% 81s 81/86 = .94 2.3 Access Radiology/Imaging Order

3 – 100% 11s 11/10 = 1.1 3.3

Create drug-drug & drug allergy interventions prior to CPOE completion

3 – 100% 118s 118/112 = 1.05

4.6

Adjustment of severity level of drug-drug & drug-allergy interventions prior to CPOE completion

3 – 100% 104s 104/109 = .95

4.6

Record Medication List 3 – 100% 210s 210/233 = .91

3.0

Change Medication List 3 – 100% 79s 79/91 = .87 3.6 Access Medication List 3 – 100% 26s 26/22 =

1.18 4.0

Record Medication Allergy List

3 – 100% 72s 72/75 = .96 2.6

Change Medication Allergy List 3 – 100% 56s 56/60 = .93 3.6 Access Medication Allergy List 3 – 100% 42s 42/48 = .87 4.6 Problem List Interventions 3 – 100% 114s 114/125 =

.91 3.2

Medication List Interventions 3 – 100% 57s 57/62 = .91 4.2 Medication Allergy Interventions

3 – 100% 62s 61/62 = .98 4.6

Demographic Interventions 3 – 100% 65s 65/72 4.6 Lab Test & Results Interventions

3 – 100% 60s 60/62 4.6

Vital Signs Interventions 3 – 100% 63s 65/63 4.3 Identify User Diagnostic and Therapeutic Reference Information

3 – 100% 71s 75/71 4.3

Configuration of CDS interventions

3 – 100% 80s 86/80 4.3

Create prescriptions 3 – 100% 72s 74/72 4.6 Reconcile patient’s active medication list with another source

3 – 100% 94s 93/94 3.6

Reconcile patient’s active problem list with another source

3 – 100% 71s 75/71 4.3

Reconcile patient’s active medication allergy list with another source

3 – 100% 59s 66/59 4.6

The results from the System Usability Scale scored the subjective satisfaction with the system based on performance with these tasks to be: 70 In addition to the performance data, the following qualitative observations were made: Major findings

• Participants liked the data gathering and accessibility of the data in ThinkHealth. • Most participants found the system overwhelming upon first entering it. • Some participants thought it took too long to do a medication order.

Areas for improvement

• Make the system a little less overwhelming when first entering into it. • Look to reduce the number of steps to do some of the tasks within the system.

INTRODUCTION The EHR tested for this study was ThinkHealth – Practice Management Software (EHR) Version 2.1.0.0. The usability testing attempted to represent realistic exercises and conditions. The purpose of this study was to test and validate the usability of the current user interface, and provide evidence of usability in the EHR. To this end, measures of effectiveness, efficiency and user satisfaction were captured during the usability testing. METHOD PARTICIPANTS A total of 3 participants were tested on the EHRUT(s). Participants in the test were [types of healthcare providers]. Participants were recruited by OrionNet Systems and were compensated $45 for their time. In addition, participants had no direct connection to the development of or organization producing the EHRUT(s). Participants were not from the testing or supplier organization. Participants were given the opportunity to have the same orientation and level of training as the actual end users would have received. For the test purposes, end-user characteristics were identified and translated into a recruitment screener used to solicit potential participants. The following is a table of participants by characteristics, including demographics, professional experience, computing experience and user needs for assistive technology. Participant names were replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.

Part ID

Gender Age Education Occupation/role Professional Experience

Computer Experience

Product Experience

Assitive Technology

Needs 1 F 25 Bachelors Nurse 3 years Intermediate None None 2 M 28 Bachelors Nurse 3 years Intermediate None None 3 F 26 Bachelors Nurse 3 years Intermediate None None Three participants (matching the demographics in the section on Participants) were recruited and three participated in the usability test. Zero participants failed to show for the study. Participants were scheduled for 29 task sessions. A spreadsheet was used to keep track of the participant schedule, and included each participant’s demographic characteristics as provided. STUDY DESIGN

During the usability test, participants interacted with one EHR. Each participant used the system in the same location, and was provided with the same instructions. The system was evaluated for effectiveness, efficiency and satisfaction as defined by measures collected and analyzed for each participant: • Number of tasks successfully completed • Time to complete the tasks • Participant’s verbalizations (comments) • Participant’s satisfaction ratings of the system TASKS A number of tasks were constructed that would be realistic and representative of the kinds of activities a user might do with this EHR, including: 170.314(a)(1) Computerized provider order entry Record Medication Order Change Medication Order Access Medication Order Record Laboratory Order Change Laboratory Order Access Laboratory Order Radiology/Imaging Order Radiology/Imaging Order Radiology/Imaging Order 170.314(a)(2) Drug-drug, drug-allergy interactions checks Create drug-drug and drug-allergy interventions prior to CPOE completion Adjustment of severity level of drug-drug interventions 170.314(a)(6) Medication list Record Medication List Change Medication List Access Medication List 170.314(a)(7) Medication allergy list Record Medication Allergy List Change Medication Allergy List Access Medication Allergy List 170.314(a)(8) Clinical decision support Problem List Interventions Medication List Interventions Medication Allergy List Interventions Demographics Interventions Lab Tests and Results Interventions Vital Signs Interventions Identify User Diagnostic and Therapeutic Reference Information

Configuration of CDS interventions by user 170.314(b)(3) Electronic prescribing Create prescriptions 170.314(b)(4) Clinical information reconciliation Reconcile patient’s active medication list with another source Reconcile patient’s active problem list with another source Reconcile patient’s active medication allergy list with another source PROCEDURES Upon arrival, participants were greeted; their identity was verified and matched with a name on the participant schedule. Participants were then assigned a participant ID. Each participant reviewed and signed an informed consent and release form. A representative from the test team witnessed the participant’s signature. To ensure that the test ran smoothly, three staff members participated in this test, the usability administrator and two data loggers. The administrator moderated the session including administering instructions and tasks. The administrator also monitored task times, obtained post-task rating data, and took notes on participant comments. The other two people served as the data logger and took notes on task success, number and type of errors, comments and time. Participants were instructed to perform the tasks (see specific instructions below):

• As quickly as possible making as few errors and deviations as possible. • Without assistance; administrators were allowed to give immaterial guidance and clarification

on tasks, but not instructions on use. • Without using a think aloud technique.

For each task, the participants were given a written copy of the task. Task timing began once the administrator finished reading the question. The task time was stopped once the participant indicated they had successfully completed the task. Following the session, the administrator gave the participant the post-test questionnaire, compensated them for their time, and thanked each individual for their participation. Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses, and post-test questionnaire were recorded into a spreadsheet. Participants were thanked for their time and compensated. Participants signed a receipt and acknowledgement form indicating that they had received the compensation.

TEST LOCATION The test facility included a quiet testing room with a table, computer for the participant. To ensure that the environment was comfortable for users, noise levels were kept to a minimum with the ambient temperature within a normal range. All of the safety instruction and evacuation procedures were valid, in place, and visible to the participants. TEST ENVIRONMENT The EHRUT would be typically be used in a healthcare office or facility. In this instance, the testing was conducted in [describe facilities]. For testing, the computer used a PC running Windows 7. The participants used a mouse and keyboard when interacting with the EHRUT. The application was set up by the vendor according to the vendor’s documentation describing the system set-up and preparation. The application itself was running on a Windows 7 PC using a test database on a LAN connection. Technically, the system performance (i.e., response time) was representative to what actual users would experience in a field implementation. Additionally, participants were instructed not to change any of the default system settings (such as control of font size). TEST FORMS AND TOOLS During the usability test, various documents and instruments were used, including: 1. Informed Consent 2. Moderator’s Guide 3. Post-test Questionnaire 4. Incentive Receipt and Acknowledgment Form The participant’s interaction with the EHRUT was captured visually and on paper. The test session were electronically transmitted to a nearby observation room where the data logger observed the test session. PARTICIPANT INSTRUCTIONS The administrator reads the following instructions aloud to the each participant: Thank you for participating in this study. Your input is very important. I will ask you to complete a few tasks using this system and answer some questions. You should complete the tasks as quickly as possible making as few errors as possible. Please try to complete the tasks on your own following the instructions very closely. Please note that we are not testing you, we are testing the system, therefore if you have

difficulty all this means is that something needs to be improved in the system. I will be here in case you need specific help, but I am not able to instruct you or provide help in how to use the application. Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be useful to you, and how we could improve it. Please be honest with your opinions. All of the information that you provide will be kept confidential and your name will not be associated with your comments at any time. Should you feel it necessary you are able to withdraw at any time during the testing. Following the procedural instructions, participants were shown the EHR and were given a time of 20 minutes to explore the system and make comments. Once this was complete, the administrator gave the following instructions: For each task, I will read the description to you and say “Begin.” At that point, please perform the task and say “Done” once you believe you have successfully completed the task. I would like to request that you not talk aloud or verbalize while you are doing the tasks. I will ask you your impressions about the task once you are done. USABILITY METRICS According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health Records, EHRs should support a process that provides a high level of usability for all users. The goal is for users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing. The goals of the test were to assess: 1. Effectiveness of ThinkHealth by measuring participant success rates and errors 2. Efficiency of ThinkHealth by measuring the average task time 3. Satisfaction with ThinkHealth by measuring ease of use ratings DATA SCORING The following table details how tasks were scored, errors evaluated, and the time data analyzed. 10 Task Success

A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance, within the time allotted on a per task basis.

The total number of successes were calculated for each task and then divided by the total number of times that task was attempted. The results are provided as a percentage.

Task times were recorded for successes. Observed task times divided by the optimal time for each task is a measure of optimal efficiency.

Task Failures

If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the end of the allotted time before successful completion, the task was counted as a “Failure.” No task times were taken for errors.

The total number of errors was calculated for each task and then divided by the total number of times that task was attempted. Not all deviations would be counted as errors.11 This should also be expressed as the mean number of failed tasks per participant.

On a qualitative level, an enumeration of errors and error types should be collected. Task Time Each task was timed from when the administrator said “Begin” until the participant

said, “Done.” If he or she failed to say “Done,” the time was stopped when the participant stopped performing the task. Only task times for tasks that were successfully completed were included in the average task time analysis. Average time per task was calculated for each task. Variance measures (standard deviation and standard error) were also calculated.

Task Rating Participant’s subjective impression of the ease of use of the application was measured by administering both a simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1 (Very Difficult) to 5 (Very Easy). These data are averaged across participants. 12

Common convention is that average ratings for systems judged easy to use should be 3.3 or above.

To measure participants’ confidence in and likeability of the [EHRUT] overall, the testing team administered the System Usability Scale (SUS) post-test questionnaire. Questions included, “I think I would like to use this system frequently,” “I thought the system was easy to use,” and “I would imagine that most people would learn to use this system very quickly.” See full System Usability Score questionnaire in Appendix 5.13

DATA ANALYSIS AND REPORTING The results of the usability test were calculated according to the methods specified in the Usability Metrics section above. Participants who failed to follow session and task instructions had their data excluded from the analyses [Provide details if there are data exclusions.]. [Provide any details of testing irregularities or issues that affected data collection or interpretation of the results.] The usability testing results for the EHRUT are detailed below (see Table [x])14. The results should be seen in light of the objectives and goals outlined in Section 3.2 Study Design. The data should yield actionable results that, if corrected, yield material, positive impact on user performance. [Furthermore, the data should be presented in forms such as the table below so that the tasks can be easily identified and their performance results examined and compared.]

Measure Task Success Task Time Task Ratings 5 = Easy

Task Mean (SD) Mean (SD)

Deviations (Observed/Optimal)

Mean (SD)

Record Medication Order 3 – 100% 150s 150/130 = 1.15

5.0

Change Medication 3 – 100% 75s 75/72 = 1.04

5.0

Access Medication Order 3 – 100% 13s 13/15 = .87

5.0

Record Lab Order 3 – 100% 135s 135/152 = .89

1.6

Change Lab Order 3 – 100% 126s 126/135 = .93

2.3

Access Lab Order 3 – 100% 27s 27/31 = .87

3.0

Record Radiology Order 3 – 100% 88s 88/95 = .92

1.6

Change Radiology/Imaging Order 3 – 100% 81s 81/86 = 94

2.3

Access Radiology/Imaging Order

3 – 100% 11s 11/10 = 1.1

3.3

Create drug-drug & drug allergy interventions prior to CPOE completion

3 – 100% 118s 118/112 = 1.05

4.6

Adjustment of severity level of drug-drug & drug-allergy interventions prior to CPOE completion

3 – 100% 104s 104/109 = .95

4.6

Record Medication List 3 – 100% 210s 210/233 = .91

3.0

Change Medication List 3 – 100% 79s 79/91 = .87

3.6

Access Medication List 3 – 100% 26s 26/22 = 1.18

4.0

Record Medication Allergy List

3 – 100% 72s 72/75 = .96

2.6

Change Medication Allergy List 3 – 100% 56s 56/60 = .93

3.6

Access Medication Allergy List 3 – 100% 42s 42/48 = .87

4.6

Problem List Interventions 3 – 100% 114s 114/125 = .91

3.2

Medication List Interventions 3 – 100% 57s 57/62 = .91

4.2

Medication Allergy Interventions

3 – 100% 62s 61/62 = .98

4.6

Demographic Interventions 3 – 100% 65s 65/72 4.6 Lab Test & Results Interventions

3 – 100% 60s 60/62 4.6

Vital Signs Interventions 3 – 100% 63s 65/63 4.3 Identify User Diagnostic and Therapeutic Reference Information

3 – 100% 71s 75/71 4.3

Configuration of CDS interventions

3 – 100% 80s 86/80 4.3

Create prescriptions 3 – 100% 72s 74/72 4.6 Reconcile patient’s active medication list with another source

3 – 100% 94s 93/94 3.6

Reconcile patient’s active problem list with another source

3 – 100% 71s 75/71 4.3

Reconcile patient’s active medication allergy list with another source

3 – 100% 59s 66/59 4.6

The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system based on performance with these tasks to be: 70. Broadly interpreted, scores under 60 represent systems with poor usability; scores over 80 would be considered above average. MAJOR FINDINGS

• Participants liked the data gathering and accessibility of the data in ThinkHealth. • Most participants found the system overwhelming upon first entering it. • Some participants thought it took too long to do a medication order.

AREAS FOR IMPROVEMENT

• Make the system a little less overwhelming when first entering into it. • Look to reduce the number of steps to do some of the tasks within the system.


4788782557R1PRA‐1.0, December 17, 2018

Appendix B: Quality Management System


Template 24 V3.7 11


4788782557R1PRA‐1.0, December 17, 2018

Appendix C: Privacy and Security


Template 24 V3.7 12


4788782557R1PRA‐1.0, December 17, 2018

Test Results Summary Document History Version

V1.0

Date

12/17/2018

END OF DOCUMENT

Description of Change

Initial release


Template 24 V3.7 13

onc hit program test results summary for 2014 …...test results summary for 2014 edition ehr...

Documents