onc hit program test results summary for 2014 edition ehr …€¦ · test results summary for 2014...

Test Results Summary for 2014 Edition EHR Certification

16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

Part 1: Product and Developer Information1.1 Certified Product Information

1.2 Developer/Vendor Information

Part 2: ONC‐Authorized Certification Body Information2.1 ONC‐Authorized Certification Body Information

11/14/2016

Signature and Date

ONC‐ACB Authorized Representative Function/Title

San Luis Obispo, CA 93401

Website: www.infogard.com

Email: [email protected]

Phone: (805) 783‐0810


Phone: (877) 901‐9990

ONC‐ACB Contact: Adam Hardcastle

This test results summary is approved for public release by the following ONC‐Authorized Certification Body

Representative:

Adam Hardcastle EHR Certification Body Manager

Test Type: Complete EHR

Developer/Vendor Name: PatientClick, Inc.

Address: 6540 Lusk Blvd, Suite C150

ONC HIT Certification Program


Product Name: PatientClick

Product Version: 4.2

Domain: Ambulatory

Developer/Vendor Contact: Ashish Mehta

ONC‐ACB Name: InfoGard Laboratories, Inc.

Address: 709 Fiero Lane Suite 25

San Diego, CA 92126

Website: http://patientclick.com

©2016 InfoGard. May be reproduced only in its original entirety, without revision 1


16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

2.2 Gap Certification

(a)(1) (a)(19) (d)(6) (h)(1)

(a)(6) (a)(20) (d)(8) (h)(2)

(a)(7) (b)(5)* (d)(9)

(a)(17) (d)(1) (f)(1)

(a)(18) (d)(5) (f)(7)*

*Gap certification allowed for Inpatient setting only

2.3 Inherited CertificationThe following identifies criterion or criteria certified via inherited certification

No inherited certification

(a)(14) (b)(9) (e)(3) Amb. only (h)(2)

(a)(15) (c)(1) (f)(1) (h)(3)

(a)(12) (b)(7) (e)(1) (g)(4)

(a)(13) (b)(8) (e)(2) Amb. only (h)(1)

(a)(10) (b)(5) (d)(8) (g)(2)

(a)(11) (b)(6) Inpt. only (d)(9) Optional (g)(3)

(a)(8) (b)(3) (d)(6) (f)(7) Amb. Only

(a)(9) (b)(4) (d)(7) (g)(1)

(d)(3)

(a)(6) (b)(1) (d)(4) (f)(6) Optional & Amb. only (a)(7) (b)(2) (d)(5)

(a)(3) (a)(18) (d)(1) (f)(4) Inpt. only

(a)(4) (a)(19) (d)(2) (f)(5) Optional & Amb. only (a)(5) (a)(20)

(a)(1) (a)(16) Inpt. only (c)(2) (f)(2)

(a)(2) (a)(17) Inpt. only (c)(3) (f)(3)

The following identifies criterion or criteria certified via gap certification

§170.314

No gap certification

§170.314



16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

Part 3: NVLAP‐Accredited Testing Laboratory Information

3.1 NVLAP‐Accredited Testing Laboratory Information

3.2 Test Information

3.2.1 Additional Software Relied Upon for Certification

No additional software required

NewCrop version 13.05.15.02 a2, a8, a10, b3, g2 Drug database

ATL Authorized Representative Function/Title

Signature and Date

Additional Software Applicable CriteriaFunctionality provided by

Additional Software

ATL Contact: Milton Padilla

For more information on scope of accreditation, please reference

http://ts.nist.gov/Standards/scopes/1004320.htm

Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory

Representative:

Milton Padilla EHR Test Body Manager

San Luis Obispo, CA 93401

Website: www.infogard.com


Phone: (805) 783‐0810

ATL Name: InfoGard Laboratories, Inc.

Accreditation Number: NVLAP Lab Code 100432‐0

Address: 709 Fiero Lane Suite 25

Report Number: 16‐3528‐R‐0027 V1.2

Test Date(s): March 19, 2015 ‐ October 11, 2016

Location of Testing: InfoGard and Vendor Site



16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

3.2.2 Test Tools

Version

2.6.1

1.0.5

1.8.2

1.7.2

1.7.2

180

3.0.3

No test tools required

3.2.3 Test Data

3.2.4 Standards

3.2.4.1 Multiple Standards Permitted

(a)(8)(ii)(A)(2)

§170.204(b)(1)

HL7 Version 3 Implementation

Guide: URL‐Based

Implementations of the Context‐

Aware Information Retrieval

(Infobutton) Domain

§170.204(b)(2)


Guide: Context‐Aware

Knowledge Retrieval

(Infobutton) Service‐Oriented

Architecture Implementation

Guide

(a)(13)

§170.207(a)(3)

IHTSDO SNOMED CT®

International Release July 2012

and US Extension to SNOMED

CT® March 2012 Release

§170.207(j)

HL7 Version 3 Standard: Clinical

Genomics; Pedigree

Direct Certificate Discovery Tool

Edge Testing Tool

Alteration (customization) to the test data was necessary and is

described in Appendix A

No alteration (customization) to the test data was necessary

The following identifies the standard(s) that has been successfully

tested where more than one standard is permitted

Criterion # Standard Successfully Tested

HL7 CDA Cancer Registry Reporting Validation Tool

HL7 v2 Electronic Laboratory Reporting (ELR) Validation ToolHL7 v2 Immunization Information System (IIS) Reporting Valdiation

Tool

HL7 v2 Laboratory Restults Intervace (LRI) Validation Tool

HL7 v2 Syndromic Surveillance Reporting Validation Tool

Transport Testing Tool

Test Tool

Cypress

ePrescribing Validation Tool



16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

None of the criteria and corresponding standards listed above are applicable

3.2.4.2 Newer Versions of Standards

No newer version of a minimum standard was tested

The following identifies the newer version of a minimum standard(s) that has

been successfully tested

Newer Version Applicable Criteria

(e)(3)(ii) Annex A of the FIPS Publication 140‐2TLS_RSA_WITH_AES_128_CBC_SHA TLS 1.0

Common MU Data

Set (15)

§170.207(a)(3)

IHTSDO SNOMED CT®




§170.207(b)(2)

The code set specified at 45 CFR

162.1002(a)(5) (HCPCS and CPT‐

4)

(e)(1)(i) Annex A of the FIPS Publication 140‐2

TLS_RSA_WITH_AES_128_CBC_SHA TLS 1.0

(e)(1)(ii)(A)(2)

§170.210(g)

Network Time Protocol Version 3

(RFC 1305)

§170. 210(g)


(RFC 5905)

(b)(2)(i)(A)

§170.207(i)


162.1002(c)(2) (ICD‐10‐CM) for

the indicated conditions

§170.207(a)(3)

IHTSDO SNOMED CT®




(b)(7)(i)

§170.207(i)


162.1002(c)(2) (ICD‐10‐CM) for

the indicated conditions

§170.207(a)(3)

IHTSDO SNOMED CT®




(a)(15)(i)

§170.204(b)(1)


Guide: URL‐Based

Implementations of the Context‐

Aware Information Retrieval

(Infobutton) Domain

§170.204(b)(2)


Guide: Context‐Aware

Knowledge Retrieval

(Infobutton) Service‐Oriented

Architecture Implementation

Guide

(a)(16)(ii)

§170.210(g)


(RFC 1305)

§170. 210(g)


(RFC 5905)



16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

3.2.5 Optional Functionality

No optional functionality tested

Common MU Data

Set (15)

Express Procedures according to the standard

specified at §170.207(b)(3) (45 CFR162.1002(a)(4):

Code on Dental Procedures and Nomenclature)

Common MU Data

Set (15)

Express Procedures according to the standard

specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD‐

10‐PCS)

(b)(2)(ii)(B)

Transmit health information to a Third Party using

the standards specified at §170.202(a) and (b) (Direct

and XDM Validation)

(b)(2)(ii)(C)

Transmit health information to a Third Party using

the standards specified at §170.202(b) and (c) (SOAP

Protocols)

(f)(3)

Ambulatory setting only – Create syndrome‐based

public health surveillance information for transmission

using the standard specified at §170.205(d)(3) (urgent

care visit scenario)

(a)(4)(iii) Plot and display growth charts

(b)(1)(i)(B)

Receive summary care record using the standards

specified at §170.202(a) and (b) (Direct and XDM

Validation)

(b)(1)(i)(C) Receive summary care record using the standards

specified at §170.202(b) and (c) (SOAP Protocols)

Criterion # Optional Functionality Successfully Tested



16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

3.2.6 2014 Edition Certification Criteria* Successfully Tested

TP** TD*** TP** TD***

1.11 2.6.1

1.2

1.2 1.4 1.6

1.4 1.3 1.3

1.4 1.3 1.2

1.3 1.2

1.3 1.3

1.2 1.4

1.3 1.9 1.5

1.3 1.2 1.6

1.2 1.3

1.2

1.5 1.3 1.3.0

1.3 1.3.0

1.7 1.4

1.4 1.6

1.4 1.2

1.3 1.4 2.0 2.0

1.4 1.7.2 1.4

1.2

1.4 1.7

1.11 2.6.1

1.11 2.6.1

*For a list of the 2014 Edition Certification Criteria, please reference

http://www.healthit.gov/certification (navigation: 2014 Edition Test Method)

**Indicates the version number for the Test Procedure (TP)

***Indicates the version number for the Test Data (TD)

(b)(8) (h)(2)

(b)(9) (h)(3)

(c)(1)

(c)(2)

(b)(5) (g)(3)

(b)(6) Inpt. only (g)(4)

(b)(7) (h)(1)

(b)(2) (f)(7) Amb. only

(b)(3) (g)(1)

(b)(4) (g)(2)

(a)(19)

(a)(20) (f)(6) Optional & Amb. only (b)(1)

(a)(16) Inpt. only (f)(3)

(a)(17) Inpt. only (f)(4) Inpt. only

(a)(18) (f)(5) Optional & Amb. only

(a)(13) (e)(3) Amb. only

(a)(14) (f)(1)

(a)(15) (f)(2)

(a)(10) (d)(9) Optional

(a)(11) (e)(1)

(a)(12) (e)(2) Amb. only

(a)(7) (d)(6)

(a)(8) (d)(7)

(a)(9) (d)(8)

(a)(4) (d)(3)

(a)(5) (d)(4)

(a)(6) (d)(5)

(a)(1) (c)(3)

(a)(2) (d)(1)

(a)(3) (d)(2)

Criteria #Version

Criteria #Version



16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

3.2.7 2014 Clinical Quality Measures*Type of Clinical Quality Measures Successfully Tested:

CMS ID Version CMS ID Version CMS ID Version CMS ID Version

2 90 136 155

22 117 v3 137 156

50 v3 122 138 v3 157

52 123 139 158

56 124 140 159

61 125 v3 141 160

62 126 142 161

64 127 143 163 v3

65 128 144 164

66 129 145 165 v3

68 130 146 166 v4

69 v3 131 147 v4 167

74 132 148 169

75 v3 133 149 177

77 134 153 179

82 135 154 182

CMS ID Version CMS ID Version CMS ID Version CMS ID Version

9 71 107 172

26 72 108 178

30 73 109 185

31 91 110 188

32 100 111 190

53 102 113

55 104 114

60 105 171

*For a list of the 2014 Clinical Quality Measures, please reference

http://www.cms.gov (navigation: 2014 Clinical Quality Measures)

Ambulatory CQMs

Inpatient CQMs

Ambulatory

Inpatient

No CQMs tested



16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

3.2.8 Automated Numerator Recording and Measure Calculation

3.2.8.1 Automated Numerator Recording

Automated Numerator Recording was not tested

3.2.8.2 Automated Measure Calculation

Automated Measure Calculation was not tested

3.2.9 Attestation

Quality Management System** C

Privacy and Security D

*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16),

(b)(3), (b)(4)

**Required for every EHR product

(a)(7) (a)(15) (b)(5)

Attestation Forms (as applicable) Appendix

Safety‐Enhanced Design* B

(a)(5) (a)(13) (b)(3) (e)(3)

(a)(6) (a)(14) (b)(4)

(a)(3) (a)(11) (a)(17) (e)(1)

(a)(4) (a)(12) (b)(2) (e)(2)

(a)(7) (a)(15) (b)(5)

Automated Numerator Recording Successfully Tested

(a)(1) (a)(9) (a)(16) (b)(6)

(a)(5) (a)(13) (b)(3) (e)(3)

(a)(6) (a)(14) (b)(4)

(a)(3) (a)(11) (a)(17) (e)(1)

(a)(4) (a)(12) (b)(2) (e)(2)

Automated Numerator Recording Successfully Tested

(a)(1) (a)(9) (a)(16) (b)(6)



16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

e1 and e2

Determined that modified Test Data had equivalent level of

robustness to NIST Test Data

Appendix A: Alteration of Test Data

Criteria Explanation

b2 Determined that modified Test Data had equivalent level of


b7

Determined that modified Test Data had equivalent level of




16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

Appendix B: Safety Enhanced Design


PatientClick EHR v4.2 Page 1 of 46

7/29/2015

PatientClick EHR v4.2

Usability Testing

Date of Usability Test: July 29, 2015 and August 11, 2016 Date of Report: Aug 4, 2015. Modified August 11, 2016 &

October 4, 2016 Report Prepared By: PatientClick

Chirag Patel, COO, PatientClick

877-901-9990 [email protected]

PO Box 26730, San Diego, CA 92196


7/29/2015

EHR Usability Test Report of PatientClick EHR Version 4.2 Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports using the ISO 9241-11 process for 170.314.g.3 Safety-Enhanced Design (a)(1), (a)(2), (a)(6),(a)(7),(a)(8),(b)(3),(b)(4).

PatientClick EHR Version 4.2

Date of Usability Test: July 29, 2015 and August 11, 2016 Date of Report: Aug 4, 2015 and modified August 11 , 2016 and October 4, 2016 Report Prepared By: PatientClick EHR

Chirag Patel, COO, PatientClick

877-901-9990

[email protected]

PO Box 26730,

San Diego, CA 92196

Table of Contents

1 EXECUTIVE SUMMARY3

2 INTRODUCTION 5

3 METHOD 5

3.1 PARTICIPANTS 5

3.2 STUDY DESIGN 6

3.3 TASKS 6

3.4 PROCEDURE 7

3.5 TEST LOCATION 8

3.6 TEST ENVIRONMENT 8

3.7 TEST FORMS AND TOOLS 8

3.8 PARTICIPANT INSTRUCTIONS 9

3.9 USABILITY METRICS 9

4 RESULTS 12

4.1 DATA ANALYSIS AND REPORTING 12

4.2 DISCUSSION OF THE FINDINGS 13

5 APPENDICES 15

5.1 APPENDIX 1: SAMPLE RECRUITING SCREENER 17

5.2 APPENDIX 2: PARTICIPANT DEMOGRAPHICS 19

5.3 APPENDIX 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM 20

5.4 APPENDIX 4: EXAMPLE MODERATOR’S GUIDE 26

5.5 APPENDIX 5: SYSTEM USABILITY SCALE QUESTIONNAIRE 49

5.6 APPENDIX 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM 50

mailto:[email protected]


7/29/2015

EXECUTIVE SUMMARY

A usability study of the PatientClick EHR Version 4.2 was conducted on 07/20/2015 - 07/27/2015 and 8/11/16

in San Diego, CA by PatientClick as part of the 2014 ONC EHR Certification. The EHR features selected for

testing are outlined in 170.314.g.3 Safety Enhanced Design requirements. The purpose of this test was to test

and validate the usability of the current user interface, and provide evidence of usability in the EHR under Test

(EHRUT). During the usability test, one (1) clinical staff and two (2) authorized medical office staff members

matching the target demographic criteria served as participants and used the EHRUT in simulated, but

representative tasks.

This study collected performance data on the following EHR features, as specified by the Safety Enhanced

Design requirements:

1. Computerized provider order entry 314.a.1

2. Drug – drug, drug allergy interaction checks 314.a.2

3. Medication list 314.a.6

4. Medication allergy list 314.a.7

5. Clinical decision support 314.a.8

6. Electronic prescribing 314.b.3

7. Clinical information reconciliation 314.b.4

TASKS

Eleven (11) task groups with twenty (20) tasks were given to interact with system, constructed so that it

would be realistic.

During the sixty (60) minute one-on-one usability test, each participant was greeted by the administrator and

asked to review and sign an informed consent/release form (included in Appendix 3); they were instructed that

they could withdraw at any time. Participants had prior experience with the EHR. The administrator introduced

the test, and instructed participants to complete a series of tasks (given one at a time) using the EHRUT.

During the testing, the administrator timed the test and, along with the data logger(s) recorded user

performance data on paper and electronically. The administrator did not give the participant assistance in how

to complete the task. Participant screens, head shots and audio were recorded for subsequent analysis.

The following types of data were collected for each participant:

• Number of tasks successfully completed within the allotted time without assistance

• Time to complete the tasks

• Number and types of errors

• Path deviations

• Participant’s verbalizations

• Participant’s satisfaction ratings with the system

All participant data was de-identified – no correspondence could be made from the identity of the participant to

the data collected. Following the conclusion of the testing, participants were asked to complete a post-test


7/29/2015

questionnaire and were not compensated for their time. Various recommended metrics, in accordance with the

examples set forth in the NIST Guide to the Processes Approach for Improving the Usability of Electronic

Health Records, were used to evaluate the usability of the EHRUT. Following is a summary of the performance

and rating data collected on the EHRUT.

Usability Test Results Summary

The results from the System Usability Scale scored the subjective satisfaction with the system based on performance

with these tasks to be: 90%

In addition to the performance data, the following qualitative observations were made:

- Major findings

The user interface, processes and features for the tasks were easy to navigate and user friendly. They

noted that it was intuitive, logical, straight-forward and required minimal clicks. The laboratory and

radiology order screens have a different flow from the rest of the site which gave participants slight

pause but were able to adapt quickly. Participants regarded the new functionality as highly desirable.

- Areas for improvement

Training for the laboratory and radiology orders screens. Since these screens have a different flow

from the rest of the site, training on how to place radiology and laboratory orders and also how to find

and modify existing orders when needed could be helpful in increasing efficiency.


7/29/2015

INTRODUCTION

The EHRUT(s) tested for this study was PatientClick EHR v4.2. Designed to present medical information to

healthcare providers in an Ambulatory Setting, the EHRUT consist of PatientClick EHR, which provides

predesigned exam templates but allows physicians and their designated staff members to customize their

templates quickly and easily with our proprietary template designer. PatientClick is designed for quick entry of

findings but yet is comprehensive. It allows doctors to design and create any type of letter or other document

using data from their exams in user-designed templates. The usability testing attempted to represent realistic

exercises and conditions.

The purpose of this study was to test and validate the usability of the current user interface, and provide

evidence of usability in the EHR under Test (EHRUT). To this end, measures of effectiveness, efficiency and

user satisfaction, such as time on task and success rate were captured during the usability testing.

METHOD

PARTICIPANTS

A total of three (3) participants were tested on the EHRUT(s). Participants in the test were one (1)

clinician (MD) and two (2) authorized medical office staff members. Participants were recruited by

PATIENTCLICK (Participant Recruiter: Steve Parker) and were not compensated for their time. In addition,

participants had no direct connection to the development of organization producing the EHRUT(s). Participants

were not from the testing or supplier organization. Participants were given the opportunity to have the same

orientation and level of training as the actual end users would have received.

For the test purposes, end-user characteristics were identified and translated into a recruitment screener used

to solicit potential participants; an example of a screener is provided in Appendix [1].

Recruited participants had a mix of backgrounds and demographic characteristics conforming to the

recruitment screener. The following is a table of participants by characteristics, including demographics,

professional experience, computing experience and user needs for assistive technology. Participant names

were replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.

Part

ID

Gender

Age

Education

Occupation/

role

Professional

Experience

Computer

Experience

Product

Experience

Assistive

Technology

Needs 1 111 Female 32 MD Medical staff 10 years Basic None Not needed

2 112 Male 36 Clinical Staff 8 Years Intermediate None Not needed

3 113 Male 38 Office staff 12 Years Pro None Not needed


7/29/2015

Three (3) participants (matching the demographics in the section on Participants) were recruited and three (3)

participated in the usability test. Zero (0) participants failed to show for the study.

Participants were scheduled sessions for 60 minutes with 10 minutes in between each session for debrief by

the administrator(s) and data logger(s), and to reset systems to proper test conditions. A spreadsheet was used

to keep track of the participant schedule, and included each participant’s demographic characteristics as

provided by the recruiting firm.

STUDY DESIGN

Overall, the objective of this test was to uncover areas where the application performed well – that is,

effectively, efficiently, and with satisfaction – and areas where the application failed to meet the needs of the

participants. The data from this test may serve as a baseline for future tests with an updated version of the

same EHR and/or comparison with other EHRs provided the same tasks are used. In short, this testing serves

as both a means to record or benchmark current usability, but also to identify areas where improvements must

be made.

During the usability test, participants interacted with one (1) EHR(s). Each participant used the system in the

same location, and was provided with the same instructions. The system was evaluated for effectiveness,

efficiency and satisfaction as defined by measures collected and analyzed for each participant:

Number of tasks successfully completed within the allotted time without assistance

Time to complete the tasks

Number and types of errors

Path deviations

Participant’s verbalizations (comments)

Participant’s satisfaction ratings of the system

Additional information about the various measures can be found in Section 3.9 on Usability Metrics.

TASKS

Eleven (11) task groups with twenty (20) tasks were given to interact with system, constructed so that it

would be realistic and representative of the kinds of activities a user might do with this EHR, including:

1. Access and change a patient allergy (3 tasks)

a. Select and record drug insurance carrier b. Select and record a patient allergy c. Access and change a patient allergy

2. Access and change a patient medication (3 tasks) a. Select a patient reported medication b. Access and change a patient reported medication c. Record a patient reported medication

3. Select a medication to prescribe and note formulary information (4 tasks) a. Select a medication to prescribe and note formulary information b. Choose a medication to prescribe and record it c. Access and change a pending prescription d. Transmit the prescription electronically

4. Change individual prescriber settings regarding less severe drug interactions (1 task) a. Change individual prescriber settings regarding less severe drug interactions

5. Access and change a laboratory order (2 tasks) a. Record a laboratory order b. Access and change a laboratory order


7/29/2015

6. Find item on Patient Summary Screen (1 task) a. Find item on Patient Summary Screen

7. Access Lab Results from Patient Chart (1 task) a. Access Lab Results from Patient Chart

8. Check Vital Sign recorded on Patient Chart (1 task) a. Check Vital Sign recorded on Patient Chart

9. Generate CDA Document (1 task)

a. Generate CDA Document for selected patient

10. Clinical decision support (2 task) a. Create and enable clinical decision support interventions rules with clinical data.

b. Check and record that clinical decision support interventions triggers electronically and occurs

automatically on patient chart for selected patient based on created CDS rule.

11. Clinical information reconciliation (1 task)

a. Create and Reconciled clinical information reconciliation for medication, allergy and problem list

for selected patient.

Tasks were selected based on their frequency of use, criticality of should always be constructed in light of the

study objectives.

PROCEDURES

Upon arrival, participants were greeted; their identity was verified and matched with a name on the participant

schedule. Participants were then informed consent and release form (See Appendix 3). A representative from

the test team witnessed the participant’s signature.

The administrator moderated the session including administering instructions and tasks. The administrator also

monitored task times, obtained post-task rating data, and took notes on participant comments. A second

person served as the data logger and took notes on task success, path deviations, number and type of errors,

and comments.

Participants were instructed to perform the tasks (see specific instructions below):

As quickly as possible making as few errors and deviations as possible.

Without assistance; administrators were allowed to give immaterial guidance and clarification

on tasks, but not instructions on use.

Without using a think aloud technique.

For each task, the participants were given a written copy of the task. Task timing began once the administrator

finished reading the question. The task time was stopped once the participant indicated they had successfully

completed the task. Scoring is discussed below in Section3.9.

Following the session, the administrator gave the participant the post-test questionnaire (e.g., the System

Usability Scale, see Appendix 5) compensated them for their time, and thanked each individual for their

participation.

Participants' demographic information, task success rate, time on task, errors, deviations, verbal responses,

and post-test questionnaire were recorded into a spreadsheet.

Participants were thanked for their time and compensated. Participants signed a receipt and acknowledgement

form (See Appendix 6) indicating that they had received the compensation.


7/29/2015

TEST LOCATION

Our test location was in our offices in San Diego, CA, but participants connected to our test computer using

GoToMeeting. The administrator and data logger watched the screen and listened to the participant. Noise

levels in the room were kept very low.

TEST ENVIRONMENT

The EHRUT would be typically be used in a healthcare office or facility. In this instance, the testing was

conducted in PatientClick’s Office (on a web-based platform on remote computer). For testing, the computer

used a Dell running Windows 7. The participants used their own computers through GoToMeeting, which had

mouse devices and keyboards, when interacting with the EHRUT.

The EHRUT used displays with 20-inchscreen size, resolution of 1280x960, color settings: sRGB

IEC61966-2.1.The application was set up by the PatientClick’s in-house Information Technology staff members

according to the vendor’s documentation describing the system set-up and preparation. The application itself

was running as Software-as-a-Service (SaaS) on a cloud-based web platform (using any web-browser) using a

test database on a WAN connection.

Technically, the system performance (i.e., response time) was representative to what actual users would

experience in a field implementation. Additionally, participants were instructed not to change any of the default

system settings (such as control of font size).

TEST FORMS AND TOOLS

During the usability test, various documents and instruments were used, including:

1. Informed Consent

2. Moderator’s Guide

3. Post-test Questionnaire

4. Incentive Receipt and Acknowledgment Form

The participant’s interaction with the EHRUT was captured and recorded digitally with screen capture software

running on the test machine. The method for recording session was a Web Camera which recorded each

participant’s facial expressions synced with the screen capture, and verbal comments were recorded nearby

observation room where the data logger observed the test session.

PARTICIPANT INSTRUCTIONS

The administrator reads the following instructions aloud to the each participant (also see the full moderator’s

guide in Appendix [B4]):

Thank you for participating in this study. Your input is very important. Our session today will last about

15 minutes. During that time you will use an instance of an electronic health record. I will ask you to

complete a few tasks using this system and answer some questions. You should complete the tasks as

quickly as possible making as few errors as possible. Please try to complete the tasks on your own

following the instructions very closely. Please note that we are not testing you we are testing the

system, therefore if you have difficulty all this means is that something needs to be improved in the


7/29/2015

system. I will be here in case you need specific help, but I am not able to instruct you or provide help in

how to use the application.

Overall, we are interested in how easy (or how difficult) this system is to use, what in it would be useful

to you, and how we could improve it. I did not have any involvement in its creation, so please be

honest with your opinions. All of the information that you provide will be kept confidential and your

name will not be associated with your comments at any time. Should you feel it necessary you are able

to withdraw at any time during the testing.

Following the procedural instructions, participants were shown the EHR and as their first task, were given time

(15minutes) to explore the system and make comments. Once this task was complete, the administrator gave

the following instructions:

For each task, I will read the description to you and say “Begin.”At that point, please perform the task

and say “Done” once you believe you have successfully completed the task. I would like to request that

you not talk loud or verbalize while you are doing the tasks. I will ask you your impressions about the

task once you are done.

Participants were then given eleven (11) task groups with twenty (20) tasks to complete. Tasks are listed in the

moderator’s guide in Appendix [B4].

USABILITY METRICS

According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health

Records, EHRs should support a process that provides a high level of usability for all users. The goal is for

users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To this end,

metrics for effectiveness, efficiency and user satisfaction were captured during the usability testing.

The goals of the test were to assess:

1. Effectiveness of PatientClick EHR by measuring Task Success and Task Failures

2. Efficiency of PatientClick EHR by measuring the Task Deviations and Task Time

3. Satisfaction with PatientClick EHR by measuring Task Ratings


7/29/2015

DATA SCORING

The following table details how tasks were scored, errors evaluated, and the time data analyzed.

Measures Rationale Scoring

Effectiveness:

Task Success

A task was counted as a “Success” if the participant was able to achieve the correct

outcome, without assistance, within the time allotted on a per task basis.

The total number of successes were calculated for each task and then divided by the total

number of times that task was attempted. The results are provided as a percentage.

Task times were recorded for successes. Observed task times divided by the optimal time

for each task is a measure of optimal efficiency.

Optimal task performance time, as benchmarked by expert performance under realistic

conditions, is recorded when constructing tasks. Target task times used for task times in

the Moderator’s Guide must be operationally defined by taking multiple measures of

optimal performance and multiplying by some factor 1.5 that allows some time buffer

because the participants are presumably not trained to expert performance. Thus, if

expert, optimal performance on a task was 10 seconds then allotted task time

performance was 10 * 1.5 seconds. This ratio should be aggregated across tasks and

reported with mean and variance scores.

Effectiveness:

Task Failures

If the participant abandoned the task, did not reach the correct answer or performed it

incorrectly, or reached the end of the allotted time before successful completion, the task

was counted as a “Failures.” No task times were taken for errors.

The total number of errors was calculated for each task and then divided by the total

number of times that task was attempted. Not all deviations would be counted as errors.

This should also be expressed as the mean number of failed tasks per participant.

On a qualitative level, an enumeration of errors and error types should be collected.

Efficiency:

Task Deviations

The participant’s path (i.e., steps) through the application was recorded. Deviations occur

if the participant, for example, went to a wrong screen, clicked on an incorrect menu item,

followed an incorrect link, or interacted incorrectly with an on-screen control. This path

was compared to the optimal path. The number of steps in the observed path is divided by

the number of optimal steps to provide a ratio of path deviation.

It is strongly recommended that task deviations be reported. Optimal paths (i.e.,

procedural steps) should be recorded when constructing tasks.

Efficiency:

Task Time

Each task was timed from when the administrator said “Begin” until the participant said,

“Done.” If he or she failed to say “Done,” the time was stopped when the participant

stopped performing the task. Only task times for tasks that were successfully completed

were included in the average task time analysis. Average time per task was calculated for


7/29/2015

each task. Variance measures (standard deviation and standard error) were also

calculated.

Satisfaction:

Task Rating

Participant’s subjective impression of the ease of use of the application was measured by

administering both a simple post-task question as well as a post-session questionnaire.

After each task, the participant was asked to rate “Overall, this task was:” on a scale of 1

(Very Difficult) to 5 (Very Easy). These data are averaged across participants.

Common convention is that average ratings for systems judged easy to use should be 3.3

or above.

To measure participant’s confidence in and likeability of the PatientClick EHR overall, the

testing team administered the System Usability Scale (SUS) post-test questionnaire.

Questions included, “I think I would like to use this system frequently,” “I thought the

system was easy to use,” and “I would imagine that most people would learn to use this

system very quickly.” See full System Usability Score questionnaire in Appendix 5

Efficiency:

TASK

TASK DEVIATION

(steps) (Observed/Optimal)

TASK

TIME(minutes) (Observed/Optimal)

1a. Select and record drug insurance carrier 2/2 0.50 / 0.45 1b. Select and record a patient allergy 4/4 0.61 / 0.55 1c. Access and change a patient allergy 6/6 0.65 / 0.55 2a. Select a patient reported medication 6/6 0.67 / 0.54 2b. Access and change a patient reported medication 3/3 0.70 / 0.61 2c. Record a patient reported medication 4/4 0.49 / 0.45 3a. Select a medication to prescribe and note formulary information 2/2 0.51 / 0.48 3b. Choose a medication to prescribe and record it 2/2 0.65 / 0.57 3c. Access and change a pending prescription 3/3 0.52 / 0.43 3d. Transmit the prescription electronically 2/2 0.50 / 0.46 4. Change individual prescriber settings regarding less severe drug interactions

3/3 0.37 / 0.33

5a. Record a laboratory order 7/7 0.55 / 0.45 5b. Access and change a laboratory order 6/6 0.45 / 0.38 6. Find item on Patient Summary Screen 4/4 0.47 / 0.40 7. Access Lab Results from Patient Chart 5/5 0.51 / 0.40 8. Check Vital Sign recorded on Patient Chart 5/5 0.61 / 0.50 9. Generate CDA Document for selected patient 3/3 0.52 / 0.45 10a. Create and enable Clinical Decision Support Interventions rule from miscellaneous menu.

3/3 0.60/0.55

10b. Check clinical decision support interventions triggers occurs automatically on patient chart

4/4 0.65/0.60

11. Create and Reconciled clinical information for medication, allergy and problem list

4/4 0.63/0.56


7/29/2015

RESULTS

DATA ANALYSIS AND REPORTING

The results of the usability test were calculated according to the methods specified in the Usability Metrics

section above. Participants who failed to follow session and task instructions had their data excluded from the

analyses.

The usability testing results for the EHRUT are detailed below. The results should be seen in light of the

objectives and goals outlined in Section 3.2 Study Design. The data should yield actionable results that, if

corrected, yield material, positive impact on user performance. Furthermore, the data should be presented in

forms such as the table below so that the tasks can be easily identified and their performance results examined and

compared.

Measure

Task

N

Task Success

Path Deviation

(steps)

Task Time (minutes)

Errors

Task Ratings 5=Easy

#

Mean (SD)

Deviations (Observed / Optimal)

Mean (SD)

Deviations (Observed / Optimal)

Mean (SD)

Mean (SD)

1a. Select and record drug insurance carrier

3 100% 2 / 2 0.80 0.50 / 0.45 0 5

1b. Select and record a patient allergy

3 100% 4 / 4 0.56 0.61 / 0.55 0 5

1c. Access and change a patient allergy

3 100% 6 / 6 0.65 0.65 / 0.55 0 5

2a. Select a patient

reported medication.

medication

3 100% 6 / 6 0.76 0.67 / 0.54 0 5

2b. Access and change a

patient reported

314.a.131314medication

3 100% 3 / 3 0.74 0.70 / 0.61 0 5

2c. Record a patient

reported medication

3 100% 4 / 4 0.84 0.49 / 0.45 0 5

3a. Select a medication to

prescribe and note

(a.2)formulary information

3 100% 2 / 2 0.60 0.51 / 0.48 0 5

3b. Choose a medication

to prescribe and record it

3 100% 2 / 2 0.65 0.65 / 0.57 0 5

3c. Access and change a

pending prescription

3 100% 3 / 3 0.62 0.52 / 0.43 0 5

3d. Transmit the

prescription electronically

3 100% 2 / 2 0.48 0.50 / 0.46 0 5

4. Change individual

prescriber settings

regarding less severe drug

interactions

3 100% 3 / 3 0.8 0.37 / 0.33 0 5

5a. Record a laboratory

order

3 94% 7 / 7 0.71 0.55 / 0.45 0 4

5b. Access and change a

laboratory order

3 96% 6 / 6 0.58 0.49 / 0.38 0 4

6. Find item on Patient

Summary Screen

3 100% 4 / 4 0.45 0.47 / 0.40 0 5


7/29/2015

The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system based on

performance with these tasks to be: [xx]. Broadly interpreted, scores under 60 represent systems with poor

usability; scores over 80 would be considered above average.

DISCUSSION OF THE FINDINGS

The test findings are discussed below.

EFFECTIVENESS

Find item on Patient Summary Screen: Participants were able to do so easily.

Access and Change Lab order: The first time any of the participants saw the laboratory ordering screens

was during the short training time during the testing session. Due to its intuitive user interface, participants

were able to learn quickly and perform the task easily.

Access Lab Results from Patient Chart: Participants were able to navigate to lab results easily.

Check Vital Sign recorded on Patient Chart: Participants were able to find and check them easily.

Generate CDA Document for selected patient: Participants were able to navigate to generate CDA

documents and were able to do so without problems.

Record a radiology order: Because radiology orders are added using the same user interface and processes

as the laboratory orders, participants had no problem performing this task. Participants learned quickly and did

not require much assistance to successfully record an order.

Select and record a patient allergy: Participants were familiar with recording allergies and were able to do so

easily.

Access and change a patient allergy: Participants were able to edit the severity and reaction and save the

changes easily.

7. Access Lab Results

from Patient Chart

3 100% 5 / 5 0.64 0.51 / 0.40 0 5

8. Check Vital Sign

recorded on Patient Chart

3 100% 5 / 5 0.20 0.61 / 0.50 0 5

9. Generate CDA

Document for selected

patient

3 100% 3 / 3 0.35 0.52 / 0.45 0 5

10a. Create and enable

Clinical Decision Support

Interventions rule from

miscellaneous menu

3 100% 3 / 3 0.75 0.60/0.55 0 5

10b. Check clinical

decision support

interventions triggers

occurs automatically on

patient chart

3

100% 4 / 4 0.80 0.65/0.60 0 5

11.Create and Reconciled

clinical information

reconciliation for

medication, allergy and

problem list

3 100% 4 / 4 0.78 0.63/0.56 0 5


7/29/2015

EFFICIENCY

Find item on Patient Summary Screen: All participants took the optimal path to successfully complete this

task.

Access and Change Lab order: All participants took the optimal path to successfully complete this task.

Access Lab Results from Patient Chart: All participants took the optimal path to successfully complete this

task.

Check Vital Sign recorded on Patient Chart: Participants took the optimal path to successfully complete this

task.

Generate CDA Document for selected patient: All participants took the optimal path to successfully

complete this task.

Record a radiology order: Participants took the optimal path to successfully complete this task.

Select and record a patient allergy: One hundred percent (100%) of the participants took the optimal path to

successfully complete this task.

Access and change a patient allergy: All participants took the optimal path to successfully complete this

task.

SATISFACTION

Find item on Patient Summary Screen: All participants rated this task as a 5.

Access and Change Lab order: All participants rated this feature as a 5 for very easy to use. Even the

participant who exceeded the time limit rated it as a 5.

Access Lab Results from Patient Chart: One hundred percent of the participants rated this task as a 5.

Check Vital Sign recorded on Patient Chart: All participants rated this feature as a 5 for very easy to use.

Even the participant who exceeded the time limit rated it as a 5.

Generate CDA Document for selected patient: One hundred percent of the participants rated this task as a

5

Record a radiology order: All participants rated this feature as a 5 for very easy to use. Even the participant

who exceeded the time limit rated it as a 5.

Select and record a patient allergy: All participants rated this task as a 5.

Access and change a patient allergy: All participants rated this feature as a 5 for very easy to use. Even the

participant who exceeded the time limit rated it as a 5.

MAJOR FINDINGS

The user interface, processes and features for the tasks were easy to navigate and user friendly. They noted

that it was intuitive, logical, straight-forward and required minimal clicks. The laboratory and radiology order

screens have a different flow from the rest of the site which gave participants slight pause but were able to

adapt quickly. Participants regarded the new functionality as highly desirable

AREAS FOR IMPROVEMENT

We found one area for improvement—training for the laboratory and radiology orders. Since these screens


7/29/2015

have a different flow from the rest of the site, training on how to place radiology and laboratory orders and also

how to find and modify existing orders when needed could be helpful in increasing efficiency task time.


7/29/2015

APPENDICES

The following appendices include supplemental data for this usability test report. Following is a list of the appendices

provided:

1: Sample recruiting screener

2: Participant demographics

3: Non-Disclosure Agreement (NDA) and Informed Consent Form

4: Example Moderator’s Guide

5: System Usability Scale Questionnaire

6: Incentive receipt and acknowledgment form


7/29/2015

Appendix 1: SAMPLE RECRUITING SCREENER

The purpose of a screener to ensure that the participants selected represent the target

user population as closely as possible. (Portions of this sample screener are taken from www.usability.gov/templates/index.html#Usability and adapted for use.) Recruiting Script for Recruiting Firm

Hello, my name is Steve, calling from PatientClick EHR. We are recruiting individuals to participate in a usability study for an electronic health record. We would like to ask you a few questions to see if you qualify and if would like to participate. This should only take a few minutes of your time. This is strictly for research purposes. If you are interested and qualify for the study, you will be paid to participate. Can I ask you a few questions?

Customize this by dropping or adding questions so that it reflects your EHR’s primary audience

1. [If not obvious] Are you male or female? [Recruit a mix of participants]

2. Have you participated in a focus group or usability test in the past xx months? [If yes, Terminate]

3. Do you, or does anyone in your home, work in marketing research, usability research, web design […etc.]? [If yes, Terminate]

4. Do you, or does anyone in your home, have a commercial or research interest in an electronic health record software or consulting company? [If yes, Terminate]

5. Which of the following best describes your age? [23 to 39; 40 to 59; 60 - to 74; 75 and older] [Recruit Mix]

6. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian, Black/African-American, Latino/a or Hispanic, etc.]

7. Do you require any assistive technologies to use a computer? [if so, please describe]

Professional Demographics Customize this to reflect your EHR’s primary audience

8. What is your current position and title? (Must be healthcare provider) RN: Specialty ________________

Physician: Specialty ________________

Resident: Specialty ________________

Administrative Staff

Other [Terminate]

15-Nov-10

http://www.usability.gov/templates/index.html#Usability


7/29/2015

9. How long have you held this position?

10. Describe your work location (or affiliation) and environment? (Recruit according to the intended users of the application) [e.g., private practice, health system, government clinic, etc.]

11. Which of the following describes your highest level of education? [e.g., high school graduate/GED, some college, college graduate (RN, BSN), postgraduate (MD/PhD), other (explain)]

Computer Expertise Customize this to reflect what you know about your EHR’s audience

12. Besides reading email, what professional activities do you do on the computer? [e.g., access EHR, research; reading news; shopping/banking; digital pictures; programming/word processing, etc.] [If no computer use at all, Terminate]

13. About how many hours per week do you spend on the computer? [Recruit according to the demographics of the intended users, e.g., 0 to 10, 11 to 25, 26+ hours per week]

14. What computer platform do you usually use? [e.g., Mac, Windows, etc.]

15. What Internet browser(s) do you usually use? [e.g., Firefox, IE, AOL, etc.]

16. In the last month, how often have you used an electronic health record?

17. How many years have you used an electronic health record?

18. How many EHRs do you use or are you familiar with?

19. How does your work environment patient records? [Recruit according to the demographics of

the intended users]

On paper

Some paper, some electronic

All electronic

Contact Information If the person matches your qualifications, ask Those are all the questions I have for you. Your background matches the people we're looking for. No Payment Done. For your participation, you will be paid 0.0. Would you be able to participate on [date, time]? [If so collect contact information] May I get your contact information?

Name of participant:

Address:

City, State, Zip:

Daytime phone number:

Evening phone number:

Alternate [cell] phone number:

Email address:

Before your session starts, we will ask you to sign a release form allowing us to videotape your session. The videotape will only be used internally for further study if needed. Will you consent to be videotaped? This study will take place at [location]. I will confirm your appointment a couple of days before your session and provide you with directions to our office. What time is the best time to reach

you?


7/29/2015

Appendix 2: PARTICIPANT DEMOGRAPHICS

The report should contain a breakdown of the key participant demographics. A representative list

is shown below.

Following is a high-level overview of the participants in this study.

Gender Men [2] Women [1] Total (participants) [3]

Occupation/Role RN/BSN [X] Physician [1] Admin Staff [2] Total (participants) [3]

Years of Experience Years experience [37] Facility Use of EHR All paper [15] Some paper, some [8] electronic All electronic [14] Total (participants) [3]

As an appendix to the report, the full participant breakdown (de-identified) should be included.


7/29/2015

Appendix 3: NON-DISCLOSURE AGREEMENT AND INFORMED CONSENT FORM

These are sample forms. The non-disclosure agreement is discretionary. Other examples may

be found at www.usability.gov.

Non-Disclosure Agreement

THIS AGREEMENT is entered into as of _DATE, between

_NAME_ (“the Participant”) and the testing organization PatientClick EHR Test Company located at 6540 Lusk

Blvd, Suite C150, San Diego, CA 92121s.

The Participant acknowledges his or her voluntary participation in today’s usability study may

bring the Participant into possession of Confidential Information. The term "Confidential

Information" means all technical and commercial information of a proprietary or confidential

nature which is disclosed by Test Company, or otherwise acquired by the Participant, in the

course of today’s study.

By way of illustration, but not limitation, Confidential Information includes trade secrets,

processes, formulae, data, know-how, products, designs, drawings, computer aided design files

and other computer files, computer software, ideas, improvements, inventions, training methods

and materials, marketing techniques, plans, strategies, budgets, financial information, or

forecasts.

Any information the Participant acquires relating to this product during this study is confidential

and proprietary to Test Company and is being disclosed solely for the purposes of the

Participant’s participation in today’s usability study. By signing this form the Participant

acknowledges that she/he will receive monetary compensation for feedback and will not disclose

this confidential information obtained today to anyone else or any other organizations. Participant’s printed name: __NAME, POSITION________________________________

Signature: _____________________________________ Date: 7/29/2015


7/29/2015

Informed Consent

Test Company would like to thank you for participating in this study. The purpose of this study is to

evaluate an electronic health records system. If you decide to participate, you will be asked to perform

several tasks using the prototype and give your feedback. The study will last about 60 minutes. At the

conclusion of the test, you will be compensated for your time. Agreement

I understand and agree that as a voluntary participant in the present study conducted by Test Company I am free to withdraw consent or discontinue participation at any time. I understand and agree to participate in the study conducted and videotaped by the Test Company. I understand and consent to the use and release of the videotape by Test Company. I understand that the information and videotape is for research purposes only and that my name and image will not be used for any purpose other than research. I relinquish any rights to the videotape and understand the videotape may be copied and used by Test Company without further permission. I understand and agree that the purpose of this study is to make software applications more useful and

usable in the future.

I understand and agree that the data collected from this study may be shared with outside of Test Company

and Test Company’s client. I understand and agree that data confidentiality is assured, because only de-

identified data – i.e., identification numbers not names – will be used in analysis and reporting of the

results.

I agree to immediately raise any concerns or areas of discomfort with the study administrator. I understand that I can leave at any time. Please check one of the following:

YES, I have read the above statement and agree to be a participant.

NO, I choose not to participate in this study.

Signature: _____________________________________ Date:_7/29/2015

Printed Name


7/29/2015

Appendix 4: EXAMPLE MODERATOR’S GUIDE

EHRUT Usability Test

Moderator’s Guide Administrator : STEVE PARKER ________________________ Data Logger : ANDY ________________________ Date : 7/29/2015__________Time : 10 AM _________ Participant # _: 3_______ Location : Web Based Platform / HIT Facility____________________________

Prior to testing

Confirm schedule with Participants Ensure EHRUT lab environment is running properly Ensure lab and data recording equipment is running properly

Prior to each participant: Reset application Start session recordings with tool

Prior to each task: Reset application to starting point for next task

After each participant: End session recordings with tool

After all testing Back up all video and data files


7/29/2015

Orientation (X minutes) Thank you for participating in this study. Our session today will last XX minutes. During that

time you will take a look at an electronic health record system.

I will ask you to complete a few tasks using this system and answer some questions. We are

interested in how easy (or how difficult) this system is to use, what in it would be useful to you,

and how we could improve it. You will be asked to complete these tasks on your own trying to do

them as quickly as possible with the fewest possible errors or deviations. Do not do anything

more than asked. If you get lost or have difficulty I cannot answer to help you with anything to do

with the system itself. Please save your detailed comments until the end of a task or the end of the

session as a whole when we can discuss freely.

I did not have any involvement in its creation, so please be honest with your opinions.

The product you will be using today is describe the state of the application, i.e., production

version, early prototype, etc. Some of the data may not make sense as it is placeholder data.

We are recording the audio and screenshots of our session today. All of the information that you

provide will be kept confidential and your name will not be associated with your comments at any

time.

Do you have any questions or concerns?

Preliminary Questions (X minutes) What is your job title / appointment? How long have you been working in this role? What are some of your main responsibilities? Tell me about your experience with electronic health records.


7/29/2015

Task Group 1: Select Drug Insurance, Record and Modify an Allergy

a. Select Anthem Blue Cross Blue Shield of Indiana POS as the Medicare Part D health plan for the selected patient

and save it. BEGIN NOW.

Success:

Easily completed

Completed with difficulty or help – describe below

Not completed

Comments:

Task Time: ________ Seconds

Optimal Path: Pt. Details>Medicare Part D dropdown>Either Save button

Correct

Minor Deviations – describe below

Major Deviations – describe below

Comments:

Observed Errors and Verbalizations:

Comments:

Rating:

Overall, this task was: 1

Show participant written scale: “Very Easy” (1) to “Very Difficult” (5)

Administrator / Note taker Comments:


7/29/2015

b. The patient says he is allergic to penicillin but doesn’t mention any severity or reaction.

Add this allergy and save it. BEGIN NOW

Success:

Easily completed


Not completed

Comments:


Optimal Path: Pt. Dashboard>Allergies>Prescribe button>Penicillins>SaveAllergy button

Correct




Comments:

Rating:





7/29/2015

c. The patient later states that his penicillin allergy results in a moderate reaction causing shortness of breath. Update this

allergy and save the changes. BEGIN NOW.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Pt. Dashboard>Allergies>Prescribe>Penicillins>Severity dropdown>Note>Save

Allergy button

Correct




Comments:

Rating:




(This satisfies task satisfies Medication allergy list (314.a.7) record, change and access medication allergy list).


7/29/2015

Task Group 2: Record and Modify Patient Reported Medications

a. Record that the patient has been taking penicillin V potassium 250 mg tablet. BEGIN NOW.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Pt. Dashboard>Medications>Prescribe>penicillin v>Drug Search button>scroll>click penicillin V

potassium 250 mg tab

Correct




Comments:

Rating:




Did the participant notice the drug-allergy interaction and close it?

(This satisfies the drug-allergy interactions checks (314.a.2) prior to CPOE completion.)


7/29/2015

b. The patient states that he has just finished the course of penicillin V potassium, so remove it from the list. BEGIN NOW.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Pt. Dashboard>Medications> Prescribe>Click X next to penicillin V potassium 250 mg Tab

Correct




Comments:

Rating:





7/29/2015

c. The patient takes clonidine. Add and save clonidine 0.1 mg tablet to the patient’s list of current medications. BEGIN

NOW

Success:

Easily completed


Not completed

Comments:


Optimal Path: Med Entry>clonidine>Drug Search button>click clonidine 0.1 mg Tab>Select to

Move to Current Meds button

Correct




Comments:

Rating:




(This satisfies medication list (314.a.6) for recording, changing and accessing the medication list.)


7/29/2015

Task Group 3: Record Medication Order and ePrescribe

a. We will now be prescribing a new drug for this patient. Search for Timoptic and note formulary information. BEGIN

NOW.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Compose Rx>Timoptic>Drug Search button

Correct




Comments:

Rating:





7/29/2015

b. Choose Timoptic 0.5% 1 drop in both eyes daily and save the prescription. BEGIN NOW.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Click Timoptic 0.5% >Daily>Save Rx

Correct




Comments:

Rating:





7/29/2015

c. Edit the prescription and change it from both eyes to the left eye. Save the prescription and take the prescription to the

review page. BEGIN NOW

Success:

Easily completed


Not completed

Comments:


Optimal Path: Click Edit button >eye, left>Save Rx>Take Complete Rx to Review Page button

Correct




Comments:

Rating:




(This satisfies task satisfies CPOE (314.a.1) record, change and access medication order.)

Did the participant notice the drug-drug interaction?

(This satisfies the Drug-drug interactions checks (314.a.2) prior to CPOE completion.)


7/29/2015

d. Transmit the prescription electronically to a test pharmacy in the list. BEGIN NOW.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Transmit Rx button>test pharmacy from list>Transmit Rx/Add to Record button

Correct




Comments:

Rating:




(This satisfies Electronic prescribing (314.b.3)


7/29/2015

Task Group 4: Change Individual Prescriber Settings

Change the Individual Prescriber Settings to hide less severe drug interactions and save the changes. BEGIN NOW.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Admin>Uncheck Hide All Less Sever Drug Interactions (Show Only

Contraindicated)>Scroll>Save Prescriber Settings button

Correct




Comments:

Rating:





7/29/2015

Task Group 5: Record Laboratory and Radiology Orders

a. Select Meaningful Use Ordering Lab and either ordering physician and order B1001, a

Complete Blood Count w/Differential (CBC), for diagnosis 366.16. BEGIN NOW.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Imaging>Order Radiology Test>Order New Test> Meaningful Use Ordering Lab from dropdown>Ordering

Physician from dropdown> search for or check B1001>search for or check 366.16>Save button

Correct




Comments:

Rating:





7/29/2015

b. Change the diagnosis on the CBC to v70.0 and validate the order. BEGIN NOW

Success:

Easily completed


Not completed

Comments:


Optimal Path: Imaging>View Radiology Orders >View Order button>check B1001>uncheck

366.16>search for or check V70.0>Save button

Correct




Comments:

Rating:




(This satisfies task satisfies CPOE (314.a.1) record, change and access laboratory and radiology/imaging orders.)


7/29/2015

Task Group 6: Find item on Patient Summary Screen

Find a patient and review lab result. BEGIN NOW.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Search Patient>Select Patient from list>Pt. Dashboard>Health Information>Select items from list

Correct




Comments:

Rating:





7/29/2015

Task Group 7: Access Lab Results from Patient Chart

Find a patient and review lab result. BEGIN NOW.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Search Patient>Select Patient from list>Pt. Dashboard>Test Result tab>Select Test>View Results button

Correct




Comments:

Rating:





7/29/2015

Task Group 8: Check Vital Sign recorded on Patient Chart.

Find a patient and check vital signs. BEGIN NOW.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Search Patient>Select Patient from list>Pt. Dashboard>Vital Signs tab>Select Chart / Tabular link

Correct




Comments:

Rating:





7/29/2015

Task Group 9: Generate CDA Document for selected patient.

Generate CDA documents for multiple patients. BEGIN NOW.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Reports>Generate CCD Report>Select Patients>Generate HL7 CCD button

Correct




Comments:

Rating:





7/29/2015

Task Group 10: Clinical decision support.

a. Create and enable clinical decision support interventions rules with clinical data.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Miscellaneous>Clinical decision rules>Create new rule with clinical data

Correct




Comments:

Rating:





7/29/2015

b. Check and record that clinical decision support interventions triggers electronically and occurs automatically on patient

chart for selected patient based on created CDS rule.

Success:

Easily completed


Not completed

Comments:


Optimal Path: Search Patient>Select Patient from list>Pt. Dashboard>CDS tab

Correct




Comments:

Rating:





7/29/2015

Task Group 11: Clinical information reconciliation.

Create and Reconciled clinical information reconciliation for medication, allergy and problem list for selected patient

Success:

Easily completed


Not completed

Comments:


Optimal Path: Reports>Clinical Reconciliation >Search Patient>Select Patient>Import CDA file>Reconcile clinical data

Correct




Comments:

Rating:





7/29/2015

Final Questions (X Minutes) What was your overall impression of this system? What aspects of the system did you like most? What aspects of the system did you like least? Were there any features that you were surprised to see? What features did you expect to encounter but did not see? That is, is there anything that is missing in this application? Compare this system to other systems you have used. Would you recommend this system to your colleagues? Administer the SUS


7/29/2015

Appendix 5: SYSTEM USABILITY SCALE QUESTIONNAIRE

In 1996, Brooke published a “low-cost usability scale that can be used for global assessments of systems usability” known as the System Usability Scale or SUS.

16 Lewis and Sauro (2009) and others have elaborated on the SUS over the years. Computation of the SUS score can be found in Brooke’s paper, in at http://www.usabilitynet.org/trump/documents/Suschapt.doc or in Tullis and Albert (2008).

Strongly Strongly disagree agree

1. I think that I would like to use this system frequently

1 2 3 4 5

2. I found the system unnecessarily

complex 1 2 3 4 5

3. I thought the system was easy to use

1 2 3 4 5

4. I think that I would need the support of a technical person to be able to use this system

5. I found the various functions in this system were well integrated

6. I thought there was too much inconsistency in this system

7. I would imagine that most people

would learn to use this system very quickly

8. I found the system very cumbersome to use

9. I felt very confident using the system

10. I needed to learn a lot of things before I could get going with this system

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

1 2 3 4 5

5

1

1

5

5

1

5

1

5

1

http://www.usabilitynet.org/trump/documents/Suschapt.doc


7/29/2015

Appendix 6: INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM

Acknowledgement of Receipt

I hereby acknowledge receipt of $______ for my participation in a research study run by Test Company. Printed Name: ___________________________________________________________

Address: _______________________________________________________________ ___________________________________________________________

Signature: _________________________________ Date: _______________ Usability Researcher: __________________________________

Signature of Usability Researcher: ___________________________

Date: _______________ Witness: ________________________________________

Witness Signature: _______________________________________

Date: _______________


16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

Appendix C: Quality Management System


Quality Management System Attestation Forin-EHR-37-V03

For reporting information related to testing of170.314(g)(4).

Vendor Name ' Scribe Genie LLC

Product Name -. Patientdhck

Type of Quality Management [] Based on Industry Standard (for example 1509001, IEC 62304, ISO System (OMS) used in the 13485, etc.). Standard:

development, testing, ® A modified or "home-grown" QMS. implementation, and Q No QMS was used. maintenance of EHR product.

Was one QMS used for all ® One QMS used, certification criteria or were Q Multiple QMS used. multiple QMS applied?

Description or documentation of QMS applied to each criteria:

Q Not Applicable.

I, the undersigned, attest that the statements in this document are completed and accurate.

Vendor Signature by an Authorized Representative

I Vish Mehta, President

Date 6/16/2015

InfoGard Laboratories, Inc. Page 1


16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

Appendix D: Privacy and Security


Privacy and Security Attestation Form-EHR-36-VO4

Vendor Name PatientClick

Product Name PatientClick ERR

110.314(d)(2) Auditable Audit Log:

events and tamper- Cannot be disabled by any user. resistance LI Audit Log can be disabled.

R Not Applicable (did not •n The EHR enforces that the audit log is enabled by default test to this criteria)

when initially configured

Audit Log Status Indicator:

LI Cannot be disabled by any user.

fl Audit Log Status can be disabled

fl The ERR enforces a default audit log status. Identify the default setting (enabled or disabled):

There is no Audit Log Status Indicator because the Audit Log cannot

be disabled.

Encryption Status Indicator (encryption of health information locally

on end user device):

Cannot be disabled by any user.

LI Encryption Status Indicator can be disabled

• LI The EHR enforces a default encryption status. Identify the

default setting (enabled or disabled):

Z There is no Encryption Status Indicator because the EHR does not allow health information to be stored locally on end user devices.

Identify the submitted documentation that describes the inability of the

EI-IR to allow users to disable the audit logs, the audit log status, and/or

the encryption status: PatientClick EHR is a SaaS based hosted solution,

no user on EHR platform has the ability to disable the Audit log, audit

log status and encryption status. All Users can either able to view audit

log or not based on who has assigned rights or access.

Identify the submitted documentation that describes the method(s) by

which the ERR protects 1) recording of actions related to electronic health information, 2) recording of audit log status, and 3) recording of

encryption status from being changed, overwritten, or deleted by the

ERR tpchnnIov: P2tientClick EHR solution Audit L0R keecs record of


Privacy and Security Attestation

what pages were viewed by which user,any changes were made by him -

in particular section as well as any data is deleted by him/her. . Audit

log will also display what all activities were done by particular user at

given date and time. No user has ability to either disable or modify

audit log in the system.

Identify the submitted documentation that describes the method(s) by

which the [HR technology detects whether the audit log has been

altered: PatientCtick Compliance team checks daily activity and audit all

actions. Any unauthorized or abnormal activity immidiately get flagged

by ERR system and then get addressed by IT department.

Z The EHR does not allow health information to be stored locally on

end-user devices.

Identify the submitted documentation that describes the functionality used to prevent health information from being

stored locally: For security we are using SHA1 and MD5

cryptography methods. We are not storing any data on the Client

End. (i.e. in Cookies). On top of this we are using SSL Certificate for web security. Whatever sensitive data we are passing in

Query string'sin encrypted format.

fl The EHR does allow health information to be stored locally on end

user devices.

Identify the RIPS 140-2 approved algorithm used for encryption:

170.314(d)(7) End-user

device encryption

Storing electronic health

information locally on end-user devices (i.e. temp files,

cookies, or other types of

cache approaches).

LI Not Applicable (did not

test to this criteria)

• Identify the submitted documentation that describes how health

information is encrypted when stored locally on end-user

devices:

fl The ERR enforces default configuration settings that either enforces

the encryption of locally stored health information or prevents health

information from being stored locally.

Identify the default setting:

Identify the hashing algorithm used for integrity (SHA-1 or higher):

SHA1

Identify the FPS 140-2 approved algorithm used for encryption:

SHA1

Identify the HIPS 140-2 approved algorithm used for hashing:

SHA1

170.314(d)(8) Integrity

fl Not Applicable (did not test to this criteria)

170.314(e)(1) View,

Download, and Transmit to

3rd Party

fl Not Applicable (did not

test to this criteria)


Privacy and Security Attestation Form-EHR-36-VU4

170.314(2)(3} Secure Identify the FPS 140-2 approved algorithm used for encryption

Messaging SHAI

fl Not Applicable did not Identify the FIPS 140-2 approved algorithm used for hashing:

I, the undersignçattest that the statements in this document are accurate

byanAuthorized Representative Vish Mehta

Date

2/13/2015



16‐3528‐R‐0027‐PRA V1.1, November 14, 2016

Test Results Summary Document History Version

V1.0

V1.1

Initial release

END OF DOCUMENT

Description of Change

10/19/2016

Date

Updated vendor name 11/14/2016


onc hit program test results summary for 2014 edition ehr …€¦ · test results summary for 2014...

Documents