Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 1 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
ONC HIT Certification Program Test Results Summary for 2014 Edition EHR Certification
Part 1: Product and Developer Information
1.1 Certified Product Information
Product Name: ezEMRxPrivate
Product Version: 9.00
Domain: Ambulatory
Test Type: Complete EHR
1.2 Developer/Vendor Information
Developer/Vendor Name: ezEMRx Inc
Address: 25W217 Lake Street
Roselle IL 60172
Website: www.ezemrx.com
Email: [email protected]
Phone: 630-307-7600
Developer/Vendor Contact: Siri Kumar
Part 2: ONC-Authorized Certification Body Information
2.1 ONC-Authorized Certification Body Information
ONC-ACB Name: InfoGard Laboratories, Inc.
Address: 709 Fiero Lane Suite 25
San Luis Obispo, CA 93401
Website: www.infogard.com
Email: [email protected]
Phone: (805) 783-0810
ONC-ACB Contact: Adam Hardcastle
This test results summary is approved for public release by the following ONC-Authorized Certification Body Representative:
Adam Hardcastle
EHR Certification Body Manager ONC-ACB Authorized Representative Function/Title
2/28/16 Signature and Date
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 2 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
2.2 Gap Certification The following identifies criterion or criteria certified via gap certification
§170.314
(a)(1) (a)(17) (d)(5) (d)(9)
(a)(6) (b)(5)* (d)(6) (f)(1)
(a)(7) (d)(1) (d)(8)
*Gap certification allowed for Inpatient setting only
No gap certification
2.3 Inherited Certification The following identifies criterion or criteria certified via inherited certification
§170.314 (a)(1) (a)(14) (c)(3) (f)(1) (a)(2) (a)(15) (d)(1) (f)(2) (a)(3) (a)(16) Inpt. only (d)(2) (f)(3) (a)(4) (a)(17) Inpt. only (d)(3) (f)(4) Inpt. only (a)(5) (b)(1) (d)(4)
(f)(5) Optional & Amb. only (a)(6) (b)(2) (d)(5)
(a)(7) (b)(3) (d)(6) (f)(6) Optional &
Amb. only (a)(8) (b)(4) (d)(7) (a)(9) (b)(5) (d)(8) (g)(1) (a)(10) (b)(6) Inpt. only (d)(9) Optional (g)(2) (a)(11) (b)(7) (e)(1) (g)(3) (a)(12) (c)(1) (e)(2) Amb. only (g)(4)
(a)(13) (c)(2) (e)(3) Amb. only
No inherited certification
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 3 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
Part 3: NVLAP-Accredited Testing Laboratory Information
Report Number: 14-2469-R-0004 V1.3 Test Dates: December 26, 27, 30, 31, 2013; January 2, 10, 15, 20, 22, 2014; March 24, 2014; May 2, 2014
3.1 NVLAP-Accredited Testing Laboratory Information
ATL Name: InfoGard Laboratories, Inc.
Accreditation Number: NVLAP Lab Code 100432-0
Address: 709 Fiero Lane Suite 25
San Luis Obispo, CA 93401
Website: www.infogard.com
Email: [email protected]
Phone: (805) 783-0810
ATL Contact: Milton Padilla
For more information on scope of accreditation, please reference http://ts.nist.gov/Standards/scopes/1004320.htm
Part 3 of this test results summary is approved for public release by the following Accredited Testing Laboratory Representative:
Milton Padilla
EHR Test Body Manager ATL Authorized Representative Function/Title
2/28/16 Signature and Date
3.2 Test Information
3.2.1 Additional Software Relied Upon for Certification
Additional Software Applicable Criteria Functionality provided by Additional Software
No additional software required
3.2.2 Test Tools
Test Tool Version
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 4 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
Test Tool Version Cypress 2.4.0 ePrescribing Validation Tool 10.6 HL7 CDA Cancer Registry Reporting Validation Tool HL7 v2 Electronic Laboratory Reporting (ELR) Validation Tool 1.7.0
HL7 v2 Immunization Information System (IIS) Reporting Validation Tool 1.8.0
HL7 v2 Laboratory Results Interface (LRI) Validation Tool 1.7.0 HL7 v2 Syndromic Surveillance Reporting Validation Tool 1.7.0 Transport Testing Tool 174 Direct Certificate Discovery Tool 2.1
No test tools required
3.2.3 Test Data
Alteration (customization) to the test data was necessary and is described in Appendix A
No alteration (customization) to the test data was necessary
3.2.4 Standards
3.2.4.1 Multiple Standards Permitted The following identifies the standard(s) that has been successfully tested where more than one standard is permitted
Criterion # Standard Successfully Tested
(a)(8)(ii)(A)(2)
§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain
§170.204(b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide
(a)(13)
§170.207(a)(3) IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
§170.207(j) HL7 Version 3 Standard: Clinical Genomics; Pedigree
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 5 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
Criterion # Standard Successfully Tested
(a)(15)(i)
§170.204(b)(1) HL7 Version 3 Implementation Guide: URL-Based Implementations of the Context-Aware Information Retrieval (Infobutton) Domain
§170.204(b)(2) HL7 Version 3 Implementation Guide: Context-Aware Knowledge Retrieval (Infobutton) Service-Oriented Architecture Implementation Guide
(a)(16)(ii) §170.210(g)
Network Time Protocol Version 3 (RFC 1305)
§170. 210(g) Network Time Protocol Version 4 (RFC 5905)
(b)(2)(i)(A)
§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions
§170.207(a)(3) IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
(b)(7)(i)
§170.207(i) The code set specified at 45 CFR 162.1002(c)(2) (ICD-10-CM) for the indicated conditions
§170.207(a)(3) IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
(e)(1)(i)
Annex A of the FIPS Publication 140-2 • Signature algorithm: sha1RSA • Signature hash algorithm: sha1 • Public key: RSA (2048 Bits)
(e)(1)(ii)(A)(2) §170.210(g)
Network Time Protocol Version 3 (RFC 1305)
§170. 210(g) Network Time Protocol Version 4 (RFC 5905)
(e)(3)(ii)
Annex A of the FIPS Publication 140-2 • Signature algorithm: sha1RSA • Signature hash algorithm: sha1 • Public key: RSA (2048 Bits)
Common MU Data Set (15)
§170.207(a)(3) IHTSDO SNOMED CT® International Release July 2012 and US Extension to SNOMED CT® March 2012 Release
§170.207(b)(2) The code set specified at 45 CFR 162.1002(a)(5) (HCPCS and CPT-4)
None of the criteria and corresponding standards listed above are applicable
3.2.4.2 Newer Versions of Standards The following identifies the newer version of a minimum standard(s) that has been successfully tested
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 6 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
Newer Version Applicable Criteria
No newer version of a minimum standard was tested
3.2.5 Optional Functionality
Criterion # Optional Functionality Successfully Tested
(a)(4)(iii) Plot and display growth charts
(b)(1)(i)(B) Receive summary care record using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(1)(i)(C) Receive summary care record using the standards specified at §170.202(b) and (c) (SOAP Protocols)
(b)(2)(ii)(B) Transmit health information to a Third Party using the standards specified at §170.202(a) and (b) (Direct and XDM Validation)
(b)(2)(ii)(C) Transmit health information to a Third Party using the standards specified at §170.202(b) and (c) (SOAP Protocols)
(f)(3) Ambulatory setting only – Create syndrome-based public health surveillance information for transmission using the standard specified at §170.205(d)(3) (urgent care visit scenario)
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(3) (45 CFR162.1002(a)(4): Code on Dental Procedures and Nomenclature)
Common MU Data Set (15)
Express Procedures according to the standard specified at §170.207(b)(4) (45 CFR162.1002(c)(3): ICD-10-PCS)
No optional functionality tested
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 7 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
3.2.6 2014 Edition Certification Criteria* Successfully Tested
Criteria # Version
Criteria # Version
TP** TD*** TP TD (a)(1) 1.2 1.5 (c)(3) 1.7.1 (a)(2) 1.2 (d)(1)
(a)(3) 1.2 1.4 (d)(2) 1.4 (a)(4) 1.4 1.3 (d)(3) 1.3 (a)(5) 1.4 1.3 (d)(4) 1.2 (a)(6) (d)(5) (a)(7) (d)(6) (a)(8) 1.2 (d)(7) 1.2 (a)(9) 1.3 1.3 (d)(8) (a)(10) 1.2 1.4 (d)(9) Optional (a)(11) 1.3
(e)(1) 1.7 1.4 (a)(12) 1.3 (e)(2) Amb. only 1.2 1.5 (a)(13) 1.2 (e)(3) Amb. only 1.3 (a)(14) 1.2 (f)(1) (a)(15) 1.5 (f)(2) 1.3 1.3 (a)(16) Inpt. only (f)(3) 1.3 1.3 (a)(17) Inpt. only (f)(4) Inpt. only
(b)(1) 1.6 1.3 (f)(5) Optional &
Amb. only (b)(2) 1.4 1.5
(b)(3) 1.4 1.3 (f)(6) Optional &
Amb. only (b)(4) 1.3 1.4
(b)(5) 1.4 1.3.0 (g)(1) 1.6 1.8 (b)(6) Inpt. only (g)(2) 1.6 1.8 (b)(7) 1.4 1.5 (g)(3) 1.3
(c)(1) 1.7.1 (g)(4) 1.2
(c)(2) 1.7.1
*For a list of the 2014 Edition Certification Criteria, please reference http://www.healthit.gov/certification (navigation: 2014 Edition Test Method) **Indicates the version number for the Test Procedure (TP) ***Indicates the version number for the Test Data (TD)
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 8 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
3.2.7 2014 Clinical Quality Measures*
Type of Clinical Quality Measures Successfully Tested: Ambulatory Inpatient No CQMs tested
*For a list of the 2014 Clinical Quality Measures, please reference http://www.cms.gov (navigation: 2014 Clinical Quality Measures)
Ambulatory CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version
2 90 136 155 V2
22 117 V2 137 156 V2
50 V2 122 V2 138 V2 157
52 123 139 158
56 124 140 159
61 125 141 160
62 126 V2 142 161
64 127 143 163
65 128 144 164
66 129 145 165 V2
68 V3 130 146 166
69 V2 131 147 167
74 132 148 V2 169
75 133 149 177
77 134 153 179
82 135 154 V2 182
Inpatient CQMs CMS ID Version CMS ID Version CMS ID Version CMS ID Version
9 71 107 172
26 72 108 178
30 73 109 185
31 91 110 188
32 100 111 190
53 102 113
55 104 114
60 105 171
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 9 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
3.2.8 Automated Numerator Recording and Measure Calculation
3.2.8.1 Automated Numerator Recording
Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6)
(a)(3) (a)(11) (a)(17) (e)(1)
(a)(4) (a)(12) (b)(2) (e)(2)
(a)(5) (a)(13) (b)(3) (e)(3)
(a)(6) (a)(14) (b)(4)
(a)(7) (a)(15) (b)(5)
Automated Numerator Recording was not tested
3.2.8.2 Automated Measure Calculation
Automated Numerator Recording Successfully Tested (a)(1) (a)(9) (a)(16) (b)(6)
(a)(3) (a)(11) (a)(17) (e)(1)
(a)(4) (a)(12) (b)(2) (e)(2)
(a)(5) (a)(13) (b)(3) (e)(3)
(a)(6) (a)(14) (b)(4)
(a)(7) (a)(15) (b)(5)
Automated Measure Calculation was not tested
3.2.9 Attestation
Attestation Forms (as applicable) Appendix
Safety-Enhanced Design* B
Quality Management System** C
Privacy and Security D
*Required if any of the following were tested: (a)(1), (a)(2), (a)(6), (a)(7), (a)(8), (a)(16), (b)(3), (b)(4) **Required for every EHR product
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 10 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
Appendix A: Alteration of Test Data
Criteria Explanation 170.314(b)(2) Determined that modified Test Data had equivalent level of robustness to
NIST Test Data
170.314(b)(7) Determined that modified Test Data had equivalent level of robustness to NIST Test Data
170.314(e)(1) Determined that modified Test Data had equivalent level of robustness to NIST Test Data
170.314(e)(2) Determined that modified Test Data had equivalent level of robustness to NIST Test Data
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 11 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
Appendix B: Safety-Enhanced Design
EHR Usability Test Report of ezEMRxPrivate Version 9.00
Report based on ISO/IEC 25062:2006 Common Industry Format for Usability Test Reports
Report Prepared By: ezEMRx Customer Teams
Balaji Venkatesh, Nick Daftari
Confidential Page 2 of 28
Table of Contents
Table of Contents ...............................................................................................................................2
EXECUTIVE SUMMARY .......................................................................................................................4
INTRODUCTION .................................................................................................................................5
UCD PROCESS OUTLINE ......................................................................................................................6
DESIGN ACTIVITIES .................................................................................................................................... 6
UCD PROCESS CHART ................................................................................................................................ 7
STUDY METHOD .................................................................................................................................8
PARTICIPANTS ........................................................................................................................................... 8
STUDY DESIGN .......................................................................................................................................... 9
DATA SCORING ........................................................................................................................................ 10
TASKS ...................................................................................................................................................... 11
TEST ENVIRONMENT ............................................................................................................................... 11
TEST FORMS AND TOOLS ........................................................................................................................ 12
USABILITY METRICS ................................................................................................................................. 12
SUMMARY OF TEST RESULTS ............................................................................................................ 13
PARTICIPANTS ......................................................................................................................................... 13
DATA ANALYSIS AND REPORTING ........................................................................................................... 14
DISCUSSION OF THE FINDINGS ............................................................................................................... 15
EFFECTIVENESS ....................................................................................................................................... 15
EFFICIENCY .............................................................................................................................................. 15
SATISFACTION ......................................................................................................................................... 15
MAJOR FINDINGS .................................................................................................................................... 16
AREAS FOR IMPROVEMENT .................................................................................................................... 16
Confidential Page 3 of 28
APPENDIX ........................................................................................................................................ 17
Sample - Initial Questionnaire ................................................................................................................ 17
Sample – Informed Consent ................................................................................................................... 19
Sample – Incentive receipt and acknowledgement ................................................................................ 20
Sample – SUS Questionnaire................................................................................................................... 21
Sample – Tasks ........................................................................................................................................ 22
Task 1: Medication list ........................................................................................................................ 22
Task 2: Medication allergy list ............................................................................................................. 23
Task 3: Interaction Check – [D-to-D and D-to-A] ................................................................................ 24
Task 4: Prescribe Medication .............................................................................................................. 25
Task 5: CPOE ........................................................................................................................................ 26
Task 6 : Clinical Information Reconciliation ........................................................................................ 27
Task 7: Clinical Decision support ......................................................................................................... 28
Confidential Page 4 of 28
EXECUTIVE SUMMARY
A usability test of ezEMRxPrivate 9.00, an Ambulatory EHR was conducted onsite at participant locations
by ezEMRx. The purpose of this test was to test and validate the usability of the current user interface,
and provide evidence of usability in the EHR Under Test (EHRUT).
During the usability test, six healthcare providers served as participants and used the EHRUT in
simulated, but representative tasks.
This study collected performance data on seven tasks conducted on an EHR:
Computerized provider order entry
Drug-drug, drug-allergy interaction checks
Medication list
Medication allergy list
Clinical decision support
Electronic prescribing
Clinical information reconciliation
During the one-on-one usability test, each participant was asked to review and sign an informed
consent/release form (template included in Appendix); they were instructed that they could withdraw at
any time. Participants all had prior experience with the usage of a previous version of the EHRUT.
The participants were instructed to complete a series of tasks using the EHRUT. During the testing, the
test time was timed and recorded as user performance data on paper. There was no assistance provided
on how to complete the task.
All participant data were de-identified – no correspondence could be made from the identity of the
participant to the data collected. Following the conclusion of the testing, participants were
compensated with $150 (physicians) and $100 (clinical staff) for their time.
Various recommended metrics, in accordance with the examples set forth in the NIST Guide to the
Processes Approach for Improving the Usability of Electronic Health Records, were used to evaluate the
usability of the EHRUT.
The UCD process applied for EHR Under Test (EHRUT) is not a industry standard process, but rather a
custom process which is broadly based on the ISO 134071 Human centred design processes for
interactive systems.
1 ISO 13407 - Human-centred design processes for interactive systems
ISO 13407 provides guidance on achieving quality in use by incorporating user centred design activities throughout
the life cycle of interactive computer-based systems. It describes user centred design as a multi-disciplinary
activity, which incorporates human factors and ergonomics knowledge and techniques with the objective of
Confidential Page 5 of 28
INTRODUCTION
The EHRUT(s) tested for this study was ezEMRxPrivate 9.00, an Ambulatory EHR. Designed to present
medical information to healthcare providers in a clinic and specialty setting, the EHRUT consists of a
workflow based approach of data capture and retrieval. The usability testing attempted to represent
realistic exercises and conditions.
The purpose of this study was to test and validate the usability of the current user interface, and provide
evidence of usability in the EHR Under Test (EHRUT).
To this end, measures of effectiveness, efficiency and user satisfaction, such as time taken to write a
prescription, enter medications, allergies, execute orders with decision support validation, were
captured during the usability testing.
The UCD process applied for EHR Under Test (EHRUT) is not an industry standard process, but rather a
custom process which is broadly based on the ISO 13407 Human centred design processes for
interactive systems.
The UCD study methods (adaptation based on ISO/IEC 25062:20062 and NIST IR 77423) and the UCD
Process (non standard custom process) described in this document were applied for the following
criteria:
· §170.314(a)(1) Computerized provider order entry
· §170.314(a)(2) Drug-drug, drug-allergy interaction checks
· §170.314(a)(6) Medication list
· §170.314(a)(7) Medication allergy list
enhancing effectiveness and productivity, improving human working conditions, and counteracting the possible
adverse effects of use on human health, safety and performance.
http://www.iso.org/iso/catalogue_detail.htm?csnumber=21197
2 ISO/IEC 25062:2006 Software engineering - Software product Quality Requirements and Evaluation (SQuaRE) --
Common Industry Format (CIF) for usability test reports.ISO/IEC 25062:2006 provides a standard method for
reporting usability test findings. The format is designed for reporting results of formal usability tests in which
quantitative measurements were collected, and is particularly appropriate for summative/comparative testing.
http://www.iso.org/iso/catalogue_detail.htm?csnumber=43046
3 NISTIR 7742 - Customized Common Industry Format Template for Electronic Health Record Usability
Testing.NISTIR 7742 provides a template for the modified version of Software engineering Software product
Quality Requirements and Evaluation(SQuaRE) Common Industry Format (CIF) for usability test reports (ISO/IEC
25062:2006(E)), the Common Industry Format (CIF) usability test report.
http://www.nist.gov/manuscript-publication-search.cfm?pub_id=907312
Confidential Page 6 of 28
· §170.314(a)(8) Clinical decision support
· §170.314(b)(3) Electronic prescribing
· §170.314(b)(4) Clinical information reconciliation
UCD PROCESS OUTLINE
The UCD process applied for EHR Under Test (EHRUT) is not an industry standard process, but rather a
custom process which is broadly based on the “ISO 13407 Human centred design processes for
interactive systems”.
ISO 13407 provides guidance on achieving quality in use by incorporating user centred design activities
throughout the life cycle of interactive computer-based systems. It describes user centred design as a
multi-disciplinary activity, which incorporates human factors and ergonomics knowledge and techniques
with the objective of enhancing effectiveness and productivity, improving human working conditions,
and counteracting the possible adverse effects of use on human health, safety and performance.
DESIGN ACTIVITIES
The User Centred Design process for EHR Under Test (EHRUT) are based on four design activities as
described in “ISO 13407 Human centred design processes for interactive systems”
The four activities initiated early into the product design, and user interface designs are:
- Understand and specify the context of use
- Specify the user and organizational requirements
- Produce design solutions
- Evaluate designs against requirements
The above activities were applied on an iterative nature for every interactive user interface element.
Confidential Page 7 of 28
UCD PROCESS CHART
The UCD process and the interdependence of the UCD activities applied on an iterative model on the
EHR Under Test (EHRUT).
Confidential Page 8 of 28
STUDY METHOD
PARTICIPANTS
Participants had a mix of backgrounds and demographic characteristics to facilitate the usability study.
The following is a table of participants by characteristics, including demographics, professional
experience, computing experience and user needs for assistive technology. Participant names were
replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.
Sl
No. Participant ID Gender Age Education
Occupation
/ Role
Professional
Experience
Computer
Experience
Product
Experience
Assistive
Technology
Needs
1 EZUSG20131501 F 23-39 College Medical
Assistant 2 Years Excellent Average None
2 EZUSG20131502 F 40-59 College Nurse 2 Years Excellent Average None
3 EZUSG20131503 M 40-59 MD Physician 10 Years Good Good None
4 EZUSG20131504 F 40-59 College LPN 15 Years Excellent Good None
5 EZUSG20131505 M 23-39 MD Physician 12 Years Good Average None
6 EZUSG20131506 M 40-59 MD/Phd Physician 27 Years Excellent Good None
A total number of Six participants were recruited (total number of participants recruited), and
participated in the usability tests on 12-16-2013 and 12-17-2013. Participants were scheduled for one on
one session(s) with a time of 60 minutes.
Usability tests were conducted on site at participant offices situated in Chicago, IL and Calumet City, IL.
Confidential Page 9 of 28
STUDY DESIGN
Overall, the objective of this test was to uncover areas where the application performed well – that is,
effectively, efficiently, and with satisfaction – and areas where the application failed to meet the needs
of the participants. The data from this test may serve as a baseline for future tests with an updated
version of the same EHR and/or comparison with other EHRs provided the same tasks are used. In short,
this testing serves as both a means to record or benchmark current usability, but also to identify areas
where improvements must be made.
The study is an adoption of the process as defined in NIST IR 7742. The methodology has been adjusted
to accommodate usability requirements of the ezEMRx EHR and a mechanism to drive user feedback
into product development lifecycles. The participants are engaged within the realm of the clinic to
simulate realistic environments; however the same is controlled with the use of a provided laptop while
under one-on-one study. The task elements and results are documented on paper while the study is in
progress. This methodology injects directed user feedback on usability into product development life
cycle.
During the usability test, the system was evaluated for effectiveness, efficiency and satisfaction as
defined by measures collected and analyzed for each participant:
Number of tasks successfully completed within the allotted time without assistance
Time to complete the tasks
Number and types of errors
Path deviations
Participant’s verbalizations (comments)
Participant’s satisfaction ratings of the system
Additional information about the various measures can be found in Section on Usability Metrics.
Confidential Page 10 of 28
DATA SCORING
The following table describes details how tasks were scored, errors evaluated, and the time data
analyzed.
Measures Rationale and Scoring
Effectiveness:
Task Success
A task was counted as a “Success” if the participant was able to achieve the correct outcome, without assistance,
within the time allotted on a per task basis.
The results are provided as a percentage. Task times were recorded for successes. Observed task times divided by
the optimal time for the task is a measure of optimal efficiency.
Optimal task performance time, as benchmarked by expert performance under realistic conditions, is recorded when
constructing tasks.
Effectiveness:
Task Failures
If the participant abandoned the task, did not reach the correct answer or performed it incorrectly, or reached the
end of the allotted time before successful completion, the task was counted as “Failures.” No task times were taken
for errors.
Not all deviations would be counted as errors. This is expressed as the mean number of failed tasks per participant.
Efficiency:
Task Deviations
The participant’s path (i.e., steps) through the application was reviewed. Deviations occur if the participant, for
example, went to a wrong screen, clicked on an incorrect menu item, followed an incorrect link, or interacted
incorrectly with an on-screen control. This path was compared to the optimal path. The number of steps in the
observed path is divided by the number of optimal steps to provide a ratio of path deviation.
Efficiency:
Task Time
Each task was timed. Only task times for tasks that were successfully completed were included in the average task
time analysis. Average time per task was calculated for each task.
Satisfaction:
Task Rating
Participant’s subjective impression of the ease of use of the application was measured by administering both a
simple post-task question as well as a post-session questionnaire. After each task, the participant was asked to rate
“Overall, this task was:” on a scale of 1 (Very Easy) to 5 (Very Difficult). These data are averaged across participants.
To measure participants’ confidence in and likeability of the EHRUT overall, the testing team administered the
System Usability Scale (SUS) post-test questionnaire. Questions included, “I think I would like to use this system
frequently,” “I thought the system was easy to use,” and “I would imagine that most people would learn to use this
system very quickly.” See full System Usability Score questionnaire in Appendix.
Confidential Page 11 of 28
TASKS
A number of tasks were constructed that would be realistic and representative of the kinds of activities
in a clinic, user might do with this EHR, including:
Computerized provider order entry
Drug-drug, drug-allergy interaction checks
Medication list
Medication allergy list
Clinical decision support
Electronic prescribing
Clinical information reconciliation
Tasks were selected based on 170.314(g)(3) Safety-enhanced design criteria as part of the ONC HIT
Certification Program.
TEST ENVIRONMENT
The EHRUT would be typically be used in a healthcare office or clinic.
In this instance, the testing was conducted at 3 different clinic locations situated in Chicago-Illinois,
Welch-Minnesota and Highland-Indiana. Apart from standard office facilities, each of the clinic locations
(test locations) comprised of comfortable and ergonomic work spaces, and a business class internet
connection with a minimum bandwidth of 25 MBPS for downloads and 10 MBPs for uploads.
For testing, the participant used a laptop computer comprising of a 15.6 inch display, Intel core i3
processor with 4GB memory, and configured with a 32 bit Microsoft Windows 7 as the operating
system. During the test, the screen resolution was set to 1152x768 with color set to “True Color (32
bit)”. The participants used a mouse and keyboard when interacting with the EHRUT. The participants
were instructed not change any of the default system settings.
The application (EHRUT) itself was setup by the vendor using a training database. The application was
setup to be accessible by desktop/laptop computers over a WAN connection using a browser and an
URL. Technically, the system performance (i.e., response time) was representative to what users would
experience with a production implementation.
During the test, the participants used the laptop configured with the Microsoft Windows 7 platform (as
stated above), and Mozilla Firefox ESR Version 17 to access the EHRUT that was setup previously with a
training database on a WAN connection.
Confidential Page 12 of 28
TEST FORMS AND TOOLS
During the usability test, the documents used were:
Informed Consent
Initial Questionnaire
Participant Instructions
SUS Questionnaire
Incentive Receipt & Acknowledgement
Examples of these documents can be found in Appendices respectively.
The participant’s interaction with the EHRUT was captured and recorded within the EHR.
USABILITY METRICS
According to the NIST Guide to the Processes Approach for Improving the Usability of Electronic Health
Records, EHRs should support a process that provides a high level of usability for all users. The goal is for
users to interact with the system effectively, efficiently, and with an acceptable level of satisfaction. To
this end, metrics for effectiveness, efficiency and user satisfaction were captured during the usability
testing.
The goals of the test were to assess:
Effectiveness of EHRUT by measuring participant success rates and errors
Efficiency of EHRUT by measuring the average task time and path deviations
Satisfaction with EHRUT by measuring ease of use ratings
Confidential Page 13 of 28
SUMMARY OF TEST RESULTS
PARTICIPANTS
A total of six participants were tested on the EHRUT(s). Participants in the test were physicians, nurses
and medical assistants. Participants were compensated $150 (physicians) and $100 (nurse, medical
assistant) for their time. In addition, participants had no direct connection to the development of the
EHRUT(s). Participants are actual end users and were familiar with the use of the EHRUT. Participants
were given the same orientation and level of training.
For the test purposes, end-user characteristics have been identified and translated into this document.
Participants had a mix of backgrounds and demographic characteristics to facilitate the usability study.
The following is a table of participants by characteristics, including demographics, professional
experience, computing experience and user needs for assistive technology. Participant names were
replaced with Participant IDs so that an individual’s data cannot be tied back to individual identities.
Sl
No. Participant ID Gender Age Education
Occupation
/ Role
Professional
Experience
Computer
Experience
Product
Experience
Assistive
Technology
Needs
1 EZUSG20131501 F 23-39 College Medical
Assistant 2 Years Excellent Average None
2 EZUSG20131502 F 40-59 College Nurse 2 Years Excellent Average None
3 EZUSG20131503 M 40-59 MD Physician 10 Years Good Good None
4 EZUSG20131504 F 40-59 College LPN 15 Years Excellent Good None
5 EZUSG20131505 M 23-39 MD Physician 12 Years Good Average None
6 EZUSG20131506 M 40-59 MD/Phd Physician 27 Years Excellent Good None
A total number of Six participants were recruited (total number of participants recruited), and
participated in the usability tests on 12-16-2013 and 12-17-2013. Participants were scheduled for one on
one session(s) with a time of 60 minutes.
Usability tests were conducted on site at participant offices situated in Chicago, IL and Calumet City, IL.
Confidential Page 14 of 28
DATA ANALYSIS AND REPORTING
The results of the usability test were calculated according to the methods specified in the Usability
Metrics section above.
The usability testing results for the EHRUT are detailed below. The results should be seen in light of the
objectives and goals outlined in Section Study Design.
Following is a summary of the performance and rating data collected on the EHRUT.
Measure
Task
N Task Success Path Deviation Task Time Errors Task Ratings
1=Very Easy
# Mean (SD)
Deviations
(Observed /
Optimal)
Mean (SD)
Secs Mean (SD) Mean (SD)
Computerized provider
order entry 6 50% 1 70 50% 2
Drug-drug, drug-allergy
interaction checks 6 67% 1 140 33% 2
Medication list 6 100% 1 63 0 1
Medication allergy list 6 100% 1 46 0 1
Clinical decision support 6 67% 1.3 105 33% 3
Electronic prescribing 6 100% 1 108 0 2
Clinical information
reconciliation 6 67% 1 90 33% 3
The results from the SUS (System Usability Scale) scored the subjective satisfaction with the system
based on performance with these tasks to be: 83%. Broadly interpreted, scores under 60 represent
systems with poor usability; scores over 80 would be considered above average.
Confidential Page 15 of 28
DISCUSSION OF THE FINDINGS
Discussions with the participants ranged in findings from slowness and usability in areas of CPOE, CDS
and medical reconciliation. Deviations were noted as some participants could not follow directions as
outlined.
It was noted that the slowness related to Drug-to-Drug interactions was due to access to staging systems
as compared to production systems.
Functional concepts of medical reconciliation and CPOE seemed to be challenging and discussions
indicated mandatory requirements of user training and education of requirements.
The presentation of CDS alerts invoked positive and negative responses from the participants. There
were indications of user looking to shut of such alerts as seemed intrusive to regular workings. On an
alternate note, some participants found the CDS alerts to be apt towards clinical decision assistance.
The overall discussions did present a positive mindset from the participants.
EFFECTIVENESS
The analysis indicated the most participants were successful in completing the tasks as presented.
Deviations were noticed on participants wherein the roles restricted functional areas (For, example,
CPOE, CDS functions were more relevant to Physicians than Nurses and Medical Assistants)
EFFICIENCY
The analysis indicated the physicians were able perform all functions successfully. This portrayed
accurate workings in realistic production environments. There were no deviations noted during the
physician based studies. Observations of participants with roles such as Nurses and Medical Assistants
did show deviations, however this is considered as exclusions since tasks such as CPOE and CDS were
not realistic events in daily working and hence the participants could not relate to the functional aspects
of such tasks.
SATISFACTION
To measure participants’ confidence in and likeability of the EHRUT overall, the testing team
administered the System Usability Scale (SUS) post-test questionnaire. The satisfaction index presented
a very positive outlook on usability. The responses have been aggregated and presented in the table
below.
Questions included, “I think I would like to use this system frequently,” “I thought the system was easy
to use,” and “I would imagine that most people would learn to use this system very quickly.” The
responses indicated very positive user usability responses.
Confidential Page 16 of 28
Aggregate SUS index -- 1 (Strongly Disagree) to 5 (Strongly Agree)
Index Parameter Index
I think that I would like to use this system frequently 4
I found the system unnecessarily complex 3
I thought the system was easy to use 4
I think that I would need the support of a technical person to be able to use this system 2
I thought there was too much inconsistency in this system 2
I would imagine that most people would learn to use this system very quickly 4
I found the system very cumbersome to use 3
I felt very confident using the system 4
I needed to learn a lot of things before I could get going with this system 3
MAJOR FINDINGS
The major finding based on the study indicated slowness of interaction checks by all participants. This
has however been discounted as this is related to slow response times from third party staging systems.
It was further determined that, slow response times are not present in production environments.
AREAS FOR IMPROVEMENT
Observations and discussions did present several areas of improvement. The highlights are:
Better alert responses to actions on the user interface
Training for users based on roles
More intuitive error descriptors
Improved directions for study
Confidential Page 17 of 28
APPENDIX
Sample - Initial Questionnaire
ezEMRx are recruiting individuals to participate in a usability study for an electronic health
record. We would like to ask you a few questions and if would like to participate. This should
only take a few minutes of your time. This is strictly for research purposes. If you are interested
and qualify for the study, you will be paid to participate.
1. Have you participated in a focus group or usability test in the past 6 months?
2. Do you, or does anyone in your home, work in marketing research, usability research, web design […etc.]?
3. Do you, or does anyone in your home, have a commercial or research interest in an electronic health record software or consulting company? [
4. Which of the following best describes your age? [23 to 39; 40 to 59; 60 - to 74; 75 and older]
5. Which of the following best describes your race or ethnic group? [e.g., Caucasian, Asian, Black/African-American, Latino/a or Hispanic, etc.]
6. Do you require any assistive technologies to use a computer? [if so, please describe]
Professional Demographics
What is your current position and title? (Must be healthcare provider)
RN: Specialty ________________
Physician: Specialty ________________
Resident: Specialty ________________
Administrative Staff
How long have you held this position?
Describe your work location (or affiliation) and environment?
Which of the following describes your highest level of education? [e.g., high school graduate/GED, some
college, college graduate (RN, BSN), postgraduate (MD/PhD), other (explain)]
Confidential Page 18 of 28
Computer Expertise
About how many hours per week do you spend on the computer? [ e.g., 0 to 10, 11 to 25, 26+ hours per
week]
What computer platform do you usually use? [e.g., Mac, Windows, etc.]
What Internet browser(s) do you usually use? [e.g., Firefox, IE, Chrome, etc.]
In the last month, how often have you used an electronic health record?
How many years have you used an electronic health record?
How many EHRs do you use or are you familiar with?
How does your work environment patient records?
On paper
Some paper, some electronic
All electronic
May I get your contact information?
Name of participant:___________________________________________
Address: ____________________________________________________
____________________________________________________
City, State, Zip: _______________________________________________
Daytime phone number: ________________________________________
Evening phone number: ________________________________________
Alternate [cell] phone number: __________________________________
Email address: _______________________________________________
Confidential Page 19 of 28
Sample – Informed Consent
Informed Consent
ezEMRx Inc would like to thank you for participating in this study. The purpose of this study is to
evaluate an electronic health records system. If you decide to participate, you will be asked to perform
several tasks using the prototype and give your feedback. The study will last about 60 minutes. At the
conclusion of the test, you will be compensated for your time.
Agreement
I understand and agree that as a voluntary participant in the present study conducted by ezEMRx Inc. I
am free to withdraw consent or discontinue participation at any time.
I understand and agree that the purpose of this study is to make software applications more useful and
usable in the future.
I understand and agree that the data collected from this study may be shared with outside of ezEMRx
Inc and ezEMRx Inc client. I understand and agree that data confidentiality is assured, because only de-
identified data – i.e., identification numbers not names – will be used in analysis and reporting of the
results.
I agree to immediately raise any concerns or areas of discomfort with the study administrator. I
understand that I can leave at any time.
Please check one of the following:
YES, I have read the above statement and agree to be a participant.
NO, I choose not to participate in this study.
Signature: _____________________________________ Date: ____________________
Confidential Page 20 of 28
Sample – Incentive receipt and acknowledgement
INCENTIVE RECEIPT AND ACKNOWLEDGMENT FORM
Company: ezEMRx Inc,
25W215 Lake Street
Roselle, IL 60125
Printed Name: ___________________________________________________________
Address: _______________________________________________________________
_______________________________________________________________
Signature: _________________________________ Date: _______________
____________________________________________________________________________
Usability Researcher: _____________________________________
Signature of Usability Researcher: ___________________________
Date: _______________
Witness: _______________________________________________
Witness Signature: _______________________________________
Date: _______________
Confidential Page 21 of 28
Sample – SUS Questionnaire
SYSTEM USABILITY SCALE QUESTIONNAIRE
Strongly Strongly
Disagree Agree
I think that I would like to use this system frequently
1 2 3 4 5
I found the system unnecessarily
complex
1 2 3 4 5
I thought the system was easy
to use
1 2 3 4 5
I think that I would need the
support of a technical person to
be able to use this system
1 2 3 4 5
I thought there was too much
inconsistency in this system
1 2 3 4 5
I would imagine that most people
would learn to use this system
very quickly
1 2 3 4 5
I found the system very
cumbersome to use
1 2 3 4 5
I felt very confident using the
system
1 2 3 4 5
I needed to learn a lot of things before
I could get going with this system
1 2 3 4 5
Confidential Page 22 of 28
Sample – Tasks
Task 1: Medication list
Please Click on the Today's Task Tab as starting point of the task.
On the current visit, patient “Jennifer Doe” indicates that there is change in his medication history. Locate
the medication tab and and complete the following task.
1) View active medication 2) Change the existing medication to “Completed” or “Discontinue” 3) Add new medication 4) Click on “Medication History” to view the medication history.
Success:
Easily completed
Completed with difficulty or help :: Describe below
_______________________________________________________________________
_______________________________________________________________________
Not completed
Task Time: __________ Seconds
Optimal Path : Today's Task -> Click on the patient “ Jennifer Doe” Encounter Task -> Click on
Medications -> Click on medication name and change status to “Discontinue” and Save -> Add new
medication and click on Add -> Click on “History” to view the patient medication history.
Correct
Minor Deviations / Cycles :: Describe below
______________________________________________________________________
______________________________________________________________________
Major Deviations :: Describe below
______________________________________________________________________
______________________________________________________________________
Observed Errors and Verbalizations:
Comments: ______________________________________________________________________________
________________________________________________________________________________________
Rating:
Overall, this task was: ______
Participant written scale: “Very Easy” (1) to “Very Difficult” (5)
Administrator Comments
Confidential Page 23 of 28
Task 2: Medication allergy list
Please Click on the Today's Task Tab as starting point of the task.
On the current visit, patient “ Jennifer Maker ” indicates that there is change in his allergy list. Locate the
allergy tab and complete the following task
1) View patient active allergy list 2) Change status of one allergy to “Inactive” 3) Add New Allergy 4) Verify patient allergy history
Success:
Easily completed
Completed with difficulty or help :: Describe below
_______________________________________________________________________
_______________________________________________________________________
Not completed
Task Time: __________ Seconds
Optimal Path : Today's Task -> Click on the patient “ Jennifer Maker” Encounter Task -> Click on
Allergy -> Click on existing Allergy and change status to “Inactive” -> Select new allergy from auto-lookup
field and add “Severity”, “Event”, “Reaction” click on Save -> Click on the “History” to view patient
allergies.
Correct
Minor Deviations / Cycles :: Describe below
______________________________________________________________________
______________________________________________________________________
Major Deviations :: Describe below
______________________________________________________________________
______________________________________________________________________
Observed Errors and Verbalizations:
Comments: ______________________________________________________________________________
________________________________________________________________________________________
Rating:
Overall, this task was: ______
Participant written scale: “Very Easy” (1) to “Very Difficult” (5)
Administrator Comments
Confidential Page 24 of 28
Task 3: Interaction Check – [D-to-D and D-to-A]
Please Click on the Today's Task Tab as starting point of the task. Ensure that the patient has following
drug-drug and a drug-food allergy to the drug chosen. This will put force the participant to find other drugs
and use other elements of the application.
Patient Allergy :
Patient Medication :
After examining the patient “ Jennifer Miller”, you have decided to put this patient on the medication ->
“Lipitor”. Check for any interaction and place an order for this medication
Success:
Easily completed
Completed with difficulty or help :: Describe below
_______________________________________________________________________
_______________________________________________________________________
Not completed
Task Time: __________ Seconds
Optimal Path : Today's Task -> Click on the patient “ Jennifer Miller” Encounter Task -> Click on
Prescription -> Click on “Drug Search” button -> Search and select the required drug -> Add “Duration”,
“Quantity”, “Sig”, “Refill” Click on Add -> Click on “Prescribe” to view D-to-D and D-to-A interaction.
Correct
Minor Deviations / Cycles :: Describe below
______________________________________________________________________
______________________________________________________________________
Major Deviations :: Describe below
______________________________________________________________________
______________________________________________________________________
Observed Errors and Verbalizations:
Comments: ______________________________________________________________________________
________________________________________________________________________________________
Rating:
Overall, this task was: ______
Participant written scale: “Very Easy” (1) to “Very Difficult” (5)
Administrator Comments
Confidential Page 25 of 28
Task 4: Prescribe Medication
Please Click on the Today's Task Tab as starting point of the task.
After examining the patient “ Jennifer Major”, you have decided to put this patient on the medication ->
“Lipitor”. Check for any interaction, Choose the pharmacy and ePrescribe for this medication.
Success:
Easily completed
Completed with difficulty or help :: Describe below
_______________________________________________________________________
_______________________________________________________________________
Not completed
Task Time: __________ Seconds
Optimal Path : Today's Task -> Click on the patient “ Jennifer Major” Encounter Task -> Click on
Prescription -> Click on “Drug Search” button -> Search and select the required drug -> Add “Duration”,
“Quantity”, “Sig”, “Refill” Click on Add -> Click on “Prescribe” to view interaction -> Select Pharmacy ->
Click on “Send” to ePrescribe.
Correct
Minor Deviations / Cycles :: Describe below
______________________________________________________________________
______________________________________________________________________
Major Deviations :: Describe below
______________________________________________________________________
______________________________________________________________________
Observed Errors and Verbalizations:
Comments: ______________________________________________________________________________
________________________________________________________________________________________
Rating:
Overall, this task was: ______
Participant written scale: “Very Easy” (1) to “Very Difficult” (5)
Administrator Comments
Confidential Page 26 of 28
Task 5: CPOE
Please Click on the Today's Task Tab as starting point of the task.
After examining the patient “ Jennifer Peter”, you have decided to create an orders for lab, radiology and
medication for this this patient. Check for any D-to-D interaction and complete the orders.
1) Select the existing order-set for this patient. 2) Add or remove any one investigation based on the patient requirement 3) Select the required action and execute the order.
Success:
Easily completed
Completed with difficulty or help :: Describe below
_______________________________________________________________________
_______________________________________________________________________
Not completed
Task Time: __________ Seconds
Optimal Path: Today's Task -> Click on the patient “ Jennifer Peter” Encounter Task -> Click on Order
set -> Select the existing order-set –> Add new lab items, -> Remove lab items -> Select action ->
execute the orders.
Correct
Minor Deviations / Cycles :: Describe below
______________________________________________________________________
______________________________________________________________________
Major Deviations :: Describe below
______________________________________________________________________
______________________________________________________________________
Observed Errors and Verbalizations:
Comments: ______________________________________________________________________________
________________________________________________________________________________________
Rating:
Overall, this task was: ______
Participant written scale: “Very Easy” (1) to “Very Difficult” (5)
Administrator Comments
Confidential Page 27 of 28
Task 6 : Clinical Information Reconciliation
Please Click on the Today's Task Tab as starting point of the task.
Patient Jennifer Carter was existing patient of yours and had recently underwent surgery and is on
multiple medication and you to reconcile the patient clinical information.
a) Check with patient and add the medication in the list provided
b) Verify the medication against his current medications
c) Add or remove the medication as required.
d) Complete reconcile
Success:
Easily completed
Completed with difficulty or help :: Describe below
_______________________________________________________________________
_______________________________________________________________________
Not completed
Task Time: __________ Seconds
Optimal Path: Today's Task -> Click on the patient “ Jennifer Carter” Encounter Task -> Click on
medication -> Select medication reconciliation –> Add new medications -> Remove medications ->
complete reconcile.
Correct
Minor Deviations / Cycles :: Describe below
______________________________________________________________________
______________________________________________________________________
Major Deviations :: Describe below
______________________________________________________________________
______________________________________________________________________
Observed Errors and Verbalizations:
Comments: ______________________________________________________________________________
________________________________________________________________________________________
Rating:
Overall, this task was: ______
Participant written scale: “Very Easy” (1) to “Very Difficult” (5)
Administrator Comments
Confidential Page 28 of 28
Task 7: Clinical Decision support
Please Click on the Today's Task Tab as starting point of the task.
Patient John Smith was existing patient of yours and has come in for the consultation and is currently on
multiple medications / has active problems and has pending results for verification.
1) Click on the encounter task 2) System will trigger an automated clinical decision support based on the patient conditions /
medication / labs 3) Click on the CDS alert and complete the required task.
Success:
Easily completed
Completed with difficulty or help :: Describe below
_______________________________________________________________________
_______________________________________________________________________
Not completed
Task Time: __________ Seconds
Optimal Path: Today's Task -> Click on the patient “Jennifer Smith” Encounter Task -> System will bring
up the CDS alert -> Complete required task (complete the orders / printing out education materials etc) -
>
Correct
Minor Deviations / Cycles :: Describe below
______________________________________________________________________
______________________________________________________________________
Major Deviations :: Describe below
______________________________________________________________________
______________________________________________________________________
Observed Errors and Verbalizations:
Comments: ______________________________________________________________________________
________________________________________________________________________________________
Rating:
Overall, this task was: ______
Participant written scale: “Very Easy” (1) to “Very Difficult” (5)
Administrator Comments
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 12 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
Appendix C: Quality Management System
���������������� ���������������������������������
�������� �!���������"��#$� %���&�
�������������������������� �������������������������������
������������������� ���������
����������� ��� ��������
������������ ��� ���������
���������������� �����
�������������������������
����������� ��������!�����
"�����#��"$����������%��
���� �������������!��
�� ���������������
���������������& ����������
�'�����������������"��������#�������� ���"(���)����*�+,-�.���"(�
)-./0������$��"�������1�� � � � � �
�2������������3%��4!��5�6���"��
������"�5���������
7���������"����������� �
��������������������������5����
� ��� ����"���� ���8�
�(�����"�������
��� ��� ����"��������
9��������������������������������"���� ����������%���������1�
� � � � � �
�����2�� ���: ���
����������� �����������
����%���������!�������������%����%����������������%������������������ �������������������
�������"�!�������:�����
2��%������ ��������������� � � � � �
9���� ;�������,<��,�).�
�
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 13 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
Appendix D: Privacy and Security
���������������� ��� �� � ��������������������
���������� ��� �����!����"� ��#��$�
�
�
������������������� ���������
������������ � � � � � ��������� �
������������� � � � � � ������������
���������������� � � � � � � ���
�������������������
�����������������������
�������������� ��!
����������
����������������������
�����������������������
�����"�#$�
� ������������������������� �
����������������������� �
�� �����������������������������������������������������
��������������������������
�
�����"�#���������������$�
� ������������������������� �
��������������������������������
�� ����������������������������������������� ��������������
������������������������������������ � � � � ���
��������������������������������������������������������������������
�������� �
�
���� ������������������������� ������ �%����%��� ����������������
�����������������$�
� ������������������������� �
�������������������������������������������
�� ������������������������������������������� ��������������
������������������������������������ � � � � ���
����������������������������������������������������������������
���������������������������������������������������������� ��
�
���������������������������������������������������������������������
��������������������������������������������������������������������
����������������������������������������� ���� � � � � ��
�
������������������������������������������������������������������
������������������������������������������������������������������
������������������������������������������������������������������������
������������������������������������������������������������������
�������������������������������� ���� � � � � ��
�
���������������� ��� �� � ��������������������
���������� ��� �����!����"� ��#����
�
������������������������������������������������������������������
������������������������������������������������������������
�������������������������� ���� � � � � ��
�����������������!����
���������� �����
������������������������
������������������������
��������������� � ������������
���������������������������
����������������� �
�
����������������������
�����������������������
�����������������������������������������������������������
���������������� �
�� ������������������������������������������������������
���������������������������������������������������������
������������������������������� ���� � � � � ��
�
�����������������������������������������������������������
������������ �
�� ���������������������������������������������������������������
� � � � � �
�� ������������������������������������������������������������
������������������������������������������������������
����������� � � � � ��
�
��������������������������������������������������������������������
������������������������������������������������������������������
��������������������������������� ��
�� ������������������������������ � � � � �������������������
�������������������������������
�����������&������#�����
����������������������
�����������������������
���������������������������������������������������������������������
� � � � � �����
����������� �'�� ������
���������������������������������������������������������������������������� �
����������������������������������������������� � � � � � �
����� � � � � � �����������������
�
Template Version 1
Test Results Summary for 2014 Edition EHR Certification 14-2469-R-0004-PRA V1.1 February 28, 2016
Page 14 of 14 ©InfoGard 2016. May be reproduced only in its original entirety, without revision
Test Results Summary Document History
Version Description of Change Date V1.0 Initial release May 13, 2014
V1.1 Updated Safety-Enhanced Design report February 28, 2016
END OF DOCUMENT