certification test plan - eac.gov...removed open primary and ranked choice voting from supported...

WYLE LABORATORIES, INC. Huntsville Facilities

CERTIFICATION TEST PLAN

Prepared for:

Manufacturer Name Dominion Voting Systems

Manufacturer System Democracy Suite Version 4.0

EAC Application No. DVS1001

Manufacturer

Address

215 Spadina Avenue, Suite 200

Toronto, Ontario, Canada


REPORT NO. Test Plan T57381.01-01, Rev. E

DATE March 23, 2012

REV DATE PAGE OR PARAGRAPH

AFFECTED DESCRIPTION OF CHANGES

--- 4-19-11 Entire Document Original Release

A 9-19-11 Entire Document Complete Document Edit Based on EAC Comments

A 9-26-11 Section 1.0 Deleted second sentence due to redundancy.

A 9-26-11 Section 1.3.1.1 Corrected typo (changed “in” to “is”).

A 9-26-11 Section 1.4.1 Removed “secure” from the second sentence of the

second paragraph of the ICE description.

A 9-26-11 Section 1.4.1 Removed “safely” from the second sentence of the first

paragraph of the ICP description.

A 9-26-11 Section 1.4.1 Added the following sentence to the second paragraph of

the ICP description: There is no paper ballot or record

produced when the ATI is utilized for voting.

A 9-26-11 Section 1.4.3 Provided definition for UPS and LAN and expanded

description for EMS Express hardware configuration.

A 9-26-11 Section 1.4.7 Changed last sentence to read as follows: This testing is

out of scope for this test campaign.

A 9-26-11 Section 2.0 Added Democracy Suite version 4.0 voting system to

TDP description.

A 9-26-11 Section 2.2 and 2.2.1 Added reference to EAC RFI for summative usability

reporting and corrected tense.

A 9-26-11 Section 3.2 Added “COTS equipment” to clarify manufacturer.

A 9-26-11 Section 4.1.2 Added “DRE” to second sentence of first paragraph.

A 9-26-11 Section 4.4.3 Second paragraph, second sentence, deleted “including

entries for invalid data”.



DATE March 23, 2012



A 9-26-11 Section 4.4.5 Deleted second sentence of second paragraph.

A 9-26-11 Section 4.5 Deleted first sentence in first paragraph and moved to

previous section.

A 9-26-11 Section 4.5 Change “…they deem as valid” to “…that violate the

standard” in the third sentence of the last paragraph.

A 9-26-11 Section 4.6 Restructured first sentence of first paragraph.

B 10-25-11 Section 1.4.5 Removed Open primary and ranked Choice Voting from

supported functionality.

B 10-25-11 Section 6.2 Removed description for PRIM-02.

B 10-25-11 Section 1.3.1.4 Reworded section to provide additional information.

B 10-25-11 Section 1.4.3 Removed “(New York State)” from Table 1-2 and 1-3

titles and corrected title for Table 1-5.

B 10-25-11 Section 1.4.3, Tables 1-3

and 1-5 Changed column titles.

B 10-25-11 Section 1.4.4

Added Note: All stated languages will be verified to be

supported; however, only English and Spanish ballots

will be cast during functional testing.

B 10-25-11 Section 2.1

Added the following to the first sentence: “…as a

complete system…” and additional wording to clarify

testing.

B 10-25-11 Section 2.2 Added additional information on reuse of previous

testing.

B 10-25-11 Section 2.3 and 2.3.1 Combined section and deleted Section 2.3.1.



DATE March 23, 2012



B 10-25-11 Section 4.1 Added “and/or evaluated” in Section 4 description.

B 10-25-11 Section 4.1.2 Reworded entire section to provide clarification on

testing.

B 10-25-11 Section 4.2 Reworded entire section to provide clarification on

testing.

B 10-25-11 Section 4.4.2 Deleted section and incorporated information into

previous section.

B 10-25-11 Section 4.4.4 Reworded first paragraph.

B 10-25-11 Section 4.5 Deleted last sentence of first paragraph.

B 10-25-11 Section 4.8 Combined last two sentences and deleted reference to

Appendix F.

B 10-25-11 Section 6.3.3 Provided additional information on Logic and Accuracy

Test.

C 11-30-11 Section 2.1 Added additional information for re-use of prior VSTL

testing.

C 11-30-11 Section 2.2 Added additional information for re-use of prior non-

VSTL testing and usability testing.

C 11-30-11 Section 2.2.1 Deleted section and incorporated information into

previous sections.

C 11-30-11 Section 4.1.2 Reworded section for clarification of not applicable

requirements.

C 11-30-11 Section 4.2 and 4.4.1

Reworded paragraphs to provide clarification on

components tested, previous state test effort, and third-

party testing, and included Temperature Power test.

C 11-30-11 Section 6.2 Updated election descriptions.



DATE March 23, 2012



C 11-30-11 Section 6.3.3 Reworded Volume/Stress/Reliability and Logic and

Accuracy test descriptions.

C 11-30-11 Appendix A Added Dominion Voting Systems Implementation

Statement.

C 11-30-11 Appendix B Added project schedule.

D 12-5-11 Sections 1.3.1, 1.3.1.1,

1.3.1.5 Changed project schedule reference to Appendix B.

D 12-5-11 Sections 1.4.6 and 4.1.1 Added VRT reference.

D 12-5-11 Section 4.2 Corrected typo from “severs” to “servers”.

D 12-5-11 Section 6.3.3 Changed formatting.

D 12-5-11 Section 6.3.3 Added paragraph on audit logs.

E 3-23-12 Entire Plan Updated to “As Run” Test Plan for Final Report.

Page No. TOC-1 of 2

Certification Test Plan T57381.01-01, Rev. E


TABLE OF CONTENTS

1. INTRODUCTION ............................................................................................................................... 1

1.1 References ................................................................................................................................ 1

1.2 Terms and Abbreviations ......................................................................................................... 2

1.3 Testing Responsibilities ........................................................................................................... 3

1.3.1 Project Schedule ......................................................................................................... 3

1.3.1.1 Owner Assignments ................................................................................ 3

1.3.1.2 Test Case Development ........................................................................... 4

1.3.1.3 Test Procedures and Validation ............................................................... 4

1.3.1.4 Third-Party Tests ..................................................................................... 4

1.3.1.5 EAC and Manufacturer Dependencies .................................................... 4

1.4 Target of Evaluation Description ............................................................................................. 4

1.4.1 System Overview ........................................................................................................ 4

1.4.2 Block Diagram ............................................................................................................ 9

1.4.3 System Limits ........................................................................................................... 10

1.4.4 Supported Languages ................................................................................................ 12

1.4.5 Supported Functionality ............................................................................................ 13

1.4.6 VVSG ............................................................................................................... 13

1.4.7 Beyond VVSG .......................................................................................................... 13

2.0 PRE-CERTIFICATION TESTING AND ISSUES ....................................................................... 13

2.1 Evaluation of Prior VSTL Testing ......................................................................................... 14

2.2 Evaluation of Prior Non-VSTL Testing ................................................................................. 14

2.3 Known Field Issues ................................................................................................................ 14

3.0 MATERIALS REQUIRED FOR TESTING .................................................................................. 14

3.1 Software ................................................................................................................................. 15

3.2 Equipment .............................................................................................................................. 24

3.3 Test Support Materials ........................................................................................................... 28

3.4 Deliverable Materials ............................................................................................................. 28

4.0 TEST SPECIFICATIONS ............................................................................................................... 30

4.1 Requirements (Strategy of Evaluations) ................................................................................ 31

4.1.1 Mapping of Requirements to Equipment Type and Features ................................... 32

4.1.2 Rationale for ‘Not Applicable’ Requirements .......................................................... 32

4.2 Hardware Configuration and Design ..................................................................................... 33

4.3 Software System Functions .................................................................................................... 34

4.4 Test Case Design ................................................................................................................... 34

4.4.1 Hardware Qualitative Examination Design .............................................................. 34

4.4.1.1 Mapping of Requirements to Specific Interfaces .................................. 38

4.4.2 Software Module Test Case Design and Data .......................................................... 38

4.4.3 Software Functional Test Case Design and Data ...................................................... 39

Page No. TOC-2 of 2

Certification Test Plan T57381.01-01, Rev. E


TABLE OF CONTENTS (continued)

4.4.4 System-Level Test Case Design ............................................................................... 40

4.5 Security Functions ............................................................................................................... 41

4.6 TDP Evaluation ............................................................................................................... 41

4.7 Source Code Review .............................................................................................................. 43

4.8 QA and CM System Review .................................................................................................. 44

5.0 TEST DATA .................................................................................................................................. 44

5.1 Test Data Recording .............................................................................................................. 44

5.2 Test Data Criteria ................................................................................................................... 45

5.3 Test Data Reduction ............................................................................................................... 45

6.0 TEST PROCEDURE AND CONDITIONS .................................................................................... 45

6.1 Facility Requirements ............................................................................................................ 45

6.2 Test Set-Up ............................................................................................................................ 46

6.3 Test Sequence ........................................................................................................................ 51

6.3.1 Hardware Test Description ....................................................................................... 52

6.3.2 Software Test Description......................................................................................... 54

6.3.3 System Testing .......................................................................................................... 55

7.0 TEST OPERATIONS PROCEDURES .......................................................................................... 59

7.1 Proprietary Data ..................................................................................................................... 59

APPENDICES

APPENDIX A DOMINION VOTING SYSTEMS DEMOCRACY SUITE 4.0 VOTING SYSTEM

IMPLEMENTATION STATEMENT ................................................................................. A-1

APPENDIX B DOMINION VOTING SYSTEMS PROJECT SCHEDULE .............................................. B-1

Page No. 1 of 59

Certification Test Plan T57381.01-01, Rev. D


1.0 INTRODUCTION

The purpose of this National Certification Test Plan (Test Plan) is to document the procedures that Wyle

Laboratories, Inc., will follow to perform certification testing of the Dominion Voting Systems, Democracy

Suite 4.0 System, to the requirements set forth for voting systems in the U.S. Election Assistance

Commission (EAC) 2005 Voluntary Voting System Guidelines (EAC 2005 VVSG). Prior to submitting the

System for certification testing, Dominion Voting Systems submitted an application to the EAC for

certification of the Democracy Suite 4.0 System to the requirements of the EAC 2005 VVSG.

At test conclusion, the results of all testing performed as part of this test program will be submitted to the

EAC in the form of a final report.

1.1 References

The documents listed below were used in the development of the Test Plan and are utilized to perform

certification testing.

Election Assistance Commission 2005 Voluntary Voting System Guidelines, Volume I, Version 1.0, “Voting System Performance Guidelines”, and Volume II, Version 1.0, “National Certification

Testing Guidelines”, dated December 2005

Election Assistance Commission Testing and Certification Program Manual, Version 1.0, effective date January 1, 2007

Election Assistance Commission Voting System Test laboratory Program Manual, Version 1.0, effective date July 2008

National Voluntary Laboratory Accreditation Program NIST Handbook 150, 2006 Edition, “NVLAP Procedures and General Requirements (NIST Handbook 150)”, dated February 2006

National Voluntary Laboratory Accreditation Program NIST Handbook 150-22, 2008 Edition, “Voting System Testing (NIST Handbook 150-22)”, dated May 2008

United States 107th Congress Help America Vote Act (HAVA) of 2002 (Public Law 107-252), dated October 2002

Wyle Laboratories’ Test Guidelines Documents: EMI-001A, “Wyle Laboratories’ Test Guidelines for Performing Electromagnetic Interference (EMI) Testing”, and EMI-002A, “Test Procedure for

Testing and Documentation of Radiated and Conducted Emissions Performed on Commercial

Products”

Wyle Laboratories’ Quality Assurance Program Manual, Revision 4

ANSI/NCSL Z540-1, “Calibration Laboratories and Measuring and Test Equipment, General Requirements”

ISO 10012-1, “Quality Assurance Requirements for Measuring Equipment”

EAC Requests for Interpretation (listed on www.eac.gov)

EAC Notices of Clarification (listed on www.eac.gov)

EAC Quality Monitoring Program residing on:

http://www.eac.gov/testing_and_certification/quality_monitoring_program.aspx

A listing of the Democracy Suite 4.0 System Technical Data Package (TDP) Documents submitted for this

certification test effort is listed in Section 3.4: Deliverable Materials.

Page No. 2 of 59



1.0 INTRODUCTION (CONTINUED)

1.2 Terms and Abbreviations

This subsection defines all terms and abbreviations applicable to the development of this Test Plan.

Table 1-1 Terms and Abbreviations

Term Abbreviation Definition

Americans with Disabilities Act of

1990 (Amended 2008) ADA

ADA is a wide-ranging civil rights law that prohibits,

under certain circumstances, discrimination based on

disability.

EMS Audio Studio AS EMS application used to record audio files.

Audio Tactile Interface ATI Voter interface designed to not require visual reading of a

ballot.

Conformité Européenne (European

Conformity) CE ---

Configuration Management CM ---

Commercial Off the Shelf COTS Commercial, readily available hardware or software

Direct Record Electronic DRE ---

United States Election Assistance

Commission EAC

Commission created per the Help America Vote Act of

2002, assigned the responsibility for setting voting system

standards and providing for the voluntary testing and

certification of voting systems.

EMS Election Event Designer EED EMS application used for election definition functionality.

Election Management System EMS The Election Management System equivalent for the

Democracy Suite System.

Equipment Under Test EUT ---

Functional Configuration Audit FCA

Exhaustive verification of every system function and

combination of functions cited in the manufacturer’s

documentation.

Federal Communications

Commission FCC ---

Help America Vote Act HAVA Act created by United States Congress in 2002.

National Institute of Standards and

Technology NIST

Government organization created to promote U.S.

innovation and industrial competitiveness by advancing

measurement science, standards, and technology in ways

that enhances economic security and improves our quality

of life.

ImageCast Central ICC High-speed central ballot scan tabulator.

ImageCast Evolution ICE Precinct-level optical scanner, ballot marker, and tabulator

with audio voting.

ImageCast Precinct ICP Precinct-level optical scanner and tabulator with audio

voting capabilities.

Page No. 3 of 59




1.2 Terms and Abbreviations (continued)

Table 1-1 Terms and Abbreviations (continued)

Term Abbreviation Definition

Physical Configuration Audit PCA

Review by accredited test laboratory to compare

voting system components submitted for

certification testing to the manufacturer’s technical

documentation, and confirmation the documentation

meets national certification requirements. A trusted

build of the executable system is performed to

ensure the certified release is built from tested

components.

Quality Assurance QA ---

EMS Results, Tally and Reporting RTR EMS application used to integrate election results

and reporting.

System Under Test SUT ---

Test Case Procedure Specifications TCPS

Wyle-developed document that specifies test items,

input specifications, output specifications,

environmental needs, special procedural

requirements, inter-case dependencies, and all

validated test cases that will be executed during the

area under test.

Technical Data Package TDP

Manufacturer documentation related to the voting

system required to be submitted as a precondition of

certification testing.

Underwriters Laboratories Inc. UL ---

Uninterruptible Power Supply UPS ---

Voluntary Voting System Guidelines EAC 2005 VVSG Published by the EAC, the third iteration of national

level voting system standards.

Wyle Operating Procedure WoP Wyle Test Method or Test Procedure.

1.3 Testing Responsibilities

All core and non-core software and hardware certification testing will be conducted under the guidance of

Wyle Laboratories, Inc., by personnel verified by Wyle to be qualified to perform the testing.

1.3.1 Project Schedule

This information is contained in a Wyle-generated Microsoft Project schedule. This schedule is presented in

Appendix B “Dominion Voting Systems Project Schedule”. The dates on the schedule are not firm dates but

planned estimates presented for informational purposes.

1.3.1.1 Owner Assignments

This information is contained in a Wyle generated Microsoft Project schedule. This schedule is presented in

Appendix B “Dominion Voting Systems Project Schedule”.

Page No. 4 of 59




1.3 Testing Responsibilities (continued)

1.3.1 Project Schedule (continued)

1.3.1.2 Test Case Development

Wyle will utilize the “Wyle Baseline Test Cases” for the Functional Configuration Audit (FCA), Usability

and System Integration Tests. These will be augmented with specially designed test cases tailored to the

Dominion Voting Systems Democracy Suite 4.0. Wyle has designed specific election definitions for the

Operational Status Check and the Accuracy Tests. The “Baseline” functional test cases, “Baseline” usability

test cases, and the election definitions are being submitted as part of this test plan package.

Throughout the test campaign, Wyle will develop and submit to the EAC Test Case Procedure Specifications

(TCPS) for major areas of testing. The TCPS documents, the test items, input specifications, output

specifications, environmental needs, special procedural requirements, inter-case dependencies, and all

validated test cases that will be executed for a given test.

1.3.1.3 Test Procedure Development and Validation

Wyle will utilize the Wyle Operating Procedures (WoPs) during the duration of this test program. These

procedures are validated and are being submitted as part of the test plan package.

1.3.1.4 Third-Party Testing

Dominion Voting Systems also submitted five hardware test reports for the ICP unit. Wyle reviewed the

reports and performed a comparison between the ICP version tested in the provided reports and the ICP

version currently submitted for testing and concluded that a portion of the hardware testing for the ICP will

be recommended for reuse to satisfy requirements for this testing campaign. Wyle determined that the ICP

shall be subjected to the following hardware tests per the EAC 2005 VVSG: Electromagnetic Radiation,

Electromagnetic Susceptibility, and all non-operational environmental testing. A listing of reports reviewed,

and Wyle’s evaluation of these reports is contained in Section 4.4.1 of this document.

Additionally, Wyle will be utilizing 3rd

party testing to perform the product safety portion of the test

campaign. Third party testing will be witnessed by Wyle personnel at MET Labs.

1.3.1.5 EAC and Manufacturer Dependencies

This information is contained in a Wyle generated Microsoft Project schedule. This schedule is presented in

Appendix B “Dominion Voting Systems Project Schedule”.

1.4 Target of Evaluation Description

The following sections address the design methodology and product description of the Democracy Suite 4.0

System, as taken from the Dominion Voting Systems technical documentation.

Page No. 5 of 59




1.4.1 Target of Evaluation Description (continued)

1.4.1 System Overview

The Dominion Voting Systems Democracy Suite 4.0 System is a paper-based optical scan voting system.

The Democracy Suite 4.0 System consists of four major components: the Election Management System

(EMS), ImageCast Evolution (ICE) precinct scanner and ballot marking device, ICP precinct scanner with

audio ballot, and ImageCast Central (ICC) central count scanner.

Election Management System

The Dominion Voting Systems Democracy Suite 4.0 EMS consists of seven components running as either a

front-end/client application or as a back-end/server application. Below is a list and brief description of each.

Democracy Suite 4.0 EMS Election Event Designer client application - integrates election definition functionality and represents a main pre-voting phase end-user application.

Democracy Suite 4.0 EMS Results Tally and Reporting client application - integrates election results acquisition, validation, tabulation, reporting and publishing capabilities and represents a main post-

voting phase end-user application.

Democracy Suite 4.0 EMS Audio Studio client application - represents an end-user helper application used to record audio files for a given election project. As such, it is utilized during the

pre-voting phase of the election cycle.

Democracy Suite 4.0 EMS Data Center Manager client application - represents a system level configuration application used in EMS back-end data center configuration.

Democracy Suite 4.0 EMS Application Server server application - represents a server side application responsible for executing long running processes, such as rendering ballots, generating

audio files and election files, etc.

Democracy Suite 4.0 EMS Network Attached Storage (NAS) Server server application - represents a server side file repository for election project file based artifacts, such as ballots, audio files, reports,

log files, election files, etc.

Democracy Suite 4.0 EMS Database Server server application - represents a server side RDBMS repository of the election project database which holds all the election project data, including pre-

voting and post-voting data.

Precinct Ballot Tabulator: ImageCast Evolution (ICE)

The ICE Ballot Counter device is a precinct-level, optical scan, ballot counter (tabulator) designed to perform

six major functions:

Ballot scanning

Tabulation

Ballot review

Second chance voting

Accessible voting

Ballot marking

Page No. 6 of 59




1.4 Target of Evaluation Description (continued)

1.4.1 System Overview (continued)

The Dominion Democracy Suite ImageCast Evolution system employs a precinct-level optical scan ballot

counter (tabulator) in conjunction with an external ballot box. This tabulator is designed to mark and/or scan

paper ballots, interpret voting marks, communicate these interpretations back to the voter (either visually

through the integrated LCD display or audibly via integrated headphones), and upon the voter’s acceptance,

deposit the ballots into the ballot box. The unit also features an Audio Tactile Interface (ATI) which permits

voters who cannot negotiate a paper ballot to generate a synchronously human and machine-readable ballot

from elector-input vote selections. In this sense, the ImageCast Evolution acts as a ballot marking device.

Photograph 1: ImageCast Evolution (ICE)

Page No. 7 of 59






Precinct Ballot Tabulator: ImageCast Precinct (ICP)

The ImageCast Precinct (ICP) Ballot Counter is a precinct-based optical scan ballot tabulator that is used in

conjunction with ImageCast compatible ballot storage boxes. The system is designed to scan marked paper

ballots, interpret voter marks on the paper ballot and store and tabulate each vote from each paper ballot. The

ICP contains a small touch-screen LCD to allow the poll worker to access diagnostic and configuration

settings.

In addition, enhanced accessibility voting may be accomplished via optional accessories connected to the

ImageCast unit. The ICP utilizes an ATI device to allow voters with disabilities to navigate and submit a

voted ballot. This is accomplished by presenting the ballot to the voter in an audio format. The ATI is

connected to the tabulator, and allows the voter to listen to an audio voting session consisting of contest and

candidate names. The ATI also allows a voter to adjust the volume and speed of audio playback. The cast

vote record is recorded electronically when the ATI is used to cast a ballot. There is no paper ballot or paper

record produced when the ATI is utilized for voting.

Photograph 2: ImageCast Precinct (ICP)

Page No. 8 of 59






Central Tabulator: ImageCast Central Count (ICC)

The Dominion Democracy Suite ICC Ballot Counter system is a high-speed, central ballot scan tabulator

based on Commercial off the Shelf (COTS) hardware, coupled with the custom-made ballot processing

application software. It is used for high speed scanning and counting of paper ballots. Central scanning

system hardware consists of a combination of two COTS devices used together to provide the required ballot

scanning processing functionality:

Canon DR-X10C Scanner: used to provide ballot scanning and image transfers to the local ImageCast Central Workstation.

ImageCast Central Workstation: a COTS computer used for ballot image and election rules processing and results transfer to the EMS Datacenter. The ImageCast Central Workstation is a

logical name for the Dominion pre-approved PC workstation hardware which executes the image

processing and election rules software application.

Photograph 3: ImageCast Central Count (ICC)

Page No. 9 of 59





1.4.2 Block Diagram

The entire system diagram is presented in Figure 1-1.

Figure 1-1 System Overview Diagram

Page No. 10 of 59





1.4.3 System Limits

The EMS platform will be tested in all two deployable physical hardware configurations:

EMS Express hardware configuration - all EMS software components are installed on a single physical PC or laptop. This is a standalone configuration.

EMS Standard hardware configuration - the EMS server components are installed on a single physical server, in addition to the Universal Power Supply (UPS) and Local Area Network (LAN)

switch devices, while the EMS client components are installed on one or more physical PCs or

laptops. All system components are interconnected in a client-server local LAN environment.

The system limits that Dominion Voting Systems has stated to be supported by the Democracy Suite 4.0 are

compiled in the table below.

Table 1-2 Democracy Suite 4.0 System Limits for Portrait Ballot Style

Limit

(Maximum Number of)

Value (by

configuration) Limiting Component

Express Standard

Ballot Positions 462 462 22 Inch Portrait Ballot

Precincts in Election 250 1000 Memory

Contests in Election 250 1000 Memory

Candidates/Counters in

Election 2500 10000 Memory


Precinct 462 462 22 Inch Portrait Ballot


Tabulator 2500 10000 Memory

Ballot Styles in Election 750 3000 Memory

Contests in a Ballot Style 156 156 22 Inch Portrait Ballot

Candidates in a Contests 231 231 22 Inch Portrait Ballot (Column

Span 3)

Ballot Styles in a Precinct 5 5 Memory

Number of Parties 30 30 No Limitation

Vote For in Contest 30 30 No Limitation

Supported Languages per

Election 5 5 Memory

Number of Write-ins 462 462 22 Inch Portrait Ballot

Page No. 11 of 59





1.4.3 System Limits (continued)

Table 1-3 Democracy Suite EMS 4.0 ImageCast Ballot Target Limits for Portrait Ballot Style

Ballot Length Maximum Positions

(Row x Column)

11 Inch 33 x 2

14 inch 45 x 3

17 Inch 57 x 3

20 Inch 69 x 3

22 Inch 77 x 3

1.4.4 Supported Languages

The following languages have been stated by Dominion Voting Systems to be supported by the Democracy

Suite 4.0:

Alaska Native

Aleut

Athabascan

Eskimo

Chinese

Filipino

French

English

Japanese

Korean

Vietnamese

Spanish

Native (other group specified)

o Apache, Jicarilla, Keres, Navajo, Seminole, Towa, Ute, Yuman

Dominion Voting Systems also states that any language that has an ISO definition file can be supported by

the Democracy Suite voting System.

Note: All stated languages will be verified to be supported; however, only English and Spanish ballots will

be cast during functional testing. However Wyle will test 1 character based language (Chinese) during

System Integration Testing.

Page No. 12 of 59





1.4.5 Supported Functionality

The Democracy Suite 4.0 is designed to support the following voting variations:

General Election

Closed Primary

Early Voting

Partisan offices

Non-Partisan offices

Write-in voting

Primary presidential delegation nominations

Straight Ticket voting

Split Precincts

Ballot Rotation

Vote for N of M

Audio Ballot

As stated in the Supported Functionality Description, the Democracy Suite 4.0 System does not include

functions for Cumulative Voting, Ranked Choice Voting (RCV), Open Primary, or Recall Issues; therefore,

testing will not be conducted on these functions.

1.4.6 VVSG

The Democracy Suite 4.0 will be tested to all applicable EAC 2005 VVSG requirements. Please refer to the

EAC online matrix tool (VRT) for further reference.

1.4.7 Beyond VVSG

Dominion Voting Systems submitted hardware test reports for Dust and Rain Tests on the ICP unit. This

testing was performed during State-level certification effort. This testing is out of scope for this test

campaign.

2.0 PRE-CERTIFICATION TESTING AND ISSUES

Currently, no pre-certification testing has been completed. Per EAC Notice of Clarification (NOC) 09-001,

Wyle views the Certification Test Plan as a living document. It will be updated with “As Run” testing and

resubmitted to the EAC as major areas of testing have been completed.

Wyle has performed the first pass review for all source code submitted by Dominion for the Democracy

Suite version 4.0 voting system. The issues with compliance to the EAC 2005 VVSG were reported back to

the manufacturer for resolution. Subsequent submissions will be reviewed by comparing the new submission

against the last submission to ensure all documented issues are resolved before the source code review is

completed.

Page No. 13 of 59



2.0 PRE-CERTIFICATION TESTING AND ISSUES (CONTINUED)

An initial Technical Data Package (TDP) review was performed on the Dominion Democracy Suite version

4.0 voting system documents submitted as their TDP to determine compliance with the EAC 2005 VVSG

and EAC requirements. Wyle found some documents were missing or included partial information, and the

existing documentation contained information which was not consistent throughout the Dominion TDP. The

results were reported to Dominion for resolution. Dominion has subsequently revised and resubmitted the

TDP. Wyle is performing a review of these documents and will submit the results to Dominion as

documented in Section 4.6 TDP Evaluation. Any incidences of non-certification issues (editing issues such

as spelling or formatting) will be noted to Dominion as informational comments for them to decide whether

to address them.

2.1 Evaluation of Prior VSTL Testing

The Dominion Voting Systems Democracy Suite 4.0 is a new voting system that has not been previously

tested as a complete system to applicable federal standards in the EAC Program. It will be fully tested and

the results will be submitted to the EAC in accordance with the requirements of the EAC Voting System

Testing and Certification Manual, Section 4 Certification Testing and Technical Review. Wyle Laboratories

performed testing to the EAC 2005 VVSG on the ICP-A configuration that consisted of an ICP unit, with

firmware version 4.5.4, and a standalone EMS configuration, version 4.5 RC9, as part of a state test effort.

Wyle will be utilizing the data obtained during that test effort to satisfy requirements for this test campaign

for the following tests: Electrical, Environmental, Usability, Security, Maintainability, Availability, Safety,

and Accuracy (performed via paper-based voting and audio voting sessions). More details of this evaluation

are provided in Section 4.4.1 of this document.

2.2 Evaluation of Prior Non-VSTL Testing

Dominion Voting Systems has submitted an ICP and ICE summative usability report per EAC Request for

Interpretation (RFI) 2007-03 “EAC Decision on Summative Usability Testing” for the Democracy Suite

Version 4.0. Summative usability testing and submission to the VSTL is required by the manufacturer as part

of the TDP. The testing focuses on the two components of the Democracy Suite that voters would use to cast

and/or print and cast ballots – the ICE and the ICP. Participants in the test assumed the role of voters who

cast ballots in person at a polling location as well as the tasks of testing the system used by the special needs

voters who required auditory, visual or physical assistance to cast their vote.

2.3 Known Field Issues

This system has never been fielded in the configuration submitted for EAC 2005 VVSG certification testing.

The ImageCast Precinct has been utilized in a small number of elections in New York State. There were no

systemic or significant issues traceable to voting system performance.

3.0 MATERIALS REQUIRED FOR TESTING

The materials required for certification testing of the Democracy Suite 4.0 voting system include software,

hardware, test materials, and deliverable materials to enable the test campaign to occur will be delivered by

Dominion Voting Systems to Wyle.

Page No. 14 of 59



3.0 MATERIALS REQUIRED FOR TESTING (CONTINUED)

3.1 Software

The tables below list the software the manufacturer must submit for testing. This section lists all software

required for operation and testing of the voting system being certified. This includes software used for

testing telecommunications, security and system integration; as well as supporting software required for the

test environment including compilers, assemblers, and database managers, etc. Both COTS and non-COTS

software components are listed in this section.

Table 3-1 Democracy Suite 4.0 EMS Software Platform Component Descriptions

Software Required For

Testing

Software

Version Filename

Democracy Suite EMS EED

Client Application 4.6

setup.exe:

EED_FED_CERT.Setup.msi

EED_FED_CERT.Setup_64b.msi

Democracy Suite EMS RTR

Client Application 4.6

setup.exe:

RTR_FED_CERT.Setup.msi

RTR_FED_CERT.Setup_x64.Setup.msi

Democracy Suite EMS

Application Server 4.6

setup.exe:

EMSApplicationServer_FED_CERT.Setup.msi

EMSApplicationServer_FED_CERT.Setup_x64.Setup.msi

Democracy Suite EMS File

System Service 4.6

Setup.exe:

DVS.Utilities.FileSystemServiceSetup.msi

Democracy Suite EMS Audio

Studio Client Application 4.6

setup.exe:

EMSAS2010_Setup.msi

Democracy Suite EMS Data

Center Manager 4.6 DemocracySuiteEMS_DCM.exe

.NET Framework 3.5 Library 4.0 dotNetFx40_Full_x86_x64.exe

NetAdvantage for .NET 2008

Vol. 1 CLR 2.0

2008 Vol.1

CLR 2.0

NetAdvantage_WinForms_20081_CLR20_Product.exe

(for details see document Components_3rdParty_1.0.xlsx)

Table 3-2 Democracy Suite 4.0 ImageCast Precinct Software Component Descriptions

Software Required

For Testing

Software

Version Filename

Election Firmware 4.6.1-US cf2xx.sig

Firmware Updater 4.6.1-US firmUp.enc

Firmware Extractor 4.6.1-US FirmwareExtract.enc

Kernel (uClinux) 4.6.1-US Image.bin.gz

Boot Loader

(COLILO) 20040221 colilo.bin

Page No. 15 of 59




3.1 Software (continued)

Table 3-3 Democracy Suite 4.0 ImageCast Evolution Software Component Descriptions

Software Required

For Testing

Software

Version Filename

VotingMachine 4.6.2 GApplication-4.6.2.vhd.7z

libAudio 0.3.7 libAudio-0.3.7.tar.bz2

MCFPGA 1.0.11 ice2_mc_p1.bit

SCFPGA 1.0.7 ice2_scb_p1.bit

Logo Platform 3.0.0 logo_platform.bmp

Logo OS 3.0.0 logo_os.bmp

Atmega Intrusion 1.0.9 logger.bin

Atmega Power 1.0.10 power.bin

Blob 1.2 mpc8347dvs.dtb

Integrated Printer 4.1.6 integratedPrinter.hex,

printerFont.hex

Table 3-4 Democracy Suite 4.0 ImageCast Central

Software Component Descriptions

Software Required

For Testing

Software

Version Filename

ImageCast Central

Application 4.0.tbd ImageCast Central.exe

Image-Analysis DLL 4.0.tbd ImgProc.dll

Table 3-5 Democracy Suite 4.0 EMS Software Platform Third Party Software Component

Descriptions

Software Required

For Testing

Software

Version Filename

NetAdvantage for

.NET Windows

Forms 2008

Subscription

2008 Vol.1

CLR 2.0

NetAdvantage_WinForms_20081_CLR2

0_Product.exe

TxText Control .NET

Version 14 16.0 tx_1600_dotnetserver_sp1.zip

Cepstral Text-to-

Speech Desktop

Voices

5.1.0 Cepstral_Allison_windows_5.1.0.msi + 3

more voices

Page No. 16 of 59





Table 3-6 Democracy Suite 4.0 EMS Client Application Software Component Descriptions

Software Required

For Testing

Software

Version Filename

Microsoft Windows 7

x64 6.1 Microsoft DVD provided

Windows Server 2008

R2 x64 6.1 Microsoft DVD provided

Microsoft SQL Server

2008 R2 x64 or

Microsoft SQL Server

2008 Express R2 x64

10.0 Microsoft DVD provided

Microsoft .NET

Framework 4.0 4.0

dotNetFx40_Full_x86_x64.exe

Microsoft Visual J#

Redistributable 2.0 x64 2.0 x64 vjredist64.exe

Adobe Acrobat Reader

9.3 or higher 9.0 AdbeRdr930_en_US.exe

Dallas 1-Wire Device

Driver version 4.0.3b

x64

4.0 install_1_wire_drivers_x64_v403beta.

msi

Cepstral Text-to-

Speech Desktop Voices 5.1.0

Cepstral_Allison_windows_5.1.0.msi +

3 more voices

Java Runtime

Environment 6.0 6.0 jre-6u18-windows-x64.exe

Microsoft IIS 7.5 7.5 Microsoft DVD provided

Table 3-7 Democracy Suite 4.0 EMS Software Platform Unmodified COTS Components Descriptions


Testing

Software

Version Filename

TX Text Control Library

for .NET 16.0

tx_1600_dotnetserver_sp1.zip


OneWire API for .NET 4.0.2.0 OneWireAPI.NET.dll


SOX – audio converter

application 14.3.1

sox.exe


Log4net 1.2.10 log4net.dll, log4net.xml


NLog – log library 1.0.0.505 NLog.dll


Page No. 17 of 59





Table 3-7 Democracy Suite 4.0 EMS Software Platform Unmodified COTS Components Descriptions

(continued)


Testing

Software

Version Filename

Cepstral Text-to-Speech 5.1.0 Cepstral_Allison_windows_5.1.0.msi + 3 more voices


iTextSharp – pdf

generation library 4.0.3

itextsharp.dll


openssl.exe, libeay32.dll,

ssleay32.dll 1.2

openssl.exe, lebeay32.dll, ssleay32.dll


SQLite 1.0.65.0

System.Data.SQLite.DLL 32-bit and 64-bit


Lame 3.98 System.Data.SQLite.DLL 32-bit and 64-bit


Speex 1.0.4 speexdec.exe and speexenc.exe


Ghostscript 8.71 gsdll32.dll – both 32-bit and 64-bit


PdfToImage.dll 1.2 PdfToImage.dll


Tamir.SharpSSH.dll,

DiffieHellman.dll,

Org.Mentalis.Security.dll

– Cryptography

SharpSSh

package

1.1.1.13

Tamir.SharpSSH.sll, Diffie.Hellman.dll,

Org.Mentalis.Security.dll


Table 3-8 Democracy Suite 4.0 ImageCast Precinct Unmodified COTS Software Component

Descriptions

Software Required

For Testing

Software

Version Filename

PNG Reference

Library 1.2.24 libpng-1.2.24.tar.gz

OpenSSL 1.1.2 Openssl-fips-1.1.2.tar.gz

Zlib 1.2.3 Zlib-1.2.3.tar.gz

Page No. 18 of 59





Table 3-9 Democracy Suite 4.0 ImageCast Evolution Unmodified COTS Software Component

Descriptions

Software Required

For Testing

Software

Version Filename

apache-log4cxx 0.10.0 apache-log4cxx-0.10.0.tar.gz

apr 1.4.4 apr-1.4.4.tar.bz2

apr-util 1.3.11 apr-util-1.3.11.tar.bz2

autoconf 2.57 autoconf-2.57.tar.bz2

bison 2.3 bison-2.3.tar.bz2

busybox 1.18.5 busybox-1.18.5.tar.bz2

ccache 2.4 ccache-2.4.tar.gz

cksum 19990607 cksum-19990607.tar.gz

cramfs 20081121 cramfs-20081121.tar.gz

distcc 2.18.3 distcc-2.18.3.tar.bz2

dtc 1.2.0 dtc-1.2.0.tar.gz

e2fsprogs 1.41.14 e2fsprogs-1.41.14.tar.gz

expat 2.0.1 expat-2.0.1.tar.gz

flex 2.5.33 flex-2.5.33.tar.gz

fontconfig 2.8.0 fontconfig-2.8.0.tar.gz

freetype 2.4.4 freetype-2.4.4.tar.bz2

genext2fs 1.4.1 genext2fs-1.4.1.tar.gz

gen_init_cpio 2.6.25-

rc7

gen_init_cpio-2.6.25-rc7.tar.gz

genromfs 0.5.1 genromfs-0.5.1.tar.gz

git 1.5.6.5 git-1.5.6.5.tar.gz

glibc 2.13 glibc-2.13.tar.bz2

glibc-ports 2.13 glibc-ports-2.13.tar.bz2

i2c-tools 3.0.3 i2c-tools-3.0.3.tar.bz2

jpegsrc v8c jpegsrc.v8c.tar.gz

libogg 1.2.2 libogg-1.2.2.tar.gz

libpng 1.5.4 libpng-1.5.4.tar.gz

libtool 1.5 libtool-1.5.tar.gz

libusb 1.0.8 libusb-1.0.8.tar.bz2

Page No. 19 of 59





Table 3-9 Democracy Suite 4.0 ImageCast Evolution Unmodified COTS Software Component

Descriptions (continued)

libusb-compat 0.1.3 libusb-compat-0.1.3.tar.bz2

linux 2.6.30.9 linux-2.6.30.9.tar.bz2

lkc 1.4 lkc-1.4.tar.gz

mkspooflinks 3.4 mkspooflinks-3.4.tar.gz

mtd-utils 20060302 mtd-utils-20060302.tar.bz2

mux_server mux_server.c

openssl-fips 1.2.3 openssl-fips-1.2.3.tar.gz

pkg-config 0.21 pkg-config-0.21.tar.gz

ppp 2.4.5 ppp-2.4.5.tar.gz

qt-everywhere 4.7.3 qt-everywhere-opensource-src-

4.7.3.tar.gz

skell 1.19 skell-1.19.tar.gz

soundtouch 1.5.0 soundtouch-1.5.0.tar.gz

sparse 0.4 sparse-0.4.tar.gz

speex 1.2rc1 speex-1.2rc1.tar.gz

sqlite 3.7.7.1 sqlite-autoconf-3070701.tar.gz

sysfsutils 2.1.0 sysfsutils-2.1.0.tar.gz

texinfo 4.8 texinfo-4.8.tar.bz2

tiff 3.9.5 tiff-3.9.5.tar.gz

tunctl 1.5 tunctl-1.5.tar.gz

tzcode 2011g tzcode2011g.tar.gz

tzdata 2011h tzdata2011h.tar.gz

u-boot-tools 1.1.6 u-boot-tools-1.1.6.tar.bz2

unifdef 1.0 unifdef-1.0.tar.gz

usb-modeswitch 1.1.7 usb-modeswitch-1.1.7.tar.bz2

usb-modeswitch-data 20110227 usb-modeswitch-data-

20110227.tar.bz2

wget 1.9.1 wget-1.9.1.tar.gz

yaffs_utils 20060418 yaffs_utils-20060418.tar.gz

zlib 1.2.5 zlib-1.2.5.tar.bz2

rpm 4.0.4 rpm-4.0.4.tar.gz

Page No. 20 of 59





Table 3-10 Democracy Suite 4.0 - ImageCast Central Build Environment Software Build Components

(Unmodified COTS)

Software Required

For Build

Software

Version Filename

Windows 7 Home

Premium

OEM installed, or full CD from

Microsoft

Visual Studio 2005 Full CD from Microsoft

Table 3-11 Democracy Suite 4.0 - ImageCast Central Build Environment Setup Software Utilities

(Unmodified COTS)

Software Required

For Build

Software

Version Filename

7-Zip 9.20 7z920.exe

Active Perl 64-bit 5.12.4.1205 ActivePerl-5.12.4.1205-

MSWin32-x64-294981.msi

Active Perl 32-bit 5.12.4.1205 ActivePerl-5.12.4.1205-

MSWin32-x86-294981.msi

Nasm 2.09.07 nasm-2.09.07-win32.zip

Table 3-12 Democracy Suite 4.0 - ImageCast Central Software Build Library Source Code

(Unmodified COTS)

Software Required

For Build

Software

Version Filename

OpenSSL Fips

1.2.3 openssl-fips-1.2.3.tar.gz

Table 3-13 Democracy Suite 4.0 - ImageCast Central Runtime Software Components

(Unmodified COTS)

Software Required

For Testing ICC

application

Software

Version Filename

Imgcomp.dll 2.11 apiman.zip

1 Wire driver 64-bit 4.03 install_1_wire_drivers_x64_v403.msi

1 Wire driver 32-bit 4.03 install_1_wire_drivers_x86_v403.msi

Kofax VRS 4.50 Full CD from Kofax

Canon Scanner driver 1.8 X10DRIT_V18.exe

VCredist 4/10/2006 vcredist_x86.exe

Page No. 21 of 59





Table 3-14 Democracy Suite 4.0 ImageCast Precinct Modified COTS Software Component

Descriptions

Software Required

For Testing

Software

Version Filename

uClinux 20070130 uClinux-dist-20070130.tar.gz

COLILO Boot Loader 20040221 Colilo20040221.tar.gz

Table 3-15 Democracy Suite 4.0 ImageCast Evolution Modified COTS Software Component

Descriptions

Software Required

For Testing

Software

Version Filename

Kernel 2.6.30.9-dvs-8 uImage

U-BOOT 1.3.4.19 u-boot.bin

Table 3-16 Democracy Suite 4.0 EMS Software Build Environment Component Descriptions


Testing

Software

Version Filename

Microsoft Windows

Server 2008 R2 x64 6.1 Microsoft DVD provided

.NET Framework 4.0 dotNetFx40_Full_x86_x64.exe

Microsoft Visual J# 2.0

Redistributable 2.0 x64 vjredist64.exe

Microsoft Visual Studio

2010 10.0

Microsoft DVD provided

(Microsoft patch KB2286556 VS10-KB2286556-x86.exe

has to be installed)

Microsoft Visual Studio

2010 Service Pack 1

10.0.30319

SP1 VS2010SP1dvd1.iso

Cruise Control 1.5 CruiseControl.NET-1.5.7256.1-Setup

Nant 0.90 nant-0.90-bin.zip

Csunit 2.1.1 csUnit.2.1.1.BETA.setup

7-Zip 9.20 x64 7z920-x64.msi

NetAdvantage

Infragistics

2008 Vol.1

CLR 2.0 NetAdvantage_WinForms_20081_CLR20_Product.exe

Tx Text Control

16.0.NET 16.0 tx_1600_dotnetserver_sp1.zip

Adobe Acrobat Reader

9.3 or higher 9.3 AdbeRdr930_en_US.exe

ImgBurn 2.5 or higher 2.5.1.0 SetupImgBurn_2.5.0.0.exe

Page No. 22 of 59





Table 3-17 Democracy Suite 4.0 ImageCast Precinct Election Firmware Compiler Descriptions


Testing

Software

Version Filename

g++ (GNU C++ compiler) gcc3.4.0-

20040603

m68k-uclinux-tools-c++-gcc3.4.0-

20040603.sh

Table 3-18 Democracy Suite 4.0 ImageCast Evolution Election Firmware Compiler Descriptions


Testing

Software

Version Filename

g++ (GNU C++ compiler) gcc-4.5.55-

eglibc-2.11.55

freescale-powerpc-linux-gnu-

2010.09-55.i686.rpm

Table 3-19 Democracy Suite 4.0 ImageCast Precinct Firmware Build Environment Component

Descriptions

Software Required

For Testing Software Version Filename

Ubuntu 10.04 LTS –

Long-term support 10.04 ubuntu-10.04.2-desktop-amd64.iso

Toolchain Installation

Script N/A Toolchain.sh

m68k uClinux tools

base gcc 3.4.0-20040603

m68k-uclinux-tools-base-gcc3.4.0-

20040603.sh

m68k uClinux tools

c++ gcc 3.4.0-20040603

m68k-uclinux-tools-c++-gcc3.4.0-

20040603.sh

m68k uClinux tools

gdb 20040603

m68k-uclinux-tools-gdb-

20040603.sh

OpenSSL 1.1.2 Openssl-fips-1.1.2.tar.gz

Table 3-20 Democracy Suite 4.0 ImageCast Evolution Firmware Build Environment Component

Descriptions

Software Required

For Testing

Software

Version Filename

Ubuntu 10.04 LTS ubuntu-10.04.2-desktop-i386.iso

LTIB 10.1.1a ltib-10-1-1a-sv.tar.gz

g++ (GNU C++

compiler)

gcc-4.5.55-

eglibc-2.11.55

freescale-powerpc-linux-gnu-2010.09-

55.i686.rpm

Page No. 23 of 59




3.2 Equipment

This subsection categorizes the equipment the manufacturer has submitted for testing. Each test element is

included in the list of the equipment required for testing of that element, including system hardware, general

purpose data processing and communications equipment, and any required test instrumentation.

Every effort is made to verify that the COTS equipment has not been modified for use. Wyle will perform

research using the COTS equipment manufacturers’ websites based on the serial and service tag numbers for

each piece of equipment and will evaluate COTS hardware, system software and communications

components for proven performance in commercial applications other than elections. For PCs, laptops, and

servers, the service tag information is compared to the system information found on each machine. Physical

external and internal examination is also performed to the best of Wyle’s abilities when the equipment is

easily accessible without the possibility of damage. Hard drives, RAM memory, and other components are

examined to verify that the components match the information found on the COTS equipment

manufacturers’ websites.

The manufacturer provided the hardware listed in Table 3-21 for the purpose of testing two documented

system configurations: Standard, and Express. This hardware consists of PCs, Application/Database Servers,

encrypted Network Attached Storage (NAS) servers, and ruggedized encrypted portable hard drives.

The system configurations consist of:

Standard: (1) PC, (1) Application Server/ Database Server, (1) encrypted NAS for Application/Database Server

Express: (1) PC and (1) Portable Hard Drive

Table 3-21 Democracy 4.0 Voting System Equipment Description

Equipment Manufacturer Version/Model Specifications Serial

Number

PC1 Dell Precision T1500

Processor: Intel Core i7-860 2.8

GHz, Memory: 4x 1GB

1333MHz DDR3, Hard Drive

Capacity: 500 GB

61VNNM1



GHz, Memory: 4x 1GB


Capacity: 500 GB

61TPNM1



GHz, Memory: 4x 1GB


Capacity: 500 GB

61YMNM1



GHz, Memory: 4x 1GB


Capacity: 500 GB

61TNNM1

Page No. 24 of 59




3.2 Equipment (continued)

Table 3-21 Democracy 4.0 Voting System Equipment Description (continued)

Equipment Manufacturer Version/Model Specifications Serial

Number

PC5 Dell Inspiron One

2305

Processor: AMD Athlon II X2

240e 2.8 GHz, Memory: 8GB

Dual Channel 1333MHz DDR3,

Hard Drive Capacity: 1 TB

564C3P1,

563F3P1

SERVER1 Dell PowerEdge

R610

Processor: Intel Xeon E5620 2.4

GHz, Memory: 8x 2GB


Capacity: 2x 500 GB

5M9NNM1


R610


GHz, Memory: 8x 2GB


Capacity: 2x 500 GB

5M8PNM1


R610


GHz, Memory: 8x 2GB


Capacity: 2x 500 GB

5M8QNM1

STORAGE1 Rocstor Guardian 4RM

Raid System

Disk space: 2 TB (Striped +

Mirrored), Processor: 400 MHz

storage I/O, Hot bus interface:

eSATA, Drive bus interface:

SATA II

ROC7326210

47/SB090101

54


Raid System

Disk space: 2 TB, Processor: 400

MHz storage I/O, Hot bus

interface: eSATA, Drive bus

interface: SATA II

ROC7326210

45/SB090101

57


Raid System

Disk space: 2 TB, Processor: 400

MHz storage I/O, Hot bus

interface: eSATA, Drive bus

interface: SATA II

ROC7326210

46/SB090101

61

STORAGE4

Rocstor

Commander

2UE Portable

Hard Drive

Hard Drive Capacity: 500 GB 5VJ4DRJP

STORAGE5 Rocstor

Commander

2UE Portable

Hard Drive

Hard Drive Capacity: 500 GB 5VJ48VFJ

Page No. 25 of 59





In order to perform the software Witness and Trusted Builds, one Personal Computer has been provided as a

build machine. The build machine is described in the table below:

Table 3-22 Build Machine Description

Equipment Manufacturer Version/Model Serial Number COTS/

Non-COTS

Build 1

Super Micro

PC w/4 Hard

Drives

PC w/4 Hard

Drives

BM-57381-001

COTS

To support the test program, Dominion has provided additional supporting hardware for the provided

Personal Computers. A list of these items is provided in Table 3-23.

Table 3-23 Dominion 4.0 COTS Voting System Support Equipment Description

Test Material Make Model Quantity Serial Number

COTS Central High

Speed Scanner Canon DR-X10C 2 ED300874, ED300880

iButton (SHA-1) with

USB Reader/Writer Maxim

USB R/W: DS9490R

iButton: DS1963S 3

4D027C, 4C9CF5,

514DFD

iButton (SHA-1) Maxim DS1963S 2 4CE4C9, 4D064A

LCD Monitor Soyo 18.5” wide LCD 1 DYLM19R6-KLE-

10202

LCD Monitor Samsung 23” wide LCD 1 MY23HVMS701197B

LCD Monitor Dell 1909W 4 07E-4EUS, 07F-071S,

07F-06US, 07F-074S

LCD Monitor Dell N445N 3 2TWC, 2UOC, 2U6C

Audio Adapter Soundwave USB Soundwave 7.1 Audio

Adapter 2

SW-57381-001, SW-

57381-002

PCI Software Soundwave Soundwave 7.1 PCI

Software 2 n/a

USB Software Soundwave USB Soundwave 7.1

Software 1 n/a

Networking Switch D-Link D-Link DES-1105 5-Port

Switch 1 DRL728A001397

Mouse Dell USB w/rollerball 4

G1A00M0M, 10203JTI,

LZA30491960,

438027372

Mouse Microsoft USB w/rollerball 1 X800898

Keyboard Kensington USB 1 D0713000487

Keyboard Microsoft USB 1 6968200717217

Page No. 26 of 59







Keyboard IBM USB 1 2162079

Compact Flash

Reader SanDisk USB 3

0171618, 0201833,

0171631

Networking

Switch D-Link DGS-2208 8-Port Switch 2

F36J69C004821,

F36J69C004824

Headphones Radio Shack 33-276-01 1 Headphones

eSATA PCI Card

(Installed into

Servers and PCs)

SIIG, Inc. eSATA II PCIe Pro Card 7 n/a

Card Reader GGI Gear Compact Flash Card Reader 4 CFRW-57381-001 thru 004

Sony Headphone MDR-G45LP-01 1 Sony

Cyber Acoustics

Headphone

Cyber

Acoustics ACM-70 2 DVS23000048



Sip & Puff Origin

Instruments Air Voter 7

AV-57381-001 thru 003, 002251,

002268, 002267

Footswitch Pair N/A Enabling Devices 4 RP-57381-001 thru 004

Compact Flash RiData CFC-14A 50 Wyle-assigned numbers: CF-XXX

The table below provides the serial numbers of the equipment submitted for testing:

Table 3-24 Democracy 4.0 Voting System Equipment

Equipment Description Serial Numbers

ICP Precinct Count Optical Scanner PCOS 320A

WLDAFBH0001, WLDAFBH0002,



WLDAFBH0023

ICE Precinct Count Optical Scanner PCOS 400A ICE2P1005, ICE2P1006, ICE2P1007,

ICE2P1008

ICP Ballot

Box Externally secure ballot box

BOX-57381-011, BOX-57381-012, BOX-

57381-013, BOX-57381-014, BOX-57381-

015

ICE Ballot

Box Externally secure ballot box

BOX-57381-01, BOX-57381-02, BOX-

57381-03, BOX-57381-04,

Page No. 27 of 59




3.3 Test Support Materials

This subsection enumerates any and all test materials needed to perform voter system testing. The scope of

testing determines the quantity of a specific material required.

The following test materials are required to support the Democracy Suite 4.0 certification testing:

Table 3-25 Democracy Suite 4.0 Test Support Materials

Test Material Quantity Make Model

Hasp Locks (red) 50 N/A N/A

Tamper Evident Seals 50 N/A SE-37

Disposable Gloves 3 N/A N/A

Gloves and Mouthpiece Kit 17 N/A N/A

Black and Clear Mouthpiece 1 N/A N/A

ATI Handsets 5 Dominion ATI-57381-001 thru 005

Black Ballot Privacy Sleeves 4 Dominion N/A

White Ballot Privacy Sleeves 4 Dominion N/A

Black Privacy Panels (set of 2

pieces) 4 Dominion

N/A

White Privacy Panels 4 Dominion N/A

Thermal Printer Rolls 100 N/A N/A

Combination Lock 2 MASTER Lock 646T

Keyed Lock 4 MASTER Lock 121Q

Security Keys 20 Maxim N/A

Ballots 8000 Dominion N/A

Dominion Cleaning Kit 1 Dominion N/A

Permanent Markers 20 p/n SHARPIE1 BK N/A

3.4 Deliverable Materials

The materials listed below are to be delivered as part of the Democracy 4.0 System to the users:

Table 3-26 Deliverable Materials

Deliverable Material Version Description

Election Event Designer 4.6 EMS client application

Results Tally and Reporting 4.6 EMS client application

Audio Studio 4.6 EMS client application

Application Server 4.6 EMS server application

Datacenter Manager 4.6 EMS server application

ImageCast Evolution

400A w/Firmware

version 4.6.2.3

loaded

Precinct ballot scanner and ADA accessible

voting device

ImageCast Precinct

320A w/Firmware

version 4.6.4

loaded

Precinct ballot scanner and ADA accessible

voting device

Page No. 28 of 59




3.4 Deliverable Materials (continued)

Table 3-26 Deliverable Materials (continued)

Deliverable Material Version Description

ImageCast Central Count

Canon DR-X10C

w/Firmware version

4.6.3 loaded

Central ballot scanner

ImageCast Evolution Metal

Ballot Box BOX-400A ICE Metal Ballot box

ImageCast Precinct Metal Ballot

Box BOX-310A ICP Metal Ballot box

ImageCast Precinct Plastic Ballot

Box BOX-330A ICP Plastic Ballot box

Rocstor Encrypted NAS Dell PowerEdge

R610

Encrypted Network Attached Storage

module for server and data backup

Rocstor Portable Hard Drive

Rocstor Commander

2UE Portable Hard

Drive

Encrypted and ruggedized external hard

drive

iButton with Reader/Writer

Maxim USB R/W:

DS9490R

iButton: DS1963S

Security authentication token with

programmer

ICE/ICP Headphones Cyber Acoustics Headphones used for audio voting

Sip/Puff Device Origin Instruments

Air Voter Binary input device for disabled voters

Footswitch Pair Enabling Devices Binary input device for disabled voters

Compact Flash Cards CFC-14A Transport Media

ATI Handset Dominion ADA voting device used in conjunction with

Binary input devices

ICP System Operation Procedures 1.1.0::147 TDP Document

EMS System Operation

Procedures 1.2.0::387 TDP Document

ICE System Operation


ICC System Operation


ICP System Maintenance Manual 1.1.0::58 TDP Document

ICE System Maintenance Manual 1.1.0::109 TDP Document

Election Event Designer User’s

Guide 1.3.3 TDP Document

Results Tally and Reporting

User’s Guide 1.2.7 TDP Document

Audio Studio User’s Guide 1.2.3 TDP Document

Page No. 29 of 59



4.0 TEST SPECIFICATIONS

Certification testing of the Democracy Suite 4.0 is the configuration submitted in the EAC application DVS-

1001. Wyle qualified personnel will ensure that all certification testing performed on the manufacturer’s

voting system follows Wyle’s procedures for testing and the specific test cases to ensure the requirements of

the EAC 2005 VVSG and EAC Testing and Certification Program Manual are met.

Below is a list of EAC Request for Interpretations (RFI) and Notice of Clarifications (NOC) that will be

incorporated in the test campaign:

Interpretations

2010-08 EAC Decision on Calling Sequence

2010-07 EAC Decision on Module Length

2010-06 EAC Decision on DRE Accessibility Requirements and Other Accessible Voting stations

2010-05 EAC Decision on Testing of Modifications to a Certified System

2010-04 EAC Decision on Functional Requirements with Respect to Security

2010-03 EAC Decision on Database Coding Conventions

2010-01 EAC Decision on Voltage Levels and ESD Test

2009-06 EAC Decision on Temperature and Power Variation

2009-05 EAC Decision on T-Coil Requirements

2009-04 EAC Decision on Audit Log Events

2009-03 EAC Decision on Battery Backup for Central Count Systems

2009-02 EAC Decision on Alternate Languages

2009-01 EAC Decision on VVPAT Accessibility New

2008-12 EAC Decision on Ballot Marking Device/Scope of Testing

2008-10 EAC Decision on Electrical Fast Transient

2008-09 EAC Decision on Safety Testing

2008-08 EAC Decision on Automatic Bar Code Readers

2008-07 EAC Decision on Zero Count to Start Election

2008-06 EAC Decision on Battery Backup for Central Count

2008-05 EAC Decision on Durability

2008-04 EAC Decision on Supported Languages

2008-03 EAC Decision on OS Configuration

2008-02 EAC Decision on Battery Backup for Optical Scan Voting Machines

2008-01 EAC Decision on Temperature and Power Variation

2007-06 EAC Decision on Recording and Reporting Undervotes

2007-05 EAC Decision on Testing Focus and Applicability

2007-04 EAC Decision on Presentation of Alternative Language

Page No. 30 of 59



4.0 TEST SPECIFICATIONS (CONTINUED)

2007-03 EAC Decision on Summative Usability Testing

2007-02 EAC Decision on Variable Names

2007-01 EAC Decision on Accessible Design

Notice of Clarifications

NOC 09-005 – Development and Submission of Test Plans for Modifications to EAC Certified Systems

NOC 09-004 – Development and Submission of Test Reports

NOC 09-003 – De Minimis Change Determination Requirement

NOC 09-002 -- Laboratory Independence Requirement

NOC 09-001 -- Requirements for Test Lab Development and Submission of Test Plans

NOC 08-003 -- EAC Conformance Testing Requirements

NOC 08-002 -- EAC Mark of Certification

NOC 08-001 -- Validity of Prior Non-core Hardware Environmental and EMC Testing

NOC 07-005 -- Voting System Test Laboratory Responsibilities in the Management and Oversight of Third

Party Testing

NOC 07-004 -- Voting System Manufacturing Facilities

NOC 07-003 -- State Testing Done in Conjunction with Federal Testing within the EAC Program

NOC 07-002 -- VSTL Work with Manufacturers Outside of Voting System Certification Engagements

NOC 07-001 -- Timely Submission of Certification Application

4.1 Requirements (Strategy of Evaluation)

To evaluate the system test requirements, each section of the EAC 2005 VVSG will be analyzed to determine

the applicable tests. The EAC 2005 VVSG Volume I Sections, along with the strategy for evaluation, are

described below:

Section 2: Functional Requirements – The requirements in this section will be tested during the FCA and System Integration test utilizing the “Wyle Baseline Test Cases” along with test cases specially

designed for the Dominion Democracy Suite 4.0 per sections 4.4.3 and 4.4.5. The data input during these

tests will be the predefined election definitions submitted as part of the Test Plan Package.

Section 3: Usability and Accessibility – The requirements in this section will be tested during the Usability Test utilizing a combination of the “Wyle Baseline Test Cases” and the “Wyle Baseline

Usability Test Cases”. The data input during this test will be the predefined election definitions

submitted as part of the Test Plan Package.

Section 4: Hardware Requirements – The requirements in this section will be tested and/or evaluated by trained Wyle personnel per sections 4.4.2 and the table in section 6.

Section 5: Software Requirements – The requirements in this section will be tested during source code review, TDP review, and FCA. A combination of review and functional testing will be performed to

ensure these requirements are met.

Page No. 31 of 59




4.1 Requirements (Strategy of Evaluation) (continued)

Section 6: Telecommunication – A test of the telecommunication technologies utilized by the Dominion Democracy Suite 4.0 will be tested for data accuracy and correctness by analyzing the packet

level information being transmitted. Section 6.2.6 will be excluded since the Democracy Suite 4.0 does

not support the use of public networks.

Section 7: Security Requirements – The requirements in this section will be tested during source code review, FCA, System Integration, and Security Tests. In addition to functional testing, the source code

for the Dominion Democracy Suite 4.0 will be analyzed utilizing Fortify ™ Source Code Analysis

(SCA) for security vulnerabilities in addition to the manual line by line review.

Section 8: Quality Assurance (QA) Requirements – The requirements in this section will be tested throughout the test campaign via various methods. TDP review will be performed on the Dominion QA

documentation to determine compliance to EAC 2005 VVSG requirements and the requirements stated

in the Dominion Voting Systems QA Program document. All source code will be checked to ensure that

proper QA documentation has been completed. All equipment received for initial testing and follow up

testing will be checked against Dominion documentation to ensure their QA process is being followed.

Wyle personnel will complete the requirements of EAC 2005 VVSG Vol. 2 Section 7, Quality Assurance

Testing and Section 1.3.1.5, Focus of Vendor Documentation that requires Wyle personnel to physically

examine documents at Dominion’s location or conduct an external evaluation utilizing equipment,

documents and support information provided by Dominion during the test campaign.

Section 9: Configuration Management (CM) Requirements – The requirements in this section will be tested throughout the test campaign. TDP review will be performed on the Dominion configuration

management documentation to determine EAC 2005 VVSG compliance and to further determine

whether Dominion is following its documented CM requirements within the TDP. During source code

review, Wyle qualified personnel will verify that Dominion Voting Systems is following EAC 2005

VVSG CM requirements as well as Dominion CM requirements. Any anomalies will be formally

reported to Dominion and the EAC. All equipment received for testing will be checked against

Dominion documentation to ensure their CM process is being followed.

4.1.1 Mapping of Requirements to Equipment Type and Features

Please refer to the EAC online matrix tool (VRT) for further reference.

4.1.2 Rationale for ‘Not Applicable’ Requirements

The Dominion Voting Systems Democracy Suite 4.0 is a paper-based precinct counting system that supports

a closed network (does not support transmission over public networks). Therefore, all EAC 2005 VVSG

requirements, with the exceptions listed below, will be evaluated as part of this test campaign.

Volume I Section 6.2.6 (Telecommunication Requirements)

Volume I Section 7.5.2 – 7.5.4 (Telecommunications and Data Transmission)

Volume I Section 7.6 (Use of Public Communication Networks)

Volume I Section 7.7 (Wireless Communications)

Volume I Section 7.9 (Voter Verifiable Paper Audit Trail Requirements)

Page No. 32 of 59




4.1 Requirements (Strategy of Evaluation) (continued)

4.1.2 Rationale for ‘Not Applicable’ Requirements (continued)

The rationale for not evaluating the Democracy Suite 4.0 to the requirements contained in the indicated

sections of the EAC 2005 VVSG is described below. Refer to the EAC online matrix tool for specific

requirements that are excluded during this test campaign.

Table 4-1 Not Applicable Requirements

EAC 2005 VVSG

Volume I Section Rationale for ‘Not Applicable’

6.2.6, 7.5.2, and 7.5.3 These requirements are written for use of public networks. The Dominion

Democracy Suite 4.0 does not use public networks.

7.5.4

This section was intended for a shared operating environment on ballot recording

and vote counting equipment. The ICE and ICP use dedicated operating

environments and will be excluded from this requirement. The EMS and ICC

components do use a shared operating environment and will be tested to this

VVSG clause.

7.6

This section pertains to “Voting systems that transmit data over public

telecommunications…” The Dominion Democracy Suite 4.0 does not support

transmission over public networks.

7.7 No wireless technology is present in the Dominion Democracy Suite 4.0.

7.9 The Dominion Democracy Suite 4.0 is a paper based system.

4.2 Hardware Configuration and Design

The Dominion Voting Systems Democracy Suite is a paper-based optical scan voting system. The

Democracy Suite system consists of four major components: the EMS, ICE precinct scanner and ballot

marking device, ICP precinct scanner, and ICC central count scanner. The Democracy Suite is comprised of

two proprietary pieces of hardware (ICE and ICP) and one piece of COTS hardware (ICC). All EMS

functions are handled by proprietary software running on COTS PC/laptops/servers. Wyle has determined

that these COTS PC/laptops/servers are not subject to hardware testing per the EAC 2005 VVSG. The

provided PC/laptops/servers documented in Section 3 Materials Required For Testing all contained CE, UL,

and FCC labeling.

ICP – Wyle Laboratories previously performed testing to the EAC 2005 VVSG on the ICP-A configuration

that consisted of an ICP unit, with firmware version 4.5.4, and a standalone EMS configuration, version 4.5

RC9, as part of a state test effort. Wyle will be utilizing the data obtained during that test effort to satisfy

requirements for this test campaign for the following tests: Electrical, Environmental, Usability, Security,

Maintainability, Availability, Safety, and Accuracy (performed via paper-based voting and audio voting

sessions).

ICE - ICE will be set on the ballot box to simulate the actual election configuration. During operational tests

the unit will be in auto feed mode (“Shoe-Shine”) and scan test ballots for the duration of the operational test.

Each unit will be loaded with the Operational Status Check Hardware election definition configured for early

voting. This will allow all the data generated for the Pre-operational, Operational, and Post-operational test to

be further analyzed, compiled and included in the Reliability and Availability Test results.

Page No. 33 of 59




4.2 Hardware Configuration and Design (continued)

ICC - ICC consists of COTS scanner and COTS Workstation PC. The Canon DR-X10C (S/N ED300874)

scanner and the Dell Inspiron One 2305 (S/N 564C3P1) Workstation PC contain CE, UL, and FCC labeling.

Due to the fact that these components are unmodified COTS equipment, as well as central count equipment,

they will be exempt from non-operational hardware testing; however the ICC will undergo Temperature

Power testing in conjunction with the ICE. Beyond the Temperature Power test, the ICC will only be utilized

in functional and system testing for this campaign.

4.3 Software System Functions

The Dominion Democracy Suite 4.0 System software is written in the C, C++, C# (C Sharp) programming

languages. The system software is broken into three areas: EMS, Precinct tabulator software acting as

firmware, and central count application running on a COTS workstation.

The Democracy Suite EMS software consists of seven applications listed below:

Election Event Designer

Results Tally and Reporting

Audio Studio

Datacenter Manager

Application Server

Network Attached Storage Server

Database Server

The Democracy Suite 4.0 contains two precinct tabulators. Both tabulators run software that is treated as

firmware. The software applications are ICP and ICE. The Democracy Suite 4.0 has an independent

workstation running proprietary software. The ICC application provides the central tabulation function for

the system.

4.4 Test Case Design

Wyle uses the V-Model Life Cycle as defined by the Institute of Electrical and Electronics Engineers (IEEE).

The IEEE definition of the V-Model Life Cycle uses two concepts “Verification” and “Validation”. Wyle’s

test approach is to use both “Verification” and “Validation” to some degree. There are four basic levels of

testing in the V-Model Life Cycle: Component, Integration, System, and Acceptance. Wyle will be

evaluating the Dominion Democracy Suite 4.0 to all four levels.

4.4.1 Hardware Qualitative Examination Design

ICP Testing

As stated previously, Wyle Laboratories performed testing to the EAC 2005 VVSG on the ICP-A

configuration that consisted of an ICP unit, with firmware version 4.5.4, and a standalone EMS

configuration, version 4.5 RC9, as part of a state test effort. Wyle will be utilizing the data obtained during

that test effort to satisfy requirements for this test campaign for the following tests: Electrical,

Environmental, Usability, Security, Maintainability, Availability, Safety (this testing was witnessed by Wyle

personnel at a third party laboratory), and Accuracy (performed via paper-based voting and audio voting

Page No. 34 of 59




4.4.1 Hardware Qualitative Examination Design (continued)