Risk-LimitingAudits:LessonsLearnedNealMcBurnett

Atthe2018-12-07MITElectionAuditSummit

NealMcBurnettConsultantColoradoRLAprojectteammemberwithDemocracyWorksandFree&FairSpeakingformyself

Workingonelectionauditsandintegritysince2003

PollworkerVolunteerIEEEP1622ViceChairCenterforElectionScienceBoardmemberSoftwaredeveloperConsultant

History,Links:

TheColoradoRisk-LimitingAuditProject(CORLA)Pilotaudits:earlyandoften

BestPractice:Ballot-LevelRLAsWithagoodsystems,processesanddata,youcandoballot-levelrisk-limitingauditswhichlimittheriskthattabulationerrorsorattacksresultingettingthewrongoutcome

Inhundredsofcontests,dozensofcountiesAcrossoverlappingdistrictsinastateEfficiently:auditedlessthantenthousandballotsstatewide

Simultaneously,opportunistically,gatherevidenceonandreportrisklevelsforalltherestofthecontests(2018stillworkinprogress)

Colorado'snewstatewidesystemisamongthemostcosteffectiveandbestforauditing:central-countscannerswithBMDsavailableforaccessibility

ResourcesAvailableFourvendorspresentedandpilotedsystemsthatcoulddoballot-levelcomparisonRLAsin2015:centralcountscanningsystemsfromDominion,Hart,ClearBallot,ES&S

OpensourceColoradoRLAcodeavailabletosupporttheseaudits,continuestobeenhanced

SupportforRLAsWidespread,transpartisanconsensusonneedforbothpaperballotsandaudits.

2003Four-partyconsensusinBoulderColorado2017EAC/NISTVoluntaryVotingSystemGuidelines(VVSG)2.02018SecuringtheVote:ProtectingAmericanDemocracy|TheNationalAcademiesPress

Hugestepsforward,stillmuchtodo

Whyisittakingsolongtoadoptrobustaudits?

ElectionsareincreasinglycomplicatedYoucan'teasilyauditthedatayou'vegotYoucan'teasilygetthedatayouneedCriticalCommonDataStandardsworkbyEAC/NIST

CommonDataFormatsWeneedformatstandards!EAC/NIST

JohnWack:OverviewofVVSG-InteroperabilityCommonDataFormats(twopresentations)

ElectionResultsCDFV1publishedasSP1500-100.

UsedinOH,NC,LACounty,otherstatesinprogress.V2synchronizeswithGoogle/VIP5.1,addsJSON.ElectionLogExportCDFsoonpublishedasSP1500-101.VoterRecordsInterchangeCDFslatedforreviewbyVRvendorsandthenpublishedasSP1500-102.

InitialuseinOHandbyOSET.CastVoteRecordsCDFschematobepublishedasSP1500-103.Continueddevelopmentanddocumentationofelectionprocessbusinessmodelsandvotingmethoddescriptions.

EvidencepresentedandcheckedPublicRLAOversightProtocol,StephanieSinger,NealMcBurnett2017

Elements:

1ChainofCustody2Tabulation3Manifest4Commitment5Randomselection6Ballotcardretrieval7BallotInterpretationanddataentry8Endingtherandomselectionandexaminationofballotscards9HandCount10AuditConclusionsAffectOutcomes

ConvincingOfficialsofElectionOutcomesColoradoRLAincludesrla_exporttooltoprovidenecessarydataforOversightProtocolincsv/jsonformats

rla_reportdemonstrationcodeinprogresstopreciselyexplainandimplementoversightsteps.

Verifiersshouldofcourseimplementorvettheirownprocesses,code,etc.

Level1:ElectionAdminstrators

Coloradocountiesandstate,basedontheirknowledgeoftheCVRsetc,dramaticallylimitedtheriskofanincorrecttabulationoutcomeinhundredsofcontests

Farmorethanmoststatescansay,veryefficient!

ConvincingOthersofElectionOutcomesLevels2and3:LosersandthePublic

MuchmoretransparencythaninthepastStillseveralcrucialholesleftinoversightprotocolSomesummarydatanotavailableyet

Wrestlingwithballotanonymityissues=>noCVRs

Can'tchecktotals,interpretations,etc.

Nevertheless,appreciateamazingongoingaccomplishmentsbystateandcountiesunderverychallengingcircumstances!

Moretocome!

DiscrepancyInvestigationsDetailedreportsstillin-progress

SoftwareshouldinformAuditBoardofeachdiscrepancyrightafterentry.Thatwouldhelpinvestigation,qualitycontrolfeedback,andtrustintheprocess.

PoorviVoraandIhaveadocumentinprogressonhowtoinvestigatediscrepancies,preservingintegrity,efficiency,flexibility

Moredatawouldbeinvaluableinfinetuningtheprocess

RemainingChallenges1Enhancereportingconvenienceandanalysis

Discrepancies,risklevelsforopportunisticallyauditedcontestsinarbitrarymixofcountiesAutomaticgeneration,publicationofAuditCenterforpublic

Modularizesoftwareforusewithexternalstatisticalmodules,additionalmethods(SUITE,Bayesian,etc.)

HandleIn-Precinct/Votecenterscanners,whichrandomizeballotsand/orCVRs:complicateprocessofmatchingpaperballotwithCVR

Supportbatch-comparisonaudits(generallypredictableriskreduction),ballot-pollingaudits(unpredictable,impracticalfortightmargins)

RemainingChallenges2Fostercollaborationbetweenclerks,privacyexperts,toolsmithsaroundpreservinganonymity,especiallyforcomplicatedsituationinColorado

Auditmoresystems:VR,signatureverification,envelopesorters

TargetedauditsEncouragecandidates,publictoidentifyadditionalinterestingballotstotargetforauditing

InadditiontofullrandomselectionauditCouldbechosenbasedonCVRs,markdensitydata,ballotimages

PublicengagementinverificationPromotepublicparticipationinauditPrintballottrackingpageswithQRcodesApptophotographballot+QRcodeAssistpublictweetslike"Iverifiedthisvote"

AcknowlegementsPaulTigerJoePezzilloPaulWalmsleyRonRivestPhilipStarkHarvieBranscombHillaryHallandBoulderCountyteamElectionVerificationNetworkColoradoLegislatorsIncrediblededicationofSOSstaffandClerks!Free&FairTeambeyondthecallofdutyGaloissupport

Updatedslides:http://bcn.boulder.co.us/~neal/elections/audit-summit.pdf