On the security of ballot marking devices

Dan S. Wallach

REVIEW 2 major objections 5 minor 20 references

On the security of ballot marking devices

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that live audits plus spoiled-ballot tracking can make ballot-marking devices trustworthy enough for accessible elections.

desk verdict A clear-headed math-backed brief for making BMDs work with live audits and spoilage monitoring, but the fleet-wide confidence claim leans on an unvalidated 'random voter' model and empirical guesses the paper itself flags. read the letter →

arxiv 1908.01897 v2 pith:E6XURPPI submitted 2019-08-05 cs.CR

Dan S. Wallach This is my paper · ORCID

classification cs.CR

keywords ballotmarkingdeviceselectionsecurityliveauditingspoilageswitched-intentattackrisk-limitingauditsaccessibilityinvotingparalleltesting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper takes on the position that ballot-marking devices (BMDs) cannot be made secure, and answers it with a concrete counterproposal: election administrators can create reasonable auditing procedures that give real confidence their BMD fleet is printing voters' true choices. The two main procedures are live audits, where an official secretly exercises a machine and checks its printed ballot, and real-time tracking of spoiled-ballot rates, where an unusual spike is treated as statistical evidence of tampering. The paper analyzes both quantitatively, showing that a few hundred live audits give high detection probability for a machine that flips a few percent of ballots, and that spoilage monitoring sharply limits how much malware can move a large jurisdiction's margin of victory without detection. If the paper is correct, jurisdictions can keep the usability and accessibility benefits of BMDs—audio ballots, large text, multiple languages, no overvotes—while retaining the security and auditability of paper ballots.

What carries the argument

The load-bearing identity is the live-audit detection probability. If a compromised BMD flips a ballot with probability $p$ on any given use, then after $n$ audits the chance of catching it at least once is $1-(1-p)^n$. This treats each audit as an independent Bernoulli trial and makes detection depend on the number of audits, not the number of ballots cast, so the relative cost of auditing falls as an electorate grows. The companion machinery is a Poisson-based spoilage alarm: given a background spoilage rate, a count above the 95% Poisson confidence bound is treated as evidence of tampering, and the paper converts that bound into the maximum margin shift malware could hope to hide. The third component is the "random voter" model, a script that tells auditors which races to vote and how to behave, built from prior ballots and event logs so an attacker cannot reliably tell an audit from a real voter.

What would settle it

A red-team experiment would settle the question: build the random-voter script from a jurisdiction's actual cast-ballot and event-log data, deploy it on machines running malware that targets a subpopulation the script rarely samples (for example, voters who use accessibility features or take unusually long), and measure whether the tampering evades both the live audits and the 95 percent spoilage confidence bound. If it does, the paper's detection probabilities do not hold for that threat model.

Watch

Extended reading notes

Core claim

The central claim is that the security of a BMD election does not depend on each voter being a vigilant auditor; it can be carried by a two-layer detection process at the election-administrator level. A malicious BMD that hides its misbehavior behind a normal screen and prints a different ballot than the voter selected—a switched-intent attack—will be caught if a live auditor happens to use that machine, and it will also cause observable voters to spoil ballots at an elevated rate. The paper argues that with enough randomized live audits and real-time spoilage data, the probability of catching such an attack is high for the large jurisdictions that actually buy BMDs, and that an emergency re-run procedure can deter or repair the residual risk. The paper does not claim BMDs are invulnerable; it claims that the combination of audits, spoilage signals, and emergency laws restores the security properties people expect from hand-marked paper while preserving BMD accessibility.

Load-bearing premise

The whole detection estimate rests on the assumption that auditors can exercise BMDs in a way indistinguishable from the real voters an attacker would target, so the "random voter" model must faithfully capture voters' preferences and behaviors; the paper calls this a tractable engineering problem but does not build or validate the model.

Editorial extensions

If this is right

Roughly 300 to 500 live audits give better than 99 percent confidence of catching a BMD that flips 1 percent of ballots, independent of the total number of votes cast.
In a 1.2-million-vote county, real-time spoilage monitoring limits undetected margin shifts to about 0.3 percentage points even under a pessimistic 10 percent voter-notice rate, and to as little as 0.06 percentage points if half of affected voters spoil their ballots.
For a small down-ballot race, such as the 9,000-vote Baytown example, focused tampering may be caught only 10 to 40 percent of the time by 300 county-wide audits, so officials must deploy extra auditors adaptively to hot races.
A single inconsistent barcode-and-text ballot is treated as incontrovertible evidence of a bug or attack, which is why attackers are unlikely to use barcode-only tampering.
Elections can treat a large cyberattack like a natural disaster: emergency laws and re-run procedures reduce the attacker's confidence of getting away with it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's detection table assumes each live audit is independent of the attacker's state; an adversary who can observe audit patterns could tune its flips to rare behaviors, which suggests a game-theoretic extension where detection probabilities degrade from Table 1.
The spoilage alarm and live audits are analyzed separately; combining them into one sequential Bayesian alarm that updates on both signals as the day progresses would likely detect a focused attack sooner than either method alone.
A natural pilot experiment would score a constructed random-voter script against real event logs; if the script under-samples slow voters, audio users, or other accessibility subpopulations, the paper's detection estimates should be discounted by exactly that under-coverage.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dan, this is a useful paper to know about. It's Wallach's response to Appel/DeMillo/Stark and Stark's parallel-testing notes, and it makes the best available case that BMDs can be used if you pair them with live auditing and real-time spoilage-rate monitoring. The probability math is correct and clearly presented; Table 1 is just 1-(1-p)^n, Table 2 generalizes Stark's Poisson logic, and Appendix A gives working Python so you can check it. The point that detection probability is independent of the number of ballots cast, which makes big jurisdictions the natural home for BMDs, is well made and under-appreciated. The paper also does something rare in this debate: it names its own weak spots. Footnote 5 says nobody knows the true baseline spoilage rate yet; Section 5.3 concedes that in a small election like Baytown, focused down-ballot tampering is caught only 10-40% of the time with 300 audits; and the emergency-procedures discussion is honest about the political difficulty of re-running an election.

The soft spot is exactly the one the stress-test flags. The entire live-audit guarantee in Table 1 and the 5.3 numbers depend on auditors sampling from the same distribution as the voters an attacker cares about. Section 5.2 says building a 'random voter' model from historical ballots and event logs is a tractable engineering problem, but it does not build it, and it does not validate it. That's an empirical premise, not a theorem. If an attacker targets slow users or accessibility-feature users or some precinct skew, the effective p in the audits is lower than assumed and the confidence numbers in the paper are not reached. The paper acknowledges the shape of this attack (Stark's point) and sidesteps it with a hand-wave. I don't think this is fatal — Wallach's larger point that live auditing plus spoilage monitoring raises the bar for attackers in large jurisdictions survives — but the abstract overreaches when it promises 'the same security and reliability properties we expect from hand-marked paper ballots.' Hand-marked ballots don't need voters to detect a systemic malware attack; the audit is on the paper itself. Here the audit is partly on the voter, and voter detection rates come from small lab studies that may not transfer.

So: the paper is a serious piece of advocacy with correct math and honest caveats, but it's a conditional case. It deserves a referee if submitted as a position paper; the referee should ask for the random-voter model to be built and evaluated, or for the scope claim to be narrowed to large jurisdictions with validated voter-detection behavior. I'd bring it to a reading group, but I wouldn't cite it as a proof of concept — only as a well-argued point of view in the debate.

Referee Report

2 major / 5 minor

Summary. The paper argues that ballot marking devices (BMDs) can be deployed securely if election administrators combine two mechanisms: live auditing, in which auditors exercise BMDs and verify the printed ballots, and real-time tracking of spoiled-ballot rates as a statistical signal of switched-intent attacks. The core probability analysis in Section 4.1 computes detection confidence 1 - (1-p)^n as a function of per-ballot attack probability p and the number of audits n, with Table 1 giving sample values. Section 5.4 extends this with a Poisson-based analysis of spoilage rates, summarized in Table 2, to bound the margin by which malware could change an election before being detected. The paper responds to Stark's criticisms by arguing that the auditor can approximate a random voter model (Section 5.2), that down-ballot tampering is hard to detect generally (Section 5.3), and that larger jurisdictions benefit from the scaling properties of the audit math. The overall claim is that reasonable procedures give confidence in a BMD fleet while preserving accessibility.

Significance. If the analysis were fully supported, the paper would make a valuable contribution to the BMD security debate: it provides transparent, machine-checkable probability calculations, a concrete staffing example for Harris County, a reproducible Python appendix (Appendix A), and direct engagement with the opposing statistical arguments. The quantitative framework is a useful starting point for evaluating live auditing and spoilage monitoring. However, the central policy claim rests on empirical assumptions about voter behavior and audit sampling that the paper does not validate. The paper is honest about several of these limitations in footnotes and in the Baytown example, but those limitations are load-bearing for the abstract's fleet-wide confidence claim.

major comments (2)

[Section 5.2] The live-audit detection probabilities in Section 4.1 and Table 1 require that each audit be drawn from the same distribution as the voters an adversary would target. The paper asserts that a 'random voter' model reflecting all voters' preferences and machine-observable behaviors is a 'tractable engineering problem,' but no construction, validation data, or error analysis is provided. Without such a model, an adversary can target subpopulations that the audit script under-samples, such as slow users or accessibility-feature users, which the paper itself acknowledges in the introduction to Section 5. This is not an abstract worry: the Baytown example in Section 5.3 shows that focused down-ballot tampering is detected with only roughly 10-40% probability, directly contradicting any claim of reliable fleet-wide detection. Because the abstract's central assertion is that reasonable auditing procedures give confidence across a fleet of BMDs, this unvalidated distributional equivalence is a load-bearing gap.
[Section 5.4 and Table 2] The spoilage-rate analysis assumes a 1% expected background spoilage rate, but footnote 5 explicitly states that 'we won't know the actual rate of spoiled ballots on these systems' until large BMD deployments exist. Table 2's margin-of-victory bounds are computed as 200.0 * (poisson.ppf(0.95, mu) - mu) / (electionSize * detectionFraction), where mu = 0.01 * electionSize. If the true spoilage rate is higher than 1%, the Poisson 95% critical value shifts upward and the undetectable margin becomes larger; if it is lower, the opposite occurs. The paper's conclusion that a county-wide attack moving the margin by more than 0.3% would be detected is therefore not robust to the stated empirical uncertainty. This is a second load-bearing empirical input that the paper flags but does not quantify as a sensitivity analysis.

minor comments (5)

[Section 5.1] The sentence 'Stark's version is more precise in its counting, representing an error in the math of Section 4 of 1.71%' is ambiguous: the 1.71% is the difference between the simple binomial approximation and Stark's hypergeometric calculation, but it reads as if Stark's calculation is the one in error. Suggest rewording to state that the binomial approximation differs from Stark's exact hypergeometric result by 1.71 percentage points.
[Section 3] There is a typo in 'prior to the start of the elction' — 'elction' should be 'election.'
[Section 5.1] In the sentence 'We're interested in how far to the right we need to get before the malware wins with probably less than 5%,' the word 'probably' should be 'probability.'
[Appendix A] The Python snippet uses Python 2 print statements and string formatting; adding a note about compatibility or updating to Python 3 would make the appendix easier for readers to run.
[Section 7] The staffing estimate for Harris County relies on 2019 early-voting center counts because the 2018 numbers could not be found, as the author notes. This is acceptable as an illustration, but the sentence in the main text could more clearly separate the assumed and the verified numbers.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; the quantitative audit analyses are self-contained probability calculations with acknowledged empirical limitations.

full rationale

The paper's quantitative results are direct probability calculations from explicitly stated scenario parameters (Sections 4.1, 5.1, 5.4), not predictions fitted to the quantities they are said to establish. The live-audit detection probability 1-(1-p)^n follows directly from the model definition; the Baytown example computes the same combinatorial probability and reports modest 10-40% detection, which contradicts rather than presupposes the paper's optimistic conclusion. Table 2 is generated from the stated Poisson 95% confidence interval plus assumed spoilage and detection rates; no target margin of victory is used as input. The 'random voter' model in Section 5.2 is explicitly described as a 'tractable engineering problem' and is not constructed or validated, and footnote 5 concedes that the actual base spoilage rate is unknown until large BMD deployments exist. These are acknowledged empirical limitations, not circular derivations. The author's self-citations (VoteBox, Rice usability studies, Kohno et al.) are contextual or empirical support, and the paper even discounts some of these studies as uncontrolled ('Neither of these were controlled studies, so their observations are unreliable for predicting verification rates'), so the central claim does not reduce to those citations. No uniqueness theorem, fitted parameter renamed as prediction, or ansatz smuggled in by citation was found. The strongest claim is weaker than its own best-case numbers in the Baytown scenario, which is the opposite of circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's quantitative claims rest on four assumptions: the per-ballot attack probability p, the fraction of voters who notice and spoil tampered ballots, the background spoilage rate, and the ability to construct an audit distribution indistinguishable from real voters. The first three are illustrative values or uncertain empirical estimates; the last is asserted without demonstration. No new physical or conceptual entities are introduced.

free parameters (3)

Per-ballot attack probability p = 1%, 5%, 10%, 15% (illustrative values in Table 1)
The detection-rate table is computed as a function of p, the probability a malicious BMD flips a given ballot. The paper does not estimate p from data; it is a threat-model parameter chosen for illustration.
Voter detection fraction d = 10%, 30%, 50% in Table 2
The spoilage-signal analysis assumes a fixed fraction of tampered ballots are noticed and spoiled by voters. These values come from limited lab studies reported in Section 3.3, but are applied without confidence intervals and are contradicted by Stark's 10% assumption.
Expected background spoilage rate = 1%
The Poisson baseline in Appendix A sets expectedSpoliage = 0.01. The paper notes in footnote 5 that the true background spoilage rate for BMDs is unknown, making this an assumption rather than a measured quantity.

assumptions (4)

domain assumption Live audits sample the same distribution of voter preferences and behaviors as real voters, so a malicious BMD cannot distinguish auditors from targeted voters.
Invoked in Section 5.2's random voter model. The paper calls building this model a tractable engineering problem without constructing or validating it, yet all detection probabilities depend on it.
domain assumption Voters who notice errors will spoil their ballots, and poll workers will record and aggregate spoilage in real time.
Sections 3.3 and 5.4 rely on this to turn spoilage rates into a detection signal. The paper acknowledges in footnote 5 that the actual background spoilage rate is unknown.
domain assumption A detected malicious BMD triggers credible emergency procedures, including the possibility of re-running the election.
Section 6.1 assumes a governor can declare an emergency and re-run an election after a cyberattack, which is a legal and political assumption not established for this scenario.
domain assumption Spoiled ballot counts follow a Poisson distribution with a known mean.
Appendix A uses Poisson quantiles to compute Table 2. The paper does not justify the distribution, account for overdispersion, or estimate the mean from BMD deployment data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On the security of ballot marking devices." pith.science (2026). https://pith.science/paper/E6XURPPI

@misc{pith2026190801897,
  author       = {Pith},
  title        = {Pith review of: On the security of ballot marking devices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E6XURPPI}},
  note         = {Machine review of arXiv:1908.01897}
}

read the original abstract

A recent debate among election experts has considered whether electronic ballot marking devices (BMDs) have adequate security against the risks of malware. A malicious BMD might produce a printed ballot that disagrees with a voter's actual intent, with the hope that voters would be unlikely to detect this subterfuge. This essay considers how an election administrator can create reasonable auditing procedures to gain confidence that their fleet of BMDs is operating correctly, allowing voters to benefit from the usability and accessibility features of BMDs while the overall election still benefits from the same security and reliability properties we expect from hand-marked paper ballots.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages

[1]

B. Adida. Helios: Web-based open-audit voting. In 17th USENIX Security Symposium , San Jose, CA, July 2008

work page 2008
[2]

Adida, O

B. Adida, O. de Marneffe, O. Pereira, and J.-J. Quisquate r. Electing a university president using open-audit voting : Analysis of real-world use of Helios. In Electronic V oting Technology W orkshop / W orkshop on Trustworthy Elections (EVT/WOTE 2009), Montreal, Canada, Aug. 2009

work page 2009
[3]

A. W. Appel. Florida is the Florida of ballot-design mist akes, Nov. 2018. https://freedom-to-tinker.com/2018/11/14/ﬂorida-is- the-ﬂorida-of-ballot-design-mistakes/

work page 2018
[4]

permission to cheat

A. W. Appel. Serious design ﬂaw in ESS ExpressV ote touchs creen: “permission to cheat”, Sept. 2018. https://freedom-to-tinker.com/2018/09/14/serious-design-ﬂaw-in-ess-expressvote-touchscreen-permission- to-cheat/

work page 2018
[5]

A. W. Appel, R. DeMillo, and P . B. Stark. Ballot-marking d evices (BMDs) cannot assure the will of the voters, Apr. 2019 . https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3375755

work page 2019
[6]

J. Benaloh. Simple veriﬁable elections. In Proceedings of the USENIX/ACCURATE Electronic V oting Technology W orkshop (EVT ’06) , V ancouver, B.C., Canada, June 2006

work page 2006
[7]

J. Benaloh. Ballot casting assurance via voter-initiat ed poll station auditing. In Proceedings of the 2nd USENIX/ACCURATE Electronic V oting Technology W orkshop (EVT ’07), Boston, MA, Aug. 2007

work page 2007
[8]

T. Burt. Protecting democratic elections through secur e, veriﬁable voting, May 2019. https://blogs.microsoft.com/on-the-issues/2019/05/06/protecting-democratic-elections-through-secure-ve riﬁable-voting/

work page 2019

Show all 20 references

[9]

B. A. Campbell and M. D. Byrne. Now do voters notice review screen anomalies? A look at voting system usability. In Electronic V oting Technology/W orkshop on Trustworthy Elections 2009, Montreal, Canada, Aug. 2009

2009
[10]

DeMillo, R

R. DeMillo, R. Kadel, and M. Marks. What voters are asked to verify affects ballot veriﬁcation: A quantitative analysis of voters’ memories of their ballots , Apr. 2019. Originally posted Dec. 2018, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3292208

2019
[11]

Everett, K

S. Everett, K. Greene, M. Byrne, D. Wallach, K. Derr, D. S andler, and T. Torous. Is newer always better? The usability of electronic voting machines versus traditional methods. In Proceedings of CHI 2008, Florence, Italy, Apr. 2008

2008
[12]

S. P . Everett. The Usability of Electronic V oting Machines and How V otes Can Be Changed Without Detection . PhD thesis, Rice University, Houston, TX, 2007

2007
[13]

J. Gilbert. Ballot marking veriﬁcation protocol, 2019 . http://www.juangilbert.com/BallotMarkingVeriﬁcation Protocol.pdf

2019
[14]

T. Kaplan. Using Hurricane Sandy as a lesson for future e lections. In The New York Times , Nov. 2013. https://www.nytimes.com/2013/11/13/nyregion/lessons-from-hurricane-sandy-being-applied-to-election-pla nning.html

2013
[15]

Kohno, A

T. Kohno, A. Stubbleﬁeld, A. D. Rubin, and D. S. Wallach. Analysis of an electronic voting system. In Proc. of IEEE Symposium on Security & Privacy , Oakland, CA, 2004

2004
[16]

State Laws & Practices for the Emergency Management of Elect ions, Apr

National Association of Secretaries of State. State Laws & Practices for the Emergency Management of Elect ions, Apr. 2017. https://www.nass.org/sites/default/ﬁles/Election%20 Cybersecurity/report-NASS-emergency-preparedness-el ections-apr2017.pdf

2017
[17]

D. R. Sandler, K. Derr, and D. S. Wallach. V oteBox: A tamp er-evident, veriﬁable electronic voting system. In Proceedings of the 17th USENIX Security Symposium (Security ’08) , San Jose, CA, July 2008

2008
[18]

D. R. Sandler and D. S. Wallach. Casting votes in the Audi torium. In Proceedings of the 2nd USENIX/ACCURATE Electronic V oting Technology W orkshop (EVT ’07), Boston, MA, Aug. 2007

2007
[19]

P . B. Stark. Is parallel testing of ballot-marking devi ces practical?, 2019. https://www.stat.berkeley.edu/~stark/Preprints/bmd-p19.pdf

2019
[20]

\\ begin { t a b u l a r }{ rrr }

P . B. Stark. There is no reliable way to detect hacked bal lot-marking devices, Aug. 2019. https://arxiv.org/abs/1908.08144. 12 A Computing detectable spoiled ballot rates This short Python program shows how to generate Table 2. In my attempts to reproduce Stark’s numbers, he...

2019 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

B. Adida. Helios: Web-based open-audit voting. In 17th USENIX Security Symposium , San Jose, CA, July 2008

work page 2008

[2] [2]

Adida, O

B. Adida, O. de Marneffe, O. Pereira, and J.-J. Quisquate r. Electing a university president using open-audit voting : Analysis of real-world use of Helios. In Electronic V oting Technology W orkshop / W orkshop on Trustworthy Elections (EVT/WOTE 2009), Montreal, Canada, Aug. 2009

work page 2009

[3] [3]

A. W. Appel. Florida is the Florida of ballot-design mist akes, Nov. 2018. https://freedom-to-tinker.com/2018/11/14/ﬂorida-is- the-ﬂorida-of-ballot-design-mistakes/

work page 2018

[4] [4]

permission to cheat

A. W. Appel. Serious design ﬂaw in ESS ExpressV ote touchs creen: “permission to cheat”, Sept. 2018. https://freedom-to-tinker.com/2018/09/14/serious-design-ﬂaw-in-ess-expressvote-touchscreen-permission- to-cheat/

work page 2018

[5] [5]

A. W. Appel, R. DeMillo, and P . B. Stark. Ballot-marking d evices (BMDs) cannot assure the will of the voters, Apr. 2019 . https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3375755

work page 2019

[6] [6]

J. Benaloh. Simple veriﬁable elections. In Proceedings of the USENIX/ACCURATE Electronic V oting Technology W orkshop (EVT ’06) , V ancouver, B.C., Canada, June 2006

work page 2006

[7] [7]

J. Benaloh. Ballot casting assurance via voter-initiat ed poll station auditing. In Proceedings of the 2nd USENIX/ACCURATE Electronic V oting Technology W orkshop (EVT ’07), Boston, MA, Aug. 2007

work page 2007

[8] [8]

T. Burt. Protecting democratic elections through secur e, veriﬁable voting, May 2019. https://blogs.microsoft.com/on-the-issues/2019/05/06/protecting-democratic-elections-through-secure-ve riﬁable-voting/

work page 2019