REVIEW 3 major objections 6 minor 1 cited by
They may look and look, yet not see: BMDs cannot be tested adequately
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The standard tests for ballot-marking devices cannot detect outcome-altering failures: even optimistic calculations require observing more than a million voters per jurisdiction to catch 5% tampering with 95% confidence.
desk verdict Strong qualitative case against BMD testing, but the headline million-voter lower bound rests on a flawed implication and needs repair before the numbers are used. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument is carried by four devices. First, a combinatorial model of a voting transaction: the paper's Table 1 catalogs roughly twenty variables — number of contests, candidates, languages, time of day, prior machine load, undervotes, changed selections, font size, audio tempo, and more — yielding on the order of $10^6$ to $10^{47}$ possible transaction types, and the quantitative analysis adopts the unrealistically small optimistic support of $S = 6.14\times 10^6$ points. Second, an adversarial-game framing in which Mallory knows Pat's testing strategy and hides altered votes in transaction types that are rare in testing but common enough in real voting to flip a margin. Third, a minimax lower bound on the $\ell^1$ error of any estimator of a discrete distribution on $S$ points, which converts 'Pat must know the voter-transaction distribution' into 'Pat must watch millions of voters.' Fourth, an oracle or 'shoulder-surfing' bound that removes model-building entirely: even with a perfect oracle, catching 15 altered ballots among 2,980 voters at 95% confidence requires checking 540 printouts, and protecting a 1%-margin contest with 13 tests per machine per day needs at least 6,580 voters and 47 machines. For passive testing, the machinery is a Poisson comparison of spoiled-ballot counts between benign and hacked operation, with the share of voters who notice errors (7% or 25%) taken from voter-behavior studies.
What would settle it
Equip a few polling places with BMDs that log every transaction detail (timestamp, language, font size, audio settings, timing per selection, changes of mind) for a full election day, and count how many distinct transaction patterns actually occur and how concentrated they are: if the effective support is small enough that a distribution estimate within $\ell^1$ error of 10% could be built from tens of thousands of voters instead of over a million, the worst-case premise fails and the required testing budgets would shrink accordingly.
Extended reading notes
Core claim
On the paper's own terms, the central claim is a statistical impossibility result: L&A testing, parallel testing, and passive testing cannot reliably detect outcome-altering BMD failures. The paper frames testing as an adversarial game between Mallory, who alters votes only on hard-to-probe transactions, and Pat, who must randomly sample from the actual distribution of voter interactions; because that distribution has a support of $S = 6.14\times 10^6$ distinguishable transaction types even under optimistic counting, minimax lower bounds on $\ell^1$ estimation error imply that a training sample of at least 1.082 million voters is needed to have a 95% chance of detecting tampering with 5% of transactions — more than the total turnout of 99.8% of jurisdictions in 2018 — and roughly 3.9 million voters to reach 99% confidence against 0.5% tampering. Passive testing fares no better: even if spoiled ballots followed a Poisson distribution with exactly known rate, detecting a 1% margin change at 5% false-positive and false-negative tolerances would require a contest with about 451,000 voters, and roughly 909,000 at 1% tolerances. The paper adds that attackers can target voters least able or least likely to check the printout — blind voters, audio-interface users, voters using the Autocast feature, voters reading non-English ballots — so real detection rates are lower than the optimistic figures, and that because the paper trail itself is untrustworthy, the only remedy once tampering is suspected is a new election.
Load-bearing premise
The load-bearing premise is that the distribution of voter interactions must be treated as completely arbitrary — any pattern at all, with no recurring structure that a few thousand observations could capture — so that guarding the worst case is the only honest way to estimate it.
Editorial extensions
If this is right
- Clean results from L&A, parallel, or passive testing would no longer be interpretable as assurance: the tests could miss outcome-altering problems by many orders of magnitude.
- Most contests are beyond the reach of these tests as a matter of arithmetic — for 2018 turnout, in 73% of U.S. states fewer than half of counties had enough voters for passive testing to work even in county-wide contests under ideal Poisson assumptions.
- Because true margins are unknown until after the election and the only remedy for detected tampering is a new election, testing budgets cannot be calibrated in advance to protect close races.
- Minimizing the number of voters who cast ballots on BMDs becomes the recommended administrative measure, with BMDs reserved mainly for voters who need them for accessibility.
Reading between the lines
- Editorial inference: the crux the paper leaves open is empirical — if real voter interactions cluster into a few typical patterns rather than filling the worst-case space, the required training samples could shrink by orders of magnitude, and a feasible sampling design might exist.
- Editorial inference: the same counting-plus-minimax geometry applies to any device whose tampering must be detected from black-box tests over a combinatorial input space, such as ATMs or self-checkout kiosks, though elections add the special constraint of secret, unlinkable voter inputs.
- Editorial inference: the paper's own discussion of cryptographic challenges suggests the constructive escape route is to change the voter's evidence relationship to the machine, so that a voter can prove a malfunction to a third party, turning an impossible testing problem into a verifiability problem.
- Editorial inference: passive spoilage monitoring may still be worth keeping as a cheap, opportunistic alarm that triggers recounts or inspections in close contests, even though it cannot by itself provide the assurance the paper demands.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript argues that three proposed methods for testing ballot-marking devices (BMDs)—pre-election logic and accuracy testing, parallel/live testing, and passive monitoring of spoiled-ballot rates—cannot reliably detect outcome-altering failures or attacks. The authors formalize testing as an adversarial game between Mallory and Pat, quantify the number of possible voting transactions (Table 1), derive an oracle bound on the number of tests needed even with a perfect model of voter behavior (Section 5.3), use minimax discrete-distribution estimation to lower-bound the training sample needed to build such a model (Section 5.4, Table 4), and compute passive-testing sample sizes under optimistic Poisson assumptions (Section 4, Tables 2 and 3). They conclude that, in practice, these methods require more observations and tests than most U.S. jurisdictions can provide, and they recommend minimizing BMD use.
Significance. The paper addresses a significant and timely policy question. If its conclusions hold, they imply that current BMD testing practices cannot provide persuasive evidence that printed votes match voter selections, which would strengthen the case for hand-marked paper ballots and risk-limiting audits. The passive-testing calculations in Section 4 are internally consistent with their stated Poisson model; the oracle bound in Section 5.3 is correct; and the paper makes its software publicly available. The main quantitative claims about model building, however, rest on a minimax argument that has a logical gap, so the strongest numbers in the abstract and Table 4 should be treated as unproved until that step is repaired.
major comments (3)
- [Section 5.4, paragraphs after Eq. (7)] The assertion that ||Phat - P||_1 >= 2r implies there is a set tau with P(tau) = r and Phat(tau) = 0 is false in general. For example, with S=2, P=(0.9,0.1), Phat=(0.1,0.9) and r=0.5, the L1 distance is 1.6 >= 1.0, but every subset has positive Phat mass. L1 error only guarantees total variation at least r, i.e., a set with |P(A)-Phat(A)| >= r, not a set that Phat assigns probability zero. Therefore the event {L1 error >= 2r} is not a subset of the event that an undetectable attack set exists, and the Cantelli step bounds the wrong event. Table 4's training-sample lower bounds do not follow from the stated proof. The authors should replace this step with a direct lower bound on the probability that some outcome-altering set has zero estimated mass (or an equivalent support-coverage argument).
- [Section 5.4, first paragraph and Table 4] The assumed support size S is inconsistent. The text says the support of P contains more than 10^8 points, then assumes P is concentrated on 1% of the smaller estimate and sets S=1.89e6; 1% of the 'optimistic' Table 1 total of 6.14e6 would be 6.14e4. Table 4 then uses S=6.14e6. Because the displayed lower bounds scale with S, the million-voter figures in the abstract and Section 7 are not reproducible without knowing which value is intended. Please reconcile these numbers and recompute the table.
- [Section 5.4, Eq. (6)] The minimax lower bound is over the entire simplex M_S of distributions on S transaction types. This establishes that no estimator works uniformly over that class, but the paper's practical conclusion—that building an accurate model requires monitoring more than a million voters in each jurisdiction—assumes the actual distribution of voting transactions is as hard as the worst case. If real transactions follow a product form or a low-dimensional parametric family, estimation would require far fewer observations. The authors should either justify the worst-case class as the appropriate model for adversarial election security, or state the quantitative conclusion as conditional on that modeling assumption.
minor comments (6)
- [Table 1] The formatting of entries such as '2 3 220' and '6 .14 x 106' suggests lost superscripts, so the total cannot be verified as printed; please ensure the table renders correctly.
- [Section 5.4, first paragraph] The phrase 'more than 10^8 points' conflicts with Table 1's 'optimistic' total of 6.14e6; clarify which support size is meant.
- [Section 5.2] The sentence 'The 13 hours of tests already described cannot not reveal such hacks' contains a double negative; it should probably read 'cannot reveal'.
- [Section 5.3] The phrase 'That is more almost triple the median turnout' should be 'That is almost triple the median turnout'.
- [Section 4.1] The cross-reference 'This section and Sections 4.1, 4.1' appears to refer to unnumbered subsections; correct the numbering.
- [References] Several references are listed as 'in press' (Appel et al.; Appel and Stark; Bernhard et al.); please update these if the publication status has changed.
Circularity Check
No significant circularity: the central quantitative claims derive from external minimax bounds and explicit assumptions, not from fitted outputs or self-citation chains.
full rationale
The paper's load-bearing numeric results are not circular. Section 5.4 invokes the external Han--Jiao--Weissman minimax lower bound for discrete distribution estimation (Eq. 7) and then applies a Cantelli-style lemma to convert expected L1 error into a training-sample lower bound; the million-voter figures in Table 4 are the result of plugging the stated support size S and confidence/error requirements into that external theorem. The passive-testing Tables 2 and 3 are explicit Poisson power calculations under stated assumptions about spoilage rates, with the 7%/25% detection rates attributed to the external Bernhard et al. study. The oracle bound in Section 5.3 is an elementary hypergeometric calculation. Self-citations (Stark 2019; Appel, DeMillo, and Stark 2020; Appel and Stark 2020) appear only as contextual support for the premises that BMDs are hackable and that voters cannot prove malfunctions; those premises are also supported by external sources and incident reports, and the testing-infeasibility conclusion is not obtained by citing them as a theorem. The possible logical weakness in Section 5.4 (L1 error greater than 2r does not by itself imply the existence of a zero-probability attack set) is a correctness concern, not a circularity: it does not make the conclusion equivalent to an input or a fitted parameter. No equation in the paper reduces to its own input by construction.
Assumptions & free parameters
free parameters (5)
- S, support size of the voting-transaction distribution =
6.14e6 (Table 4) or 1.89e6 (text)
- Detection rate for passive testing (fraction of affected voters who spoil) =
7% and 25%
- Benign spoilage base rate in passive testing =
0.5%, 1%, 1.5%
- False positive and false negative rate targets =
5% and 1%
- Contest margin scenarios =
1%, 2%, 3%, 4%, 5%
assumptions (7)
- domain assumption Mallory knows Pat's testing strategy before the election and can use past voter behavior data.
- domain assumption The space of voting transactions has at least the cardinality given in Table 1 (for example 6.14e6 under optimistic assumptions).
- ad hoc to paper The true distribution P is concentrated on 1% of the optimistic support size (S = 1.89e6 in text).
- domain assumption Voting transactions are independent, identically distributed draws from an unknown distribution P over a finite support of size S.
- domain assumption For passive testing, spoiled ballots follow a Poisson distribution with exactly known benign rate, and the attack increases the rate by (margin/2) times the detection rate.
- standard math Han, Jiao, and Weissman (2015) minimax lower bound for discrete distribution estimation is valid and applicable.
- standard math Cantelli's inequality and the variance bound Var(||Phat-P||1) <= 1.
Cite this review
Pith. "Pith review of They may look and look, yet not see: BMDs cannot be tested adequately." pith.science (2026). https://pith.science/paper/QKSTSK7Y
@misc{pith2026190808144,
author = {Pith},
title = {Pith review of: They may look and look, yet not see: BMDs cannot be tested adequately},
year = {2026},
howpublished = {\url{https://pith.science/paper/QKSTSK7Y}},
note = {Machine review of arXiv:1908.08144}
}
read the original abstract
Bugs, misconfiguration, and malware can cause ballot-marking devices (BMDs) to print incorrect votes. Several approaches to testing BMDs have been proposed. In logic and accuracy testing (LAT) and parallel or live testing, auditors input known test votes into the BMD and check the printout. Passive testing monitors the rate of "spoiled" BMD printout, on the theory that if BMDs malfunction, the rate will increase noticeably. We show that these approaches cannot reliably detect outcome-altering problems, because: (i) The number of possible interactions with BMDs is enormous, so testing interactions uniformly at random is hopeless. (ii) To probe the space of interactions intelligently requires an accurate model of voter behavior, but because the space of interactions is so large, building an accurate model requires observing a huge number of voters in every jurisdiction in every election--more voters than there are in most jurisdictions. (iii) Even with a perfect model of voter behavior, the number of tests needed exceeds the number of voters in most jurisdictions. (iv) An attacker can target interactions that are expensive to test, e.g., because they involve voting slowly; or interactions for which tampering is less likely to be noticed, e.g., because the voter uses the audio interface. (v) Whether BMDs misbehave or not, the distribution of spoiled ballots is unknown and varies by election and possibly by ballot style: historical data do not help much. Hence, there is no way to calibrate a threshold for passive testing, e.g., to guarantee at least a 95% chance of noticing that 5% of the votes were altered, with at most a 5% false alarm rate. (vi) Even if the distribution of spoiled ballots were known to be Poisson, the vast majority of jurisdictions do not have enough voters for passive testing to have a large chance of detecting problems but only a small chance of false alarms.
Figures
Forward citations
Cited by 1 Pith paper
-
On the security of ballot marking devices
The paper argues that live auditing plus spoiled ballot tracking can give election officials a statistically measurable chance of catching malicious ballot marking devices, especially in large jurisdictions.
Reference graph
Works this paper leans on
-
[5]
EVT’06. USA: USENIX Association. Bernhard, M., A. McDonald, H. Meng, J. Hwa, N. Bajaj, K. Chang, and J. A. Halderman
-
[2020]
Can Voters Detect Malicious Manipulation of Ballot Marking Devices?
“Can Voters Detect Malicious Manipulation of Ballot Marking Devices?” In 41st IEEE Symposium on Security and Privacy , (in press). IEEE. California Secretary of State. 2016. “June 7, 2016 - Presidential Primary Election Voter Participation Statistics by County.”https://elections.cdn.sos.ca.gov/sov/2016-primary/ 03-voter-participation-stats-by-county.pdf ....
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.