{"id":"1ecf4897-486e-40ee-825f-925becb1c5b0","arxiv_id":"1908.08144","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Logic and accuracy, parallel, and passive testing cannot reliably detect BMDs that alter election outcomes, because the space of possible voter-machine interactions is enormous and the required sample sizes exceed typical jurisdiction turnout.","lead":"Ballot-marking devices can print ballots that do not match what voters selected, and this paper argues that no practical testing scheme can reliably catch such errors. It shows that effective testing would require monitoring more voters than most U.S. jurisdictions have, so hand-marked paper ballots are the safer choice.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5.4's minimax argument conflates large L1 error with existence of a zero-probability attack set; the million-voter training-sample numbers rest on an unproved implication.","rationale":"The reader's weakest_assumption was that the worst-case simplex M_S may be too pessimistic because real voting transactions might have low-dimensional structure. That is a model-choice concern, but it is not the most load-bearing issue: even granting M_S, the minimax derivation in Section 5.4 has an internal gap. The claim that L1 error exceeding 2r implies the existence of a zero-probability attack set is not a theorem; L1 error is total variation, which measures how much probability mass can be shifted among sets, not whether an entire P-mass-r set is assigned zero probability by P_hat. If P_hat has full support, no such set exists, so the subsequent probability calculation and Table 4 lower bounds are unsupported. This is more damaging than the low-dimensional critique because it does not rely on any empirical assumption about voter behavior; it is a logical discontinuity in the proof as written. The reader did flag the general area in the rationale, so agreement is partial rather than full. I would not move the verdict to REJECT: the paper contains independent support for a weaker but still policy-relevant conclusion. The oracle bound in Section 5.3 shows that even perfect knowledge of P requires testing roughly 18% of voters to detect a 1% margin change, and the combinatorial argument in Section 5.2 and Table 1 shows that the transaction space is enormous and attacks can be conditioned on mutually exclusive events. The passive-testing analysis, while optimistic in its Poisson assumptions, is internally self-consistent and shows that many jurisdictions are too small for meaningful passive testing. Thus the qualitative claim that current testing approaches cannot reliably detect outcome-altering BMD failures is plausible and partially supported. But the specific numeric claim that training sets must exceed a million voters in each jurisdiction is the headline quantitative result, and it currently rests on the faulty L1-to-zero-probability step. The verdict should remain CONDITIONAL, contingent on repairing or replacing that step; hence UNCHANGED relative to the reader's verdict.","tokens_in":19681,"tokens_out":8320,"duration_ms":81099,"concrete_test":"Analytically or computationally test the key implication: exhibit P and P_hat with ||P - P_hat||_1 >= 2r for r=0.05 (e.g., S=2, P=(0.9,0.1), P_hat=(0.1,0.9), or P_hat with strictly positive support) and check whether there exists a subset A with P(A)=r and P_hat(A)=0. If no such subset exists, as in the example, the paper's \"zero chance of detection\" step is false. To test the numerical impact, replace the zero-detection assumption with the real detection probability for the worst attack set A of P-mass r, namely 1 - (1 - P_hat(A))^T under T tests drawn from P_hat, and recompute the training sample sizes in Table 4. If the required n is materially smaller than the reported million-voter bounds (e.g., by an order of magnitude), the headline quantitative claim needs revision; if it remains comparable, the conclusion survives despite the faulty step.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The quantitative core of the parallel-testing claim is the lower bound in Section 5.4. The paper asserts that if ||P_hat - P||_1 >= 2r then \"there can be a set tau... P(tau)=r but P_hat(tau)=0\", so tests drawn from P_hat have zero chance of sampling tau; hence if Pr{||P_hat - P||_1 >= 2r} > alpha, even infinite testing cannot guarantee 1-alpha detection. This implication is false in general. L1 distance >= 2r only guarantees total variation >= r, i.e., sets A with |P(A)-P_hat(A)| >= r; it does not guarantee that any such A satisfies P_hat(A)=0. Example: S=2, P=(0.9,0.1), P_hat=(0.1,0.9), r=0.5 gives L1 distance 1.6 >= 1.0, yet every subset has positive P_hat mass, so no attack set is missed with probability zero. The event {L1 error >= 2r} is broader than the event \"there exists an undetectable attack set\"; bounding the former by alpha does not bound the latter. The Han-Jiao-Weissman theorem bounds expected L1 error, not the probability of a zero-estimated-mass set. Table 4's figures (e.g., 1.08 million training voters for 95% detection of 5% tampering) therefore do not follow from the stated proof. The qualitative conclusion may be salvageable through the oracle bound and the combinatorial transaction-space argument, but the strongest quantitative support needs a separate argument relating L1 error (or some other estimation error) to the probability that an outcome-altering attack set is sampled with probability zero under P_hat. An internal inconsistency in S (1.89e6 in the text vs 6.14e6 in Table 4) compounds the issue but is secondary.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript argues that three proposed methods for testing ballot-marking devices (BMDs)—pre-election logic and accuracy testing, parallel/live testing, and passive monitoring of spoiled-ballot rates—cannot reliably detect outcome-altering failures or attacks. The authors formalize testing as an adversarial game between Mallory and Pat, quantify the number of possible voting transactions (Table 1), derive an oracle bound on the number of tests needed even with a perfect model of voter behavior (Section 5.3), use minimax discrete-distribution estimation to lower-bound the training sample needed to build such a model (Section 5.4, Table 4), and compute passive-testing sample sizes under optimistic Poisson assumptions (Section 4, Tables 2 and 3). They conclude that, in practice, these methods require more observations and tests than most U.S. jurisdictions can provide, and they recommend minimizing BMD use.","tokens_in":20114,"tokens_out":15578,"duration_ms":145691,"significance":"The paper addresses a significant and timely policy question. If its conclusions hold, they imply that current BMD testing practices cannot provide persuasive evidence that printed votes match voter selections, which would strengthen the case for hand-marked paper ballots and risk-limiting audits. The passive-testing calculations in Section 4 are internally consistent with their stated Poisson model; the oracle bound in Section 5.3 is correct; and the paper makes its software publicly available. The main quantitative claims about model building, however, rest on a minimax argument that has a logical gap, so the strongest numbers in the abstract and Table 4 should be treated as unproved until that step is repaired.","major_comments":[{"comment":"The assertion that ||Phat - P||_1 >= 2r implies there is a set tau with P(tau) = r and Phat(tau) = 0 is false in general. For example, with S=2, P=(0.9,0.1), Phat=(0.1,0.9) and r=0.5, the L1 distance is 1.6 >= 1.0, but every subset has positive Phat mass. L1 error only guarantees total variation at least r, i.e., a set with |P(A)-Phat(A)| >= r, not a set that Phat assigns probability zero. Therefore the event {L1 error >= 2r} is not a subset of the event that an undetectable attack set exists, and the Cantelli step bounds the wrong event. Table 4's training-sample lower bounds do not follow from the stated proof. The authors should replace this step with a direct lower bound on the probability that some outcome-altering set has zero estimated mass (or an equivalent support-coverage argument).","section":"Section 5.4, paragraphs after Eq. (7)"},{"comment":"The assumed support size S is inconsistent. The text says the support of P contains more than 10^8 points, then assumes P is concentrated on 1% of the smaller estimate and sets S=1.89e6; 1% of the 'optimistic' Table 1 total of 6.14e6 would be 6.14e4. Table 4 then uses S=6.14e6. Because the displayed lower bounds scale with S, the million-voter figures in the abstract and Section 7 are not reproducible without knowing which value is intended. Please reconcile these numbers and recompute the table.","section":"Section 5.4, first paragraph and Table 4"},{"comment":"The minimax lower bound is over the entire simplex M_S of distributions on S transaction types. This establishes that no estimator works uniformly over that class, but the paper's practical conclusion—that building an accurate model requires monitoring more than a million voters in each jurisdiction—assumes the actual distribution of voting transactions is as hard as the worst case. If real transactions follow a product form or a low-dimensional parametric family, estimation would require far fewer observations. The authors should either justify the worst-case class as the appropriate model for adversarial election security, or state the quantitative conclusion as conditional on that modeling assumption.","section":"Section 5.4, Eq. (6)"}],"minor_comments":[{"comment":"The formatting of entries such as '2 3 220' and '6 .14 x 106' suggests lost superscripts, so the total cannot be verified as printed; please ensure the table renders correctly.","section":"Table 1"},{"comment":"The phrase 'more than 10^8 points' conflicts with Table 1's 'optimistic' total of 6.14e6; clarify which support size is meant.","section":"Section 5.4, first paragraph"},{"comment":"The sentence 'The 13 hours of tests already described cannot not reveal such hacks' contains a double negative; it should probably read 'cannot reveal'.","section":"Section 5.2"},{"comment":"The phrase 'That is more almost triple the median turnout' should be 'That is almost triple the median turnout'.","section":"Section 5.3"},{"comment":"The cross-reference 'This section and Sections 4.1, 4.1' appears to refer to unnumbered subsections; correct the numbering.","section":"Section 4.1"},{"comment":"Several references are listed as 'in press' (Appel et al.; Appel and Stark; Bernhard et al.); please update these if the publication status has changed.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The passive-testing section is the most defensible part and could support the policy conclusion even if the minimax argument in Section 5.4 is replaced. The L1-to-zero-support implication is not a presentation issue; it is the load-bearing step for Table 4, so I could not recommend acceptance in the current form. I would encourage the authors to either repair the minimax step with a direct support-coverage lower bound or clearly mark the million-voter estimates as conditional on a worst-case modeling assumption."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Stan, here's my take on 1908.08144. The paper's qualitative case is strong: because the space of BMD interactions is enormous and the attacker can target rare or untestable transaction types, L&A and parallel testing can't give meaningful assurance. The passive-testing analysis is also careful: the Poisson power tables are internally consistent, and the conclusion that most jurisdictions lack enough voters for 1% margins is hard to argue with. The oracle 'shoulder-surfing' bound is correct and already shows that even perfect knowledge of voter behavior demands testing a large fraction of ballots.\n\nThe soft spot is the minimax lower bound in Section 5.4, and it's a real one. The paper claims that if ||Phat - P||_1 >= 2r then there exists a set tau with P(tau)=r and Phat(tau)=0, so no test sampled from Phat can hit it. That implication is false. L1 distance only implies a set where the probability difference is at least r; it doesn't imply Phat assigns zero mass. Simple two-point example: P=(0.9,0.1), Phat=(0.1,0.9), r=0.5 has L1=1.6 >= 1.0, but every subset has positive Phat mass. So the event {L1 >= 2r} is broader than 'there is a zero-probability attack set,' and bounding the former doesn't bound the latter. The table's 1.08-million-voter numbers don't follow from the stated proof. The S inconsistency (1.89e6 in text vs 6.14e6 in the table) is secondary but should be fixed.\n\nThat said, this is one flawed step in an otherwise serious paper. The passive-testing impossibility and the combinatorial arguments stand on their own. The conclusion may be salvageable by a correct missing-mass or coupon-collector argument, but the authors need to supply it. I'd send this to peer review; the referee should push hard on Section 5.4. The paper is worth reading for anyone working on election security, but don't cite the million-voter parallel-testing bound until the proof is repaired.","headline":"Strong qualitative case against BMD testing, but the headline million-voter lower bound rests on a flawed implication and needs repair before the numbers are used.","tokens_in":20627,"tokens_out":3305,"would_cite":true,"duration_ms":30274,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62C20","62G05"],"pacs":[],"model":"deepseek-v4-flash","headline":"The standard tests for ballot-marking devices cannot detect outcome-altering failures: even optimistic calculations require observing more than a million voters per jurisdiction to catch 5% tampering with 95% confidence.","keywords":["elections","ballot-marking devices","logic and accuracy testing","parallel testing","passive testing","minimax estimation","spoiled-ballot rate","voter behavior model"],"falsifier":"Equip a few polling places with BMDs that log every transaction detail (timestamp, language, font size, audio settings, timing per selection, changes of mind) for a full election day, and count how many distinct transaction patterns actually occur and how concentrated they are: if the effective support is small enough that a distribution estimate within $\\ell^1$ error of 10% could be built from tens of thousands of voters instead of over a million, the worst-case premise fails and the required testing budgets would shrink accordingly.","tokens_in":19459,"feed_emoji":"🗳️","tokens_out":17527,"duration_ms":147823,"temperature":0.7,"pith_summary":"The paper claims that the three standard ways of checking whether ballot-marking devices (BMDs) printed what voters chose — pre-election logic and accuracy tests, parallel or 'live' tests on election day, and passive monitoring of spoiled-ballot rates — cannot reliably detect problems big enough to change an election outcome. The obstacle is combinatorial: a voting 'transaction' includes not just the selections but dozens of contextual variables (language, font size, audio settings, time taken, changes of mind, time of day), so the space of possible interactions is astronomically large and testing it uniformly is hopeless. Testing intelligently requires knowing how real voters interact with the machines, but the paper shows that estimating that distribution accurately enough would require observing well over a million voters per jurisdiction, under optimistic assumptions. Most U.S. jurisdictions have far fewer voters than that — the 2018 median county turnout was just under 3,000 — so the required samples are unattainable, and even a perfect model of voter behavior would leave most contests too small to protect. If the paper is right, no feasible testing program can provide persuasive public evidence that BMD-based election outcomes were not altered, and minimizing the number of ballots cast on BMDs is the prudent course.","feed_headline":"Testing cannot verify ballot-marking devices","feed_subtitle":"Logic, parallel, and spoiled-ballot tests need multi-million-voter samples most jurisdictions can't reach.","key_machinery":"The argument is carried by four devices. First, a combinatorial model of a voting transaction: the paper's Table 1 catalogs roughly twenty variables — number of contests, candidates, languages, time of day, prior machine load, undervotes, changed selections, font size, audio tempo, and more — yielding on the order of $10^6$ to $10^{47}$ possible transaction types, and the quantitative analysis adopts the unrealistically small optimistic support of $S = 6.14\\times 10^6$ points. Second, an adversarial-game framing in which Mallory knows Pat's testing strategy and hides altered votes in transaction types that are rare in testing but common enough in real voting to flip a margin. Third, a minimax lower bound on the $\\ell^1$ error of any estimator of a discrete distribution on $S$ points, which converts 'Pat must know the voter-transaction distribution' into 'Pat must watch millions of voters.' Fourth, an oracle or 'shoulder-surfing' bound that removes model-building entirely: even with a perfect oracle, catching 15 altered ballots among 2,980 voters at 95% confidence requires checking 540 printouts, and protecting a 1%-margin contest with 13 tests per machine per day needs at least 6,580 voters and 47 machines. For passive testing, the machinery is a Poisson comparison of spoiled-ballot counts between benign and hacked operation, with the share of voters who notice errors (7% or 25%) taken from voter-behavior studies.","core_discovery":"On the paper's own terms, the central claim is a statistical impossibility result: L&A testing, parallel testing, and passive testing cannot reliably detect outcome-altering BMD failures. The paper frames testing as an adversarial game between Mallory, who alters votes only on hard-to-probe transactions, and Pat, who must randomly sample from the actual distribution of voter interactions; because that distribution has a support of $S = 6.14\\times 10^6$ distinguishable transaction types even under optimistic counting, minimax lower bounds on $\\ell^1$ estimation error imply that a training sample of at least 1.082 million voters is needed to have a 95% chance of detecting tampering with 5% of transactions — more than the total turnout of 99.8% of jurisdictions in 2018 — and roughly 3.9 million voters to reach 99% confidence against 0.5% tampering. Passive testing fares no better: even if spoiled ballots followed a Poisson distribution with exactly known rate, detecting a 1% margin change at 5% false-positive and false-negative tolerances would require a contest with about 451,000 voters, and roughly 909,000 at 1% tolerances. The paper adds that attackers can target voters least able or least likely to check the printout — blind voters, audio-interface users, voters using the Autocast feature, voters reading non-English ballots — so real detection rates are lower than the optimistic figures, and that because the paper trail itself is untrustworthy, the only remedy once tampering is suspected is a new election.","pith_inferences":["Editorial inference: the crux the paper leaves open is empirical — if real voter interactions cluster into a few typical patterns rather than filling the worst-case space, the required training samples could shrink by orders of magnitude, and a feasible sampling design might exist.","Editorial inference: the same counting-plus-minimax geometry applies to any device whose tampering must be detected from black-box tests over a combinatorial input space, such as ATMs or self-checkout kiosks, though elections add the special constraint of secret, unlinkable voter inputs.","Editorial inference: the paper's own discussion of cryptographic challenges suggests the constructive escape route is to change the voter's evidence relationship to the machine, so that a voter can prove a malfunction to a third party, turning an impossible testing problem into a verifiability problem.","Editorial inference: passive spoilage monitoring may still be worth keeping as a cheap, opportunistic alarm that triggers recounts or inspections in close contests, even though it cannot by itself provide the assurance the paper demands."],"forward_implications":["Clean results from L&A, parallel, or passive testing would no longer be interpretable as assurance: the tests could miss outcome-altering problems by many orders of magnitude.","Most contests are beyond the reach of these tests as a matter of arithmetic — for 2018 turnout, in 73% of U.S. states fewer than half of counties had enough voters for passive testing to work even in county-wide contests under ideal Poisson assumptions.","Because true margins are unknown until after the election and the only remedy for detected tampering is a new election, testing budgets cannot be calibrated in advance to protect close races.","Minimizing the number of voters who cast ballots on BMDs becomes the recommended administrative measure, with BMDs reserved mainly for voters who need them for accessibility."],"supporting_citations":[{"why":"Supplies the minimax lower bound on the $\\ell^1$ error of estimating a discrete distribution on $S$ points, from which the multi-million-voter training-sample sizes are derived.","marker":"Han, Jiao, and Weissman 2015"},{"why":"Supplies the measured 7% rate at which voters notice and spoil erroneously printed ballots, the key input to the passive-testing power calculations.","marker":"Bernhard et al. 2020"},{"why":"Supplies the 2018 turnout data (median county turnout 2,980 voters; two-thirds of jurisdictions below 43,000) that show required sample sizes exceed available voters.","marker":"U.S. Election Assistance Commission 2018"},{"why":"The proposal to test BMDs by monitoring spoiled-ballot rates, which the paper argues cannot be calibrated.","marker":"Wallach 2019"},{"why":"The proposal to run parallel tests on a fixed schedule during the election, which the paper shows malware can evade.","marker":"Gilbert 2019"},{"why":"Establishes the premise that BMD printout is a hackable paper trail and that voters cannot prove a machine malfunctioned.","marker":"A. W. Appel, DeMillo, and Stark 2020"},{"why":"Provides the notion of software independence used to classify BMD elections as not strongly software independent, so the only remedy is a new election.","marker":"Rivest and Wack 2006"}],"fun_headline_variants":["BMD testing: a statistical impossibility","Why BMD verification fails at scale","No way to reliably test ballot-marking devices","Ballot-marking devices: untestable by design"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the distribution of voter interactions must be treated as completely arbitrary — any pattern at all, with no recurring structure that a few thousand observations could capture — so that guarding the worst case is the only honest way to estimate it.","fun_headline_variants_meta":{"raw":{"variants":["BMD testing: a statistical impossibility","Why BMD verification fails at scale","No way to reliably test ballot-marking devices","Ballot-marking devices: untestable by design"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000464,"raw_usage":{"total_tokens":2463,"prompt_tokens":1234,"completion_tokens":1229,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":850,"completion_tokens_details":{"reasoning_tokens":1172}},"tokens_in":850,"tokens_out":1229,"duration_ms":11979,"temperature":1.0,"reasoning_tokens":1172,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:49:39.148719+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Equip a few polling places with BMDs that log every transaction detail (timestamp, language, font size, audio settings, timing per selection, changes of mind) for a full election day, and count how many distinct transaction patterns actually occur and how concentrated they are: if the effective support is small enough that a distribution estimate within $\\ell^1$ error of 10% could be built from tens of thousands of voters instead of over a million, the worst-case premise fails and the required testing budgets would shrink accordingly.","supporting_citations":[],"review_version":1}