{"id":"609ff671-6da2-4bd2-900e-b26b943178e2","arxiv_id":"1908.01897","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper argues that live auditing plus spoiled ballot tracking can give election officials a statistically measurable chance of catching malicious ballot marking devices, especially in large jurisdictions.","lead":"This paper argues that ballot marking devices can be made adequately secure through live audits during voting and real-time tracking of spoiled ballots. It is a readable contribution to an active debate over whether such devices can protect voters' intent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The live-audit guarantee depends on an unvalidated 'random voter' model; if adversaries target subpopulations the audit script under-samples, the Table 1 detection probabilities are not achieved.","rationale":"The reader's weakest-assumption analysis correctly identifies the random-voter model as the critical dependency. The Section 4 mathematics, 1-(1-p)^n, is correct under its sampling model, and the Appendix A code does reproduce Table 2, so the statistical mechanics are not the problem. The chain breaks earlier: the 'reasonable auditing procedures' promised in the abstract must produce audits that are effectively draws from the voter distribution an adversary would target. Section 5.2 asserts this is a tractable engineering problem but gives no construction, no validation data, and no sensitivity analysis. The paper itself flags related unknowns: footnote 5 admits the true spoilage baseline is unknown until large BMD deployments exist, and Section 5.3 concedes that focused down-ballot tampering in a small jurisdiction is detected with only 10-40% probability. These are not internal contradictions, but they make the central claim conditional on empirical inputs that have not been established. The proposed concrete test would turn the central assumption into a measured quantity: simulate a targeted adversary against an audit script generated from real event logs. If the script samples the targeted subpopulation at approximately the same rate as real voters, the Table 1 guarantee is meaningful; if not, the guarantee is overstated. Because this is exactly the condition the reader attached, the verdict remains CONDITIONAL and no adjustment is needed.","tokens_in":12747,"tokens_out":4400,"duration_ms":105690,"concrete_test":"Use real BMD event-log data from a pilot deployment or usability study to fit a joint distribution of observable voter behaviors: interaction duration, use of accessibility modes, number of back-tracks, and contests selected. Generate an audit script by sampling from this fitted model as Section 5.2 proposes. Define an adversary that targets a rare, machine-observable subpopulation (e.g., the slowest 5% of users or button-box users) and flips votes with probability p. Simulate n audits drawn from the script and compute the detection probability. Compare it with 1-(1-p)^n from Table 1. If the achieved probability falls materially below the stated 95% or 99% target, the live-audit guarantee is not established for selective attackers. If no event-log data are available, collect such data in a mock election before relying on the model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing empirical assumption appears in Section 5.2: the live-audit procedure requires a 'random voter' model that reflects all voters across each precinct, including preferences and machine-observable behaviors, and the paper asserts this is a 'tractable engineering problem' without constructing or validating it. Table 1 and the Section 5.3 detection estimates assume each audit is drawn from the same distribution as the voters an adversary would target. If an attacker targets a subpopulation the audit script under-samples, such as slow users, accessibility-feature users, or voters in a particular precinct, the effective cheating probability observed per audit is lower than the nominal p, so the advertised 99% confidence is not reached. The paper itself concedes the closely related failure mode in its Baytown example: focused down-ballot tampering is detected with only roughly 10-40% probability. Because the abstract's policy claim is that reasonable auditing procedures give confidence across the fleet, this unvalidated distributional equivalence is the least secure link in the argument. It is not an internal inconsistency; it is an empirical premise that the paper flags as a challenge rather than supporting with data. The same category of concern appears in footnote 5, which acknowledges that the true baseline spoilage rate is unknown until large BMD deployments exist.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that ballot marking devices (BMDs) can be deployed securely if election administrators combine two mechanisms: live auditing, in which auditors exercise BMDs and verify the printed ballots, and real-time tracking of spoiled-ballot rates as a statistical signal of switched-intent attacks. The core probability analysis in Section 4.1 computes detection confidence 1 - (1-p)^n as a function of per-ballot attack probability p and the number of audits n, with Table 1 giving sample values. Section 5.4 extends this with a Poisson-based analysis of spoilage rates, summarized in Table 2, to bound the margin by which malware could change an election before being detected. The paper responds to Stark's criticisms by arguing that the auditor can approximate a random voter model (Section 5.2), that down-ballot tampering is hard to detect generally (Section 5.3), and that larger jurisdictions benefit from the scaling properties of the audit math. The overall claim is that reasonable procedures give confidence in a BMD fleet while preserving accessibility.","tokens_in":12954,"tokens_out":3159,"duration_ms":34106,"significance":"If the analysis were fully supported, the paper would make a valuable contribution to the BMD security debate: it provides transparent, machine-checkable probability calculations, a concrete staffing example for Harris County, a reproducible Python appendix (Appendix A), and direct engagement with the opposing statistical arguments. The quantitative framework is a useful starting point for evaluating live auditing and spoilage monitoring. However, the central policy claim rests on empirical assumptions about voter behavior and audit sampling that the paper does not validate. The paper is honest about several of these limitations in footnotes and in the Baytown example, but those limitations are load-bearing for the abstract's fleet-wide confidence claim.","major_comments":[{"comment":"The live-audit detection probabilities in Section 4.1 and Table 1 require that each audit be drawn from the same distribution as the voters an adversary would target. The paper asserts that a 'random voter' model reflecting all voters' preferences and machine-observable behaviors is a 'tractable engineering problem,' but no construction, validation data, or error analysis is provided. Without such a model, an adversary can target subpopulations that the audit script under-samples, such as slow users or accessibility-feature users, which the paper itself acknowledges in the introduction to Section 5. This is not an abstract worry: the Baytown example in Section 5.3 shows that focused down-ballot tampering is detected with only roughly 10-40% probability, directly contradicting any claim of reliable fleet-wide detection. Because the abstract's central assertion is that reasonable auditing procedures give confidence across a fleet of BMDs, this unvalidated distributional equivalence is a load-bearing gap.","section":"Section 5.2"},{"comment":"The spoilage-rate analysis assumes a 1% expected background spoilage rate, but footnote 5 explicitly states that 'we won't know the actual rate of spoiled ballots on these systems' until large BMD deployments exist. Table 2's margin-of-victory bounds are computed as 200.0 * (poisson.ppf(0.95, mu) - mu) / (electionSize * detectionFraction), where mu = 0.01 * electionSize. If the true spoilage rate is higher than 1%, the Poisson 95% critical value shifts upward and the undetectable margin becomes larger; if it is lower, the opposite occurs. The paper's conclusion that a county-wide attack moving the margin by more than 0.3% would be detected is therefore not robust to the stated empirical uncertainty. This is a second load-bearing empirical input that the paper flags but does not quantify as a sensitivity analysis.","section":"Section 5.4 and Table 2"}],"minor_comments":[{"comment":"The sentence 'Stark's version is more precise in its counting, representing an error in the math of Section 4 of 1.71%' is ambiguous: the 1.71% is the difference between the simple binomial approximation and Stark's hypergeometric calculation, but it reads as if Stark's calculation is the one in error. Suggest rewording to state that the binomial approximation differs from Stark's exact hypergeometric result by 1.71 percentage points.","section":"Section 5.1"},{"comment":"There is a typo in 'prior to the start of the elction' — 'elction' should be 'election.'","section":"Section 3"},{"comment":"In the sentence 'We're interested in how far to the right we need to get before the malware wins with probably less than 5%,' the word 'probably' should be 'probability.'","section":"Section 5.1"},{"comment":"The Python snippet uses Python 2 print statements and string formatting; adding a note about compatibility or updating to Python 3 would make the appendix easier for readers to run.","section":"Appendix A"},{"comment":"The staffing estimate for Harris County relies on 2019 early-voting center counts because the 2018 numbers could not be found, as the author notes. This is acceptable as an illustration, but the sentence in the main text could more clearly separate the assumed and the verified numbers.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"This is a position paper for a law journal rather than a security conference, so the bar for empirical validation might be lower than in systems research. Even so, the abstract's claims are strong and general, and the two empirical assumptions identified in the major comments — the random-voter audit model and the unknown spoilage baseline — are central to the quantitative conclusions. The paper would be substantially strengthened by a sensitivity analysis over plausible spoilage rates and an explicit statement of what evidence would validate the random-voter model. I see no indication of inappropriate citation or novelty suppression; the author engages directly with the opposing papers and cites his own prior work where it is actually relevant."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dan, this is a useful paper to know about. It's Wallach's response to Appel/DeMillo/Stark and Stark's parallel-testing notes, and it makes the best available case that BMDs can be used if you pair them with live auditing and real-time spoilage-rate monitoring. The probability math is correct and clearly presented; Table 1 is just 1-(1-p)^n, Table 2 generalizes Stark's Poisson logic, and Appendix A gives working Python so you can check it. The point that detection probability is independent of the number of ballots cast, which makes big jurisdictions the natural home for BMDs, is well made and under-appreciated. The paper also does something rare in this debate: it names its own weak spots. Footnote 5 says nobody knows the true baseline spoilage rate yet; Section 5.3 concedes that in a small election like Baytown, focused down-ballot tampering is caught only 10-40% of the time with 300 audits; and the emergency-procedures discussion is honest about the political difficulty of re-running an election.\n\nThe soft spot is exactly the one the stress-test flags. The entire live-audit guarantee in Table 1 and the 5.3 numbers depend on auditors sampling from the same distribution as the voters an attacker cares about. Section 5.2 says building a 'random voter' model from historical ballots and event logs is a tractable engineering problem, but it does not build it, and it does not validate it. That's an empirical premise, not a theorem. If an attacker targets slow users or accessibility-feature users or some precinct skew, the effective p in the audits is lower than assumed and the confidence numbers in the paper are not reached. The paper acknowledges the shape of this attack (Stark's point) and sidesteps it with a hand-wave. I don't think this is fatal — Wallach's larger point that live auditing plus spoilage monitoring raises the bar for attackers in large jurisdictions survives — but the abstract overreaches when it promises 'the same security and reliability properties we expect from hand-marked paper ballots.' Hand-marked ballots don't need voters to detect a systemic malware attack; the audit is on the paper itself. Here the audit is partly on the voter, and voter detection rates come from small lab studies that may not transfer.\n\nSo: the paper is a serious piece of advocacy with correct math and honest caveats, but it's a conditional case. It deserves a referee if submitted as a position paper; the referee should ask for the random-voter model to be built and evaluated, or for the scope claim to be narrowed to large jurisdictions with validated voter-detection behavior. I'd bring it to a reading group, but I wouldn't cite it as a proof of concept — only as a well-argued point of view in the debate.","headline":"A clear-headed math-backed brief for making BMDs work with live audits and spoilage monitoring, but the fleet-wide confidence claim leans on an unvalidated 'random voter' model and empirical guesses the paper itself flags.","tokens_in":13498,"tokens_out":2856,"would_cite":false,"duration_ms":29164,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that live audits plus spoiled-ballot tracking can make ballot-marking devices trustworthy enough for accessible elections.","keywords":["ballot marking devices","election security","live auditing","ballot spoilage","switched-intent attack","risk-limiting audits","accessibility in voting","parallel testing"],"falsifier":"A red-team experiment would settle the question: build the random-voter script from a jurisdiction's actual cast-ballot and event-log data, deploy it on machines running malware that targets a subpopulation the script rarely samples (for example, voters who use accessibility features or take unusually long), and measure whether the tampering evades both the live audits and the 95 percent spoilage confidence bound. If it does, the paper's detection probabilities do not hold for that threat model.","tokens_in":12508,"feed_emoji":"🗳️","tokens_out":7004,"duration_ms":70972,"temperature":0.7,"pith_summary":"The paper takes on the position that ballot-marking devices (BMDs) cannot be made secure, and answers it with a concrete counterproposal: election administrators can create reasonable auditing procedures that give real confidence their BMD fleet is printing voters' true choices. The two main procedures are live audits, where an official secretly exercises a machine and checks its printed ballot, and real-time tracking of spoiled-ballot rates, where an unusual spike is treated as statistical evidence of tampering. The paper analyzes both quantitatively, showing that a few hundred live audits give high detection probability for a machine that flips a few percent of ballots, and that spoilage monitoring sharply limits how much malware can move a large jurisdiction's margin of victory without detection. If the paper is correct, jurisdictions can keep the usability and accessibility benefits of BMDs—audio ballots, large text, multiple languages, no overvotes—while retaining the security and auditability of paper ballots.","feed_headline":"Audits can make ballot-marking devices safe enough for elections","feed_subtitle":"Live audit math plus spoiled-ballot tracking can catch BMD tampering while keeping paper-ballot security.","key_machinery":"The load-bearing identity is the live-audit detection probability. If a compromised BMD flips a ballot with probability $p$ on any given use, then after $n$ audits the chance of catching it at least once is $1-(1-p)^n$. This treats each audit as an independent Bernoulli trial and makes detection depend on the number of audits, not the number of ballots cast, so the relative cost of auditing falls as an electorate grows. The companion machinery is a Poisson-based spoilage alarm: given a background spoilage rate, a count above the 95% Poisson confidence bound is treated as evidence of tampering, and the paper converts that bound into the maximum margin shift malware could hope to hide. The third component is the \"random voter\" model, a script that tells auditors which races to vote and how to behave, built from prior ballots and event logs so an attacker cannot reliably tell an audit from a real voter.","core_discovery":"The central claim is that the security of a BMD election does not depend on each voter being a vigilant auditor; it can be carried by a two-layer detection process at the election-administrator level. A malicious BMD that hides its misbehavior behind a normal screen and prints a different ballot than the voter selected—a switched-intent attack—will be caught if a live auditor happens to use that machine, and it will also cause observable voters to spoil ballots at an elevated rate. The paper argues that with enough randomized live audits and real-time spoilage data, the probability of catching such an attack is high for the large jurisdictions that actually buy BMDs, and that an emergency re-run procedure can deter or repair the residual risk. The paper does not claim BMDs are invulnerable; it claims that the combination of audits, spoilage signals, and emergency laws restores the security properties people expect from hand-marked paper while preserving BMD accessibility.","pith_inferences":["The paper's detection table assumes each live audit is independent of the attacker's state; an adversary who can observe audit patterns could tune its flips to rare behaviors, which suggests a game-theoretic extension where detection probabilities degrade from Table 1.","The spoilage alarm and live audits are analyzed separately; combining them into one sequential Bayesian alarm that updates on both signals as the day progresses would likely detect a focused attack sooner than either method alone.","A natural pilot experiment would score a constructed random-voter script against real event logs; if the script under-samples slow voters, audio users, or other accessibility subpopulations, the paper's detection estimates should be discounted by exactly that under-coverage."],"forward_implications":["Roughly 300 to 500 live audits give better than 99 percent confidence of catching a BMD that flips 1 percent of ballots, independent of the total number of votes cast.","In a 1.2-million-vote county, real-time spoilage monitoring limits undetected margin shifts to about 0.3 percentage points even under a pessimistic 10 percent voter-notice rate, and to as little as 0.06 percentage points if half of affected voters spoil their ballots.","For a small down-ballot race, such as the 9,000-vote Baytown example, focused tampering may be caught only 10 to 40 percent of the time by 300 county-wide audits, so officials must deploy extra auditors adaptively to hot races.","A single inconsistent barcode-and-text ballot is treated as incontrovertible evidence of a bug or attack, which is why attackers are unlikely to use barcode-only tampering.","Elections can treat a large cyberattack like a natural disaster: emergency laws and re-run procedures reduce the attacker's confidence of getting away with it."],"supporting_citations":[{"why":"States the opposing argument that BMDs cannot assure the will of voters, which the paper's live-audit and spoilage-monitoring analysis is designed to answer.","marker":"[5]"},{"why":"Supplies the first verification-rate study showing that one-third to one-half of voters notice introduced errors on a voting machine display.","marker":"[12]"},{"why":"Follow-up usability study supporting the paper's claim that voters detect screen-to-ballot mismatches at meaningful rates.","marker":"[9]"},{"why":"Sets out the oracle-bound objection that live audits must probe every subset of voter preferences and behaviors.","marker":"[19]"},{"why":"Formulates the down-ballot tampering and spoiled-ballot-signal objections that Sections 5 and 6 answer.","marker":"[20]"},{"why":"An independent proposal for a ballot-marking verification protocol, cited as precedent for the live-auditing approach.","marker":"[13]"},{"why":"Origin of the challenge-based live-auditing mechanism that this paper adapts to BMD fleets.","marker":"[7]"},{"why":"Documents a BMD design flaw where printing and casting are integrated, motivating the paper's focus on stateless machines with manual casting.","marker":"[4]"}],"fun_headline_variants":["Live audits + spoiled ballots catch BMD attacks","Two-layer audit catches hidden BMD ballot switches","Audits and spoilage tracking secure BMD elections","Catch ballot-switching malware with live audits and spoilage","BMD security via audits and spoiled-ballot signals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole detection estimate rests on the assumption that auditors can exercise BMDs in a way indistinguishable from the real voters an attacker would target, so the \"random voter\" model must faithfully capture voters' preferences and behaviors; the paper calls this a tractable engineering problem but does not build or validate the model.","fun_headline_variants_meta":{"raw":{"variants":["Live audits + spoiled ballots catch BMD attacks","Two-layer audit catches hidden BMD ballot switches","Audits and spoilage tracking secure BMD elections","Catch ballot-switching malware with live audits and spoilage","BMD security via audits and spoiled-ballot signals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000587,"raw_usage":{"total_tokens":2697,"prompt_tokens":822,"completion_tokens":1875,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":438,"completion_tokens_details":{"reasoning_tokens":1799}},"tokens_in":438,"tokens_out":1875,"duration_ms":13842,"temperature":1.0,"reasoning_tokens":1799,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T15:00:10.622718+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A red-team experiment would settle the question: build the random-voter script from a jurisdiction's actual cast-ballot and event-log data, deploy it on machines running malware that targets a subpopulation the script rarely samples (for example, voters who use accessibility features or take unusually long), and measure whether the tampering evades both the live audits and the 95 percent spoilage confidence bound. If it does, the paper's detection probabilities do not hold for that threat model.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"States the opposing argument that BMDs cannot assure the will of voters, which the paper's live-audit and spoilage-monitoring analysis is designed to answer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the first verification-rate study showing that one-third to one-half of voters notice introduced errors on a voting machine display."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Follow-up usability study supporting the paper's claim that voters detect screen-to-ballot mismatches at meaningful rates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Sets out the oracle-bound objection that live audits must probe every subset of voter preferences and behaviors."},{"cited_title":"They may look and look, yet not see: BMDs cannot be tested adequately","cited_arxiv_id":"1908.08144","evidence_quote":"Formulates the down-ballot tampering and spoiled-ballot-signal objections that Sections 5 and 6 answer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"An independent proposal for a ballot-marking verification protocol, cited as precedent for the live-auditing approach."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Origin of the challenge-based live-auditing mechanism that this paper adapts to BMD fleets."},{"cited_title":"permission to cheat","cited_arxiv_id":null,"evidence_quote":"Documents a BMD design flaw where printing and casting are integrated, motivating the paper's focus on stateless machines with manual casting."}],"review_version":1}