{"id":"1c177faf-9516-47dc-8c5e-a219196174df","arxiv_id":"2509.11068","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An LLM output can be verified by regenerating a few randomly chosen segments under identical hardware, with a tunable detection probability and 12.4x speedup over full regeneration.","lead":"This paper proposes a way for one AI agent to check that another agent really used the claimed language model: re-run a small random piece of the output on identical hardware and compare. It shows such spot checks can be much cheaper than regenerating the whole answer, but only if every agent uses the exact same machines and software.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Positive same-hardware determinism is asserted but never tested: Section IV-A explicitly declines to run the check, and only the negative cross-GPU failure is reported. The entire detection formula of Eq. (3) is moot if token-exact replication fails on identical stacks.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the positive case of same-hardware deterministic replication is never tested. My analysis confirms this. The combinatorial model in Eq. (3) is correct, but it only matters if token-exact replication actually holds. Since the paper explicitly acknowledges the homogeneity restriction and lists it as a prerequisite, the appropriate editorial decision is to require a positive determinism test before accepting the central claim. This is consistent with the reader's CONDITIONAL verdict, so I do not recommend changing it. If the proposed test fails, the paper should be rejected, but that is an empirical outcome we cannot know without the experiment. The paper otherwise makes a useful, if narrow, contribution under its stated assumptions.","tokens_in":8476,"tokens_out":2129,"duration_ms":28167,"concrete_test":"Run a controlled same-hardware determinism experiment: on the same GPU instance, load the same Llama-3-8B-Instruct checkpoint with the same software container, fixed seed, and generation parameters (e.g., temperature=0, do_sample=False). Generate the same 792-token CoT prompt at least 100 times across separate process launches. For each output, take a prefix of length j (e.g., j=50, 100, 200) and regenerate the remaining tokens from that prefix under identical conditions, comparing token-by-token. Report the exact-match rate. Additionally, repeat the Table I timing measurements at least 10 times and report mean and standard deviation. If the exact-match rate is less than 100%, the central mechanism fails and the claimed asymmetric effort is not achievable even in a homogeneous environment.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The framework's central premise is that an autoregressive LLM under identical hardware/software/seed conditions deterministically reproduces the exact same tokens from a given prefix (Section III-B). On this premise rests the targeted validation mechanism and the detection probability in Eq. (3). Yet the paper never positively tests this premise. Section IV-A states: 'the output of a targeted validation will always align with the corresponding portion of the original sequence in the controlled conditions. Hence, we will not revisit it here.' The only experimental evidence about determinism is negative: Section IV-C shows that different GPU models (RTX 4000 Ada vs. A40) produce non-identical outputs. That does not establish that the same GPU and software stack produce identical outputs. Non-determinism can arise even on identical hardware: nondeterministic CUDA kernels, atomic operations, thread scheduling, or version drift in PyTorch/HuggingFace. If even one token differs, a validator cannot distinguish a tampered segment from a nondeterministic output, and the entire verification scheme collapses. The citations [25]-[27] are about stochastic decoding choices or reproducibility in general, not a direct demonstration that a HuggingFace pipeline is token-exact on the same GPU. Thus the load-bearing empirical claim is untested, and the reported 12.41x asymmetric effort ratio depends on a premise that may be false.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a verification framework for LLM outputs in multi-agent systems. It relies on deterministic replicability: under identical hardware/software/seed conditions, an autoregressive LLM reproduces the same tokens from a given prefix. The framework lets a validator construct a prompt by concatenating the original prompt with a prefix of the output, regenerate the remaining tokens, and compare them to the original output. To avoid exhaustive checking, the scheme divides an output into k segments, assumes f tampered segments, and has q validators each randomly checking r segments. The paper derives the detection probability P(Detect) = 1 - (C(k-f,r)/C(k,r))^q and validates it with Monte Carlo simulation. It also reports timing experiments on Llama 3 showing that verifying the last 50 tokens of a 792-token response takes 2.59 s versus 32.13 s for full generation, a 12.41x asymmetric effort ratio. The paper further reports that validation across two different GPU models fails, and concludes that a homogeneous computational environment is required.","tokens_in":8771,"tokens_out":4294,"duration_ms":55437,"significance":"If the deterministic-replicability premise holds, the framework offers a simple, parameter-free method for probabilistic output auditing at substantially lower cost than full regeneration. Equation (3) is mathematically correct for the stated sampling scheme, and the Monte Carlo validation in Figure 3 is consistent. The approach is notable for avoiding fitted parameters and for providing an explicit, tunable trade-off between validation effort and detection probability. However, the central premise is external to the derivation and is not positively tested. The paper's own text in Section IV-A explicitly declines to test same-hardware determinism, and Section IV-C only demonstrates the negative cross-GPU case. Since the entire mechanism collapses if token-exact replication fails on identical stacks, the contribution is conditional on an unverified empirical claim. The stress-test concern lands squarely on this gap. The combinatorial derivation itself is sound but does not, by itself, establish practical security.","major_comments":[{"comment":"The load-bearing assumption is deterministic replicability on identical hardware/software, stated in Section III-B. Section IV-A explicitly declines to test it: 'the output of a targeted validation will always align with the corresponding portion of the original sequence in the controlled conditions. Hence, we will not revisit it here.' This is not acceptable for a central premise. The only empirical evidence in Section IV-C is a negative result on different GPU models (RTX 4000 Ada vs. A40), which does not establish that the same GPU and software stack are token-exact. Nondeterministic CUDA kernels, atomic operations, thread scheduling, or version drift can create differences even on identical hardware. Citations [25]–[27] are about stochastic decoding choices and reproducibility reporting, not a direct demonstration of token-exact determinism. The paper needs a positive experiment: rep","section":"Section III-B and IV-A"},{"comment":"The detection probability formula assumes the tampered segments are fixed before validators sample, and that the generator cannot adapt to the validators' choices. The paper does not state this as a security assumption or specify a commitment mechanism. If a malicious generator can choose which f segments to tamper after observing the validators' random queries, or if validators' random seeds are predictable, the uniform-sampling model in Eq. (3) no longer applies and the detection probability can drop to zero. A practical deployment needs a delayed-randomness or commit-reveal protocol. Also, Figure 3 is a Monte Carlo simulation of the same combinatorial sampling scheme; it validates the algebra and implementation, but it is not an empirical security test against an actual adversary. The paper should distinguish these and state what security guarantees are proved versus assumed.","section":"Section III-C, Eq. (3)"},{"comment":"The asymmetric effort ratio of 12.41x is based on a single run with no variance, confidence intervals, or specification of how prompt processing and model loading are counted. Verification of the last 50 tokens requires processing a 742-token prefix and generating 50 tokens; full generation processes a short prompt and generates 792 tokens. The cost composition differs, and the reported ratio may depend on the specific implementation, GPU, batch size, and whether KV-cache reuse or graph compilation is used. I recommend multiple trials with mean/standard deviation and a clear statement of the experimental conditions (decode strategy, temperature, seed, max new tokens, prompt-processing times).","section":"Section IV-A, Table I"}],"minor_comments":[{"comment":"Typographical and notation issues: 'r distinct segment' should be 'r distinct segments'; the set notation for tampered segments is garbled by LaTeX braces ('n{˙y}1 ,· · ·,{˙y}f o' renders poorly). The text says 'tuneable' and 'tunable' inconsistently; 'tunable' is fine. The parameter f is used both as the count of tampered segments and as an index in the notation; please clarify.","section":"Section III-C"},{"comment":"The experimental section does not state the decoding parameters used for Llama 3 (temperature, top-p, seed, max_new_tokens, batch size). Since deterministic replicability depends on these, please report them. Also note whether the final 50-token verification was run from the exact prefix of the original generation, and whether the output was produced in a single generation call or with any control-flow features that could affect determinism.","section":"Section IV-A"},{"comment":"The phrase 'we posit that a standardised execution environment ... is a prerequisite' is appropriate, but the paragraph conflates 'different GPU models produce non-identical outputs' with 'identical stacks are deterministic.' Please separate the negative cross-hardware result from the unverified same-hardware claim, and consider stating explicitly that the same-hardware case remains an open empirical question in this paper.","section":"Section IV-C"},{"comment":"References [25]–[27] are not strong evidence for the 'deterministic nature' of LLMs under controlled conditions. [25] discusses nucleus sampling and degeneration; [26] is about reproducibility reporting; [27] contrasts stochastic and deterministic decision patterns in LLMs. None demonstrate token-exact replication on identical hardware. Either add direct experiments or replace these citations with sources that actually establish the property (e.g., reproducibility studies of HuggingFace transformers on fixed hardware).","section":"Section III-B"}],"recommendation":"major_revision","confidential_remarks":"The combinatorial portion of the paper is correct but small; the main contribution is the framing of tractable asymmetric verification through deterministic replicability. The missing same-hardware determinism experiment is feasible within the manuscript's scope and should be added before publication. I do not see a circularity problem in the derivation itself; the issue is an omitted load-bearing empirical test. The paper's current form overclaims empirical validation, since Figure 3 only validates the sampling formula. I recommend major revision rather than rejection because the premise is potentially testable and the conditional framework is clearly presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one. First, the core mechanism is exactly what the title says: because an autoregressive LLM is assumed to replicate token-for-token on identical hardware/software, a validator can re-generate a random segment of the output and compare. The detection probability is just the hypergeometric law, Eq. (3), and the paper validates it with a Monte Carlo simulation that matches almost perfectly. That part is fine. Second, the load-bearing assumption—that the same GPU and software stack actually produce identical tokens—is never tested. Section IV-A explicitly says they will not revisit it, and the only empirical evidence about determinism is the negative result that different GPUs produce different outputs. That is a control experiment, not a test of the premise.\n\nWhat is genuinely new here is modest but real: the specific combination of deterministic re-generation and distributed random segment sampling for verification does not appear in the prior work they cite. The timing measurement, a 12.4x asymmetry for a 50-token check, is illustrative rather than rigorous (single run, no error bars, no released code or prompt), but it does demonstrate the intended effect. The paper is clearly written and the math is correct.\n\nThe soft spots are the ones you would expect. The untested same-hardware determinism is the big one; if even occasional nondeterminism occurs from CUDA kernels, thread scheduling, or version drift, the whole scheme collapses. The citations [25]-[27] do not establish token-exact replicability; they are about stochastic decoding and reproducibility in general. The language about being a 'foundational layer' and 'cornerstone' for responsible AI is inflated relative to what is delivered. The distributed protocol is only sketched, and the threat model ignores the possibility that the validator itself is on slightly different software.\n\nIs this paper worth your time? For someone working on LLM verification, it is a useful pointer and a clean formulation of an idea that could be made to work in a tightly controlled cloud environment. But it needs a positive determinism test on identical hardware, repeated timing measurements, and released artifacts before it carries much weight. I would not cite it in my own work yet, though I would bring it to a reading group as a starting point for discussing what 'deterministic' actually means in practice.\n\nAs for peer review: I would send it out. The question is substantive, the framework is not obviously wrong, and a referee could push the authors to run the missing experiment. It deserves a serious look, but the verdict should be conditional on demonstrating the determinism premise rather than assuming it.","headline":"A clean but thinly tested idea: verify LLM outputs by re-generating random segments on identical hardware, with the entire mechanism resting on an untested determinism assumption.","tokens_in":9251,"tokens_out":1621,"would_cite":false,"duration_ms":22110,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Autoregressive LLM outputs are cheaply auditable by regenerating random segments under identical hardware.","keywords":["LLM verification","asymmetric computation","deterministic replicability","probabilistic verification","multi-agent trust","targeted validation","tamper detection"],"falsifier":"Run the same autoregressive model twice on the same GPU with the same seed, same prompt, and the same software versions, and compare the full output token by token; if any token differs, deterministic replicability fails. A more targeted falsifier is to reproduce the 792-token response and the 50-token validation on identical hardware and verify both the exact token match and the claimed 12.41x time ratio.","tokens_in":8364,"feed_emoji":"🔍","tokens_out":3532,"duration_ms":39363,"temperature":0.7,"pith_summary":"This paper tries to establish that LLM outputs in multi-agent systems can be verified with asymmetric effort, meaning checking a response costs much less than generating it. The mechanism relies on deterministic replicability: the same model, with the same prompt and environment, should reproduce the same continuation tokens. By having validators regenerate small random chunks of the output and compare them, the framework detects tampering with a tunable probability. Because the check only regenerates a slice of the sequence, it is far cheaper than full re-generation; the paper reports a 12.41x speedup for a 50-token check. The framework is confined to strictly homogeneous execution environments, which the authors identify as a prerequisite.","feed_headline":"12x faster LLM output checks via random segment replay","feed_subtitle":"Validators regenerate small random chunks of a response; identical hardware makes tampering detectable.","key_machinery":"The load-bearing mechanism is deterministic replicability combined with random subsampling. Deterministic replicability states that an identical LLM under identical conditions regenerates the same continuation given the prompt plus prior output tokens; this turns validation into a comparison of regenerated tokens. Distributed probabilistic verification turns the otherwise intractable search for a tampered position into a sampling problem: each validator randomly selects a small number of segments, and the probability of missing all tampered segments drops exponentially with the number of validators. The central identity is the detection probability formula, which shows how detection scales w","core_discovery":"In the paper's own terms, the central discovery is the combination of targeted validation with distributed probabilistic verification. Starting from the proposition that an autoregressive LLM, given a prompt concatenated with the first j tokens of its own output, will deterministically produce the exact remaining tokens when external conditions are fixed, the authors derive a detection probability for tamper detection: P(Detect) = 1 - (C(k-f, r)/C(k, r))^q, where the output is split into k segments, f are tampered, each of q validators checks r random distinct segments. They demonstrate empirically that the detection probability matches this formula and that verifying the final 50 tokens of","pith_inferences":["Editorial inference: the framework's usefulness hinges on the untested positive case of same-hardware determinism; if even identical GPUs with the same software stack produce token differences (e.g., due to nondeterministic kernel execution), the whole approach fails, so a natural next experiment is a bitwise reproducibility test on identical hardware.","Editorial inference: the same probabilistic scheme could be adapted to fuzzy verification that tolerates small numeric differences, which would relax the strict homogeneity requirement the paper names as future work.","Editorial inference: the 12.41x speedup is for a particular model and hardware, but the cost ratio should generalize roughly linearly with the fraction of tokens regenerated, so longer checks trade off cost against detection probability in a predictable way.","Editorial inference: the framework only verifies that a claimed model was literally run on the stated prompt; it does not validate reasoning quality, so it is complementary to, not a replacement for, output-quality checks."],"forward_implications":["If the central claim is correct, validators can audit LLM outputs without re-running full inference, making multi-agent systems practically auditable in controlled deployments.","The detection probability formula gives system designers a direct way to tune security: increase q or r to reach desired detection rates, as shown for k=20, f=2.","The framework specifically counters cost evasion (cheaper model substitution) and malicious content injection (small tampered segments), the two threats in its threat model.","Because verification is independent per validator, the workload spreads across agents without any single validator bearing full regeneration cost.","The requirement of a homogeneous execution environment means deployments should use standardized cloud instances or version-controlled containers for the framework to work."],"fun_headline_variants":["12x faster LLM verification via random segment checks","LLM audit 12x faster by replaying random output chunks","Sample and check: 12x faster LLM verification","Deterministic replicability enables cheap LLM verification","Fast, cheap LLM audits via deterministic replay"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The framework stands or falls on the claim that an identical LLM on identical hardware and software will regenerate the exact same tokens for a given prefix; this positive case is asserted but not directly tested in the paper.","fun_headline_variants_meta":{"raw":{"variants":["12x faster LLM verification via random segment checks","LLM audit 12x faster by replaying random output chunks","Sample and check: 12x faster LLM verification","Deterministic replicability enables cheap LLM verification","Fast, cheap LLM audits via deterministic replay"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000503,"raw_usage":{"total_tokens":2286,"prompt_tokens":728,"completion_tokens":1558,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":1479}},"tokens_in":472,"tokens_out":1558,"duration_ms":12204,"temperature":1.0,"reasoning_tokens":1479,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T17:07:07.211432+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same autoregressive model twice on the same GPU with the same seed, same prompt, and the same software versions, and compare the full output token by token; if any token differs, deterministic replicability fails. A more targeted falsifier is to reproduce the 792-token response and the 50-token validation on identical hardware and verify both the exact token match and the claimed 12.41x time ratio.","supporting_citations":[],"review_version":1}