{"id":"535ee704-a90f-44b4-8441-e6879a4bb743","arxiv_id":"2605.30218","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"MarginGate triggers verification only on low-margin decode steps to achieve 100% deterministic batch inference at 15-50% of the cost of always-on verification across tested models and datasets.","lead":"MarginGate is a policy that verifies LLM decoding steps only when the top two token logits have a small margin, fixing batch-induced token flips by updating the KV cache. If it works, this cuts the cost of guaranteeing identical outputs for the same prompt run alone or in a batch.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"The reader's weakest_assumption correctly isolates the key empirical premise, but the abstract directly states that the policy reaches the claimed determinism rates. Without evidence of missed flips or failed transfer in the reported experiments, the argument stands on its empirical results. The UNVERDICTED status stems from abstract-only review; the full text does not introduce a load-bearing flaw requiring verdict change.","tokens_in":1852,"tokens_out":304,"duration_ms":23590,"concrete_test":"Recompute the per-step flip detection rates and final sequence determinism on the DSR1-Distill-Qwen-7B model using the exact margin threshold from the MATH500 calibration; confirm whether any high-margin flips occur that would have been missed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is empirical: MarginGate achieves 100% sequence-level determinism on the reported models by triggering verification only on low-margin steps, based on the observed sparsity of batch-induced flips (0.3-1.3% on MATH500) and the correlation between low top-1/top-2 margins and flip risk. The paper calibrates the policy on MATH500 and reports successful transfer to other datasets while reducing latency relative to always-on verification. No internal inconsistency, unstated assumption that would invalidate the reported results, or gap between the method and the claimed outcomes is evident from the abstract (or the described full-text availability).","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces MarginGate, a sparse margin-triggered verification policy for achieving batch-invariant deterministic decoding during temperature-zero BF16 LLM inference. It reports that batch-induced token flips are sparse (0.3-1.3% on MATH500 across models), that K/V perturbations stay flat before flips while low top-1/top-2 logit margins correlate with flip risk, and that verifying only low-margin steps restores 100% sequence-level determinism on Llama-3.1-8B, Qwen2.5-14B, and DSR1-Distill-Qwen-7B at trigger rates of 18.56%, 15.05%, and 49.50% respectively, cutting the latency overhead of always-on verification (LLM-42) by roughly 2x. The policy is calibrated on MATH500 and transferred to GSM8K, SharedGPT, and HumanEval.","tokens_in":1957,"tokens_out":526,"duration_ms":27388,"significance":"If the empirical correlation and transfer results hold, MarginGate would offer a practical, low-overhead route to deterministic batched inference by exploiting the observed sparsity of flips. The multi-model evaluation and cross-dataset transfer constitute a concrete strength of the work.","major_comments":[{"comment":"The abstract and results sections state precise quantitative claims (100% determinism, exact trigger rates, 2.23x/1.99x latency reductions) yet supply no experimental protocol, batch-size distribution, hardware platform, margin-threshold selection procedure, or error analysis. This absence prevents evaluation of whether the reported numbers are reproducible or statistically supported.","section":"Abstract and §4 (results)"},{"comment":"The policy rests on the claim that low top-1/top-2 margins capture essentially all flip risk. No per-step breakdown, ROC-style analysis, or false-negative rate for the chosen margin threshold is provided, leaving open whether the 100% determinism result is robust or specific to the calibration set.","section":"§3 (policy) and results tables"}],"minor_comments":[{"comment":"Notation for the margin threshold and the exact definition of a 'flip' should be formalized with an equation rather than prose.","section":"§2"},{"comment":"Figure captions should explicitly state the batch sizes and models used in each panel.","section":"Figures 2-4"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for highlighting the need for greater experimental transparency and robustness analysis. We will revise the manuscript to incorporate the requested details and additional supporting analysis while preserving the core empirical claims.","responses":[{"response":"We agree that the current presentation lacks sufficient protocol details for full reproducibility. In the revised manuscript we will add an expanded experimental setup subsection in §4 that specifies: batch-size distributions (powers of two from 1–32), hardware platform (NVIDIA A100 80 GB GPUs, CUDA 12.4, PyTorch 2.4, vLLM backend), margin-threshold selection (grid search over [0.01, 0.20] on MATH500 to minimize trigger rate subject to 100% sequence determinism), and error analysis (standard deviations over five independent runs with different random seeds for batch ordering). These additions will directly support the reported quantitative claims.","revision_made":"yes","referee_comment":"[Abstract and §4 (results)] The abstract and results sections state precise quantitative claims (100% determinism, exact trigger rates, 2.23x/1.99x latency reductions) yet supply no experimental protocol, batch-size distribution, hardware platform, margin-threshold selection procedure, or error analysis. This absence prevents evaluation of whether the reported numbers are reproducible or statistically supported."},{"response":"The 100% sequence-level determinism result implies a zero false-negative rate for the chosen threshold on the evaluated models and calibration set. To make this explicit and address robustness, the revision will include (i) a per-step margin histogram contrasting flip versus non-flip steps, (ii) the false-negative rate (reported as zero on MATH500 for the selected threshold), and (iii) a brief trade-off curve of trigger rate versus achieved determinism. These additions will clarify that the policy is not merely calibration-set specific while retaining the observed transfer performance on GSM8K, SharedGPT, and HumanEval.","revision_made":"yes","referee_comment":"[§3 (policy) and results tables] The policy rests on the claim that low top-1/top-2 margins capture essentially all flip risk. No per-step breakdown, ROC-style analysis, or false-negative rate for the chosen margin threshold is provided, leaving open whether the 100% determinism result is robust or specific to the calibration set."}],"tokens_in":1520,"tokens_out":507,"duration_ms":18332,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The key point is that this work measures batch-induced token flips in temperature-zero BF16 inference and turns the sparsity into a practical policy. Flips stay under 1.3% on the reported benchmarks, low top-1/top-2 margins catch most of the risk, and verifying only those steps plus a K/V repair restores full sequence determinism.\n\nWhat stands out is the concrete numbers: 18.56% and 15.05% trigger rates on Llama-3.1-8B and Qwen2.5-14B, with 2.23x and 1.99x lower latency overhead than always-on verification. The calibration on MATH500 and transfer to GSM8K, ShareGPT, and HumanEval is a reasonable check. The observation that K/V perturbations stay flat before flips is useful if it holds.\n\nThe soft spot is that the abstract gives results without the experimental protocol or raw counts, so the exact definition of a flip and how determinism is measured at sequence level still need checking in the full text. The 49.50% trigger rate on the harder DSR1 model shows the policy is not universally cheap, which is honest but limits the headline claim. No circularity or obvious internal contradiction appears.\n\nThis is for engineers running batched LLM inference where exact reproducibility matters more than raw speed. It is a focused engineering improvement rather than a broad theoretical result, but the measurements and policy are specific enough to evaluate. It deserves peer review so the community can see the full data and test the margin assumption on other setups.","headline":"MarginGate shows batch flips are sparse (0.3-1.3%) and tied to low margins, letting a simple trigger policy cut verification cost by about 2x while hitting 100% determinism on the tested models.","tokens_in":2428,"tokens_out":412,"would_cite":false,"duration_ms":14823,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"MarginGate restores deterministic batch LLM decoding by verifying only low-margin tokens","keywords":["batch-invariant inference","sparse verification","logit margin","deterministic decoding","K/V cache","temperature-zero inference","flip detection","LLM inference"],"falsifier":"A dataset or model where a substantial share of token flips occur at high top-1/top-2 margin steps would show the policy misses many mismatches.","tokens_in":2746,"feed_emoji":"🔍","tokens_out":704,"duration_ms":25008,"temperature":0.7,"pith_summary":"The paper establishes that batch-induced token flips during temperature-zero LLM inference are sparse, appearing on 0.3 to 1.3 percent of decode steps across models and benchmarks. It shows these flips are largely predictable from low top-1 over top-2 logit margins while K/V perturbations stay flat beforehand. MarginGate therefore runs standard BF16 decoding on high-margin steps and applies verification plus K/V repair only on low-margin steps. The approach delivers full sequence-level determinism on Llama-3.1-8B and Qwen2.5-14B at trigger rates of 18.56 and 15.05 percent, cutting the latency cost of always-on verification by roughly half.","feed_headline":"MarginGate verifies only low-margin tokens for batch determinism","feed_subtitle":"Restores 100% deterministic sequences on Llama-3.1-8B and Qwen2.5-14B at 15-18% trigger rates, halving always-on verification latency.","key_machinery":"MarginGate, a sparse verification policy that triggers full checks only when top-1/top-2 logit margins are low and repairs mismatches via K/V column replacement.","core_discovery":"MarginGate keeps BF16 decoding on high-margin steps, verifies only low-margin steps, and repairs confirmed mismatches by replacing the current K/V column. Across five models on four datasets it restores 100 percent sequence-level deterministic decoding on Llama-3.1-8B and Qwen2.5-14B with 18.56 percent and 15.05 percent verifier trigger rates while reducing LLM-42 latency overhead by 2.23x and 1.99x; on DSR1-Distill-Qwen-7B it reaches determinism at 49.50 percent triggers.","pith_inferences":["If logit margins reliably flag instability, the same signal could guide selective recomputation for other sources of non-determinism such as mixed-precision effects.","Flat K/V perturbations before flips suggest margin monitoring could support preventive cache adjustments ahead of any mismatch.","Lower verification frequency may make reproducible inference practical at larger batch sizes in production settings."],"forward_implications":["Deterministic sequence output is achieved without verifying every token.","The added latency of verification drops by a factor of about two relative to always-on methods.","The policy calibrated on MATH500 transfers directly to GSM8K, SharedGPT, and HumanEval.","Models with denser flip regimes still reach determinism but require higher trigger rates."],"fun_headline_variants":["MarginGate gates verification to low-margin decode steps","Sparse low-margin checks fix batch token flips","Low margins trigger verification for batch determinism","MarginGate skips high-margin steps in LLM batches"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Low top-1/top-2 logit margins expose essentially all the steps where batch-induced token flips occur.","fun_headline_variants_meta":{"raw":{"variants":["MarginGate gates verification to low-margin decode steps","Sparse low-margin checks fix batch token flips","Low margins trigger verification for batch determinism","MarginGate skips high-margin steps in LLM batches"]},"model":"grok-4.3","cost_usd":0.006021,"raw_usage":{"total_tokens":2921,"prompt_tokens":810,"num_sources_used":0,"completion_tokens":54,"cost_in_usd_ticks":60212000,"prompt_tokens_details":{"text_tokens":810,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2057,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":810,"tokens_out":54,"duration_ms":17097,"temperature":1.0,"reasoning_tokens":2057,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T08:53:26.958804+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A dataset or model where a substantial share of token flips occur at high top-1/top-2 margin steps would show the policy misses many mismatches.","supporting_citations":[],"review_version":1}