{"id":"ad64884f-afb3-4e0b-926b-0ed5e513e846","arxiv_id":"2505.13204","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AASD combines alignment sampling from the prefilling distribution with entropy-based conditional verification to improve both the accuracy and speed of retrieval-based speculative decoding.","lead":"This paper introduces AASD, a training-free decoding method that uses the model's own prefilling probabilities and prompt text to draft tokens, then accepts high-quality prompt fragments through an adaptive probability threshold. It reports better generation scores on average and up to 2.23x faster decoding across eight question answering, summarization, and code tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (5) produces nonpositive thresholds whenever entropy exceeds β/α ≈ 1 nat, so in typical high-entropy next-token distributions conditional verification accepts every draft token, making the reported 3.3-point accuracy gain attributable to unconditional context copying rather than the described…","rationale":"The reader's weakest_assumption correctly identifies Eq. (5) as the key risk, but stops at questioning whether the monotonic relationship between uncertainty and permissiveness holds across tasks. The more specific and more damaging issue is that, as written, the formula makes the threshold negative for entropy above β/α = 1 nat, so the acceptance condition is trivially satisfied and no entropy-dependent filtering occurs in the typical operating regime. This is an internal inconsistency: the prose says low-confidence tokens should get stricter thresholds, whereas the formula relaxes them. It also changes the interpretation of the central empirical claim: if δ is negative for most tokens, AASD's accuracy gain over greedy is essentially the gain from copying verbatim spans out of the prompt, not from a calibrated verification policy. This does not necessarily invalidate the observed score improvements, but it means the paper's explanation of why the method works is suspect, and the comparison against greedy sampling is not an apples-to-apples evaluation of generation quality. The recommended verdict remains CONDITIONAL because a straightforward fix - correcting the sign or clipping δ to be positive, and reporting the distribution of δ - could resolve the concern, and the empirical tables might still stand. However, the condition is now more specific: the authors must either correct Eq. (5) (e.g., to δ = min(αH + β, Δ) or δ = min(max(-αH + β, 0), Δ)) and re-run the experiments, or provide evidence that the original implementation actually produces positive, discriminating thresholds. The reader's verdict already calls for conditional acceptance, and this concern substantiates that condition with a concrete, checkable defect.","tokens_in":14234,"tokens_out":6435,"duration_ms":67654,"concrete_test":"Run the released AASD code on NQ with LLaMA3.1-8B-Instruct (α=0.1, β=0.1) and log, for every verified token, the entropy H of the verification distribution, the threshold δ from Eq. (5), and the draft probability p. Compute the fraction of accepted tokens with δ ≤ 0; if this fraction exceeds 50%, the threshold is vacuous in the majority of cases. Then rerun AASD with the threshold clipped to a small positive value, e.g., δ' = max(δ, 1e-4), keeping everything else fixed, and compare the accuracy (Table 1) and speed (Table 2) to the reported AASD numbers. If accuracy drops substantially, the reported gain is due to unconstrained acceptance; if not, Eq. (5) as written is not what the implementation computes.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central accuracy claim depends on conditional verification (Eq. 4-6) filtering low-quality draft tokens while accepting high-quality ones. With the stated α=0.1 and β=0.1, Eq. (5) gives δ = min(-0.1H + 0.1, Δ), where H is the entropy of the verification distribution and Δ is its maximum probability. For a next-token distribution with H > 1 nat, -0.1H + 0.1 < 0, and since Δ ≥ 0, δ is negative. The acceptance condition p ≥ δ then holds for every token, since all probabilities are nonnegative. LLM next-token distributions for open-domain tasks routinely have entropy well above 1 nat (vocabularies of 32k-128k tokens, predictive entropy often 2-6 nats), so for the majority of verification steps the 'adaptive threshold' is vacuous: every retrieved and alignment-sampled token passes unconditionally. This contradicts the paper's stated design that 'low-confidence tokens can be processed with stricter thresholds' - high entropy (low confidence) actually yields the most permissive threshold. The ablation in Table 4 shows adaptive verification (47.98) outperforms a fixed threshold of 0.1 (46.80), but if δ is mostly negative this simply means accepting more context tokens, not a calibrated entropy-dependent tradeoff. The case study (Table 7) confirms that relaxed thresholds degrade quality, so the method's success in Table 1 may reflect that retrieving and copying long prompt n-grams inflates F1/ROUGE/Edit-Sim scores, especially for datasets with high source-target overlap (Appendix B, e.g., NQ R=1.00). Thus the central claim that AASD 'improves generation accuracy' via an adaptive verification threshold is not supported as written; the mechanism as specified degenerates to unconditional acceptance in the common high-entropy regime.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Alignment-Augmented Speculative Decoding (AASD), a training-free speculative decoding method that retrieves n-gram drafts from the input prompt, augments them via alignment sampling from the prefilling distribution, and verifies them with an entropy-based adaptive probability threshold. The authors claim that AASD improves generation accuracy over greedy sampling (average score 44.69 to 47.98 on LLaMA3.1-8B-Instruct across eight datasets) while speeding up decoding by up to 2.23x, and that it outperforms REST and PLD in mean acceptance length and throughput. The method is evaluated on QA, summarization, and code completion tasks with two instruction-tuned models, with ablations showing both alignment sampling and conditional verification contribute to efficiency and accuracy gains.","tokens_in":14589,"tokens_out":3954,"duration_ms":37384,"significance":"If the results hold, AASD would be a useful training-free plug-in for speculative decoding that improves both accuracy and efficiency by exploiting prompt–output overlap, a setting where existing retrieval-based methods achieve lower acceptance rates. The paper's strengths include a coherent algorithmic design, a comparison to relevant retrieval-based baselines (REST, PLD) on efficiency, an ablation study isolating the two proposed components, an evaluation on SpecBench, and an honest discussion of limitations including safety concerns and the trade-off between accuracy and acceptance threshold. The central accuracy claim, however, is weakened by the absence of statistical rigor, an apples-to-oranges comparison against standard sampling methods, and a concern that Eq. (5) may reduce to unconditional acceptance for typical high-entropy token distributions.","major_comments":[{"comment":"The section/equation reference is given in the section field.","section":"§3.3, Eq. (5)"},{"comment":"This is the second major comment in the array.","section":"§4.1, Table 1"},{"comment":"This is the third major comment.","section":"§4.1, Table 1"},{"comment":"This is the fourth major comment.","section":"§4.1"}],"minor_comments":[{"comment":"In the text following Eq. (5), 'α a is the factor' is a typo; please correct it to 'α is the factor'.","section":"§3.3"},{"comment":"The pseudocode is underspecified: the Sample function in line 12 is not defined, and the construction of the draft tree T and its attention mask is not explained in the algorithm. Please provide details or a reference to a formal definition.","section":"Appendix C, Algorithm 1"},{"comment":"The header 'ADR' in Table 6 appears to be a typo for 'ARD' (autoregressive decoding); please fix it.","section":"Table 6"},{"comment":"The NQ dataset in Table 8 has a mean target length of 145 tokens, while the other QA datasets have target lengths around 5 tokens. This large discrepancy is not discussed; please explain whether it affects the comparability of the accuracy and efficiency results across QA datasets.","section":"§4.1 and Table 8"},{"comment":"The sentence 'We use unified hyperparameters for AASD on all datasets' is immediately followed by α=0.1 for LLaMA3.1 and α=0.2 for Qwen2.5. Please rephrase to distinguish per-model hyperparameters from per-dataset hyperparameters.","section":"§4.1"},{"comment":"The terms 'strict verification,' 'non-strict verification,' and 'conditional verification' are used without a precise definition of the acceptance rule in the strict case; please define them early in the paper to avoid ambiguity.","section":"§2.2 and §3.3"}],"recommendation":"major_revision","confidential_remarks":"For the editor: The paper is within the journal's scope and the proposed method is interesting, but the central accuracy claim hinges on the behavior of Eq. (5), which may be vacuous in typical high-entropy settings. The authors need to provide an empirical analysis of the threshold distribution and, if necessary, revise the verification rule. The absence of statistical significance testing and the apples-to-oranges comparison in Table 1 also need to be addressed before the paper can be considered for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look for the alignment-sampling idea, but the conditional-verification mechanism trips over its own equation. With α=0.1 and β=0.1, Eq. (5) gives δ = 0.1 − 0.1H. For any verification distribution with entropy above 1 nat—which is most next-token distributions for LLMs—δ is negative, so every token passes the acceptance test. That is the opposite of the paper's stated design: low-confidence tokens are supposed to get stricter thresholds, but high entropy actually makes the threshold permissive. Unless the authors intended normalized entropy or a different log base and forgot to say so, the adaptive piece does no filtering in the common regime. The ablation contrast between adaptive (47.98) and fixed 0.1 (46.80) then just means 'accept more context tokens,' not that the entropy calibration works.\n\nGive credit where it is due: alignment sampling from the prefill distribution to expand retrieved n-grams into draft trees is new relative to REST, PLD, and LLMA, and it is a plausible, training-free way to raise acceptance rates. The ablations show both components contribute, and the efficiency numbers (MAL 2.39, speedup 2.23×) are reasonable for retrieval-based decoding. The limitations section is honest about safety and about applying alignment sampling to REST. The paper is not circular and it invents no entities.\n\nThe soft spots beyond Eq. (5): no error bars or significance tests; α is set per model with no described selection procedure; comparing a decoding algorithm against greedy, top-k, nucleus, and beam mixes method with sampling strategy; and several datasets have high source-target overlap (NQ R=1.00) where copying context inflates F1/ROUGE/Edit-Sim. The case study in Table 7 actually shows that overly relaxed thresholds degrade output, which makes the reported success of the adaptive threshold look suspicious unless something else is going on.\n\nWho gets value from this: people working on training-free speculative decoding could reuse the alignment-sampling trick, and the paper would be a good discussion piece for a reading group. But I would not cite the accuracy claim in its current form. A serious editor should send it to review—the flaw is concrete and fixable, and the core idea has merit. I would ask the authors to report the distribution of δ over real verification steps, clarify the entropy normalization, add error bars, describe how α is chosen, and compare against a no-verification version. If the entropy threshold turns out to be genuinely active, this becomes a solid paper.","headline":"The alignment-sampling idea is genuinely new, but Eq. (5) makes the claimed adaptive verification almost always vacuous, so the central accuracy claim is not supported as written.","tokens_in":15171,"tokens_out":4628,"would_cite":false,"duration_ms":49256,"reading_group":"yes","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AASD, a training-free speculative decoding algorithm, lifts LLaMA3.1 average generation scores by 3.3 points over greedy sampling while speeding decoding by up to 2.23x.","keywords":["speculative decoding","training-free drafting","alignment sampling","conditional verification","draft-target alignment","entropy-based threshold","retrieval-based decoding","LLM inference acceleration"],"falsifier":"Take a task where the prompt contains fluent but misleading text that resembles the expected answer, run AASD with conditional verification enabled and disabled, and compare F1 or ROUGE-L; if enabling conditional verification raises the acceptance of prompt-fragment tokens while lowering the score below greedy, the entropy-based threshold is accepting the wrong fragments.","tokens_in":14038,"feed_emoji":"⚡","tokens_out":9378,"duration_ms":90508,"temperature":0.7,"pith_summary":"Speculative decoding normally accelerates generation only when draft tokens match the target model's own distribution, and existing methods train extra modules to force that match. This paper argues that when drafts are retrieved from the prompt itself, a training-free algorithm can improve both speed and accuracy by attacking alignment from both sides. Alignment sampling adds alternative high-probability tokens from the prefill distribution to poorly aligned draft positions, and conditional verification relaxes or tightens the acceptance threshold per token according to the entropy of its verification distribution. On eight long-context datasets, AASD raises LLaMA3.1-8B-Instruct's average score from 44.69 to 47.98 over greedy sampling while reaching a mean acceptance length of 2.39 and a 2.23x decoding speedup. If the result holds, inference acceleration need not cost quality, and prompt context can serve as a free, task-adaptive draft source.","feed_headline":"A training-free decoder lifts scores 3.3 points and speeds up 2.23x","feed_subtitle":"Drafting from the prompt and checking token entropy makes decoding faster and more accurate at once.","key_machinery":"Two mechanisms carry the argument. Alignment sampling starts from a sliding-window draft pool built from the prompt, retrieves n-grams by the longest matching key, and then, for positions where the retrieved token is not top-ranked in the model's prefill distribution, adds the next most probable tokens as sibling branches; the resulting draft tree is verified in one parallel forward pass. Conditional verification replaces strict or fixed-threshold acceptance with per-token thresholds $\\delta=\\min(-\\alpha H+\\beta,\\Delta)$, where $H$ is the entropy of the target model's verification distribution, $\\Delta$ is its maximum probability, and $\\alpha,\\beta$ are fixed hyperparameters; the $\\Delta$ cap guarantees the most probable token always passes. The design lets the model copy useful text directly from its input context while still refusing low-confidence borrowings.","core_discovery":"On its own terms, the paper claims that poor draft-target alignment, not draft quality, is what limits retrieval-based speculative decoding, and that AASD fixes this without training. First, alignment sampling uses the output distribution the target model already computed during prefilling: at positions where the retrieved n-gram disagrees with the model's top choices, the algorithm appends additional high-probability tokens as extra branches of the draft tree, so the draft becomes more likely to contain the token the target model would emit. Second, conditional verification makes the target model more accommodating toward high-quality but non-aligned drafts: a draft token is accepted when its probability under the target model is at least $\\delta=\\min(-\\alpha H+\\beta,\\Delta)$, where $H$ is the entropy of the verification distribution and $\\Delta$ the highest token probability, so confident predictions use relaxed thresholds and uncertain ones use strict thresholds. With LLaMA3.1-8B-Instruct, this combination improves the average F1, ROUGE-L, and Edit-Sim score across eight datasets from 44.69 under greedy sampling to 47.98, and it surpasses REST and PLD in mean acceptance length and throughput, reaching a 2.23x speedup on LCC code completion.","pith_inferences":["The paper leaves alignment sampling for database-driven drafters such as REST as future work; a testable extension is to precompute top-token distributions for the draft pool and apply AASD unchanged there.","Because a single fixed pair of $\\alpha,\\beta$ is used across all eight datasets, the 3.3-point average likely understates what a per-task or per-model calibration of the entropy threshold could achieve.","The reported batch-size trend suggests the method's practical edge is largest for interactive, low-batch serving rather than high-throughput offline batches.","The safety passage implies AASD effectively raises the model's willingness to trust context, so a safety-aligned variant that filters prompt fragments before acceptance is a natural next direction."],"forward_implications":["AASD can be dropped onto existing autoregressive LLMs without retraining, extra parameters, or an external draft model, so the acceleration transfers when model weights change.","Retrieval from the prompt alone can beat both autoregressive greedy decoding and database-based retrieval such as REST on acceptance length, so the input context is a sufficient draft source for long-context generation.","Accuracy and speed need not trade off: accepting high-quality non-aligned fragments can improve task scores while raising throughput.","Entropy-based adaptive thresholds outperform fixed thresholds and top-k verification in the reported ablations, which implies acceptance decisions should depend on the shape of the verification distribution.","Because the decoder is encouraged to copy context fragments, the paper's limitations state that deployment must filter harmful or untrusted prompt content to avoid propagating it into responses."],"supporting_citations":[{"why":"REST supplies the retrieval-based speculative decoding baseline and the draft-pool and tree-draft design AASD builds on.","marker":"He et al., 2024"},{"why":"PLD is the prompt-lookup baseline that retrieves n-grams directly from the input context, and AASD is compared against it.","marker":"Saxena, 2023"},{"why":"LLMA defines the prompt- and reference-based retrieval drafting family that AASD extends.","marker":"Yang et al., 2023"},{"why":"This work motivates accepting correct but non-aligned draft candidates, the premise behind conditional verification.","marker":"Bachmann et al., 2025"},{"why":"This establishes the draft-and-verify framework and the lossless acceleration ideal that AASD modifies with non-strict verification.","marker":"Leviathan et al., 2023"},{"why":"EAGLE represents the training-based alignment approach that AASD deliberately avoids, framing the training-free contribution.","marker":"Li et al., 2024"},{"why":"SpecBench is the benchmark used to measure and compare decoding speedup ratios.","marker":"Xia et al., 2024"},{"why":"LLaMA3.1-8B-Instruct is the primary target model on which the accuracy and efficiency claims are measured.","marker":"Meta-AI, 2024"}],"fun_headline_variants":["Training-free decoding: +3.3 points, 2.23x speedup","Entropy-driven verification speeds decoding, lifts quality 3.3 pts","Align draft tree with prompt to accelerate LLM output","Prompt-derived drafts and conditional checks boost decoding","AASD: more accurate decoding without training, 2.23x faster"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy gain rests on the heuristic that lower model uncertainty should permit more permissive acceptance thresholds, with a single formula applied across all tasks; if that relation fails on a task, borrowed prompt fragments will degrade generation quality instead of improving it.","fun_headline_variants_meta":{"raw":{"variants":["Training-free decoding: +3.3 points, 2.23x speedup","Entropy-driven verification speeds decoding, lifts quality 3.3 pts","Align draft tree with prompt to accelerate LLM output","Prompt-derived drafts and conditional checks boost decoding","AASD: more accurate decoding without training, 2.23x faster"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1440,"prompt_tokens":990,"completion_tokens":450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":359}},"tokens_in":606,"tokens_out":450,"duration_ms":4960,"temperature":1.0,"reasoning_tokens":359,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:17:04.337665+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a task where the prompt contains fluent but misleading text that resembles the expected answer, run AASD with conditional verification enabled and disabled, and compare F1 or ROUGE-L; if enabling conditional verification raises the acceptance of prompt-fragment tokens while lowering the score below greedy, the entropy-based threshold is accepting the wrong fragments.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LLaMA3.1-8B-Instruct is the primary target model on which the accuracy and efficiency claims are measured."}],"review_version":1}