Pith. sign in

REVIEW 3 major objections 6 minor 76 references

CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding

T0 review · 3 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Block-parallel drafting errors concentrate at low-confidence tokens, and repairing only those positions extends accepted prefixes by 4.2-7.5%.

desk verdict CURE is an honest, incremental repair mechanism for block-parallel speculative decoding that currently costs more wall-clock time than the baseline it builds on, and the repair-branch suffix construction is underspecified enough to undermine the accepted-length gains. read the letter →

arxiv 2608.00531 v1 pith:P5724VTB submitted 2026-08-01 cs.AI

Aofan Liu , Jingxiang Meng , Fangxin Liu , Yongbiao Chen This is my paper
classification cs.AI
keywords speculativedecodingblock-paralleldraftinguncertaintyfocalpointsrepairtreeKV-cacheresynchronizationlargelanguagemodelinferencecodegenerationlatencyoptimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CURE claims that block-parallel draft models usually fail at a few localized low-confidence tokens, and that spending a bounded verification budget on alternative branches at exactly those positions can extend the accepted prefix without full-tree overhead. The method reads the gap between each position's top-1 and top-2 token scores, flags positions below a threshold, attaches a small number of alternative continuations there, and verifies all paths in one target-model pass. Across code-generation and math benchmarks, the repaired tree raises accepted tokens per step by 4.2-7.5% over a single-path parallel draft, and yields 2.66-3.49x end-to-end speedups over target-only autoregressive decoding. The paper is explicit that this is an acceptance gain, not a cheaper draft: tree verification adds 1.40-1.61x TPOT overhead over the parallel backend, so repair pays off only where target-draft disagreement is concentrated.

What carries the argument

The load-bearing object is the Uncertainty Focal Point (UFP) gate: for each draft position, m_i = l_{i,1} - l_{i,2}, the gap between the top two log-probabilities. Positions with m_i below tau_margin = 1.0 are the only places where CURE spends its verification budget, and they are prioritized in ascending margin order. Around these nodes it builds a sparse repair tree: branch 0 is always the original parallel draft, alternative high-scoring tokens are sampled at the UFP, and the suffix is resampled under the substitution. The second essential mechanism is cache resynchronization: when a repair branch beats branch 0, the winning tokens are replayed through the draft backend so the draft model

What would settle it

Take a fixed repair-tree budget and choose repair positions by random selection (or by the inverse of the margin, i.e., highest-confidence positions) instead of low-margin positions, keeping everything else identical. If accepted length per step stays at 4.2-7.5% above the single-path parallel baseline, then the margin gate is not the cause of the gain; the gain would be attributable to extra branching alone. The paper's own precision number (35.9% at a 10% budget) makes this test pointed.

Watch

Extended reading notes

Core claim

CURE's central claim is that drafting errors are not spread evenly across a block; they concentrate in uncertainty focal points, and repairing only those positions recovers most of what a uniform tree or longer draft would recover. It keeps the original parallel path as branch 0, computes the per-position predictive margin m_i = l_{i,1} - l_{i,2}, selects positions with margin below tau_margin = 1.0 in ascending order, and spawns a bounded number of alternative branches (B=5 maximum, depth 15) at those nodes. All candidate paths are verified together in one tree-attention forward pass, and the path with the longest accepted prefix is kept. If a repair path wins, the draft model's KV cache is

Load-bearing premise

The whole method rests on the assumption that the gap between the draft model's top two token scores marks the positions where extra branches will actually extend the accepted prefix; the paper's own offline measure gives that gate only 35.9% precision at a 10% block budget.

Editorial extensions

If this is right

  • Existing block-parallel draft backends can be upgraded with local repair without retraining or added parameters; all new components run at inference time.
  • The gain in accepted tokens does not automatically mean wall-clock speedup: extra tree verification costs 1.40-1.61x TPOT over the parallel baseline, so repair is a tunable knob between arithmetic intensity and memory-bound latency.
  • Cache resynchronization is essential: removing it drops average accepted length by about 40%, so any practical deployment must keep the draft state aligned with the verified output.
  • Repair preserves measured task quality: pass@1 matches target-only autoregressive decoding on the tested code benchmarks.
  • The approach is plug-and-play with standard parallel drafting frameworks rather than a replacement draft architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The offline diagnostic implies a low-cost pre-verification filter could harvest a large share of the gains: selecting 10% of blocks already captures 39.6% of extra accepted tokens, so a static or learned pruning rule might make repair nearly overhead-free.
  • Because the margin gate's precision is only 35.9% at that 10% budget, the reported end-to-end improvement probably depends on the gate being cheap and on tree verification being batched; systems with different backend-model pairs may need a recalibrated threshold.
  • The large GSM8K acceptance gain (32%) hints that local uncertainty repair transfers beyond code, but the paper calls that run a legacy verification protocol, so a matched-protocol replication is needed before treating cross-domain transfer as established.
  • A work-adaptive policy that switches between plain parallel drafting and CURE based on block-level uncertainty would be a natural extension, since on easy spans the repair overhead can exceed its acceptance benefit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes CURE, an inference-time module for block-parallel speculative decoding. CURE identifies 'uncertainty focal points' in a draft block using top-1/top-2 logit margins, attaches a bounded repair tree at those positions, verifies all paths in a single Tree Attention pass, and resynchronizes the draft KV-cache when a repair branch wins. The claimed results are a 4.2–7.5% increase in average accepted length over a single-path parallel draft and an end-to-end speedup of 2.66–3.49× over target-only autoregressive decoding on code benchmarks, with extra experiments on GSM8K. The paper also includes ablations of branch 0 and cache resynchronization, an offline analysis of repair selectivity, and a discussion of the acceptance–compute trade-off.

Significance. If the central claims hold, CURE is a useful plug-in for block-parallel drafters, since it requires no retraining and is architecture-agnostic in principle. The paper makes a fair attempt to separate token-level acceptance from functional correctness, and it explicitly acknowledges that its external comparisons are not controlled. The offline analysis of repair utility is a useful diagnostic. However, the significance is currently limited by several load-bearing gaps: the repair-branch construction is underspecified for a block-parallel drafter, the measured 'fast' path is not target-preserving (exact rate 0.854 in Table 4), and the main configuration ('fast_margin100') and the draft backend are never defined. These issues must be resolved before the headline gains can be trusted as reported.

major comments (3)
  1. [Candidate Tree Construction; Algorithm 1] The construction of repair-branch suffixes is not specified at the level required by the central claim. The draft model q is a block-parallel model emitting one block of K tokens and K logit distributions in a single forward pass; it does not provide conditional per-token logits for a suffix after substituting a token at position i unless q is rerun for each repair prefix. If CURE reuses branch-0 suffix logits for repair branches, then the Tree Attention verification is not evaluating the true q(prefix, substituted-token) distribution, so the speculative acceptance test is not the standard one and the method is not distribution-preserving. If, instead, q is rerun, the effective generation-rate formula in the Methodology (which treats C_d as a constant draft overhead) omits the extra draft forward passes, and the reported speedups may be overestimated. The paper must specify precisely wha
  2. [Scope and Numerical Precision; Table 4] The manuscript acknowledges token-level mismatches in the optimized bfloat16/FlashAttention path and states that only a 'strict replay path' restores target consistency at additional latency. Table 4 reports an exact rate of only 0.854 for the full method, i.e., 14.6% of generated sequences differ from target-only AR. The headline speedups in Table 1 are measured on this fast, non-consistent path. Since speculative decoding is normally advertised as lossless, and the central claim is that CURE improves acceptance over a parallel baseline, the paper must either (a) report the same speedup and acceptance metrics on the strict replay path, or (b) clearly and prominently reframe the claims as applying to a non-target-preserving approximation. As written, the fast path's speedup is not established for a target-preserving decoder.
  3. [Experimental Setup; Table 1] The configuration 'fast_margin100' used for all main results is never defined. The reader cannot tell what margin threshold, branch budget, tree depth, verification beam, or margin scale it uses, nor how it relates to the stated defaults (τ_margin = 1.0, B = 5, depth = 15, beam = 5). The block-parallel draft backend q is described only as a 'pretrained Qwen3-8B-based block-parallel draft backend (block size K = 16)' without naming the architecture, training data, or implementation. All results appear to be single runs with no error bars or multiple seeds, and the hyperparameters (τ, B, depth, beam, s) are selected by hand without sensitivity analysis. These omissions make the 4.2–7.5% acceptance gain and the 2.66–3.49× speedup hard to interpret or reproduce, and they leave open the possibility that the reported gains are specific to one unstated configuration.
minor comments (6)
  1. [Equation (3)] The margin scale s in the dynamic branching formula is never given a value or a sensitivity analysis. Please specify s and state how it was chosen.
  2. [GSM8K paragraph] The phrase 'under the legacy verification protocol' is undefined. Please specify what protocol was used for GSM8K and how it differs from the code-benchmark protocol.
  3. [Table 4] The column 'Accept ratio' is not defined. The caption says 'Ratios use the parallel draft as reference,' but it is unclear whether this is a ratio of accepted lengths, acceptance probabilities, or something else.
  4. [Table 2 caption] The caveat about different hardware, generation lengths, and implementations is appreciated, but the caption 'Reported Qwen3-8B speedups' could mislead. Consider renaming it 'Context from prior reported results' to match the body text.
  5. [Figure 1] Figure 1 lists 'depth = 15' and 'verification beam = 5' without defining how tree depth and beam width constrain the candidate tree in the algorithm. A formal definition would help.
  6. [General] The paper does not describe any code or artifact release. For an empirical systems paper, naming the parallel backend and releasing the implementation would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; CURE's speedup claims are empirical, though the accepted-length gain is partly structural.

full rationale

Most of CURE's derivation chain is empirical rather than formal. The margin gate (Eqs. 2-3) selects uncertain positions, tree construction substitutes alternative tokens and resamples suffixes, Tree Attention verifies against the target, and the longest accepted prefix (Eq. 4) is selected. The reported speedups are measured against target-only autoregressive decoding in the same run protocol and are not forced by any fitted parameter or self-citation. One structural caveat exists: since branch 0 (pi_0) is always kept in the verification tree and the framework selects argmax a(pi), the accepted length relative to branch 0 is guaranteed to be non-decreasing by construction; the 4.2-7.5% magnitude is still empirical, and the end-to-end speedup depends on verification overhead, which the paper reports as a 1.40-1.61x TPOT increase over the parallel baseline. The paper also self-cites DART (Liu et al. 2026) in Related Work, but that citation is contextual and not load-bearing. The 'Scope and Numerical Precision' passage admits token-level mismatches on the optimized bfloat16/FlashAttention path and states that only a strict replay path restores target consistency; this is a validity limitation, not circularity, and it is explicitly acknowledged. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors, and no central claim reduces to a self-citation chain. Score 2 reflects the mild by-construction positivity of the accepted-length comparison and the minor non-load-bearing self-citation, not a circular derivation.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method rests on the empirical reliability of logit-margin gating, the availability of top-2 logits from an unnamed backend, and exact KV replay; all are domain assumptions that are only weakly validated.

free parameters (6)
  • margin threshold tau_margin = 1.0
    Positions with m_i < 1.0 are marked as UFPs; no sensitivity analysis is reported.
  • branch budget B (bmin/bmax) = B=5, bmin=1, bmax=5
    At most 5 alternative branches per selected position; dynamic mapping Eq. (3).
  • max tree depth = 15
    Candidate paths can extend to depth 15.
  • verification beam width = 5
    Active tree frontier is constrained to beam 5.
  • margin scale s = not specified
    Used in Eq. (3) to normalize u_i; value never given.
  • block size K = 16
    Length of block-parallel draft; from experimental setup.
assumptions (4)
  • domain assumption Top-2 logit margin is a valid uncertainty signal for token error location
    The entire UFP detection rests on Eq. (2); the paper provides only indirect support (offline Table 5) and low precision.
  • domain assumption Block-parallel draft backend exposes per-position top-2 log-probabilities in a single forward pass
    The unnamed Qwen3-8B-based backend must return these logits; not proven or cited.
  • domain assumption Replaying the winning repair path through the draft backend exactly restores the KV cache state
    Cache resynchronization (Section 'Joint Verification and Cache Resynchronization') assumes replay is exact; ablation shows it is necessary but not that it is exact.
  • standard math Tree Attention verification over the sparse tree yields identical acceptance decisions as independent path verification
    Standard property of tree verification used to justify single-pass verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding." pith.science (2026). https://pith.science/paper/P5724VTB

@misc{pith2026260800531,
  author       = {Pith},
  title        = {Pith review of: CURE: Local Uncertainty Repair for Block-Parallel Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P5724VTB}},
  note         = {Machine review of arXiv:2608.00531}
}
abstract

Speculative decoding mitigates the latency of sequential generation in autoregressive Large Language Models (LLMs) by interleaving draft generation with target verification. However, existing parallel drafting backends often suffer from rapid accuracy degradation over long horizons, leading to high rejection rates during verification and suboptimal wall-clock speedups. We observe that drafting errors are not uniformly distributed but typically stem from localized high-uncertainty tokens that destabilize downstream generation trajectories. Motivated by this token error pattern, we propose CURE, a budget-aware dynamic repair tree designed to repair errors at uncertainty focal points without incurring prohibitive tree-verification overheads. Specifically, our method uses predictive confidence margins to dynamically locate candidate error tokens within a block-parallel draft, expands bounded repair paths only at these fragile nodes, and employs a novel repair resynchronization mechanism to realign draft states post-verification. Evaluations on code-generation benchmarks (HumanEval, MBPP, and LiveCodeBench-lite) and mathematical reasoning benchmark (GSM8K) demonstrate that CURE increases the average accepted length by 4.2-7.5% over parallel baselines without repair, translating to an end-to-end speedup of $2.66-3.49\times$ over target-only decoding. Furthermore, we provide a plug-and-play repair module compatible with standard parallel drafting frameworks. We also characterize the trade-off between draft compute and verification efficiency.

Figures

Figures reproduced from arXiv: 2608.00531 by the authors.

Figure 1
Figure 1. Overview of one decoding iteration of CURE: a block-parallel draft produces branch 0 alongside per-position uncertainty margins. Low-margin positions receive a bounded repair budget, and the target model verifies branch 0 together with the resulting repair paths in a single pass. The framework selects the longest accepted prefix and resynchronizes the draft cache only when a repair path wins. calized ambiguity that … view at source ↗
Figure 2
Figure 2. Reported Qwen3-8B speedups from Table 2. Filled markers denote CURE; hollow markers denote source-specific [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

76 extracted references · 60 canonical work pages

  1. [1]

    Proceedings of the 40th International Conference on Machine Learning , year=

    Fast Inference from Transformers via Speculative Decoding , author=. Proceedings of the 40th International Conference on Machine Learning , year=

  2. [3]

    Proceedings of the 41st International Conference on Machine Learning , pages=

    Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads , author=. Proceedings of the 41st International Conference on Machine Learning , pages=

    work page

  3. [4]

    Proceedings of the 41st International Conference on Machine Learning , pages=

    EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty , author=. Proceedings of the 41st International Conference on Machine Learning , pages=

    work page

  4. [6]

    Chen, Jian and Liang, Yesheng and Liu, Zhijian , journal=

    work page

  5. [7]

    Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems , pages=

    SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification , author=. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems , pages=

    work page

  6. [8]

    Advances in Neural Information Processing Systems , year=

    SEQUOIA: Scalable and Robust Speculative Decoding , author=. Advances in Neural Information Processing Systems , year=

    work page

  7. [9]

    Proceedings of the 34th International Conference on Software Engineering , pages=

    On the Naturalness of Software , author=. Proceedings of the 34th International Conference on Software Engineering , pages=

    work page

  8. [10]

    Cognitive Science , volume=

    Do Programmers Prefer Predictable Expressions in Code? , author=. Cognitive Science , volume=

    work page

Show all 76 references
  1. [11]

    Advances in Neural Information Processing Systems , volume=

    Blockwise Parallel Decoding for Deep Autoregressive Models , author=. Advances in Neural Information Processing Systems , volume=

  2. [12]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , pages=

    Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics , pages=

  3. [13]

    He, Zhenyu and Zhong, Zexuan and Cai, Tianle and Lee, Jason and He, Di , booktitle=

  4. [14]

    Break the Sequential Dependency of

    Fu, Yichao and Bailis, Peter and Stoica, Ion and Zhang, Hao , booktitle=. Break the Sequential Dependency of

  5. [15]

    Wang, Jikai and Su, Yi and Li, Juntao and Xia, Qingrong and Ye, Zi and Duan, Xinyu and Wang, Zhefeng and Zhang, Min , journal=

  6. [16]

    Li, Yuhui and Wei, Fangyun and Zhang, Chao and Zhang, Hongyang , booktitle=

  7. [17]

    Elhoushi, Mostafa and others , booktitle=

  8. [18]

    International Conference on Learning Representations , year=

    Learning Harmonized Representations for Speculative Sampling , author=. International Conference on Learning Representations , year=

  9. [19]

    First Conference on Language Modeling , year=

    Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding , author=. First Conference on Language Modeling , year=

  10. [20]

    Advances in Neural Information Processing Systems , volume=

    Scaling Speculative Decoding with Lookahead Reasoning , author=. Advances in Neural Information Processing Systems , volume=

  11. [21]

    Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation , pages=

    How Speculative Can Speculative Decoding Be? , author=. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation , pages=

    2024

  12. [22]

    Sun, Ziteng and Suresh, Ananda Theertha and Ro, Jae Hun and Beirami, Ahmad and Jain, Himanshu and Yu, Felix , booktitle=

  13. [23]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Graph-Structured Speculative Decoding , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  14. [24]

    Lin, Chi-Heng and Tuli, Shikhar and Smith, James and Hsu, Yen-Chang and Shen, Yilin and Jin, Hongxia , booktitle=

  15. [25]

    Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit

    Li, Zhen and Su, Yupeng and Wang, Songmiao and Yang, Runming and Xie, Congkai and Liu, Aofan and Li, Ming and Cao, Jiannong and Xie, Yuan and Wong, Ngai and Yang, Hongxia , journal=. Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit

  16. [26]

    Yang, Cehao and Lin, Xueyuan and Xu, Chengjin and Jiang, Xuhui and Ma, Shengjie and Liu, Aofan and Xiong, Hui and Guo, Jian , booktitle=

  17. [27]

    2025 , doi=

    Wang, Bin and Li, Hui and Liu, Aofan and Yang, Botao and Yang, Ao and Zhong, Yilu and Huang, Weixiang and Zhang, Yanping and Huang, Runhuai and Zeng, Weimin , booktitle=. 2025 , doi=

    2025

  18. [28]

    Liu, Steven and Luo, Jane and Zhang, Xin and Liu, Aofan and Liu, Hao and Wu, Jie and Huang, Ziyang and Huang, Yangyu and Kang, Yu and Li, Scarlett , journal=

  19. [29]

    Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation , author=. Findings of the Association for Computational Linguistics: EMNLP 2023 , pages=

    2023

  20. [31]

    International Conference on Learning Representations , year=

    Zhou, Yongchao and Lyu, Kaifeng and Rawat, Ankit Singh and Menon, Aditya Krishna and Rostamizadeh, Afshin and Kumar, Sanjiv and Kagy, Jean-Fran. International Conference on Learning Representations , year=

  21. [32]

    Proceedings of the 41st International Conference on Machine Learning , pages=

    Online Speculative Decoding , author=. Proceedings of the 41st International Conference on Machine Learning , pages=

  22. [33]

    Huang, Kaixuan and Guo, Xudong and Wang, Mengdi , journal=

  23. [34]

    Zhang, Situo and Wang, Hankun and Ma, Da and Zhu, Zichen and Chen, Lu and Lan, Kunyao and Yu, Kai , journal=

  24. [35]

    Xiong, Yunfan and Zhang, Ruoyu and Li, Yanzeng and Wu, Tianhao and Zou, Lei , journal=

  25. [36]

    Li, Yuhui and Wei, Fangyun and Zhang, Chao and Zhang, Hongyang , journal=

  26. [37]

    Liu, Fuliang and Li, Xue and Zhao, Ketai and Gao, Yinxi and Zhou, Ziyan and Zhang, Zhonghui and Wang, Zhibin and Dou, Wanchun and Zhong, Sheng and Tian, Chen , journal=

  27. [38]

    Shi, Weijie and Xu, Qiang and Deng, Fan and Wu, Yaguang and Liu, Jiarun and Xu, Yehong and Chen, Hao and Zhu, Jia and Xu, Jiajie and Huang, Xiangjun and Yang, Jian and Zhou, Xiaofang , journal=

  28. [39]

    Speculative Decoding for

    Xu, Changran and Liu, Yi and Zhou, Yunhao and Huang, Shan and Xu, Ningyi and Xu, Qiang , journal=. Speculative Decoding for

  29. [42]

    Jain, Naman and Han, King and Gu, Alex and Li, Wen-Ding and Yan, Fanjia and Zhang, Tianjun and Wang, Sida and Solar-Lezama, Armando and Sen, Koushik and Stoica, Ion , journal=

  30. [44]

    Ankner, Z.; Parthasarathy, R.; Nrusimha, A.; Rinard, C.; Ragan-Kelley, J.; and Brandon, W. 2024. Hydra: Sequentially-Dependent Draft Heads for Medusa Decoding. In First Conference on Language Modeling

    2024

  31. [45]

    Austin, J.; Odena, A.; Nye, M.; Bosma, M.; Michalewski, H.; Dohan, D.; Jiang, E.; Cai, C.; Terry, M.; Le, Q.; and Sutton, C. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732

    2021 arXiv

  32. [46]

    Cai, T.; et al. 2024. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. In Proceedings of the 41st International Conference on Machine Learning, 5209--5235

    2024

  33. [47]

    Casalnuovo, C.; Lee, K.; Wang, H.; Devanbu, P.; and Morgan, E. 2020. Do Programmers Prefer Predictable Expressions in Code? Cognitive Science, 44(12): e12921

    2020

  34. [48]

    Chen, C.; et al. 2023. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv preprint arXiv:2302.01318

    2023 arXiv

  35. [49]

    Chen, J.; Liang, Y.; and Liu, Z. 2026. DFlash : Block Diffusion for Flash Speculative Decoding. arXiv preprint arXiv:2602.06036

    2026 arXiv

  36. [50]

    Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. d. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374

    2021 arXiv

  37. [51]

    Chen, Z.; et al. 2024. SEQUOIA: Scalable and Robust Speculative Decoding. In Advances in Neural Information Processing Systems

    2024

  38. [52]

    Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; Hesse, C.; and Schulman, J. 2021. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  39. [53]

    Elhoushi, M.; et al. 2024. LayerSkip : Enabling Early Exit Inference and Self-Speculative Decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 12622--12642

    2024

  40. [54]

    Fu, Y.; Bailis, P.; Stoica, I.; and Zhang, H. 2024. Break the Sequential Dependency of LLM Inference Using Lookahead Decoding. In Proceedings of the 41st International Conference on Machine Learning, 14060--14079

    2024

  41. [55]

    Fu, Y.; Ge, R.; Shao, Z.; Deng, Z.; and Zhang, H. 2025. Scaling Speculative Decoding with Lookahead Reasoning. In Advances in Neural Information Processing Systems, volume 38

    2025

  42. [56]

    Gong, Z.; Liu, J.; Wang, Z.; Wu, P.; Wang, J.; Cai, X.; Zhao, D.; and Yan, R. 2024. Graph-Structured Speculative Decoding. In Findings of the Association for Computational Linguistics: ACL 2024, 11404--11415

    2024

  43. [57]

    He, Z.; Zhong, Z.; Cai, T.; Lee, J.; and He, D. 2024. REST : Retrieval-Based Speculative Decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics, 1582--1595

    2024

  44. [58]

    T.; Gabel, M.; Su, Z.; and Devanbu, P

    Hindle, A.; Barr, E. T.; Gabel, M.; Su, Z.; and Devanbu, P. 2012. On the Naturalness of Software. In Proceedings of the 34th International Conference on Software Engineering, 837--847

    2012

  45. [59]

    Huang, J.; Zhang, Y.; Zhang, Q.; Lin, H.; Xu, H.; and Zhang, L. 2026. Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding. arXiv preprint arXiv:2605.29707

    2026 arXiv

  46. [60]

    Huang, K.; Guo, X.; and Wang, M. 2024. SpecDec++ : Boosting Speculative Decoding via Adaptive Candidate Lengths. arXiv preprint arXiv:2405.19715

    2024 arXiv

  47. [61]

    Jain, N.; Han, K.; Gu, A.; Li, W.-D.; Yan, F.; Zhang, T.; Wang, S.; Solar-Lezama, A.; Sen, K.; and Stoica, I. 2024. LiveCodeBench : Holistic and Contamination Free Evaluation of Large Language Models for Code. arXiv preprint arXiv:2403.07974

    2024 arXiv

  48. [62]

    Leviathan, Y.; Kalman, M.; and Matias, Y. 2023. Fast Inference from Transformers via Speculative Decoding. Proceedings of the 40th International Conference on Machine Learning

    2023

  49. [63]

    Li, Y.; Wei, F.; Zhang, C.; and Zhang, H. 2024 a . EAGLE -2: Faster Inference of Language Models with Dynamic Draft Trees. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 7421--7432

    2024

  50. [64]

    Li, Y.; Wei, F.; Zhang, C.; and Zhang, H. 2025. EAGLE -3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test. arXiv preprint arXiv:2503.01840

    2025 arXiv

  51. [65]

    Li, Y.; et al. 2024 b . EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. In Proceedings of the 41st International Conference on Machine Learning, 28935--28948

    2024

  52. [66]

    Lin, C.-H.; Tuli, S.; Smith, J.; Hsu, Y.-C.; Shen, Y.; and Jin, H. 2024. SLiM : Speculative Decoding with Hypothesis Reduction. In Findings of the Association for Computational Linguistics: NAACL 2024, 1005--1017

    2024

  53. [67]

    Liu, F.; Li, X.; Zhao, K.; Gao, Y.; Zhou, Z.; Zhang, Z.; Wang, Z.; Dou, W.; Zhong, S.; and Tian, C. 2026. DART : Diffusion-Inspired Speculative Decoding for Fast LLM Inference. arXiv preprint arXiv:2601.19278

    2026

  54. [68]

    Liu, X.; Hu, L.; Bailis, P.; Cheung, A.; Deng, Z.; Stoica, I.; and Zhang, H. 2024. Online Speculative Decoding. In Proceedings of the 41st International Conference on Machine Learning, 31131--31146

    2024

  55. [69]

    Liu, Z.; Zhang, C.; and Song, D. 2024. How Speculative Can Speculative Decoding Be? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, 8265--8275

    2024

  56. [70]

    Miao, X.; et al. 2024. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verification. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, 932--949

    2024

  57. [71]

    Shi, W.; Xu, Q.; Deng, F.; Wu, Y.; Liu, J.; Xu, Y.; Chen, H.; Zhu, J.; Xu, J.; Huang, X.; Yang, J.; and Zhou, X. 2026. SpecBlock : Block-Iterative Speculative Decoding with Dynamic Tree Drafting. arXiv preprint arXiv:2605.07243

    2026 arXiv

  58. [72]

    Stern, M.; Shazeer, N.; and Uszkoreit, J. 2018. Blockwise Parallel Decoding for Deep Autoregressive Models. In Advances in Neural Information Processing Systems, volume 31

    2018

  59. [73]

    T.; Ro, J

    Sun, Z.; Suresh, A. T.; Ro, J. H.; Beirami, A.; Jain, H.; and Yu, F. 2023. SpecTr : Fast Speculative Decoding via Optimal Transport. In Advances in Neural Information Processing Systems, volume 36

    2023

  60. [74]

    Wang, J.; Su, Y.; Li, J.; Xia, Q.; Ye, Z.; Duan, X.; Wang, Z.; and Zhang, M. 2025. OPT -Tree: Speculative Decoding with Adaptive Draft Tree Structure. Transactions of the Association for Computational Linguistics, 13: 188--199

    2025

  61. [75]

    Xia, H.; Ge, T.; Wang, P.; Chen, S.-Q.; Wei, F.; and Sui, Z. 2023. Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, 3909--3925

    2023

  62. [76]

    Xiong, Y.; Zhang, R.; Li, Y.; Wu, T.; and Zou, L. 2024. DySpec : Faster Speculative Decoding with Dynamic Token Tree Structure. arXiv preprint arXiv:2410.11744

    2024 arXiv

  63. [77]

    Xu, C.; Liu, Y.; Zhou, Y.; Huang, S.; Xu, N.; and Xu, Q. 2025. Speculative Decoding for Verilog : Speed and Quality, All in One. arXiv preprint arXiv:2503.14153

    2025 arXiv

  64. [78]

    Yan, M.; Agarwal, S.; and Venkataraman, S. 2024. Decoding Speculative Decoding. arXiv preprint arXiv:2402.01528

    2024 arXiv

  65. [79]

    Zhang, J.; Wang, J.; Li, H.; Shou, L.; Chen, K.; Chen, G.; and Mehrotra, S. 2024 a . Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, 11263--11282

    2024

  66. [80]

    Zhang, L.; Wang, X.; Huang, Y.; and Xu, R. 2025. Learning Harmonized Representations for Speculative Sampling. In International Conference on Learning Representations

    2025

  67. [81]

    Zhang, S.; Wang, H.; Ma, D.; Zhu, Z.; Chen, L.; Lan, K.; and Yu, K. 2024 b . AdaEAGLE : Optimizing Speculative Decoding via Explicit Modeling of Adaptive Draft Structures. arXiv preprint arXiv:2412.18910

    2024 arXiv

  68. [82]

    S.; Menon, A

    Zhou, Y.; Lyu, K.; Rawat, A. S.; Menon, A. K.; Rostamizadeh, A.; Kumar, S.; Kagy, J.-F.; and Agarwal, R. 2024. DistillSpec : Improving Speculative Decoding via Knowledge Distillation. In International Conference on Learning Representations

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.