Pith. sign in

REVIEW 2 major objections 6 minor 68 references

A small autoregressive adapter turns factorized draft marginals into high-acceptance proposal trees, delivering 4.37 imes decoding speedup.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 21:47 UTC pith:ZAGE5H3V

load-bearing objection Solid hybrid systems paper: Weaver lifts the factorized acceptance ceiling and the GDN tree kernel fills a real gap; 24.7% over tuned DFlash is well measured. the 2 major comments →

arxiv 2607.06763 v1 pith:ZAGE5H3V submitted 2026-07-07 cs.LG cs.CL

Trees from Marginals: Autoregressive drafting with factorized priors

Yuma Oda , Ryan Mathieu , Roman Knyazhitskiy , Artur Chakhvadze This is my paper
classification cs.LG cs.CL
keywords speculative decodingfactorized draft modelsautoregressive residual adapterproposal treesGated Delta Nettree verificationacceptance rateLLM inference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Speculative decoding speeds up language-model generation by drafting several candidate tokens and checking them in one verifier pass. Factorized drafters are very fast because they emit many future-token marginals in parallel, but the independence assumption makes acceptance collapse as draft length grows. This paper claims that a lightweight autoregressive adapter, Weaver, can restore the missing conditional dependencies by building trees only over the top-K tokens of those marginals, without ever multiplying by the full vocabulary matrix. Combined with a new rollback-free tree-verification algorithm for Gated Delta Net layers, the hybrid method (DFlash-TfM) raises mean accepted length enough to reach a 4.37 imes speedup over ordinary autoregressive decoding and a 24.7% gain over a tuned factorized baseline on a 27B model. A sympathetic reader cares because the same independence ceiling that has limited diffusion-style drafters can be lifted with only tens of millions of extra parameters and a systems kernel that works for modern hybrid architectures.

Core claim

The acceptance ceiling of factorized drafters is structural, not a capacity limit: any pure-marginal oracle is eventually beaten by a small autoregressive residual that conditions on realized draft tokens inside the top-K support. Weaver implements that residual, constructs maximum-probability trees from it, and, with a fused GDN tree kernel, yields higher tokens-per-step than either chain factorized drafting or prior tree methods built on the same marginals.

What carries the argument

Weaver: a single-layer 56.7M-parameter autoregressive residual adapter that adds corrections only over the K=512 top marginal tokens of a frozen factorized drafter, then builds proposal trees that are verified by a masked triangular-solve kernel for Gated Delta Net layers.

Load-bearing premise

Almost all of the verifier’s probability mass must stay inside the frozen top-512 tokens of the factorized drafter; if mass systematically leaves that pool, the residual adapter cannot recover it.

What would settle it

Measure verifier mass outside the top-512 DFlash candidates on held-out long-horizon or out-of-domain sequences; if that mass rises substantially and mean accepted length of DFlash-TfM falls below the pure-marginal oracle, the hybrid claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes DFlash-TfM (Trees from Marginals): a hybrid speculative-decoding pipeline that takes top-K future-token marginals from a frozen factorized DFlash drafter and feeds them as a constrained prior to Weaver, a 56.7M-parameter autoregressive residual adapter that builds proposal trees without a full-vocabulary projection. For Qwen3.6-style targets with Gated Delta Net layers, the authors derive a rollback-free tree verification algorithm based on a partial-order chunked delta-rule algebra and implement a fused CUDA kernel in SGLang. Empirically, on Qwen3.6-27B (bf16, batch size 1) across chat/math/code workloads, tree-verified DFlash-TfM reports a 4.37× speedup over autoregressive decoding and a 24.7% interactivity gain over an optimized DFlash chain baseline, with ablations isolating the residual adapter (vs DDTree) and the tree structure (vs chain).

Significance. If the reported gains hold under broader deployment, the work is a clear advance for local and interactivity-sensitive LLM inference: it shows that the independence ceiling of factorized drafters is structural rather than capacity-limited, and that a small AR residual over top-K marginal support is enough to lift long-depth acceptance while preserving the parallel draft cost. The GDN tree kernel is independently useful for any non-diagonal linear-attention target and is backed by microbenchmarks (Tables 1/3, Figure 5) and a precision claim relative to a double-precision reference. Strengths include multi-workload Table 2, acceptance-vs-depth curves with CIs (Figure 4), verification ablations (Table 4, Figure 6), budget sweeps (Appendix B), and released Weaver weights plus SGLang kernels—reproducible systems artifacts that raise the bar for speculative-decoding papers.

major comments (2)
  1. §3.5 and Figure 4: the claim that DFlash-TfM exceeds the acceptance ceiling of any marginal / argmax-marginal drafter is load-bearing for the theoretical framing. The plotted oracle is estimated under chain-style position-wise TV (Eqs. 15–17) with finite-M Monte Carlo. Please state explicitly that this ceiling applies to independent (or argmax) marginal proposals under the analyzed coupling, not to all tree constructions from marginals (e.g. DDTree), and report how sensitive the oracle curves are to M and to the conditioning event A (prefix acceptance). Without that scope statement, readers may over-read the figure as ruling out all marginal-based tree methods rather than pure factorized chains.
  2. §4.1.3 footnote and the free parameter K=512: the hybrid’s validity rests on the claim that DFlash’s top-K pool retains ~97.8% of verifier mass on held-out data. This average is not broken down by draft depth, temperature, or domain. Because the paper’s main qualitative claim is long-depth superiority (Figure 4; §5.1 MAL gains), please report mass-in-pool vs position (and ideally vs dataset) and, if mass leakage grows with depth, quantify the effect on residual logits / acceptance. A short sensitivity sweep over K would also make the free-parameter choice less ad hoc.
minor comments (6)
  1. Abstract / Figure 1 caption: abstract reports 24.7% over DFlash; Figure 1 caption says “25% on average.” Align the headline number.
  2. §5.1 vs Table 2: prose cites 392.8 tok/s and chain 121.5 / tree 296.9 tok/s; these absolute rates are not in Table 2 (which reports speedup and τ). Either add a column or cite the figure/appendix source.
  3. §3.4.1–3.4.2: the GDN tree algebra (Eqs. 9–13) is clear, but a one-line statement that Traversal verification [10] is applied after the fused forward (and that commit replays only the accepted path) would help readers who jump to the kernel section.
  4. Bibliography: several entries carry 2026 dates (e.g. DFlash, PARD-2, Traversal Verification). If these are preprints, mark them as such for archival clarity.
  5. Algorithm 1 (Appendix A): “Depth-l candidates” indexing in the residual line is slightly ambiguous relative to node depth; a short comment on how cands_node is chosen per depth would help reimplementation.
  6. Typos / polish: e.g. “full ­vocabulary” spacing artifacts, “in­block”, and the decorative word cloud on page 1 do not aid readability in a journal version.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper whose acceptance bounds, GDN algebra, and speedups are independently measured or derived, not forced by construction or self-citation.

full rationale

The paper is a hybrid drafting + systems contribution. Weaver is trained with an LK surrogate (KL + TV) that targets acceptance, then evaluated on held-out multi-workload interactivity (Table 2) and against an independently estimated marginal-oracle ceiling obtained by sampling target-model continuations (Eqs. 14–17, Fig. 4); the reported exceedance of that ceiling is therefore an empirical observation, not a tautology. The tree-construction algorithm (Alg. 1) and residual logits are ordinary best-first expansion over a truncated vocabulary; nothing equates the output tree probability to the training loss by definition. The GDN tree-verification algebra (Eqs. 7–13) is a direct, non-circular extension of the known chunked delta-rule recurrence to a partial order, implemented as a fused kernel whose timings are measured against a recurrent baseline (Table 3, Fig. 5). No uniqueness theorem, ansatz, or load-bearing premise is imported solely via overlapping-author citation. All central claims rest on released weights, open kernels, and external benchmarks, so the derivation chain is self-contained.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central speedup claim rests on standard speculative-decoding theory, the published DFlash checkpoint, the GDN recurrence, and a handful of engineering hyper-parameters chosen for the Qwen3.6-27B setting. No new physical entities are postulated; free parameters are ordinary ML knobs.

free parameters (5)
  • candidate pool size K = 512
    Fixed at 512; controls both residual projection cost and mass coverage. Reported 97.8% average mass retention is measured, not derived.
  • tree expansion width w = 2-8
    Chosen in {2–8} to trade draft quality for parallel Weaver throughput; optimal value is empirical.
  • LK-loss mixing schedule η = 2
    Set to 2 (paper notes default 3 works less well); controls KL-to-TV transition during training.
  • greedy-match auxiliary weight γ = 0.1
    Set to 0.1 to encourage argmax agreement under greedy verification.
  • Weaver architecture size = 56.7M params
    Single transformer layer, dim 2048, 16 heads, MLP 2048 → 56.7 M parameters; chosen for latency budget.
axioms (4)
  • domain assumption Speculative sampling acceptance probability equals 1 − TV(p_draft, p_verifier) and is optimal for single-step drafts (Leviathan et al., Chen et al.).
    Used throughout §2.2 and §3.5 to bound factorized drafters and to justify the LK training objective.
  • domain assumption Gated Delta Net state update S_t = α_t (I − β_t k_t k_t^⊤) S_{t−1} + β_t k_t v_t^⊤ admits a chunked triangular-solve form that can be extended to ancestor-masked trees.
    Core of the novel kernel derivation in §3.4; rests on the published GDN recurrence of Yang et al.
  • ad hoc to paper Top-K marginal tokens of a well-trained factorized drafter retain nearly all verifier probability mass for the evaluated workloads.
    Measured 97.8% on held-out data; required for the truncated-vocabulary residual to be lossless in practice.
  • domain assumption Maximum-draft-probability tree construction (DySpec-style best-first with batched width-w expansion) is a sufficiently good proxy for expected acceptance.
    Adopted from DySpec; paper notes it is imperfect but empirically effective.
invented entities (2)
  • Weaver independent evidence
    purpose: Lightweight AR residual adapter that conditions on DFlash top-K marginals and verifier hidden states to restore token dependencies without full-vocabulary projection.
    New model architecture introduced and trained by the authors; independent evidence is the released weights and measured acceptance gains.
  • Rollback-free GDN tree-verification kernel independent evidence
    purpose: Single-pass ancestor-masked triangular solve that scores an entire draft tree without speculative state writes, followed by a short commit replay.
    New systems primitive derived and implemented in CUDA for SGLang; independent evidence is the open-source kernels and micro-benchmarks.

pith-pipeline@v1.1.0-grok45 · 25045 in / 3293 out tokens · 36264 ms · 2026-07-10T21:47:42.918206+00:00 · methodology

0 comments
read the original abstract

Speculative decoding greatly increases the interactivity of autoregressive language models by trading off computation for extra tokens generated in a single forward pass. Factorized draft models are especially efficient because they predict future-token marginals in parallel, but their independence assumption causes acceptance rates to degrade sharply as the speculative budget grows. We analyze this limitation and introduce Weaver, a lightweight autoregressive adapter that constructs proposal trees from the top-K marginals of a factorized drafter. Weaver restores conditional dependencies between proposed tokens while avoiding a full-vocabulary projection. To support fast verification for models with Gated Delta Net layers, we derive a rollback-free tree-verification algorithm and implement optimized CUDA kernels in SGLang. By combining these model and systems contributions we achieve a 4.37-fold speedup over autoregressive decoding, and outperform a highly optimized DFlash baseline by 24.7%.

Figures

Figures reproduced from arXiv: 2607.06763 by Artur Chakhvadze, Roman Knyazhitskiy, Ryan Mathieu, Yuma Oda.

Figure 1
Figure 1. Figure 1: Comparison of decoding speed for Qwen3.6 [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the DFlash-TfM drafting procedure. A factorized drafter emits marginal distributions for several positions in a single forward pass. Because the factorized drafter ignores conditional dependencies between positions, the acceptance rate falls as the draft length grows ( [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Tree verification modifies the GDN chunk algebra to use a partial [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Acceptance probability for a token at a specific future position [PITH_FULL_IMAGE:figures/full_fig_p017_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The scaling behaviour of verification time vs the number of draft [PITH_FULL_IMAGE:figures/full_fig_p022_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Mean accepted length versus draft length for chain proposals on [PITH_FULL_IMAGE:figures/full_fig_p024_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Interactivity versus speculative budget with temperature 1.0 and [PITH_FULL_IMAGE:figures/full_fig_p034_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Acceptance length (τ) versus speculative budget temperature 1.0 and reasoning on. 34 [PITH_FULL_IMAGE:figures/full_fig_p034_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Interactivity versus speculative budget with temperature 1.0 and [PITH_FULL_IMAGE:figures/full_fig_p035_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Acceptance length (τ) versus speculative budget temperature 1.0 and reasoning off. 35 [PITH_FULL_IMAGE:figures/full_fig_p035_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Interactivity versus speculative budget with greedy decoding and [PITH_FULL_IMAGE:figures/full_fig_p036_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Acceptance length (τ) versus speculative budget greedy decoding and reasoning on. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Interactivity versus speculative budget with greedy decoding and [PITH_FULL_IMAGE:figures/full_fig_p037_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Acceptance length (τ) versus speculative budget greedy decoding and reasoning off. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: DFlash: acceptance by draft token position on MTBench with tem [PITH_FULL_IMAGE:figures/full_fig_p038_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: DFlash-TfM: acceptance by draft token position on MTBench with temperature 1.0 38 [PITH_FULL_IMAGE:figures/full_fig_p038_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 68 canonical work pages · 36 internal anchors

  1. [1]

    Attention Is All You Need

    A. Vaswani et al., “Attention Is All You Need,” p. 1, Jun. 2017, [Online]. Available: https://arxiv.org/abs/1706.03762v7

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  2. [2]

    ISSCC 2014 / SESSION 1 / PLENARY / 1.1 Computing's Energy Problem (and what we can do about it),

    M. Horowitz, “ISSCC 2014 / SESSION 1 / PLENARY / 1.1 Computing's Energy Problem (and what we can do about it),” doi: 10.1109/ ISSCC.2014.6757323

    work page arXiv 2014

  3. [3]

    Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models

    M. Noukhovitch, S. Huang, S. Xhonneux, A. Hosseini, R. Agarwal, and A. Courville, “Asynchronous RLHF: Faster and More Efficient Off­Policy RL for Language Models,” 13th International Conference on Learning Repre - sentations, ICLR 2025 , pp. 35297–35323, Oct. 2024, [Online]. Available: https://arxiv.org/abs/2410.18252v3 25

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  4. [4]

    AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning

    W. Fu et al., “AReaL: A Large­Scale Asynchronous Reinforcement Learn­ ing System for Language Reasoning,” May 2025, [Online]. Available: https://arxiv.org/abs/2505.24298v5

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    Fast Inference from Transformers via Speculative Decoding

    Y. Leviathan, M. Kalman, and Y. Matias, “Fast Inference from Transform­ ers via Speculative Decoding,” Proceedings of Machine Learning Research, vol. 202, pp. 19274–19286, Nov. 2022, [Online]. Available: https://arxiv. org/abs/2211.17192v2

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  6. [6]

    PARD: Accelerating LLM Inference with Low ­Cost PARallel Draft Model Adaptation,

    Z. An, H. Bai, Z. Liu, D. Li, and E. Barsoum, “PARD: Accelerating LLM Inference with Low ­Cost PARallel Draft Model Adaptation,” Apr. 2025, [Online]. Available: https://arxiv.org/abs/2504.18583v4

    work page arXiv 2025

  7. [7]

    Introspective Diffusion Language Models

    Y. Yu et al., “Introspective Diffusion Language Models,” Apr. 2026, [On ­ line]. Available: https://arxiv.org/abs/2604.11035v1

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  8. [8]

    PARD-2: Target-Aligned Parallel Draft Model for Dual-Mode Speculative Decoding

    Z. An, T. Liu, Z. Liu, D. Li, R. Liu, and E. Barsoum, “PARD ­2: Target ­ Aligned Parallel Draft Model for Dual­Mode Speculative Decoding,” May 2026, [Online]. Available: https://arxiv.org/abs/2605.08632v1

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  9. [9]

    DFlash: Block Diffusion for Flash Speculative Decoding

    J. Chen, Y. Liang, and Z. Liu, “DFlash: Block Diffusion for Flash Specu ­ lative Decoding,” Feb. 2026, [Online]. Available: https://arxiv.org/abs/ 2602.06036v2

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  10. [10]

    Traversal Verification for Speculative Tree Decoding,

    Y. Weng et al., “Traversal Verification for Speculative Tree Decoding,” May 2025, [Online]. Available: https://arxiv.org/abs/2505.12398v2

    work page arXiv 2025

  11. [11]

    Accelerating Speculative Decoding with Block Diffusion Draft Trees

    L. Ringel and Y. Romano, “Accelerating Speculative Decoding with Block Diffusion Draft Trees,” Apr. 2026, [Online]. Available: https://arxiv.org/ abs/2604.12989v1

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  12. [12]

    SGLang: Efficient Execution of Structured Language Model Programs,

    L. Zheng et al. , “SGLang: Efficient Execution of Structured Language Model Programs,” Advances in Neural Information Processing Systems, vol. 37, Dec. 2023, doi: 10.52202/079017-2000

    work page doi:10.52202/079017-2000 2023

  13. [13]

    Accelerating Large Language Model Decoding with Speculative Sampling

    C. Chen, S. Borgeaud, G. Irving, J.­B. Lespiau, L. Sifre, and J. Jumper, “Ac­ celerating Large Language Model Decoding with Speculative Sampling,” pp. 2023–2025, Feb. 2023, [Online]. Available: https://arxiv.org/abs/ 2302.01318v1

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  14. [14]

    Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads

    T. Cai et al., “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads,” Proceedings of Machine Learning Re - 26 search, vol. 235, pp. 5209–5235, Jan. 2024, [Online]. Available: https:// arxiv.org/abs/2401.10774v3

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  15. [15]

    EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty

    Y. Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty,” Proceedings of Machine Learn- ing Research, vol. 235, pp. 28935–28948, Jan. 2024, [Online]. Available: https://arxiv.org/pdf/2401.15077

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  16. [16]

    PLD+: Accelerating LLM inference by leveraging Language Model Artifacts

    S. Somasundaram, A. Phukan, and A. Saxena, “PLD+: Accelerating LLM inference by leveraging Language Model Artifacts,” Dec. 2024, [Online]. Available: https://arxiv.org/abs/2412.01447v1

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  17. [17]

    SAM Decoding: Speculative Decoding via Suffix Automaton

    Y. Hu et al., “SAM Decoding: Speculative Decoding via Suffix Automaton,” Nov. 2024, [Online]. Available: https://arxiv.org/abs/2411.10666v2

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  18. [18]

    Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling,

    X. Luo et al. , “Turning Trash into Treasure: Accelerating Inference of Large Language Models with Token Recycling,” Proceedings of the Annual Meeting of the Association for Computational Linguistics , vol. 1, pp. 6816– 6831, Aug. 2024, doi: 10.18653/v1/2025.acl-long.338

    work page doi:10.18653/v1/2025.acl-long.338 2024

  19. [19]

    Draft & Verify: Lossless Large Language Model Acceler ­ ation via Self ­Speculative Decoding,

    J. Zhang et al., “Draft & Verify: Lossless Large Language Model Acceler ­ ation via Self ­Speculative Decoding,” Proceedings of the Annual Meeting of the Association for Computational Linguistics , vol. 1, pp. 11263–11282, May 2024, doi: 10.18653/v1/2024.acl-long.607

    work page doi:10.18653/v1/2024.acl-long.607 2024

  20. [20]

    LayerSkip: Enabling Early Exit Inference and Self ­ Speculative Decoding,

    M. Elhoushi et al. , “LayerSkip: Enabling Early Exit Inference and Self ­ Speculative Decoding,” Proceedings of the Annual Meeting of the Associa - tion for Computational Linguistics, vol. 1, pp. 12622–12642, Apr. 2024, doi: 10.18653/v1/2024.acl-long.681

    work page doi:10.18653/v1/2024.acl-long.681 2024

  21. [21]

    Hydra: Sequentially ­Dependent Draft Heads for Medusa Decoding,

    Z. Ankner, R. Parthasarathy, A. Nrusimha, C. Rinard, J. Ragan­Kelley, and W. Brandon, “Hydra: Sequentially ­Dependent Draft Heads for Medusa Decoding,” Feb. 2024, [Online]. Available: https://arxiv.org/abs/2402. 05109v2

    work page 2024

  22. [22]

    EAGLE ­2: Faster Inference of Language Models with Dynamic Draft Trees,

    Y. Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE ­2: Faster Inference of Language Models with Dynamic Draft Trees,” EMNLP 2024 - 2024 Confer- ence on Empirical Methods in Natural Language Processing, Proceedings of the Conference, pp. 7421–7432, Jun. 2024, doi: 10.18653/v1/2024.emnlp- main.422. 27

    work page doi:10.18653/v1/2024.emnlp- 2024

  23. [23]

    EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test

    Y. Li, F. Wei, C. Zhang, and H. Zhang, “EAGLE ­3: Scaling up Inference Acceleration of Large Language Models via Training ­Time Test,” Mar. 2025, [Online]. Available: https://arxiv.org/abs/2503.01840v3

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  24. [24]

    Multi­token­prediction in Gemma 4

    “Multi­token­prediction in Gemma 4.” [Online]. Available: https://blog.google/innovation-and-ai/technology/developers- tools/multi-token-prediction-gemma-4/

    work page

  25. [25]

    Markov Chains and Mixing Times, second edition

    D. A. Levin, Y. Peres, and E. L. Wilmer, “Markov Chains and Mixing Times, second edition.”

    work page

  26. [26]

    Block Verification Accelerates Speculative Decoding

    Z. Sun et al., “Block Verification Accelerates Speculative Decoding,” Apr. 2025, [Online]. Available: http://arxiv.org/abs/2403.10444

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  27. [27]

    SpecInfer: Accelerating Generative Large Language Model Serving with Tree­based Speculative Inference and Verification,

    X. Miao et al., “SpecInfer: Accelerating Generative Large Language Model Serving with Tree­based Speculative Inference and Verification,” Interna- tional Conference on Architectural Support for Programming Languages and Operating Systems - ASPLOS , vol. 3, pp. 932–949, Apr. 2024, doi: 10.1145/3620666.3651335

    work page doi:10.1145/3620666.3651335 2024

  28. [28]

    Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding

    Z. Chen et al., “Sequoia: Scalable, Robust, and Hardware ­aware Specula­ tive Decoding,” Feb. 2024, [Online]. Available: https://arxiv.org/abs/ 2402.12374v3

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  29. [29]

    SpecTr: Fast Speculative Decoding via Optimal Transport

    Z. Sun, A. Beirami, A. T. Suresh, H. Jain, J. H. Ro, and F. Yu, “SpecTr: Fast Speculative Decoding via Optimal Transport,” Advances in Neural Information Processing Systems , vol. 36, Oct. 2023, [Online]. Available: https://arxiv.org/abs/2310.15141v2

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  30. [30]

    Multi-Draft Speculative Sampling: Canonical Decomposition and Theoretical Limits

    A. Khisti, M. R. Ebrahimi, H. Dbouk, A. Behboodi, † R. Memisevic, and C. Louizos, “Multi­Draft Speculative Sampling: Canonical Decomposition and Theoretical Limits,” Oct. 2024, [Online]. Available: https://arxiv. org/abs/2410.18234v2

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  31. [31]

    DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure

    Y. Xiong, R. Zhang, Y. Li, T. Wu, and L. Zou, “DySpec: Faster Speculative Decoding with Dynamic Token Tree Structure,” Oct. 2024, [Online]. Available: https://arxiv.org/abs/2410.11744v1

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  32. [32]

    The Mamba in the Llama: Distilling and Accelerating Hybrid Models,

    J. Wang, D. Paliotta, A. May, A. M. Rush, and T. Dao, “The Mamba in the Llama: Distilling and Accelerating Hybrid Models,” Advances 28 in Neural Information Processing Systems , vol. 37, Aug. 2024, doi: 10.52202/079017-1996

    work page doi:10.52202/079017-1996 2024

  33. [33]

    Snakes and Ladders: Accelerating State Space Model Inference with Speculative Decoding

    Y. Wu, Y. Dukler, † M. Trager, A. Achille, W. Xia, and S. Soatto, “Snakes and Ladders: Accelerating State Space Model Inference with Speculative Decoding.”

    work page

  34. [34]

    STree: Speculative Tree Decoding for Hybrid State­Space Models,

    Y. Wu, Z. Qin, A. Wong, and S. Soatto, “STree: Speculative Tree Decoding for Hybrid State­Space Models,” May 2025, [Online]. Available: https:// arxiv.org/abs/2505.14969v2

    work page arXiv 2025

  35. [35]

    Gated Delta Networks: Improving Mamba2 with Delta Rule

    S. Yang, J. Kautz, and A. Hatamizadeh, “Gated Delta Networks: Improving Mamba2 with Delta Rule,” 13th International Conference on Learning Rep- resentations, ICLR 2025, pp. 73311–73331, Dec. 2024, [Online]. Available: https://arxiv.org/abs/2412.06464v3

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  36. [36]

    Parallelizing Linear Transformers with the Delta Rule over Sequence Length,

    S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim, “Parallelizing Linear Transformers with the Delta Rule over Sequence Length,” Advances in Neural Information Processing Systems , vol. 37, Jun. 2024, doi: 10.52202/079017-3668

    work page doi:10.52202/079017-3668 2024

  37. [37]

    MT ­Bench­101: A Fine ­Grained Benchmark for Evalu ­ ating Large Language Models in Multi ­Turn Dialogues,

    G. Bai et al. , “MT ­Bench­101: A Fine ­Grained Benchmark for Evalu ­ ating Large Language Models in Multi ­Turn Dialogues,” Nov. 2024, doi: 10.18653/v1/2024.acl-long.401

    work page doi:10.18653/v1/2024.acl-long.401 2024

  38. [38]

    LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding

    A. Samarin, S. Krutikov, A. Shevtsov, S. Skvortsov, F. Fisin, and A. Golubev, “LK Losses: Direct Acceptance Rate Optimization for Speculative Decod­ ing,” Feb. 2026, [Online]. Available: https://arxiv.org/abs/2602.23881 v2

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  39. [39]

    Nemotron ­Post­Training­Dataset­v2

    D. Nathawani et al. , “Nemotron ­Post­Training­Dataset­v2.” [Online]. Available: https://huggingface.co/datasets/nvidia/Nemotron-Post- Training-Dataset-v2

    work page

  40. [40]

    LMSYS­Chat­1M: A Large ­Scale Real­World LLM Con­ versation Dataset

    L. Zheng et al., “LMSYS­Chat­1M: A Large ­Scale Real­World LLM Con­ versation Dataset.” 2023

    work page 2023

  41. [41]

    OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants

    Teknium, “OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants.” [Online]. Available: https://huggingface. co/datasets/teknium/OpenHermes-2.5 29

    work page

  42. [42]

    Code Alpaca: An Instruction­following LLaMA model for code generation

    S. Chaudhary, “Code Alpaca: An Instruction­following LLaMA model for code generation.” GitHub, 2023

    work page 2023

  43. [43]

    Muon is Scalable for LLM Training

    J. Liu et al. , “Muon is Scalable for LLM Training,” Feb. 2025, [Online]. Available: https://arxiv.org/pdf/2502.16982

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  44. [44]

    Decoupled Weight Decay Regularization

    I. Loshchilov and F. Hutter, “Decoupled Weight Decay Regularization,” 7th International Conference on Learning Representations, ICLR 2019, Nov. 2017, [Online]. Available: https://arxiv.org/pdf/1711.05101

    work page internal anchor Pith review Pith/arXiv arXiv 2019

  45. [45]

    ShareChat: A Dataset of Chatbot Conversations in the Wild

    Y. Yan, T. Nguyen, B. Su, M. Lieffers, and T. Le, “ShareChat: A Dataset of Chatbot Conversations in the Wild.” [Online]. Available: https://arxiv. org/abs/2512.17843

    work page internal anchor Pith review Pith/arXiv arXiv

  46. [46]

    Training Verifiers to Solve Math Word Problems

    K. Cobbe et al., “Training Verifiers to Solve Math Word Problems,” arXiv preprint arXiv:2110.14168, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  47. [47]

    Let's Verify Step by Step

    H. Lightman et al. , “Let's Verify Step by Step,” arXiv preprint arXiv:2305.20050, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  48. [48]

    American Invitational Mathematics Examina­ tion (AIME) 2025

    Y. Zhang and T. Math­AI, “American Invitational Mathematics Examina­ tion (AIME) 2025.” 2025

    work page 2025

  49. [49]

    Evaluating Large Language Models Trained on Code,

    M. Chen et al., “Evaluating Large Language Models Trained on Code,” 2021

    work page 2021

  50. [50]

    Program Synthesis with Large Language Models

    J. Austin et al., “Program Synthesis with Large Language Models,” arXiv preprint arXiv:2108.07732, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  51. [51]

    LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code

    N. Jain et al., “LiveCodeBench: Holistic and Contamination Free Evalu ­ ation of Large Language Models for Code,” 13th International Conference on Learning Representations, ICLR 2025 , pp. 25479–25519, Mar. 2024, [Online]. Available: https://arxiv.org/pdf/2403.07974

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  52. [52]

    A* Sampling

    C. J. Maddison, D. Tarlow, and T. Minka, “A* Sampling,” Advances in Neural Information Processing Systems, vol. 4, no. January, pp. 3086–3094, Oct. 2014, [Online]. Available: https://arxiv.org/abs/1411.0030v2

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  53. [53]

    Accelerating LLM Inference with Staged Speculative Decoding

    B. Spector and C. Re, “Accelerating LLM Inference with Staged Specula ­ tive Decoding,” Aug. 2023, [Online]. Available: https://arxiv.org/abs/ 2308.04623v1 30

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  54. [54]

    Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models

    M. Arriola et al., “Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models,” 13th International Conference on Learn- ing Representations, ICLR 2025 , pp. 84192–84219, Mar. 2025, [Online]. Available: https://arxiv.org/abs/2503.09573v3

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  55. [55]

    Coupling without Communication and Drafter-Invariant Speculative Decoding

    M. Daliri, C. Musco, A. Theertha, and S. G. Research, “Coupling without Communication and Drafter­Invariant Speculative Decoding,” Aug. 2024, [Online]. Available: https://arxiv.org/abs/2408.07978v4

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  56. [56]

    Fast-dllm v2: Efficient block-diffusion llm.arXiv preprint arXiv:2509.26328, 2025

    C. Wu et al., “Fast­dLLM v2: Efficient Block ­Diffusion LLM,” pp. 2025– 2035, Sep. 2025, [Online]. Available: https://arxiv.org/abs/2509.26328 v1

    work page arXiv 2025

  57. [57]

    Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding

    C. Wu et al. , “Fast­dLLM: Training­free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding,” pp. 2025–2032, May 2025, [Online]. Available: https://arxiv.org/abs/2505.22618v3

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  58. [58]

    FR ­Spec: Accelerating Large ­Vocabulary Language Models via Frequency­Ranked Speculative Sampling,

    W. Zhao et al. , “FR ­Spec: Accelerating Large ­Vocabulary Language Models via Frequency­Ranked Speculative Sampling,” Proceedings of the Annual Meeting of the Association for Computational Linguistics, vol. 1, pp. 3909–3921, Feb. 2025, doi: 10.18653/v1/2025.acl-long.198

    work page doi:10.18653/v1/2025.acl-long.198 2025

  59. [59]

    Future Lens: Antici ­ pating Subsequent Tokens from a Single Hidden State,

    K. Pal, J. Sun, A. Yuan, B. C. Wallace, and D. Bau, “Future Lens: Antici ­ pating Subsequent Tokens from a Single Hidden State,” CoNLL 2023 - 27th Conference on Computational Natural Language Learning, Proceedings , pp. 548–560, Nov. 2023, doi: 10.18653/v1/2023.conll-1.37

    work page doi:10.18653/v1/2023.conll-1.37 2023

  60. [60]

    interpreting GPT: the logit lens

    nostalgebraist, “interpreting GPT: the logit lens.” [Online]. Avail ­ able: https://www.lesswrong.com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens

    work page

  61. [61]

    Introducing DiffusionGemma

    “Introducing DiffusionGemma.” [Online]. Available: https://blog. google/innovation-and-ai/technology/developers-tools/diffusion- gemma-faster-text-generation/

    work page

  62. [62]

    Lost in the Middle: How Language Models Use Long Contexts

    J. Wang et al., “OPT­Tree: Speculative Decoding with Adaptive Draft Tree Structure,” Transactions of the Association for Computational Linguistics , vol. 13, pp. 188–199, Jun. 2024, doi: 10.1162/tacl\_a\_00735

    work page internal anchor Pith review doi:10.1162/tacl 2024

  63. [63]

    Qwen3.5

    “Qwen3.5.” [Online]. Available: https://qwen.ai/blog?id=qwen3.5 31

    work page

  64. [64]

    Speculative Decoding with a Speculative Vocabulary,

    M. Williams, Y. D. Kwon, R. Li, A. Kouris, and S. I. Venieris, “Speculative Decoding with a Speculative Vocabulary,” Feb. 2026, [Online]. Available: https://arxiv.org/pdf/2602.13836

    work page arXiv 2026

  65. [65]

    The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models

    G. Cui et al., “The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models,” May 2025, [Online]. Available: https:// arxiv.org/abs/2505.22617v1

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  66. [66]

    Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

    T. Dao and A. Gu, “Transformers are SSMs: Generalized Models and Effi­ cient Algorithms Through Structured State Space Duality,” Proceedings of Machine Learning Research , vol. 235, pp. 10041–10071, May 2024, [Online]. Available: https://arxiv.org/abs/2405.21060v1

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  67. [67]

    Understanding the Effects of RLHF on LLM Generalisation and Diversity,

    R. Kirk et al., “Understanding the Effects of RLHF on LLM Generalisation and Diversity,” 12th International Conference on Learning Representations, ICLR 2024, Oct. 2023, [Online]. Available: https://arxiv.org/abs/2310. 06452v3

    work page 2024

  68. [68]

    VOCABTRIM: Vocabulary Pruning for Efficient Specula ­ tive Decoding in LLMs,

    R. Goel et al., “VOCABTRIM: Vocabulary Pruning for Efficient Specula ­ tive Decoding in LLMs,” Jun. 2025, [Online]. Available: https://arxiv. org/pdf/2506.22694 32 Appendix A. Pseudocode for tree construction Algorithm 1: Adaptive tree construction. The marginal prior and the Weaver context cache are computed once per speculation step; the tree is then gr...

    work page arXiv 2025