Pith. sign in

REVIEW 3 major objections 7 minor 58 references

Speculative decoding can speed long-context LLM inference by up to 2.79× using the model's own sparse KV cache for drafting and full-context verification, with no retraining.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 02:20 UTC pith:US3QXFH4

load-bearing objection A useful engineering contribution with a clean efficiency analysis, but the distribution-preservation claim needs a precise acceptance rule and longer-context evidence. the 3 major comments →

arxiv 2607.27735 v1 pith:US3QXFH4 submitted 2026-07-30 cs.CL

A Sparse Glimpse of the Whole: Train-Free Self-Speculative Decoding

Yuesong Liu , Yuan Zeng , Min Lyu , Ruilin Liu , Yu Guo , Yinlong Xu This is my paper
classification cs.CL
keywords speculative decodinglong-context inferenceKV cache compressionself-speculationadaptive speculation lengthentropy-based controltraining-free accelerationattention importance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that long-context LLM inference can be accelerated by having the model itself draft tokens from a small, recallable subset of its key-value memory, then verify those drafts against the full memory, with no retraining and no change to the output distribution. It also claims a general rule: extending the number of speculative tokens improves speed only while each additional token's acceptance probability exceeds the relative drafting cost; beyond that, longer speculation actively reduces speed. If right, the work offers a practical speedup of up to 2.79× on 3B–14B models across retrieval, reasoning, and synthesis tasks, and explains why fixed-length speculative decoding is inherently suboptimal.

Core claim

The paper's central claim is that a single target model can serve as both drafter and verifier in a training-free speculative decoding loop. During drafting, the model attends only to a per-head sparse set of key-value positions—sink tokens, recent tokens, and top-ranked historical tokens—where the ranking comes from attention statistics already computed in the previous full-context verification pass, so no extra forward pass is needed. During verification, the full dense KV cache is used, which preserves the target distribution exactly. An online entropy-based controller then chooses the speculation length that maximizes estimated step-wise efficiency. The paper reports end-to-end speedups

What carries the argument

The central object is the unified speedup identity S = (αk+1)/(kγ+1), where k is speculation length, α is token acceptance rate, and γ is the relative drafting cost; its derivative shows that speedup improves with k only when α > γ. The second mechanism is the recallable sparse KV index, which for each attention head retains sink tokens, recent tokens, and the top-scoring historical tokens according to attention weights reused from the prior verification pass, allowing cheap drafts without permanently discarding dense KV states. The third is the entropy-based controller that estimates per-token acceptance probabilities and selects k* by maximizing expected tokens per step divided by estimate

Load-bearing premise

The reported speedups assume that the drafting cost and acceptance behavior measured on a single GPU with contexts capped at 64K and generation capped at 128 tokens carry over to the longer contexts and longer outputs that motivate long-context inference.

What would settle it

Run the same implementation on a 100K+ token context and generate at least 512 tokens; if end-to-end speedup falls below 1× or well below the 64K results because the sparse-index update and non-fused attention overhead grow with context, or if acceptance starts to decay after 128 generated tokens, the central long-context claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Speculative decoding methods should adapt their speculation length rather than fix it, because fixed horizons inevitably cross the point where marginal acceptance falls below drafting cost.
  • A target model can accelerate itself without auxiliary draft models or training, as long as drafting uses a sparse but recallable view of its own KV cache.
  • Full-context verification preserves the output distribution exactly, so the reported speedups come without changing model behavior.
  • The speedup persists across 3B–14B models and multiple long-context task types, suggesting the mechanism is not tied to one architecture or benchmark.
  • Drafting compression ratio presents a concrete trade-off: the empirical optimum sits around 10% KV retention, balancing acceptance against drafting overhead.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The measured 41.1 ms per-iteration sparse-index update may grow with context length, so the true long-context ceiling depends on whether this overhead scales sublinearly; fused kernels or hardware-aware gathering could push real-world speedups higher than reported, or reveal a scaling limit hidden by the 64K cap.
  • The entropy controller is only moderately predictive, so replacing or augmenting it with a stronger confidence signal—such as agreement between sparse and full attention—could yield larger speedups and more stable acceptance.
  • The α > γ efficiency-inversion condition doubles as a diagnostic for any drafting mechanism: if a drafter's tail acceptance decays below its relative cost, the speculator should stop early rather than lengthen the draft.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper proposes SparseSpec-L, a training-free self-speculative decoding framework for long-context LLM inference. The authors first derive a unified speedup formula S=(αk+1)/(kγ+1) and show that when the acceptance rate α falls below the drafting overhead γ, increasing the speculation horizon k reduces speedup. The method uses a sparsified, per-head KV cache for drafting, refreshes the sparse indices with attention statistics from the previous full-context verification pass, and selects k online with an entropy-based controller. On LongBench v2 and InfiniteBench, the paper reports 1.55–2.79× end-to-end speedups over autoregressive decoding on 3B–14B models while claiming that the target model's output distribution is preserved.

Significance. If the correctness issues are resolved, the system is a practical and attractive addition to the sparse self-speculative decoding literature. Its training-free design and recallable sparse index are well motivated, and the closed-form speedup identity in Eq. (1)–(2) provides a transparent basis for the adaptive controller. The empirical results are internally consistent: the Table 3 ablation shows the predicted efficiency inversion, and the acceptance rates are high enough to explain the reported speedups. However, the current manuscript does not state a precise acceptance rule and therefore does not establish the distribution-preservation claim; without that, the headline contribution is unproven. The evaluation also covers only a limited operating envelope. The theoretical part is derivative, but the empirical/design contribution could be valuable after a major revision.

major comments (3)
  1. [Methodology — Full Verification Stage; Eq. (8)] The paper's central correctness claim — 'preserving the target model's output distribution' — is not established because the acceptance rule is never specified. The text says the verifier 'identifies the longest prefix that satisfies the verification criteria' and later refers to a 'strict prefix-matching criterion,' but it does not state the sampling rule. For a draft distribution q and target distribution p, distribution preservation requires the standard rejection-sampling acceptance probability min(1, p(x)/q(x)) and a correction sample from (p−q)_+ on rejection. If acceptance instead means exact equality with the full model's greedy or sampled token, the output is an approximation, not the target distribution. This is load-bearing: the abstract and contributions advertise exact distribution preservation. The authors must specify the rule, provide a proof, and reconcile Eq. (8)'s pref
  2. [Adaptive Speculation Length; Eq. (8)] The controller claims to select k* that 'strictly maximizes the expected step-wise efficiency.' The expression ∏_{i=1}^m p_i assumes the m draft-token acceptance events are conditionally independent. The paper acknowledges this assumption in one clause but gives no empirical evidence of its validity or its effect on the selected k*. If acceptance is correlated across positions, the expected accepted-prefix length is mis-estimated, so the controller is not necessarily optimal. This does not affect Eq. (1)–(2), but it tempers the adaptivity claim. Please either reframe the controller as a heuristic or validate the independence assumption on the tested workloads.
  3. [Experimental Setup; Limitations] The empirical basis for the long-context claim is narrow. All end-to-end results use a single A40, contexts capped at 60–64K, and generation length 128 tokens. The paper's motivation is long-context and long-output inference; the sparse-index update cost (41.1 ms/iteration) and non-fused kernel overhead may scale differently with context length, and the entropy controller's behavior on longer generations is untested. No error bars or repeated runs are reported, and no code or data are provided. Please add at least one 100K+ context experiment and a longer-generation evaluation, or provide a scaling model showing γ remains bounded; also report standard deviations.
minor comments (7)
  1. [Abstract] The phrase 'up to speedup over autoregressive decoding' is missing the numeric value; '2.79×' should be inserted.
  2. [Experimental Setup] The paper uses both '60K setting' and '64K cap' inconsistently. Clarify the exact maximum context length used in each main table.
  3. [Table 1] '∞-Bench' is not defined at first use; use the standard name InfiniteBench. RULER QA1/NIAH are also undefined in the text.
  4. [Figure 1 caption] The caption references '† Tab. 1, §Appendix, and ‡ Fig. 2' but the appendix is not present in the manuscript. Remove or fill these references.
  5. [Adaptive Speculation Length] The exponential moving averages in Eq. (6) require initialization details. State how H_acc and H_rej are initialized and how the first draft step is handled.
  6. [Methodology — Hyperparameter Selection] The values of K_sink and K_recent are said to follow 'established empirical constants' but are never given. Include the exact values used in the experiments.
  7. [Implementation Details] The 41.1 ms sparse-index update cost should state the exact context length and hardware, and clarify whether it is included in the reported throughput/speedup numbers.

Circularity Check

0 steps flagged

No circular derivation: the speedup identity is algebraic and the entropy controller is an online optimizer, not a fitted prediction; no load-bearing self-citations.

full rationale

The paper's derivation chain is self-contained. Eq.1 defines speedup as S=(αk+1)/(kγ+1) directly from the components α, k, γ, and Eq.2 is the derivative of that identity, so the efficiency inversion (α<γ ⇒ ∂S/∂k<0) is an algebraic consequence rather than a fitted or self-referential input. Figure 2 and Tables 1–3 report direct measurements against autoregressive decoding and external baselines; the speedups are not computed from the paper's own formulas. The entropy-based controller (Eqs.6–8) does fit EMA class centers to observed accept/reject labels, but it is used as an online control signal to choose k*, not as evidence for the paper's speedup or acceptance claims; the reported acceptance rates, average accepted tokens, and throughput are measured after the fact. The adaptive ablation in Table 3 is an optimization comparison, not a prediction-from-fit. There are no load-bearing self-citations: the only related self-speculative works mentioned (prior SparseSpec, Vegas) are external and used for positioning, not to justify correctness. The strongest caveat—that distribution preservation is asserted from full-KV verification without a fully specified acceptance rule (rejection sampling vs exact-match)—is a missing proof or correctness risk, not a circular reduction, so it does not affect the circularity score.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on a small number of tuned hyperparameters (compression ratio, W, Ksink/Krecent, β, candidate set) and on domain assumptions about attention transfer, conditional independence, and stable costs. No new physical or conceptual entities are introduced.

free parameters (5)
  • Sparse KV budget Ktotal (compression ratio) = ~10% of tokens (empirical optimum in Fig. 4)
    The 10% compression ratio is chosen as the empirical optimum balancing acceptance rate and drafting cost; it is tuned, not derived.
  • Aggregation window W = 16
    Set to W=16 because it empirically yields stable importance signals; no sensitivity analysis is shown for W.
  • Sink/recent token counts Ksink, Krecent = not reported
    Follows established empirical constants from StreamingLLM/SnapKV, but exact values are not given, affecting the sparse index.
  • Entropy smoothing coefficient β and class-center initialization = not reported
    Eq.6 updates moving averages with β; no value or initialization procedure is specified.
  • Candidate speculation set K = not reported (ablation uses {4,8,12,16,20})
    The controller selects from a pre-defined candidate set whose composition is not specified in the main text.
axioms (5)
  • standard math Speculative rejection verification with full KV cache preserves the target model's output distribution exactly
    Standard property of speculative decoding (Chen et al. 2023; Leviathan et al. 2023), invoked in the introduction and verification design.
  • ad hoc to paper The acceptance probability of successive draft tokens is conditionally independent
    Invoked to write expected accepted tokens as a sum of products in Eq.8; this is not justified and is generally false for autoregressive generation.
  • domain assumption Per-head attention weights from the last W=16 verification queries are a sufficient importance signal for future drafting queries
    This is the core heuristic of Eq.3; if future queries attend to different tokens, the sparse index will be mis-specified.
  • domain assumption Draft cost Cd and verification cost Cv are stable enough to be treated as constant within a decoding run
    Eqs.1 and 8 treat Cd/Cv as fixed, but implementation overhead such as the 41.1 ms sparse-index update may vary with context length.
  • domain assumption Empirical α decay with k observed on two benchmarks transfers to other long-context workloads
    Figure 2 uses only LongBench v2 and XSum with Llama-3.1-8B; the efficiency-inversion motivation may not hold for all models and tasks.

pith-pipeline@v1.3.0-daily-deepseek · 11235 in / 15360 out tokens · 160089 ms · 2026-08-01T02:20:33.557139+00:00 · methodology

0 comments
read the original abstract

Speculative decoding alleviates the memory-bandwidth bottleneck in large language model inference, but its acceleration is jointly constrained by drafting overhead, token acceptance, and speculation length. We present a unified efficiency analysis showing that extending the speculation horizon can reduce rather than improve speedup when the marginal acceptance probability falls below the relative drafting cost. Guided by this analysis, we introduce SparseSpec-L, a training-free self-speculative decoding framework for long-context inference. SparseSpec-L generates lightweight drafts directly from the target model using a dynamically sparsified and recallable KV cache. It recycles per-head attention statistics produced during full-context verification as a no-extra-forward importance signal, allowing critical historical tokens to be recalled without permanently discarding the dense KV cache. An online entropy-based controller further selects the speculation length according to expected step-wise efficiency. Experiments across multiple long-context tasks and model scales show consistent end-to-end acceleration, with up to speedup over autoregressive decoding while preserving the target model's output distribution.

Figures

Figures reproduced from arXiv: 2607.27735 by Min Lyu, Ruilin Liu, Yinlong Xu, Yuan Zeng, Yuesong Liu, Yu Guo.

Figure 1
Figure 1. Figure 1: Comparison of existing speculative decoding paradigms ( [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Performance analysis of Eagle-3 and Mag [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Overview of SparseSpec-L’s sparse-to-full self-speculative decoding pipeline. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Sensitivity analysis on LongBench v2 with Llama [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 21 linked inside Pith

  1. [1]

    Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education

    Clancey, William J. Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education. Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)

  2. [2]

    Classification Problem Solving

    Clancey, William J. Classification Problem Solving. Proceedings of the Fourth National Conference on Artificial Intelligence

  3. [3]

    , title =

    Robinson, Arthur L. , title =. 1980 , doi =. https://science.sciencemag.org/content/208/4447/1019.full.pdf , journal =

    1980

  4. [4]

    New Ways to Make Microcircuits Smaller---Duplicate Entry

    Robinson, Arthur L. New Ways to Make Microcircuits Smaller---Duplicate Entry. Science

  5. [5]

    Clancey and Glenn Rennels , abstract =

    Diane Warner Hasling and William J. Clancey and Glenn Rennels , abstract =. Strategic explanations for a diagnostic consultation system , journal =. 1984 , issn =. doi:https://doi.org/10.1016/S0020-7373(84)80003-6 , url =

    doi:10.1016/s0020-7373(84)80003-6 1984

  6. [6]

    and Rennels, Glenn R

    Hasling, Diane Warner and Clancey, William J. and Rennels, Glenn R. and Test, Thomas. Strategic Explanations in Consultation---Duplicate. The International Journal of Man-Machine Studies

  7. [7]

    Poligon: A System for Parallel Problem Solving

    Rice, James. Poligon: A System for Parallel Problem Solving

  8. [8]

    Transfer of Rule-Based Expertise through a Tutorial Dialogue

    Clancey, William J. Transfer of Rule-Based Expertise through a Tutorial Dialogue

  9. [9]

    The Engineering of Qualitative Models

    Clancey, William J. The Engineering of Qualitative Models

  10. [10]

    2023 , eprint=

    Attention Is All You Need , author=. 2023 , eprint=

    2023

  11. [11]

    Pluto: The 'Other' Red Planet

    NASA. Pluto: The 'Other' Red Planet

  12. [12]

    arXiv preprint arXiv:2601.03267 , year=

    Openai gpt-5 system card , author=. arXiv preprint arXiv:2601.03267 , year=

    Pith/arXiv arXiv

  13. [13]

    arXiv preprint arXiv:2601.11659 , year=

    The Llama 4 Herd: Architecture, Training, Evaluation, and Deployment Notes , author=. arXiv preprint arXiv:2601.11659 , year=

    arXiv

  14. [14]

    Advances in neural information processing systems , volume=

    Chain-of-thought prompting elicits reasoning in large language models , author=. Advances in neural information processing systems , volume=

  15. [15]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  16. [16]

    arXiv preprint arXiv:1911.02150 , year=

    Fast transformer decoding: One write-head is all you need , author=. arXiv preprint arXiv:1911.02150 , year=

    Pith/arXiv arXiv 1911

  17. [17]

    Proceedings of the 29th symposium on operating systems principles , pages=

    Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=

  18. [18]

    arXiv preprint arXiv:2302.01318 , year=

    Accelerating large language model decoding with speculative sampling , author=. arXiv preprint arXiv:2302.01318 , year=

    Pith/arXiv arXiv

  19. [19]

    International Conference on Machine Learning , pages=

    Fast inference from transformers via speculative decoding , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  20. [20]

    Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 , pages=

    Specinfer: Accelerating large language model serving with tree-based speculative inference and verification , author=. Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3 , pages=

  21. [21]

    Advances in Neural Information Processing Systems , volume=

    H2o: Heavy-hitter oracle for efficient generative inference of large language models , author=. Advances in Neural Information Processing Systems , volume=

  22. [22]

    arXiv preprint arXiv:2406.10774 , year=

    Quest: Query-aware sparsity for efficient long-context llm inference , author=. arXiv preprint arXiv:2406.10774 , year=

    Pith/arXiv arXiv

  23. [23]

    Advances in neural information processing systems , volume=

    Infllm: Training-free long-context extrapolation for llms with an efficient context memory , author=. Advances in neural information processing systems , volume=

  24. [24]

    arXiv preprint arXiv:2309.17453 , year=

    Efficient streaming language models with attention sinks , author=. arXiv preprint arXiv:2309.17453 , year=

    Pith/arXiv arXiv

  25. [25]

    Advances in Neural Information Processing Systems , volume=

    Snapkv: Llm knows what you are looking for before generation , author=. Advances in Neural Information Processing Systems , volume=

  26. [26]

    18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages=

    \ InfiniGen \ : Efficient generative inference of large language models with dynamic \ KV \ cache management , author=. 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24) , pages=

  27. [27]

    Advances in Neural Information Processing Systems , volume=

    Arkvale: Efficient generative llm inference with recallable key-value eviction , author=. Advances in Neural Information Processing Systems , volume=

  28. [28]

    arXiv preprint arXiv:2401.15077 , year=

    Eagle: Speculative sampling requires rethinking feature uncertainty , author=. arXiv preprint arXiv:2401.15077 , year=

    Pith/arXiv arXiv

  29. [29]

    arXiv preprint arXiv:2503.01840 , year=

    Eagle-3: Scaling up inference acceleration of large language models via training-time test , author=. arXiv preprint arXiv:2503.01840 , year=

    Pith/arXiv arXiv

  30. [30]

    arXiv preprint arXiv:2404.19737 , year=

    Better & faster large language models via multi-token prediction , author=. arXiv preprint arXiv:2404.19737 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2401.10774 , year=

    Medusa: Simple llm inference acceleration framework with multiple decoding heads , author=. arXiv preprint arXiv:2401.10774 , year=

    Pith/arXiv arXiv

  32. [32]

    arXiv preprint arXiv:2310.07177 , year=

    Online speculative decoding , author=. arXiv preprint arXiv:2310.07177 , year=

    Pith/arXiv arXiv

  33. [33]

    arXiv preprint arXiv:2502.20330 , year=

    RAPID: Long-Context Inference with Retrieval-Augmented Speculative Decoding , author=. arXiv preprint arXiv:2502.20330 , year=

    Pith/arXiv arXiv

  34. [34]

    arXiv preprint arXiv:2408.11049 , year=

    Magicdec: Breaking the latency-throughput tradeoff for long context generation with speculative decoding , author=. arXiv preprint arXiv:2408.11049 , year=

    Pith/arXiv arXiv

  35. [35]

    Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Longbench v2: Towards deeper understanding and reasoning on realistic long-context multitasks , author=. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  36. [36]

    Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

    Longbench: A bilingual, multitask benchmark for long context understanding , author=. Proceedings of the 62nd annual meeting of the association for computational linguistics (volume 1: Long papers) , pages=

  37. [37]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    ∞ Bench: Extending long context evaluation beyond 100K tokens , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  38. [38]

    Advances in neural information processing systems , volume=

    Flashattention: Fast and memory-efficient exact attention with io-awareness , author=. Advances in neural information processing systems , volume=

  39. [39]

    arXiv preprint arXiv:2307.08691 , year=

    Flashattention-2: Faster attention with better parallelism and work partitioning , author=. arXiv preprint arXiv:2307.08691 , year=

    Pith/arXiv arXiv

  40. [40]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  41. [41]

    arXiv preprint arXiv:2407.21783 , year=

    The llama 3 herd of models , author=. arXiv preprint arXiv:2407.21783 , year=

    Pith/arXiv arXiv

  42. [42]

    5-coder technical report , author=

    Qwen2. 5-coder technical report , author=. arXiv preprint arXiv:2409.12186 , year=

    Pith/arXiv arXiv

  43. [43]

    Advances in neural information processing systems , volume=

    Language models are few-shot learners , author=. Advances in neural information processing systems , volume=

  44. [44]

    Advances in neural information processing systems , volume=

    Are sixteen heads really better than one? , author=. Advances in neural information processing systems , volume=

  45. [45]

    Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations , pages=

    Transformers: State-of-the-art natural language processing , author=. Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations , pages=

    2020

  46. [46]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Draft& verify: Lossless large language model acceleration via self-speculative decoding , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  47. [47]

    Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

    Layerskip: Enabling early exit inference and self-speculative decoding , author=. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages=

  48. [48]

    arXiv preprint arXiv:2406.02069 , year=

    Pyramidkv: Dynamic kv cache compression based on pyramidal information funneling , author=. arXiv preprint arXiv:2406.02069 , year=

    Pith/arXiv arXiv

  49. [49]

    Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    Pyramidinfer: Pyramid kv cache compression for high-throughput llm inference , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=

    2024

  50. [50]

    Advances in Neural Information Processing Systems , volume=

    Scissorhands: Exploiting the persistence of importance hypothesis for llm kv cache compression at test time , author=. Advances in Neural Information Processing Systems , volume=

  51. [51]

    Advances in Neural Information Processing Systems , volume=

    Kangaroo: Lossless self-speculative decoding for accelerating llms via double early exiting , author=. Advances in Neural Information Processing Systems , volume=

  52. [52]

    arXiv preprint arXiv:2404.11912 , year=

    Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding , author=. arXiv preprint arXiv:2404.11912 , year=

    Pith/arXiv arXiv

  53. [53]

    arXiv preprint arXiv:2512.01278 , year=

    Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding , author=. arXiv preprint arXiv:2512.01278 , year=

    arXiv

  54. [54]

    arXiv preprint arXiv:2602.07223 , year=

    Vegas: Self-Speculative Decoding with Verification-Guided Sparse Attention , author=. arXiv preprint arXiv:2602.07223 , year=

    Pith/arXiv arXiv

  55. [55]

    Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    Draft model knows when to stop: Self-verification speculative decoding for long-form generation , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages=

    2025

  56. [56]

    arXiv preprint arXiv:2410.18351 , year=

    Adaedl: Early draft stopping for speculative decoding of large language models via an entropy-based lower bound on token acceptance probability , author=. arXiv preprint arXiv:2410.18351 , year=

    Pith/arXiv arXiv

  57. [57]

    arXiv preprint arXiv:2405.04304 , year=

    Dynamic speculation lookahead accelerates speculative decoding of large language models , author=. arXiv preprint arXiv:2405.04304 , year=

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2404.06654 , year=

    RULER: What's the real context size of your long-context language models? , author=. arXiv preprint arXiv:2404.06654 , year=

    Pith/arXiv arXiv