Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read PipeSpec breaks the draft-and-verify lockstep of speculative decoding by arranging several LLMs into an asynchronous pipeline, claiming up to 2.54x speedup over autoregressive decoding.

desk verdict A genuinely new asynchronous hierarchical speculation design with plausible speedups, but the 'guaranteed improvement' theorem is about tokens per step, not time, and the headline comparison uses extra GPUs. read the letter →

arxiv 2505.01572 v1 pith:5BCOFNCM submitted 2025-05-02 cs.AI cs.DC

Bradley McDanel , Sai Qian Zhang , Yunhai Hu , Zining Liu This is my paper · ORCID
classification cs.AIcs.DC
keywords speculativedecodingLLMinferenceaccelerationpipelineparallelismasynchronousexecutiontokenverificationrollbackthroughputhierarchical
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that speculative decoding's usual draft-then-verify lockstep, where a small model fills a batch while the large model waits and then the large model verifies while the small model waits, wastes most of the available hardware, and that the waste can be eliminated by arranging several models of increasing size into an asynchronous pipeline. It presents PipeSpec, where each model continuously produces tokens into a buffer, the next model verifies them against its own predictions, and a rejection rolls back earlier buffers. The paper derives an expected-token-per-step formula for $k$ stages, proves that this expectation is always greater than one for any nonzero acceptance rate, and derives closed-form steady-state verification probabilities. Experiments on LLaMA-2 and LLaMA-3 across summarization and code-generation benchmarks report up to 2.54x wall-clock speedup over autoregressive decoding, with three-model pipelines beating two-model ones. If correct, this turns speculative decoding from a paired-model trick into a scalable multi-device pipeline whose gains grow with depth.

What carries the argument

The machinery is a $k$-model producer-consumer pipeline operating without global synchronization. Model $M_0$ continuously writes draft tokens to buffer $O_0$; each later model $M_i$ reads from $O_{i-1}$, generates its own single-token prediction, appends the longest prefix that matches to $O_i$, and if a mismatch occurs, signals all earlier stages to roll back their buffers to the last verified token. The analytical engine is a recurrence for $\rho_i(t_j)$, the probability that model $M_i$ performs verification at its $j$-th time step: $\rho_i(t_j) = \rho_i(t_{j-1})\alpha_{i-1,i}^{\gamma_i+1} + (1-\rho_i(t_{j-1}))\alpha_{i-1,i}$, whose limit gives the closed-form steady-state verification probability $\rho_i = \alpha_{i-1,i}/(1-\alpha_{i-1,i}^{\gamma_i+1}+\alpha_{i-1,i})$. Substituting this $\rho_i$ into the expected-token formula $E[N(M_i)] = (1-\rho_i)\cdot 1 + \rho_i\cdot(1-\alpha^{\gamma_i+1})/(1-\alpha)$ is what produces the claimed guarantee, and the comparison against standard speculative decoding's wait time $\gamma_t/c_{d,t}$ is what locates the speedup in asynchronous execution.

What would settle it

Record the final verifier's buffer occupancy over many generated sequences: if the verifier ever begins a verification step with fewer than its full $\gamma$-token window available, the closed-form steady-state probability $\rho_i = \alpha_{i-1,i}/(1-\alpha_{i-1,i}^{\gamma_i+1}+\alpha_{i-1,i})$ is not the true steady state, and the per-step guarantee in Theorem 1 does not by itself imply the wall-clock speedup. A direct experiment would measure average wall-clock tokens per second at very low acceptance rates, for example $\alpha \approx 0.05$; if PipeSpec ever falls below autoregressive decoding there, the claimed guarantee fails in practice.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is Theorem 1: for any two consecutive models with acceptance probability $0<\alpha<1$ and verification window size $0<\gamma$, PipeSpec's expected number of accepted tokens per verification step is $E[N(M_k)] = (1-\rho_k)\cdot 1 + \rho_k\cdot (1-\alpha^{\gamma+1})/(1-\alpha) > 1$, where $\rho_k = \alpha/(1-\alpha^{\gamma+1}+\alpha)$ is the steady-state probability that the final model verifies a draft window. Because this expectation always exceeds one, the paper states that PipeSpec is guaranteed to produce more tokens per step than autoregressive decoding. It then derives the corresponding speedup expression for standard speculative decoding, $SD(\mathcal{P}_a) = (1-\alpha^{\gamma_t+1})/((1-\alpha_{d,t})(\gamma_t/c_{d,t}+1))$, and argues that PipeSpec avoids the $\gamma_t/c_{d,t}$ waiting term through asynchronous execution. The empirical counterpart is the measured speedup of up to 2.54x relative to autoregressive decoding with a $\{1B,8B,70B\}$ LLaMA-3.1 pipeline on HumanEval, alongside ablation results attributing most of the gain to asynchronous execution (2.54x vs 1.37x when synchronization is forced).

Load-bearing premise

The proof assumes that whenever the final verifier is ready to check a window, a full window of draft tokens is already waiting in its buffer, so the expected tokens produced per step depends only on acceptance rates and window size and not on whether earlier stages can supply tokens fast enough or recover after rollbacks.

Editorial extensions

If this is right

  • PipeSpec's expected tokens per verification step is greater than one for every nonzero acceptance rate, so even a weak draft model should not reduce throughput below autoregressive decoding on a per-step basis.
  • Asynchronous execution is the main source of speedup: in the HumanEval ablation, forcing the pipeline back into synchronous lockstep drops the speedup from 2.54x to 1.37x.
  • Intermediate models improve throughput twice, by filtering bad drafts before they reach the expensive verifier and by raising the acceptance rate into the final stage, so the {1B,8B,70B} pipeline reaches 2.54x versus 2.27x for {8B,70B}.
  • Because each stage is a producer-consumer pair, adding more stages scales the benefit; three-model configurations beat two-model configurations on XSum and HumanEval, and deeper pipelines are predicted to continue this trend.
  • Energy per token drops from 16.5J (autoregressive) to 5.8J (PipeSpec) in the reported LLaMA-3.1-70B measurement, because average GPU utilization rises and idle periods shrink.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test combinations, but its orchestration layer is orthogonal to tree-based drafting, Medusa-style heads, and layer-skipping, so stacking PipeSpec with those methods is a natural next experiment.
  • The closed-form steady-state probability suggests a design rule the paper leaves implicit: choose intermediate stages to maximize the acceptance-rate gain into the next stage rather than simply to minimize per-token time, since $\rho_i$ depends on $\alpha$ through the ratio $\alpha/(1-\alpha^{\gamma+1}+\alpha)$.
  • A queueing model with finite buffers would convert the per-step token guarantee into a wall-clock guarantee; the paper's Theorem 1 does not model production-rate limits or rollback cascades, so its guaranteed throughput improvement is a per-step statement, not a wall-clock proof.
  • The reported speedups use greedy decoding (temperature 0); sampling-based verification, where rejection is random rather than deterministic, is an untested regime that could either dampen or amplify the long acceptance tails the paper observes.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes PipeSpec, a framework for hierarchical speculative LLM decoding in which k models of increasing size operate asynchronously in a producer-consumer pipeline: the smallest model continuously drafts tokens, intermediate models refine and partially verify them, and the largest model performs final verification. The authors derive an analytical model (Section 3.3) with a claimed guarantee that the expected number of tokens emitted per verification step exceeds that of autoregressive decoding for any nonzero acceptance rate, and they report experiments on LLaMA-2 and LLaMA-3 showing up to 2.54x speedup over an autoregressive baseline, with ablations isolating the effect of asynchronous execution and hierarchical refinement.

Significance. The asynchronous pipelining idea is practically motivated, and the ablation comparing synchronous versus asynchronous execution is a useful experimental contribution. The GPU utilization and energy-per-token measurements in Section 4.5 provide system-level insight that is often missing from speculative decoding papers. However, the central theoretical claim is about expected tokens per verification step, not wall-clock throughput, and the headline empirical comparison uses more GPUs for PipeSpec than for the autoregressive baseline. If a time model and a same-resource comparison are added, the paper could be a solid systems contribution; as written, the key speedup claim is not fully established. The theorem itself is close to a definitional consequence of the model and does not by itself justify the phrase 'guaranteed throughput improvements.'

major comments (4)
  1. [Section 3.3, Eq. (4) and Theorem 1] Theorem 1 proves E[N(M_K)] > 1 for any nonzero acceptance rate when gamma > 0, but this is a statement about expected tokens per verification step of the final model, not about tokens per second. A wall-clock speedup requires a time model: the per-step duration of the final verifier, the production rates of earlier stages, buffer sizes, and rollback costs. The implicit assumption that M_{K-1} always has gamma_K draft tokens buffered when M_K is ready to verify is not guaranteed; indeed, Section 4.4 states that PipeSpec 'degrades at larger sizes as verification must wait for draft tokens,' so the assumption can fail in the implemented system. The abstract and contributions claim 'guaranteed throughput improvements,' but Eq. (4) alone does not establish any tokens-per-second number. Please either add a queueing/time model connecting Eq. (4) to wall-clock throughput, or weaken the claim to expected tokens per verification step.
  2. [Section 4.1, Table 2, Figure 5] The headline 2.54x speedup compares PipeSpec {1B, 8B, 70B} running on four A100 GPUs (the 70B model split across two GPUs plus one GPU each for the 1B and 8B drafts) against an autoregressive 70B baseline using only two GPUs. This conflates algorithmic pipelining with a doubling of hardware resources. The comparison against speculative decoding in Table 2 is fair in terms of GPU count, but the abstract's central 'up to 2.54x speedup' is relative to the two-GPU autoregressive baseline. A fair comparison requires matching the total number of GPUs or reporting per-GPU normalized throughput; otherwise the claimed advantage over the baseline is not established. The Limitations section acknowledges extra hardware and energy, but the abstract and conclusion present the speedup unconditionally.
  3. [Section 4.4 and Eq. (1)] The paper states that the authors 'consistently used lookahead sizes of 8 for SD and 0 for PS.' Under the model in Eq. (1), setting gamma = 0 makes the speculative term (1 - alpha^{gamma+1})/(1 - alpha) equal to 1, so the expected number of tokens per step becomes exactly 1, eliminating the theoretical advantage claimed by Theorem 1. If 'lookahead 0' instead means an unbounded or adaptive buffer rather than a zero-token window, this needs to be stated explicitly and incorporated into the model. As written, the empirical configuration is inconsistent with the positive-gamma assumption used in the theoretical analysis, and the claimed 2.54x speedup cannot be explained by Eq. (4) without further clarification.
  4. [Section 4.2, Tables 1 and 2] The empirical results are reported as single point estimates without error bars, confidence intervals, or standard deviations, and no code is released. Because the headline claim is a quantitative speedup, it is important to know the variance across prompts, even under greedy decoding, and to enable reproducibility. Please provide per-configuration variance or raw results, and ideally release the implementation, so that the 2.54x figure can be independently assessed.
minor comments (6)
  1. [Table 2] There is a typo: 'AR Basline' should be 'AR Baseline.'
  2. [Section 4.2] The phrase 'additionaly 12% speedup' should be 'additional 12% speedup.'
  3. [References] The citation for Spector and Ré is incomplete: the reference gives a workshop name but no paper title or full author list; please complete the entry and reconcile the in-text spelling 'Ré' with the reference list.
  4. [Section 3.3, Eq. (2)] Equation (2) uses the symbol alpha without the subscript i-1,i, even though the text defines alpha_{i-1,i} as the per-pair acceptance rate; please clarify whether alpha in Eq. (2) is intended to be alpha_{i-1,i} and whether it is assumed constant across token positions.
  5. [Section 4.1 and Figure 5] The figure caption and Section 4.1 do not define the time unit for 'tokens/unit' used in Figures 1 and 2; since the paper makes throughput claims, the definition of a time unit should be explicit.
  6. [Abstract and Table 2] The abstract says PipeSpec 'outperforming state-of-the-art methods,' but several comparisons in Table 2 (LayerSkip, Draft&Verify) use speedup numbers quoted from the original papers rather than numbers measured in the same harness; please either run these baselines under identical conditions or qualify the claim.

Circularity Check

1 steps flagged · score 2.0 of 10

The theoretical 'guaranteed throughput improvement' in Theorem 1 is a definitional tautology, while the headline 2.54x speedup is measured independently; overall circularity is minor and confined to the analytical overclaim.

  1. self definitional [Section 3.3, Eq. (4), Theorem 1]
    "PipeSpec(P)=(1−ρ_k)·1+ρ_k·(1−α^{γ+1})/(1−α)>1 (4) ... It is obvious that PipeSpec(P) is greater than 1 for any α and γ greater than 0, so the pipeline specification is definitely better than autoregressive decoding."

    The theorem's conclusion is already contained in the definition of the random variable. Eq. (1) defines E[N(M_K)] as a mixture: a rejection contributes exactly one token, and an accepted window contributes (1−α^{γ+1})/(1−α), which is greater than 1 for 0<α<1 and γ>0. With the steady-state ρ_k>0, Eq. (4) is just the statement that a convex combination of 1 and a value greater than 1 exceeds 1. No queueing, production-rate, or wall-clock time enters; hence 'higher number of tokens per step' does not by itself imply 'throughput improvement' over autoregressive decoding. The theorem is therefore a tautology about the chosen metric, not an independent derivation of a speedup.

full rationale

The paper's central empirical claim, up to 2.54x speedup, is supported by end-to-end timing measurements in Table 2 against an autoregressive baseline. These measurements are independent of the analytical model: no fitted parameter is relabeled as a prediction, and the speedup numbers are not derived from Eq. (4). The only clear circular step is Theorem 1, where the 'guaranteed throughput improvement' reduces to the definition of the per-step token count: because a rejection is counted as one token and an acceptance contributes a positive geometric-series bonus, the expected count per step is greater than one by construction. That is a real self-definitional overclaim, but it is not load-bearing for the measured speedups. The paper does not rely on a self-citation chain; the only related-work citations used in the derivation, notably Leviathan et al. for the speculative-decoding acceptance formula, are external. The Limitations section and Section 4.4 acknowledge that verification can wait for draft tokens and that extra hardware and energy are required, which are correctness and scope concerns rather than additional circularity. Overall, the derivation chain is mostly self-contained, with the theoretical guarantee being the notable exception.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The theoretical model inherits the standard speculative decoding assumption of a fixed acceptance rate and reuses the Leviathan et al. expected-token formula. Its main unsupported addition is the assumption of uninterrupted token supply to each verification stage, which makes the 'guaranteed improvement' a statement about an ideal pipeline rather than a physically realizable system. No new physical entities are introduced.

free parameters (1)
  • PipeSpec lookahead window size = 0 (vs 8 for SD)
    Section 4.4 states 'We consistently used lookahead sizes of 8 for SD and 0 for PS.' This hand-selected asymmetry favors PipeSpec by avoiding wasted speculation at large windows, and the reported speedups depend on it.
assumptions (5)
  • domain assumption Acceptance rate α_{i-1,i} is constant across token positions and independent of previous tokens.
    Section 3.3 defines α_{i-1,i} as a single probability per model pair. This ignores the well-known variability of acceptance rates (e.g., harder tokens later in a sequence).
  • domain assumption The expected number of accepted tokens per verification round follows the standard speculative decoding formula (1-α^{γ+1})/(1-α) from Leviathan et al. (2023).
    Eq. (1) reuses this formula without re-deriving it for asynchronous multi-stage buffers.
  • domain assumption The recurrence for verification probability ρ_i(t_j) converges to its stationary value and the Cesàro average equals that value.
    Eq. (3) states the limit without proving convergence or stationarity conditions.
  • ad hoc to paper The verifier always has γ_i draft tokens buffered when it begins verification; no queueing delays or buffer underruns are modeled.
    Eq. (1) and Theorem 1 implicitly assume the pipeline is never starved, which is essential to the throughput claim.
  • ad hoc to paper Tokens per step is equated with throughput; step duration is not included in Theorem 1.
    Theorem 1 compares expected tokens per step with autoregressive decoding, but a step for the large model takes the same wall-clock time in both schemes only if the large model never waits, which requires the previous axiom.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding." pith.science (2026). https://pith.science/paper/5BCOFNCM

@misc{pith2026250501572,
  author       = {Pith},
  title        = {Pith review of: PipeSpec: Breaking Stage Dependencies in Hierarchical LLM Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5BCOFNCM}},
  note         = {Machine review of arXiv:2505.01572}
}
abstract

Speculative decoding accelerates large language model inference by using smaller draft models to generate candidate tokens for parallel verification. However, current approaches are limited by sequential stage dependencies that prevent full hardware utilization. We present PipeSpec, a framework that generalizes speculative decoding to $k$ models arranged in a hierarchical pipeline, enabling asynchronous execution with lightweight coordination for prediction verification and rollback. Our analytical model characterizes token generation rates across pipeline stages and proves guaranteed throughput improvements over traditional decoding for any non-zero acceptance rate. We further derive closed-form expressions for steady-state verification probabilities that explain the empirical benefits of pipeline depth. Experimental results show that PipeSpec achieves up to 2.54$\times$ speedup while outperforming state-of-the-art methods. We validate PipeSpec across text summarization and code generation tasks using LLaMA 2 and 3 models, demonstrating that pipeline efficiency increases with model depth, providing a scalable approach to accelerating LLM inference on multi-device systems.

Figures

Figures reproduced from arXiv: 2505.01572 by the authors.

Figure 1
Figure 1. Comparison of different LLM decoding ap [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Comparison of different decoding approaches showing token generation over time. From top to bottom: (1) [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Analysis of token acceptance patterns and [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Impact of lookahead window size on token [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Collaborative Inference and Learning between Edge SLMs and Cloud LLMs: A Survey of Algorithms, Execution, and Open Challenges

    cs.DC 2025-07 conditional novelty 4.0 of 10

    A survey that builds a taxonomy of edge-cloud LLM-SLM collaboration for inference and training, claiming to be the first to unify both phases.

Reference graph

Works this paper leans on

27 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Reza Yazdani Aminabadi, Samyam Rajbhandari, Ammar Ahmad Awan, Cheng Li, Du Li, Elton Zheng, Olatunji Ruwase, Shaden Smith, Minjia Zhang, Jeff Rasley, et al. 2022. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis...

    2022

  4. [4]

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774

    arXiv 2024

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  6. [6]

    Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691

    arXiv 2023

  7. [7]

    Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . 2022. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 35:16344--16359

    2022

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

Show all 27 references
  1. [9]

    Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. 2024. Layer skip: Enabling early exit inference and self-speculative decoding. arXiv preprint arXiv:2404.16710

    2024 arXiv

  2. [10]

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057

    2024 arXiv

  3. [11]

    Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. 2024. Rest: Retrieval-based speculative decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers...

    2024

  4. [12]

    Ke Hong, Guohao Dai, Jiaming Xu, Qiuli Mao, Xiuhong Li, Jun Liu, Kangdi Chen, Yuhan Dong, and Yu Wang. 2023. Flashdecoding++: Faster large language model inference on gpus. arXiv preprint arXiv:2311.01282

    2023 arXiv

  5. [13]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Princip...

    2023

  6. [14]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. https://arxiv.org/abs/2211.17192 Fast inference from transformers via speculative decoding . Preprint, arXiv:2211.17192

    2023 arXiv

  7. [15]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077

    2024 arXiv

  8. [16]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. 2024. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings...

    2024

  9. [17]

    Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. arXiv preprint arXiv:1602.06023

    2016 arXiv

  10. [18]

    Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1797--1807

    2018

  11. [19]

    Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, \'I \ n igo Goiri, Saeed Maleki, and Ricardo Bianchini. 2024. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), p...

    2024

  12. [20]

    Ying Sheng, Lianmin Zheng, Binhang Yuan, Zhuohan Li, Max Ryabinin, Beidi Chen, Percy Liang, Christopher R \'e , Ion Stoica, and Ce Zhang. 2023. Flexgen: High-throughput generative inference of large language models with a single gpu. In International Conference on Machine Lear...

    2023

  13. [21]

    Accelerating llm inference with staged speculative decoding

    Benjamin Frederick Spector and Christopher Re. Accelerating llm inference with staged speculative decoding. In Workshop on Efficient Systems for Foundation Models@ ICML2023

  14. [22]

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. In Advances in Neural Information Processing Systems, volume 31, pages 10107--10116

    2018

  15. [23]

    Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. 2024. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.11912

    2024 arXiv

  16. [24]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  17. [25]

    Neal Vaidya, Fred Oh, and Nick Comly. 2023. Optimizing inference on large language models with nvidia tensorrt-llm, now publicly available. https://github.com/NVIDIA/TensorRT-LLM. [Online]

    2023

  18. [26]

    Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. 2023. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168

    2023 arXiv

  19. [27]

    Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-Fran c ois Kagy, and Rishabh Agarwal. 2024. Distillspec: Improving speculative decoding via knowledge distillation. In The Twelfth International Conference on Learning ...

    2024

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.