Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Hierarchy Drafting accelerates lossless LLM inference by fetching draft tokens from context, model, and statistics databases in temporal-locality order, reporting up to 1.70x speedup over autoregressive decoding on Vicuna and Llama-2.

desk verdict Real speedups and a clear empirical method, undermined by an ablation that contradicts the paper's own claim that the third database helps. read the letter →

arxiv 2502.05609 v1 pith:J7VL6VSD submitted 2025-02-08 cs.CL

Sukmin Cho , Sangjin Choi , Taeho Hwang , Jeongyeon Seo , Soyeong Jeong , Huije Lee , Hoyun Song , Jong C. Park
show 1 more author
Youngjin Kwon
This is my paper · ORCID
classification cs.CL
keywords speculativedecodinghierarchicaldraftingtemporallocalitydatabaselosslessaccelerationLLMinferencen-gramretrievalSpec-Bench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language model inference is bottlenecked by generating one token at a time. This paper argues that a speculative decoding drafter can be made both fast and accurate without any fine-tuning by drawing draft tokens from three databases ordered by temporal locality: a small context database built from the current generation, a model database of phrases the LLM family repeats across generations, and a large statistics database of common corpus language. On Spec-Bench it reports 1.51x–1.70x speedups over autoregressive decoding at temperature 0.0 across Vicuna-7B/13B/33B and Llama-2-7B/13B, consistently above single-source drafters such as PLD, LADE, and REST, and it keeps the largest gains under temperature sampling. The method is lossless because the target model verifies every drafted token, and it requires no parameter updates, so it can be dropped into an existing serving stack.

What carries the argument

The mechanism is a hierarchy of n-gram databases indexed by prefix-key to draft-value, with each source assigned by temporal locality: Dc is a small LRU-updated lookup table (~1K sequences) that stores tokens from the current prompt and from parallel decoding; Dm holds 100K frequent token sequences sampled from 39,283 texts generated by the 7B model of the target family; Ds is a 200M-sequence corpus database (about 12GB) searched with a suffix array. Drafting retrieves up to N=7 candidate sequences of length m=4, starting from Dc and moving to Dm then Ds only when more candidates are needed. Because higher-locality tokens are rarer, this access order keeps retrieval latency low while raising acceptance; the ablation shows the order matters, with non-hierarchical orders raising drafting latency to as much as 12ms.

What would settle it

Run HD on a larger target model in the same family, for example a 70B model, with Dm built from 7B-generated texts, and compare speedup against REST on Spec-Bench; if the acceptance ratio falls toward the statistics-only level or the speedup drops below REST, the transfer premise is refuted. A cheaper check is to ablate Dm on the 13B and 33B targets and see whether the acceptance gap over REST disappears.

Watch

Extended reading notes

Core claim

The central claim is that draft-token sources have different degrees of temporal locality—how often a token sequence reappears within the current generation versus across generations—and that organizing them into databases ranked by that locality lets one drafter be accurate and cheap at the same time. HD builds a context-dependent database (tokens tied to the current prompt and generation), a model-dependent database (frequent n-grams from model-generated texts), and a statistics-dependent database (common corpus n-grams), and for every decoding step it accesses them in that order until seven candidate continuations are collected. The target LLM verifies all candidates in one forward pass, which makes the method lossless. On Spec-Bench, HD reports the highest speedup among database drafting methods on all five tested models at both temperatures, with a 1.51x–1.70x speedup at T=0.0 versus 1.17x–1.38x for REST, and acceptance ratios around 73–80%.

Load-bearing premise

The load-bearing premise is that word-sequence patterns learned from a 7B model of a family transfer to the 13B and 33B targets well enough to keep acceptance high; if that transfer fails, the model-dependent database contributes little and the reported speedups on larger models would not generalize.

Editorial extensions

If this is right

  • At T=0.0, HD reports 1.51x–1.70x speedup versus 1.17x–1.38x for REST, making it the strongest training-free database drafter in these comparisons.
  • HD stays above the other database drafters at T=1.0 and on each of the six Spec-Bench tasks, so it is less task-sensitive than PLD, LADE, or REST.
  • HD beats the non-database method SpS and narrows the gap to Medusa without retraining, suggesting training-free drafting can approach trained drafters on these benchmarks.
  • The framework is plug-and-play: future sources can be added to the appropriate database by locality, for example recycled tokens into the context-dependent database.
  • Losslessness is preserved because the target model verifies every draft candidate before any token is emitted.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the temporal-locality ordering generalizes, new draft sources such as retrieved documents, user-specific prefixes, or code-completion context could be slotted into Dc or Dm without retraining; the paper sketches this but does not test it.
  • The 7B-to-13B/33B transfer of Dm suggests a cheap recipe for new model families—generate a few tens of thousands of texts with the smallest member and reuse the same statistics database—but this recipe is untested beyond these two families.
  • The paper's own comparison shows that a compact statistics database (ShareGPT, 465MB) yields a higher speedup (1.57x) than the 12GB UltraChat database (1.51x) despite a lower accepted length, which suggests retrieval latency, not coverage, may be the practical bottleneck to attack next.
  • A direct testable prediction is that adding a fourth database with even longer-horizon patterns, such as document-level repetition, would further lift acceptance on summarization and RAG tasks without a proportional latency penalty.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Hierarchy Drafting (HD), a training-free speculative decoding method that constructs three token databases—context-dependent (Dc), model-dependent (Dm), and statistics-dependent (Ds)—ordered by temporal locality, and during drafting accesses them sequentially until the draft set is filled. On Spec-Bench, HD is reported to outperform PLD, LADE, and REST across Vicuna-7B/13B/33B and Llama-2-7B/13B at T=0.0 and T=1.0, with speedups up to 1.70x. The paper also provides analyses of per-database success rates, access order, temperature robustness, ablations, and alternative token sources.

Significance. If the empirical claims hold, HD is a useful practical contribution: it is training-free, lossless, and consistently faster than single-source database drafters, while narrowing the gap with retraining-based methods such as Medusa. The paper ships a public code repository, uses the standard Spec-Bench evaluation, and includes ablations and per-database analyses, which are strengths. The hierarchical ordering idea is simple and plausible. However, the ablation evidence currently undercuts the specific claim that the three-database hierarchy is what delivers the reported gains, and the single-run evaluation limits confidence in the robustness claims.

major comments (3)
  1. [Appendix C.4, Table 3] The ablation contradicts the central design claim. On Llama-2-7b, full HD with (Dc, Dm, Ds) achieves 79.72% acceptance, 2.18 ms drafting latency, and 1.64x speedup, while (Dc, Dm) alone achieves 75.60% acceptance, 0.03 ms latency, and 1.71x speedup. Adding Ds therefore raises acceptance by 4.12 percentage points but lowers end-to-end speedup by 0.07x, so the statement in C.4 that 'combining multiple databases ... lead[s] to more robust and efficient performance' is not supported by the reported numbers. The main text (Section 5.1) attributes HD's acceleration to the hierarchical integration of all three sources; the authors need either to recommend (Dc, Dm) as the default configuration or to identify settings (e.g., larger targets or longer generations) where Ds improves end-to-end speedup and report those settings. The adjacent sentence calling Dc 'the largest database' is also incorrect, since Ds is the 200M-sequence database, and this mislabeling obscures the interpretation.
  2. [Section 4, Experimental Setup] The entire evaluation is a single run, with no error bars or seeds reported for Table 2. This matters because the paper's stated contribution is 'consistent acceleration across diverse tasks and temperatures,' and the T=1.0 results use sampling, which introduces run-to-run variability. The sentence 'we observed only marginal differences between runs' is not accompanied by data. Please report means and standard deviations over at least three seeds (or provide per-seed tables) for speedup, acceptance ratio, and drafting latency, or explicitly restrict the claims to the single-run setting.
  3. [Section 4, Implementation Detail; Appendix C.5] The model-dependent database is built from 39,283 texts generated by a 7B model of the same family as the target, and the paper evaluates on Vicuna-7B/13B/33B and Llama-2-7B/13B. The reported speedups on 13B and 33B therefore assume that n-gram transfer from the smaller family member to larger members is sufficiently strong, but this assumption is not tested: Table 4 varies Dm and Ds sources only for Vicuna-7B, and Figure 5's per-database success statistics are given only for Llama-2-7b. Please report Dm's draft-success and verify-success rates broken down by target model size, or run the (Dc, Dm) vs. (Dc, Dm, Ds) ablation on at least one 13B model, to establish the contribution of Dm on larger targets.
minor comments (5)
  1. [Abstract; Table 2] The abstract says the experiments use LLMs with 7B and 13B parameters, but Section 4 and Table 2 also include Vicuna-33B; please align the abstract with the actual evaluation.
  2. [Section 3.2] The description of the context-dependent database uses the notation x1:l and xl:l+m for keys and values, while Algorithm 1 and the retrieval equation use xn-l:n; please clarify the indexing convention.
  3. [Appendix B; Section 4] Appendix B says the prefix length used as the key is 1 for Dc and Dm, whereas Section 4 sets l=2 for the previous-token window; please explain how these two settings interact.
  4. [Table 2] Table 2 does not report standard deviations for drafting latency, although Table 1 does; please include them for consistency and to support the comparison across methods.
  5. [Figure 3] The caption of Figure 3 refers to '4-gram statistics for 100 generations,' but the lower plot is described as a frequency analysis of two sampled 4-grams; please label the axes of the lower plot and state what the colors denote.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; reported speedups are measured end-to-end, with same-family Dm and ablation inconsistency being empirical design concerns rather than self-referential reasoning.

full rationale

The paper's speedup claims rest on measured Spec-Bench runs (Table 2) rather than on derived constants; no equation defines the reported speedup in terms of the database contents. The model-dependent database is generated by 'a 7B model from the targeted LLM family' (Section 4), which for the 7B targets is the target model itself; this is a mild self-matching design choice, but acceptance ratios and latencies are measured end-to-end, not fitted, and the 13B/33B results test transfer. Ds reuses REST's retriever and corpus with explicit attribution, and LADE's parallel decoding is credited; these are external components, not self-citations. No uniqueness theorem or ansatz is imported from the authors' prior work (there are no such self-citations). The paper's own ablation (Appendix C.4, Table 3) shows full HD (1.64x, 2.18ms) is slower than (Dc,Dm) (1.71x, 0.03ms), which weakens the hierarchical-integration claim empirically, but this is an internal-consistency question, not a circularity: the full method's numbers are not equal to its inputs by construction. Hence no circular step is exhibited.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim relies on several hand-picked hyperparameters (l, m, N, 100k top-k, 12GB corpus) and a domain assumption about n-gram temporal locality. No new physical or mathematical entities are introduced; the 'three databases' are engineering components, not theoretical postulates. The losslessness guarantee is inherited from prior speculative sampling results.

free parameters (6)
  • l (prefix/query length) = 2 (main text), key length 1 for Dc/Dm in Appendix B
    Used in Algorithm 1 and Section 4; retrieval hit rate depends on this value, and the paper gives conflicting values.
  • m (draft sequence length) = 4
    Length of each fetched draft sequence; chosen in Section 4 without sensitivity analysis.
  • N (draft set size) = 7
    Number of parallel draft sequences verified per step; chosen in Section 4 without sensitivity analysis.
  • T (max generation length) = 1024
    Generation cap in experiments; standard setting.
  • Dm top-k token sequences = 100k
    Sampled from 39,283 OASST generations; hand-chosen capacity.
  • Ds database size = about 12GB UltraChat (200M sequences)
    Adopted from REST; affects latency and accuracy.
assumptions (4)
  • domain assumption N-gram temporal locality: tokens in LLM text repeat within and across generations.
    Figure 3 analyzes 100 generations; the whole method is built on this observation.
  • domain assumption Draft tokens can be retrieved from prefix n-gram keys and accepted often enough to accelerate decoding.
    Equation (3) and Algorithm 1; same premise as PLD, LADE, and REST.
  • domain assumption Using LADE's n-gram verification and parallel decoding preserves losslessness and is compatible with Dm and Ds.
    Appendix B says HD is based on the LADE implementation; this is borrowed, not re-proven.
  • standard math Speculative sampling with draft probability 1.0 preserves the target distribution.
    Cited from Chen et al. (2023a); not re-derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding." pith.science (2026). https://pith.science/paper/J7VL6VSD

@misc{pith2026250205609,
  author       = {Pith},
  title        = {Pith review of: Lossless Acceleration of Large Language Models with Hierarchical Drafting based on Temporal Locality in Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J7VL6VSD}},
  note         = {Machine review of arXiv:2502.05609}
}
read the original abstract

Accelerating inference in Large Language Models (LLMs) is critical for real-time interactions, as they have been widely incorporated into real-world services. Speculative decoding, a fully algorithmic solution, has gained attention for improving inference speed by drafting and verifying tokens, thereby generating multiple tokens in a single forward pass. However, current drafting strategies usually require significant fine-tuning or have inconsistent performance across tasks. To address these challenges, we propose Hierarchy Drafting (HD), a novel lossless drafting approach that organizes various token sources into multiple databases in a hierarchical framework based on temporal locality. In the drafting step, HD sequentially accesses multiple databases to obtain draft tokens from the highest to the lowest locality, ensuring consistent acceleration across diverse tasks and minimizing drafting latency. Our experiments on Spec-Bench using LLMs with 7B and 13B parameters demonstrate that HD outperforms existing database drafting methods, achieving robust inference speedups across model sizes, tasks, and temperatures.

Figures

Figures reproduced from arXiv: 2502.05609 by the authors.

Figure 1
Figure 1. Result of database drafting methods on Spec￾Bench (Xia et al., 2024) with Vicuna-7B (Zheng et al., 2023). The values in the plot denote the speedup against autoregres￾sive decoding. (Left) QA and summarization task perfor￾mance. (Right) Acceptance ratio and drafting latency. LLM to generate multiple accepted tokens in the verification phase, speculative decoding improves both the throughput and the latency of the LL… view at source ↗
Figure 2
Figure 2. Overview of database drafting and our proposed method, Hierarchy Drafting (HD). A. Previous database drafting methods retrieve draft tokens from a single database constructed from a single source, leading to inconsistent acceleration gains across different scenarios. B. HD, however, leverages multiple databases encompassing diverse sources to improve token coverage, ensuring consistent performance. B-1. During the d… view at source ↗
Figure 3
Figure 3. (Upper) 4-gram statistics for 100 generations of Llama-2-7b. The x-axis shows the order of 4-grams across 100 generations, with major ticks marking generation steps. The y-axis represents unique 4-gram indices. Red dots indicate 4-grams from previous processes, while blue dots represent those from the current process. (Lower) Frequency analysis of two 4-grams, represented by red and blue dots, respectively. be repea… view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: (Left) Speedup comparison with non-database drafting methods with Vicuna-7B on Spec-Bench. (Right) Speedup comparison of database drafting methods across six tasks of Spec-Bench [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: (Left) Verify success and draft latency for the databases Dc, Dm, and Ds in HD. Verify success represents the proportion of accepted accesses relative to the total accesses. (Right) Verify success density plots for each database across six tasks in Spec-Bench. Both res…
Figure 6
Figure 6. Figure 6: Impact of access order in HD with Llama-2-7b on Spec-Bench. Blue and red bars depict the acceptance ratio and draft latency, respectively. The value over the bars denotes the speedup against autoregressive decoding. turn Conversation or Summarization, where the context…
Figure 7
Figure 7. Figure 7: Correlation between generated token length and elapsed latency using Llama-2-7b-chat on Spec-Bench. Dots in the plot represent acceleration results for individual gener￾ations, while the lines show the linear regression results for each method. Llama-2 from the officia…
Figure 9
Figure 9. Figure 9: Analysis of database access using Llama-2-7b-chat on Spec-Bench. The total bar height represents the overall database access ratio, with each color indicating draft failure, draft success, and draft & verify success. We conducted a breakdown study of database access pa…
Figure 10
Figure 10. Figure 10: Venn diagram of accepted tokens using a single database. Beyond the ablation study, we performed an in￾depth analysis to verify the unique token distribu￾tions of each database by examining the accepted tokens when using only a single database, as shown on the right s…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Faster and Better LLMs via Latency-Aware Test-Time Scaling

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Test-time scaling should be tuned for wall-clock latency rather than token count; branches plus speculative decoding win under time budgets.

Reference graph

Works this paper leans on

47 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Chandra, and Marc Snir

    Alok Aggarwal, Bowen Alpern, Ashok K. Chandra, and Marc Snir. 1987. https://doi.org/10.1145/28395.28428 A model for hierarchical memory . In Proceedings of the 19th Annual ACM Symposium on Theory of Computing, 1987, New York, New York, USA , pages 305--314. ACM

    work page arXiv 1987

  2. [2]

    Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan - Kelley, and William Brandon. 2024. https://doi.org/10.48550/ARXIV.2402.05109 Hydra: Sequentially-dependent draft heads for medusa decoding . arXiv preprint arXiv:2402.05109

    arXiv doi:10.48550/arxiv.2402.05109 2024

  3. [3]

    Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint - Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. 2014. https://doi.org/10.3115/V1/W14-3302 Findings of the 2014 workshop on statistical machine translation . In Proceedings of the Ninth Workshop on S...

    doi:10.3115/v1/w14-3302 2014

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    2020

  5. [5]

    Fenglong Cai, Dong Yuan, Zhe Yang, and Lizhen Cui. 2024 a . https://doi.org/10.1109/ICWS62655.2024.00099 Edge-llm: A collaborative framework for large language model serving in edge computing . In IEEE International Conference on Web Services, ICWS 2024, Shenzhen, China, July 7-13, 2024 , pages 799--809. IEEE

    arXiv 2024

  6. [6]

    Lee, Deming Chen, and Tri Dao

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D. Lee, Deming Chen, and Tri Dao. 2024 b . https://openreview.net/forum?id=PEpbUobfJv Medusa: Simple LLM inference acceleration framework with multiple decoding heads . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net

    work page 2024

  7. [8]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean - Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023 b . https://doi.org/10.48550/ARXIV.2302.01318 Accelerating large language model decoding with speculative sampling . arXiv preprint arXiv:2302.01318

    arXiv doi:10.48550/arxiv.2302.01318 2023

  8. [9]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. http://arxiv.org/abs/2110.14168 Training verifiers to solve math word problems . arXiv preprint arXiv:2110.14168

    arXiv 2021

Show all 47 references
  1. [10]

    Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. https://doi.org/10.18653/V1/2023.EMNLP-MAIN.183 Enhancing chat language models by scaling high-quality instructional conversations . In Proceedings of the 2023 Conference ...

    2023 doi

  2. [11]

    Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023. https://openreview.net/forum?id=tcbBPnfwxS OPTQ : Accurate quantization for generative pre-trained transformers . In The Eleventh International Conference on Learning Representations

    2023

  3. [12]

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. https://openreview.net/forum?id=eDjvSFOkXw Break the sequential dependency of LLM inference using lookahead decoding . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,...

    2024

  4. [13]

    Mahoney, and Kurt Keutzer

    Amir Gholami, Zhewei Yao, Sehoon Kim, Coleman Hooper, Michael W. Mahoney, and Kurt Keutzer. 2024. https://doi.org/10.1109/MM.2024.3373763 Ai and memory wall . IEEE Micro, 44(3):33--39

    2024

  5. [14]

    Zhenyu He, Zexuan Zhong, Tianle Cai, Jason Lee, and Di He. 2024. https://aclanthology.org/2024.naacl-long.88 REST : Retrieval-based speculative decoding . In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Huma...

    2024

  6. [15]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen - tau Yih. 2020. https://doi.org/10.18653/V1/2020.EMNLP-MAIN.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on...

    2020 doi

  7. [16]

    Mahoney, and Kurt Keutzer

    Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W. Mahoney, and Kurt Keutzer. 2023. https://doi.org/10.48550/ARXIV.2306.07629 Squeezellm: Dense-and-sparse quantization . arXiv preprint arXiv:23056.07629

    2023 doi arXiv

  8. [17]

    Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Yacine Jernite, Margaret Mitchell, Carlos Mu \ n oz Ferrandis, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. 2023. https://openreview.net/forum?id=pxpbTdUEpD The stack: 3 TB...

    2023

  9. [18]

    o pf, Yannic Kilcher, Dimitri von R \

    Andreas K \"o pf, Yannic Kilcher, Dimitri von R \"u tte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Minh Nguyen, Oliver Stanley, Rich \'a rd Nagyfi, Shahul ES, Sameer Suri, David Alexandrovich Glushkov, Arnav Varma Dantuluri, Andrew Maguire, Christ...

    2023

  10. [19]

    Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming - Wei Chang, Andrew M

    Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur P. Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming - Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and...

    2019 doi

  11. [20]

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. https://doi.org/10.1145/3600006.3613165 Efficient memory management for large language model serving with pagedattention . In Proceedings of the 29...

    2023

  12. [21]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. https://proceedings.mlr.press/v202/leviathan23a.html Fast inference from transformers via speculative decoding . In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA , volume 2...

    2023

  13. [22]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 a . http://arxiv.org/abs/2406.16858 EAGLE-2: faster inference of language models with dynamic draft trees . arXiv preprint arXiv:2406.16858

    2024 arXiv

  14. [23]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 b . https://openreview.net/forum?id=1NdN7eXyb4 EAGLE: speculative sampling requires rethinking feature uncertainty . In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27,...

    2024

  15. [24]

    Xianzhen Luo, Yixuan Wang, Qingfu Zhu, Zhiming Zhang, Xuanyu Zhang, Qing Yang, Dongliang Xu, and Wanxiang Che. 2024. https://arxiv.org/pdf/2408.08696 Turning trash into treasure: Accelerating inference of large language models with token recycling . arXiv preprint arXiv:2408.08696

    2024 arXiv

  16. [25]

    Udi Manber and Eugene W. Myers. 1993. https://doi.org/10.1137/0222058 Suffix arrays: A new method for on-line string searches . SIAM J. Comput. , 22(5):935--948

    1993 doi

  17. [26]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, Chunan Shi, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. 2024. https://doi.org/10.1145/3620666.3651335 Specinfer: Acc...

    2024

  18. [27]

    Ramesh Nallapati, Bowen Zhou, C \' cero Nogueira dos Santos, C aglar G \" u l c ehre, and Bing Xiang. 2016. https://doi.org/10.18653/V1/K16-1028 Abstractive text summarization using sequence-to-sequence rnns and beyond . In Proceedings of the 20th SIGNLL Conference on Computat...

    2016 doi

  19. [28]

    Patterson

    David A. Patterson. 2004. https://doi.org/10.1145/1022594.1022596 Latency lags bandwith . Commun. ACM , 47(10):71--75

    2004

  20. [29]

    Reiner Pope, Sholto Douglas, Aakanksha Chowdhery, Jacob Devlin, James Bradbury, Jonathan Heek, Kefan Xiao, Shivani Agrawal, and Jeff Dean. 2023. https://proceedings.mlsys.org/paper\_files/paper/2023/hash/c4be71ab8d24cdfb45e3d06dbfca2780-Abstract-mlsys2023.html Efficiently scal...

    2023

  21. [30]

    Guillem Ram \' rez, Alexandra Birch, and Ivan Titov. 2024. https://openreview.net/forum?id=T9cOYH0wGF Optimising calls to large language models with uncertainty-based two-tier selection . In First Conference on Language Modeling

    2024

  22. [31]

    Andrea Santilli, Silvio Severino, Emilian Postolache, Valentino Maiorca, Michele Mancusi, Riccardo Marin, and Emanuele Rodol \` a . 2023. https://doi.org/10.18653/V1/2023.ACL-LONG.689 Accelerating transformer inference for translation via parallel decoding . In Proceedings of ...

    2023 doi

  23. [32]

    Apoorv Saxena. 2023. https://github.com/apoorvumang/prompt-lookup-decoding/ Prompt lookup decoding

    2023

  24. [33]

    Noam Shazeer. 2019. http://arxiv.org/abs/1911.02150 Fast transformer decoding: One write-head is all you need . arXiv preprint arXiv:1911.02150

    2019 arXiv

  25. [34]

    Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph Gonzalez, and Ion Stoica. 2024. https://proceedings.mlsys.org/paper_files/paper/2024/hash/906419cd502575b617cc489a1a696a67-Abstract-Co...

    2024

  26. [35]

    Benjamin Spector and Christopher R \' e . 2023. https://doi.org/10.48550/ARXIV.2308.04623 Accelerating LLM inference with staged speculative decoding . arXiv preprint arXiv:2308.04623

    2023 doi arXiv

  27. [36]

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. https://proceedings.neurips.cc/paper/2018/hash/c4127b9194fe8562c64dc0f5bf2c93bc-Abstract.html Blockwise parallel decoding for deep autoregressive models . In Advances in Neural Information Processing Systems 31: Annual C...

    2018

  28. [37]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton - Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu,...

    2023 doi arXiv

  29. [38]

    Heming Xia, Tao Ge, Peiyi Wang, Si - Qing Chen, Furu Wei, and Zhifang Sui. 2023. https://doi.org/10.18653/V1/2023.FINDINGS-EMNLP.257 Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation . In Findings of the Association for Computational Li...

    2023 doi

  30. [39]

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. 2024. https://doi.org/10.18653/V1/2024.FINDINGS-ACL.456 Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding . In Findin...

    2024 doi

  31. [40]

    Nan Yang, Tao Ge, Liang Wang, Binxing Jiao, Daxin Jiang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. https://doi.org/10.48550/ARXIV.2304.04487 Inference with reference: Lossless acceleration of large language models . arXiv preprint arXiv:2304.04487, abs/2304.04487

    2023 doi arXiv

  32. [41]

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. 2022. http://papers.nips.cc/paper\_files/paper/2022/hash/adf7fa39d65e2983d724ff7da57f00ac-Abstract-Conference.html Zeroquant: Efficient and affordable post-training quantization for larg...

    2022

  33. [42]

    Euiin Yi, Taehyeon Kim, Hongseok Jeung, Du - Seong Chang, and Se - Young Yun. 2024. https://doi.org/10.48550/ARXIV.2406.16758 Towards fast multilingual LLM inference: Speculative decoding and specialized drafters . arXiv preprint arXiv:2406.16758

    2024 doi arXiv

  34. [43]

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. 2024. https://doi.org/10.48550/ARXIV.2401.02385 Tinyllama: An open-source small language model . arXiv preprint arXiv:2401.02385

    2024 doi arXiv

  35. [44]

    Barrett, Zhangyang Wang, and Beidi Chen

    Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher R \' e , Clark W. Barrett, Zhangyang Wang, and Beidi Chen. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/6ceefa7b15572587b78ecfcebb2827f8-Abstract-C...

    2023

  36. [45]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei - Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. http://papers.nips.cc/paper\_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstr...

    2023

  37. [46]

    Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Rostamizadeh, Sanjiv Kumar, Jean-Fran c ois Kagy, and Rishabh Agarwal. 2024. https://openreview.net/forum?id=rsY6J3ZaTF Distillspec: Improving speculative decoding via knowledge distillation . In The T...

    2024

  38. [47]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...

  39. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.