Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CTC training for draft tokens lifts speculative decoding to 2.8x faster inference.

desk verdict A genuinely novel CTC-based draft model for speculative decoding with promising but unverifiable numbers, and a mechanism claim that oversells what the inference procedure actually does. read the letter →

arxiv 2412.00061 v1 pith:UJON6HLI submitted 2024-11-25 cs.LG cs.AIcs.CL

Zhuofan Wen , Shangtong Gui , Yang Feng This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords speculativedecodingCTClossdraftmodelLLMinferenceaccelerationnon-autoregressivetokentreeverificationknowledgedistillationacceptancerate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speculative decoding speeds up LLMs by having a small draft model propose several future tokens that the base model then verifies in parallel; the speed depends on how often those proposed tokens are accepted. This paper claims that the draft model's acceptance rate can be raised by training it with Connectionist Temporal Classification (CTC), a sequence-level objective that considers all ways a proposed token sequence could collapse to the target text. The resulting CTC-drafter uses a transformer layer over the base model's hidden states, adds blank and repeated tokens to the candidate set, and removes them before verification with an adjusted attention mask. On Vicuna and LLaMA-2-Chat models, the paper reports 3.4 to 3.6 accepted tokens per step and 2.2x to 2.8x speedups, beating Medusa and Hydra on MT-bench and GSM8K. The paper's central claim is that CTC's probability allocation over full alignments, not per-position prediction, is what makes the drafts more sequentially coherent and therefore more acceptable.

What carries the argument

The central mechanism is the pairing of a sequence-level CTC loss with a CTC Transform Module. CTC expands the output space with a blank symbol and treats consecutive duplicates as collapsible; the loss sums the probability of every alignment that reduces to the target label, using dynamic programming rather than per-token cross-entropy. The draft head itself is a single transformer layer that reads the base model's hidden states and outputs per-position distributions over the augmented vocabulary. At inference, raw candidates are built by taking the top-k tokens per position, and the CTC Transform Module deletes blanks and duplicated tokens and masks those positions in the base model's attention map, so the tree verification only scores the collapsed sequence. The paper attributes the acceptance-rate gain to the way this objective allocates probability across full token sequences, thereby strengthening correlations between draft tokens during drafting.

What would settle it

Train the same transformer draft module and inference pipeline with ordinary per-position cross-entropy instead of the CTC loss on the same distilled labels; if the acceptance rate and speedup match CTC-drafter's reported numbers, the CTC objective is not the source of the gain.

Watch

Extended reading notes

Core claim

CTC-drafter is a draft module that attaches to a frozen base LLM and is trained with the CTC objective on labels produced by the base model itself. At inference, it selects top-k tokens at each of several future positions, combines them into a token tree, removes blank symbols and consecutive duplicate tokens through a CTC Transform Module, and masks the removed positions in the attention map so that parallel verification sees only the surviving sequence. The paper's claim is that the sequence-level CTC training objective, which sums probability over all alignments that collapse to the target, makes the draft model favor candidate sequences with coherent internal token dependencies, and that this is why the acceptance rate is higher than that of Medusa or Hydra. On MT-bench and GSM8K with Vicuna-7B, Vicuna-13B, Vicuna-33B, and LLaMA-2-Chat base models, it reports between 3.4 and 3.6 accepted tokens per decoding step and speedups of 2.2x to 2.8x over vanilla autoregressive decoding.

Load-bearing premise

Everything rests on the premise that training with the CTC sequence-level objective actually makes the independently chosen per-position top-k picks land on jointly more acceptable token sequences.

Editorial extensions

If this is right

  • If CTC-drafter works as described, a frozen base LLM can be accelerated by adding a small CTC-trained module, with no fine-tuning of the base model itself.
  • The draft module transfers to other base models from the same family with only a modest drop, since it learns from hidden states rather than from a separate autoregressive draft model.
  • The reported numbers imply 3.4 to 3.6 tokens accepted per decoding step, more than Medusa's roughly 2.6 and Hydra's roughly 3.0 on the same Vicuna benchmarks.
  • The CTC transform gives variable-length candidate sequences after removing blanks and repeats, so the framework can be combined with other candidate-verification and sampling criteria.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The factorization in Equation 7 means the claimed correlations are learned only through the training objective; a direct way to test this is to score candidate sequences by the CTC-trained joint probability rather than by per-position top-k ranks.
  • Because the CTC transform makes candidate length adaptive, the same draft module could be composed with tree-based drafters or with verification criteria such as nucleus sampling, which the paper lists as future work.
  • If the mechanism is real, the gains should track text predictability: larger on code and math, where the paper reports the best category results, and smaller on open-ended roleplay; that asymmetry is testable across more benchmarks.
  • A natural extension is to replace the independence assumption in Equation 7 with a sequence model such as a CRF or autoregressive head, to see how much of the remaining gap is due to the factorization.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes CTC-drafter, a draft model for speculative decoding that inserts a transformer layer on top of the base LLM's hidden states and trains it with a connectionist temporal classification (CTC) loss on distilled labels from the base model. At inference, the draft module independently selects top-k tokens at each position, combines them into candidate sequences, applies a CTC transform that removes blanks and consecutive duplicates, modifies the attention map for verification, and verifies all candidates in parallel with the base model. Experiments on MT-bench and GSM8K with Vicuna-7B/13B/33B and LLaMA-2-Chat models report higher average accepted tokens per step and higher speedup than Medusa and Hydra, with a reported 3.4-3.6 accepted tokens per step and 2.2x-2.8x speedup on the tested models. The paper argues that CTC training strengthens correlations between draft tokens and thereby improves acceptance rate.

Significance. If the empirical results hold, the paper would make a useful contribution by introducing a sequence-level training objective to non-autoregressive draft models in speculative decoding, an area where most existing draft heads are trained with per-position cross-entropy. The held-out evaluation on MT-bench and GSM8K, the comparison across multiple base-model sizes and families, and the use of knowledge-distilled labels are all appropriate design choices and are strengths of the paper. The main significance, however, rests on the claim that the improvement comes from CTC-induced correlations among draft tokens, and that claim is not currently supported by the paper's own inference procedure or ablation design. The reported speedup over Hydra is also weakened by the fact that Hydra's numbers are taken from the original paper rather than measured in the same setup. The central idea is plausible and the results are potentially interesting, but the causal attribution and the fairness of the comparison need substantial additional evidence before the claims can be accepted.

major comments (4)
  1. [Section 3.2, Eq. (7); Section 3.3] The central attribution is unsupported by the paper's own inference procedure. Equation (7) defines P(A|X,theta) as a product of per-position probabilities p(a_t|X,theta), and Section 3.3 selects top-k tokens independently at each position before combining them into candidate sequences. The draft model therefore never conditions later draft tokens on earlier sampled draft tokens; the only post-hoc operation is the CTC transform that deletes blanks and repeats after all positions have been chosen. CTC training can improve per-position marginal calibration, but the paper provides no analysis or experiment showing how the sequence-level CTC objective transfers to greedy top-k decoding so as to create joint coherence among draft tokens. The claim that CTC 'strengthens the correlations between draft tokens during the draft phase' (Abstract, Section 3.3) is thus not realized by the described mechanism. To support the central claim, the authors should either derive why CTC training improves top-k candidate quality under the independence factorization, or add an experiment that isolates the effect of the CTC objective on candidate-sequence quality, or revise the claimed mechanism to a per-position calibration effect.
  2. [Table 2, Section 4.3] The ablation in Table 2 is confounded and cannot support the paper's attribution of the gains to CTC loss. The row 'Linear layer + Cross Entropy Loss' differs from 'Transformer layer + CTC Loss' in both the network architecture and the training objective, and the 'CTC Verify' column adds the CTC transform and attention-map modification as an additional inference-time change. Consequently, the improvement in beta from 2.58 to 3.56 cannot be attributed specifically to the CTC loss, the transformer layer, or the CTC transform. The paper needs a factorial ablation, for example a transformer layer trained with cross-entropy loss and a linear layer trained with CTC loss, both evaluated with and without CTC verify, to separate the contributions of architecture, objective, and verification transform. Without this, the central claim that CTC training is responsible for the acceptance-rate improvement is not established.
  3. [Section 4.2, Table 1] The comparison against Hydra is not apples-to-apples. Section 4.2 states that the Hydra results on Vicuna models are 'acquired from its corresponding paper' rather than measured on the same hardware, software, decoding hyperparameters, and implementation. Since the paper's main claim is that CTC-drafter outperforms Hydra by achieving a higher acceptance rate and speedup, taking Hydra's numbers from the original publication is insufficient; differences in GPU type, batch size, tree size, top-k, implementation efficiency, and evaluation protocol can materially change speedup and accepted-token ratios. The authors should re-run Hydra under the same experimental conditions as their own method, and report the exact configuration used for both methods. In addition, the paper reports no error bars, confidence intervals, or multiple-seed runs for any of the speedup or acceptance-rate numbers, so it is unclear whether the reported differences are statistically reliable.
  4. [Section 3.1, Section 3.3; Figure 1] The attention-map modification after the CTC transform is described only at a high level ('Positions in the attention map that corresponds to tokens been removed in CTC transform will be masked') with no derivation or analysis showing that this masking preserves the base model's conditional probability estimates for the remaining tokens. Speculative decoding's correctness relies on the verification distribution matching the base model's true conditional distribution; if masking changes the conditioning context in an ad hoc way, the acceptance criterion may no longer correspond to the base model's greedy decoding. The paper should either provide a precise mathematical definition of the modified attention map and a proof or argument that it yields the correct conditional distribution, or explicitly present the masking as a heuristic and evaluate its effect on verification correctness. The current description leaves a load-bearing step underspecified.
minor comments (5)
  1. [Section 4.1 and Table 1] The values of top_k, the number of reserved candidate combinations, and the tree sizes for Medusa and CTC-drafter are not reported. Since these parameters directly control the trade-off between draft quality and verification cost, omitting them makes the speedup comparison difficult to reproduce and to interpret.
  2. [Appendix A, Figure 4] The LLaMA-2-Chat results are presented only as bar charts without numeric values, standard deviations, or the number of evaluation runs. To support the claim of generality across base-model families, the exact accepted-token counts and speedup ratios should be given in a table.
  3. [Section 4.3 and Figure 3] The time-consumption breakdown in Figure 3 is presented as percentages without absolute timings or error bars. Because the speedup comparison depends on these timing measurements, reporting only relative percentages makes it hard to assess the variability of the draft overhead across inputs and runs.
  4. [Section 6] The conclusion correctly acknowledges limitations, including uncertainty about whether the draft model structure is optimal and the need to test on more model types. These limitations should be addressed or explicitly scoped in the main body rather than only in the conclusion, since they bear on the generality claims made earlier.
  5. [Equation (13)] The definition of speedup as (T_vanilla/N_vanilla)/(T_spec/N_spec) is clear in intent but the notation is slightly ambiguous: T is total inference time and N is total token count, so the ratio is average time per token. Clarifying this in the text would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: acceptance rate and speedup are measured on held-out benchmarks, and no prediction reduces to a fitted input or a self-citation chain.

full rationale

The paper's derivation chain is: train a draft module on ShareGPT with a CTC loss over labels distilled from the fixed base model (Eqs. 2-11), then at inference select top-k tokens per position, apply the CTC transform, and measure accepted tokens and speedup on MT-bench and GSM8K (Eqs. 12-13). The reported beta and gamma are empirical measurements on unseen benchmarks, not fitted constants. No parameter is fit to the evaluation outcome. The distilled labels come from the base model being accelerated, which is the intended training signal for a drafter rather than an imported prediction. No load-bearing self-citation or uniqueness theorem is used; Hydra numbers are cited from its paper, and Medusa is retrained by the authors. The claim that CTC 'strengthens correlations between draft tokens' is a mechanistic explanation that may be under-supported (Eq. 7 factorizes per-position, so the inference-time top-k selection does not condition later tokens on earlier sampled tokens), but that is a correctness or attribution concern, not circularity: the speedup result would remain an empirical finding even if the stated mechanism were incomplete. The ablation in Table 2 changes architecture and loss jointly, so the specific contribution of CTC is not isolated, but this again is experimental confounding rather than a prediction reducing by construction. Accordingly, no circular step meets the evidentiary bar in the hard rules.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central results rest on standard CTC machinery, a conditional-independence factorization, distilled labels from the base model, and an unverified attention-mask modification. No new physical entities are introduced; the free parameters are the unreported top-k and candidate-count choices that control the speedup trade-off.

free parameters (3)
  • top_k (draft token candidates per position) = not reported
    Section 3.3 says top k tokens are selected per position with k predefined, but the value is never given; it controls the size of the token tree and directly trades draft quality against verification cost.
  • kept_combinations (number of raw candidate sequences reserved) = not reported
    Section 3.3 says 'a group of the most valuable combinations are reserved' without specifying how many or how they are selected; acceptance rate and speedup depend on this choice.
  • max_length_training_data = 2048
    Training hyperparameter reported in Section 4.1; it is part of the model configuration but not central to the inference-speedup claim.
assumptions (4)
  • standard math CTC loss computed by dynamic programming over all alignments (Eq. 1, Eq. 6) is a valid training objective for a non-autoregressive draft model.
    The paper relies on Graves et al.'s CTC algorithm without reproving it; this is background from prior literature.
  • domain assumption Per-position draft token probabilities are conditionally independent given the base model hidden states (Eq. 7).
    This factorization is stated in Section 3.2 and used to compute the CTC objective; it is the mechanism by which the draft module generates tokens in parallel, and it limits the correlations the draft can express.
  • domain assumption Greedy-decoded outputs of the base model (Y_distill, Eq. 5) are good training labels for the draft model.
    Section 3.2 follows the knowledge distillation approach of DistillSpec; if distilled labels diverge from the distribution the base model uses during verification, acceptance rates could be lower.
  • ad hoc to paper Masking removed positions in the attention map after CTC transform preserves the base model's conditional probability estimates for the remaining tokens.
    Section 3.3 states the attention map is modified after removing duplicates and blanks, but no derivation or implementation detail is given; if the modified mask does not match the correct tree attention, verification acceptance is not well-founded.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration." pith.science (2026). https://pith.science/paper/UJON6HLI

@misc{pith2026241200061,
  author       = {Pith},
  title        = {Pith review of: Speculative Decoding with CTC-based Draft Model for LLM Inference Acceleration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UJON6HLI}},
  note         = {Machine review of arXiv:2412.00061}
}
read the original abstract

Inference acceleration of large language models (LLMs) has been put forward in many application scenarios and speculative decoding has shown its advantage in addressing inference acceleration. Speculative decoding usually introduces a draft model to assist the base LLM where the draft model produces drafts and the base LLM verifies the draft for acceptance or rejection. In this framework, the final inference speed is decided by the decoding speed of the draft model and the acceptance rate of the draft provided by the draft model. Currently the widely used draft models usually generate draft tokens for the next several positions in a non-autoregressive way without considering the correlations between draft tokens. Therefore, it has a high decoding speed but an unsatisfactory acceptance rate. In this paper, we focus on how to improve the performance of the draft model and aim to accelerate inference via a high acceptance rate. To this end, we propose a CTC-based draft model which strengthens the correlations between draft tokens during the draft phase, thereby generating higher-quality draft candidate sequences. Experiment results show that compared to strong baselines, the proposed method can achieve a higher acceptance rate and hence a faster inference speed.

Figures

Figures reproduced from arXiv: 2412.00061 by the authors.

Figure 1
Figure 1. Illustration of CTC-drafter model training and inference strategy. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Average number of tokens accepted per decoding step in different question categories on [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The percentage of time consumed for different processes based on CTC-drafter(left) and [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: The bar charts of speedup ratio relative to vanilla method [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs

    cs.CL 2025-06 reject novelty 5.0 of 10

    ASTS extends locally typical sampling with semantic scoring and dynamic thresholds, reporting improved perplexity, MAUVE, and diversity on story and summarization tasks.

Reference graph

Works this paper leans on

28 extracted references · 6 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Hydra: Sequentially-dependent draft heads for medusa decoding

    Zachary Ankner, Rishab Parthasarathy, Aniruddha Nrusimha, Christopher Rinard, Jonathan Ragan-Kelley, and William Brandon. Hydra: Sequentially-dependent draft heads for medusa decoding. arXiv preprint arXiv:2402.05109, 2024

    arXiv 2024

  3. [3]

    Medusa: Simple framework for accelerating llm generation with multiple decoding heads, 2023

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, and Tri Dao. Medusa: Simple framework for accelerating llm generation with multiple decoding heads, 2023

    work page 2023

  4. [4]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    arXiv 2023

  5. [5]

    Cascade speculative drafting for even faster llm inference

    Ziyi Chen, Xiaocong Yang, Jiacheng Lin, Chenkai Sun, Jie Huang, and Kevin Chen- Chuan Chang. Cascade speculative drafting for even faster llm inference. arXiv preprint arXiv:2312.11462, 2023

    arXiv 2023

  6. [6]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023

    2023

  7. [7]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  8. [8]

    Tutorial on directed acyclic graphs

    Jean C Digitale, Jeffrey N Martin, and Medellena Maria Glymour. Tutorial on directed acyclic graphs. Journal of Clinical Epidemiology, 142:264–267, 2022

    work page 2022

Show all 28 references
  1. [9]

    Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks

    Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning , pages 369–376, 2006

    2006

  2. [10]

    Speed: Speculative pipelined execution for efficient decoding

    Coleman Hooper, Sehoon Kim, Hiva Mohammadzadeh, Hasan Genc, Kurt Keutzer, Amir Gholami, and Sophia Shao. Speed: Speculative pipelined execution for efficient decoding. arXiv preprint arXiv:2310.12072, 2023

    2023 arXiv

  3. [11]

    Mistral 7b

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023

    arXiv

  4. [12]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning , pages 19274–19286. PMLR, 2023

    2023

  5. [13]

    Eagle: Speculative sampling requires rethinking feature uncertainty

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077, 2024

    2024 arXiv

  6. [14]

    Specinfer: Accelerating generative llm serving with speculative inference and token tree verification

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative llm serving with speculative inference and token tree verification. arXiv preprint arXiv:2...

    2023 arXiv

  7. [15]

    Accelerating llm inference with staged speculative decoding

    Benjamin Spector and Chris Re. Accelerating llm inference with staged speculative decoding. arXiv preprint arXiv:2308.04623, 2023

    2023 arXiv

  8. [16]

    Insertion transformer: Flexible sequence generation via insertion operations

    Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion transformer: Flexible sequence generation via insertion operations. In International Conference on Machine Learning, pages 5976–5985. PMLR, 2019

    2019

  9. [17]

    Blockwise parallel decoding for deep autoregressive models

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems , 31, 2018

    2018

  10. [18]

    Instantaneous grammatical error correction with shallow aggressive decoding

    Xin Sun, Tao Ge, Furu Wei, and Houfeng Wang. Instantaneous grammatical error correction with shallow aggressive decoding. arXiv preprint arXiv:2106.04970, 2021. 10

    2021 arXiv

  11. [19]

    Spectr: Fast speculative decoding via optimal transport

    Ziteng Sun, Ananda Theertha Suresh, Jae Hun Ro, Ahmad Beirami, Himanshu Jain, and Felix Yu. Spectr: Fast speculative decoding via optimal transport. Advances in Neural Information Processing Systems, 36, 2024

    2024

  12. [20]

    An introduction to conditional random fields

    Charles Sutton, Andrew McCallum, et al. An introduction to conditional random fields. F oundations and Trends® in Machine Learning, 4(4):267–373, 2012

    2012

  13. [21]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  14. [22]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  15. [23]

    Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 3909–3925, 2023

    2023

  16. [24]

    Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding

    Heming Xia, Zhe Yang, Qingxiu Dong, Peiyi Wang, Yongqi Li, Tao Ge, Tianyu Liu, Wenjie Li, and Zhifang Sui. Unlocking efficiency in large language model inference: A comprehensive survey of speculative decoding. arXiv preprint arXiv:2401.07851, 2024

    2024 arXiv

  17. [25]

    Predictive pipelined decoding: A compute-latency trade-off for exact llm decoding

    Seongjun Yang, Gibbeum Lee, Jaewoong Cho, Dimitris Papailiopoulos, and Kangwook Lee. Predictive pipelined decoding: A compute-latency trade-off for exact llm decoding. arXiv preprint arXiv:2307.05908, 2023

    2023 arXiv

  18. [26]

    Draft & verify: Lossless large language model acceleration via self-speculative decoding

    Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding. arXiv preprint arXiv:2309.08168, 2023

    2023 arXiv

  19. [27]

    Judging llm-as-a-judge with mt-bench and chatbot arena

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024

    2024

  20. [28]

    Distillspec: Improving speculative decoding via knowledge distillation

    Yongchao Zhou, Kaifeng Lyu, Ankit Singh Rawat, Aditya Krishna Menon, Afshin Ros- tamizadeh, Sanjiv Kumar, Jean-François Kagy, and Rishabh Agarwal. Distillspec: Improving speculative decoding via knowledge distillation. arXiv preprint arXiv:2310.08461, 2023. 11 Figure 4: The ba...

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.