Pith. sign in

REVIEW 3 major objections 5 minor 49 references

Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A two-layer drafter that foresees multiple tokens yields a lossless 2.9x-3.5x speedup for LLMs.

desk verdict Useful SAR speculative decoding result with a real training contribution, but the headline speedups may be in-sample because the tree and k are tuned on the same benchmarks. read the letter →

arxiv 2412.12639 v3 pith:6TG7BLW4 submitted 2024-12-17 cs.CL cs.AI

Xiangxiang Gao , Weisheng Xie , Yiwei Xiang , Feng Ji This is my paper · ORCID
classification cs.CLcs.AI
keywords speculativedecodingsemi-autoregressivelargelanguagemodelsinferenceaccelerationglancingdistillationtreelosslessspeedup
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a small draft model, only two Transformer layers deep, can make an LLM generate text 2.9 to 3.5 times faster without altering the output distribution. The trick is to have the drafter predict several future tokens at once, then have the LLM verify them in a single pass. Two components make this work: a training objective called Coupled Sequential Glancing Distillation that preserves dependencies between tokens inside one block, and a custom decoding tree that lets the drafter make several passes and propose many candidate continuations. If the claim holds, LLM serving can be accelerated with minimal extra memory, which matters for real-time and resource-constrained deployments.

What carries the argument

The machinery has two parts. Coupled Sequential Glancing Distillation (CSGD) is a training method: it replaces a number of consecutive ground-truth tokens and features proportional to the current Hamming distance between the drafter's prediction and the LLM's output, so the drafter learns the mutual information that ties tokens within a block together. The Custom-Designed Decoding Tree is a decoding-time structure that lets the drafter run several forward passes and, with a relaxed causal mask (tokens inside the same k-token block may attend to each other), propose many candidate branches that the LLM verifies in parallel.

What would settle it

Run Falcon on a model family not used in tuning (e.g., a 7B model from a different lineage) with the Vicuna tree and k=2 under the same greedy protocol; if the speedup falls below the best baseline's speedup, the advantage is specific to the tuned tree shapes.

Watch

Extended reading notes

Core claim

Falcon's central claim is that semi-autoregressive speculative decoding can be made both fast and accurate if the drafter learns inter-token dependencies within a block. CSGD training replaces a Hamming-distance-determined number of consecutive incorrectly predicted tokens and features with the LLM's ground truth, with the replacement count shrinking as training progresses. The information-theoretic argument decomposes the two-token SAR loss as $H(X|C)+I(X;C)+H(Y|X)+I(X;Y)$ and contrasts it with the standard SAR loss $H(Y|X)$, showing that CSGD aligns the SAR objective more closely with the autoregressive one and thus improves acceptance. The custom decoding tree, using a relaxed causal mask, lets the drafter generate $n$ candidate tokens per node across $m$ forward passes, producing $k\cdot m\cdot n$ drafts per phase. On greedy decoding over MT-Bench, HumanEval, and GSM8K, Falcon reports 2.91x-3.51x wall-time speedups over vanilla autoregression, with acceptance rates 3-5% above EAGLE and 15-17% above Medusa.

Load-bearing premise

The decoding tree shapes and hyperparameters (k=2, m child counts, four forward passes) were tuned on the same benchmarks where speedups are measured, so the reported 2.91x-3.51x numbers may not transfer to other models or datasets without re-tuning.

Editorial extensions

If this is right

  • Falcon's drafter needs only two Transformer layers, so the memory and compute overhead of speculation is small enough for latency-critical serving.
  • Decoding is lossless, meaning the generated text is distributionally identical to vanilla autoregressive generation, allowing drop-in deployment.
  • The CSGD training recipe could be reused to raise the acceptance rate of any semi-autoregressive decoder that currently suffers from low draft accuracy.
  • The tree-construction rule of thumb (deeper and wider branches for higher-probability tokens) is a transferable heuristic for designing decoding-time search trees.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The tree shapes in Figures 5 and 6 were tuned on the same benchmarks where speedups are reported, so the claimed advantage may narrow on unseen models or datasets; a clean transfer test would freeze the Vicuna tree and evaluate on a model family not used during tuning.
  • The information-theoretic analysis suggests that any training signal that increases mutual information between tokens in a block should improve SAR acceptance, not only ground-truth replacement; this could be checked by comparing CSGD to other intra-block dependency regularizers.
  • The replacement-count schedule $\lambda = 0.4(e_{pt}-e_{pc})/e_{pt}$ is a hand-set curve; making it adaptive to validation acceptance rate might improve robustness, though the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Falcon, a semi-autoregressive (SAR) speculative decoding framework for LLM inference. Falcon combines a Coupled Sequential Glancing Distillation (CSGD) training objective for a lightweight SAR drafter with a custom-designed decoding tree that supports multiple candidate branches and multiple forward passes per drafting phase. The authors report lossless speedups of 2.91x-3.51x over vanilla autoregressive decoding on Vicuna-7B/13B and LLaMA2-Chat-7B/13B across MT-Bench, HumanEval, and GSM8K, claiming improvements over Eagle, Medusa, Lookahead, SPS, and PLD. The paper also offers an information-theoretic explanation of CSGD in Section 3.3. Experiments include wall-clock speedup, acceptance rate, average acceptance length, and ablations for tree attention, CSGD, and the block size k.

Significance. If the reported speedups transfer to held-out settings and to comparisons with the current strongest autoregressive baselines, Falcon would be a practically valuable contribution: it achieves these results with a drafter of only two transformer-layer equivalents, reports wall-clock measurements, and provides code and detailed ablation settings. The acceptance-rate gains over Eagle (3%-5%) and over Medusa (15-17 percentage points) are nontrivial. However, the current evidence is weakened by in-sample hyperparameter tuning of the decoding tree and k, and by the absence of EAGLE-2/EAGLE-3 baselines. The theoretical section is currently not load-bearing for the empirical claim but is presented as a contribution and needs either rigorous derivation or explicit reframing.

major comments (3)
  1. [Section 3.3, Eqs. (11)-(12)] The theoretical analysis does not derive the claimed entropy decomposition. Equations (11) and (12) are obtained by replacing H(X|Y) with H(X|C) and one occurrence of I(X;Y) with I(X;C) in Eq. (10), but no argument is given for why CSGD's training loss should equal these expressions, and the substitution changes the conditioning structure without justification. Additionally, Eq. (10) contains 2I(X;Y), while the text refers to 'a mutual information term I(X;Y)'. Since the paper advertises a 'comprehensive theoretical analysis', the derivation should be made rigorous or the section should be explicitly reframed as intuitive motivation.
  2. [Section 4.4, Table 1] The comparison omits EAGLE-2 and EAGLE-3, which are stronger autoregressive drafting baselines than the EAGLE-1 version cited in the paper. The abstract and Section 1 claim Falcon 'outstrips existing speculative decoding methods for LLMs', but the table only supports superiority over the listed methods. Please add these baselines or explicitly state why they are not applicable before making the SOTA claim.
  3. [Section 4.5, Table 5 and Appendix A] The speedup numbers are likely in-sample: Table 5 selects k=2 on Vicuna-7B by highest MT-Bench speedup among k={2,3,4}, and Appendix A states that the tree structures are 'optimized based on intuition and findings during experiments' and are model-dependent (Figures 5 and 6 differ for Vicuna and LLaMA). No held-out data is used. Since acceptance rate and wall-clock speedup depend strongly on tree shape and the number of forward passes, the reported 2.91x-3.51x range should be validated on benchmarks or datasets not used for tuning, or the authors should report results with a fixed tree and fixed k across all settings.
minor comments (5)
  1. [Section 4.5] The reference 'Table ??' should read 'Table 5'.
  2. [Table 1 and Section 4.4] SpS, PLD, and Lookahead results are copied from their technical reports rather than measured on the same H800 server; the text should make this limitation explicit in the table caption or in Section 4.4.
  3. [Section 3.3, Eq. (10)] The equation shows 2I(X;Y), but the surrounding text describes the mutual information term as I(X;Y); align the notation.
  4. [Appendix A, Figures 5-6] The tree diagrams are difficult to read and the exact branching factors (m) for each node are not given textually; please provide the full tree specification in text or a table.
  5. [Section 3.4 and Section 4.4] Section 3.4 says the drafter performs m forward passes, while Section 4.4 repeatedly states 'four forward passes'; clarify that m=4 for the main configuration.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: the reported speedups are empirical measurements, and the tree/k tuning on the same benchmarks is a validity risk rather than a circular reduction.

full rationale

The paper's central claims are empirical measurements rather than derivations that reduce to their inputs. The 2.91x-3.51x speedups are wall-clock measurements against vanilla autoregressive decoding on MT-Bench, HumanEval, and GSM8K (Table 1), with acceptance rate and average acceptance length reported separately in Tables 2 and 3. The CSGD training objective (Equations 1-8) is a concrete loss combining regression, KL-divergence, and cross-entropy terms; its effect on acceptance rate is measured in the ablation study (Table 4), not assumed. The information-theoretic discussion in Section 3.3 is a post-hoc entropy decomposition illustrating why conditioning on features and tokens from one time step ahead could help; it is not used to fit any parameter, so it cannot be circular. The decoding-tree shape and k=2 hyperparameter are admittedly tuned on the same benchmarks: Appendix A states 'the choices of tree structure are optimized based on intuition and findings during experiments,' and Table 5 compares k factors using MT-Bench speedup. This is an in-sample selection or generalizability risk, but it is not a circular reduction because the reported speedup is a measured outcome of the chosen configuration, not a fitted value that is identical to the input by construction. There are no load-bearing self-citations: the EAGLE feature-concatenation idea is attributed to external prior work, and no uniqueness theorem is invoked to force the design. Therefore no circularity is identified.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

Falcon's central speedup rests on several hand-chosen hyperparameters (k, lambda, alpha, omega_dist, noise, tree shape) and on domain assumptions derived from EAGLE and standard speculative decoding. The theoretical explanation in Section 3.3 introduces an ad hoc substitution whose validity is unproven, so the explanation is best treated as a conjecture rather than a derivation.

free parameters (7)
  • k (block size) = 2
    Number of tokens drafted per forward pass; ablation shows k=2 gives the best speedup, so it is chosen by hand from experiments on MT-Bench.
  • lambda schedule coefficient = 0.4
    Glancing replacement fraction N = lambda * hamming distance, with lambda = 0.4*(ept-epc)/ept; this constant is set without sensitivity analysis.
  • alpha = 0.9
    Weight for soft distillation loss in Ldist = alpha*Lsoft + (1-alpha)*Lhard; set to 0.9 without ablation.
  • omega_dist = 0.1
    Weight on distillation loss in L = Lreg + omega_dist*Ldist; set to 0.1 without ablation.
  • noise range = U(-0.1, 0.1)
    Data augmentation noise added to features to avoid error accumulation; chosen by hand, not ablated.
  • tree structure (m, depth, shape) = model-dependent, see Figures 5 and 6
    The decoding tree shape and child counts are optimized based on intuition and findings during experiments and are different for Vicuna and LLaMA, so they are fitted to the test models.
  • forward passes per draft phase = 4
    The paper states Falcon performs four forward passes each drafting phase; this design choice is not ablated independently of the tree.
assumptions (5)
  • domain assumption Standard rejection sampling in speculative decoding is lossless, so greedy decoding output equals vanilla AR output.
    Invoked implicitly in Section 4 to call Falcon lossless; it is standard from Leviathan et al., but not restated.
  • standard math The information-theoretic identity H(X)+H(Y)=H(Y|X)+2I(X;Y)+H(X|Y) is the correct decomposition for 2-token SAR loss.
    Used in Section 3.3 equation (10); the identity is correct under standard entropy definitions.
  • ad hoc to paper The CSGD replacement strategy can be modeled by replacing H(X|Y) with H(X|C) and I(X;Y) with I(X;C) in the loss decomposition.
    Section 3.3 equations (11) and (12) assume this without derivation; this is the paper's core explanatory claim and is not established.
  • domain assumption Features from one time step ahead carry richer semantic context that improves drafter predictions.
    Adopted from EAGLE (Li et al., 2024), Section 3.1; Falcon relies on this for its input construction.
  • domain assumption The Relaxed Causal Mask, which allows within-block attention, preserves the causality needed for correct verification.
    Section 3.4; if within-block attention leaked future target information into the draft, acceptance could be inflated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree." pith.science (2026). https://pith.science/paper/6TG7BLW4

@misc{pith2026241212639,
  author       = {Pith},
  title        = {Pith review of: Falcon: Faster and Parallel Inference of Large Language Models through Enhanced Semi-Autoregressive Drafting and Custom-Designed Decoding Tree},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6TG7BLW4}},
  note         = {Machine review of arXiv:2412.12639}
}
read the original abstract

Striking an optimal balance between minimal drafting latency and high speculation accuracy to enhance the inference speed of Large Language Models remains a significant challenge in speculative decoding. In this paper, we introduce Falcon, an innovative semi-autoregressive speculative decoding framework fashioned to augment both the drafter's parallelism and output quality. Falcon incorporates the Coupled Sequential Glancing Distillation technique, which fortifies inter-token dependencies within the same block, leading to increased speculation accuracy. We offer a comprehensive theoretical analysis to illuminate the underlying mechanisms. Additionally, we introduce a Custom-Designed Decoding Tree, which permits the drafter to generate multiple tokens in a single forward pass and accommodates multiple forward passes as needed, thereby boosting the number of drafted tokens and significantly improving the overall acceptance rate. Comprehensive evaluations on benchmark datasets such as MT-Bench, HumanEval, and GSM8K demonstrate Falcon's superior acceleration capabilities. The framework achieves a lossless speedup ratio ranging from 2.91x to 3.51x when tested on the Vicuna and LLaMA2-Chat model series. These results outstrip existing speculative decoding methods for LLMs, including Eagle, Medusa, Lookahead, SPS, and PLD, while maintaining a compact drafter architecture equivalent to merely two Transformer layers.

Figures

Figures reproduced from arXiv: 2412.12639 by the authors.

Figure 1
Figure 1. Framework of Falcon. It illustrates the computational process and displays the corresponding generation results of [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The training procedure of CSGD. Yˆ is the initial predicted feature representation sequence of the draft model, Y t is the ground-truth feature calculated by LLMs, ti is the original token, t t i is the target token generated by LLMs, hi is the original feature sequence, and h t i is the target feature generated by LLMs. tion from LLMs Y t . Then, we replace a certain number of continuously predicted token sequence … view at source ↗
Figure 3
Figure 3. SAR decoding tree attention illustrated. This visu [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Speedup ratio of Vicuna and LLaMA2-Chat on MT-bench for greedy (temperature=0). [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The tree structure used for Vicuna series. [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: The tree structure used for LLaMA series. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

49 extracted references · 22 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Bao, Y.; Zhou, H.; Huang, S.; Wang, D.; Qian, L.; Dai, X.; Chen, J.; and Li, L. 2022. latent-GLAT : Glancing at Latent Variables for Parallel Text Generation. In Muresan, S.; Nakov, P.; and Villavicencio, A., eds., Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 8398--8409. Dublin, Ireland: ...

    work page 2022

  4. [4]

    D.; Chen, D.; and Dao, T

    Cai, T.; Li, Y.; Geng, Z.; Peng, H.; Lee, J. D.; Chen, D.; and Dao, T. 2024. Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads. arXiv:2401.10774

    arXiv 2024

  5. [5]

    Chen, C.; Borgeaud, S.; Irving, G.; Lespiau, J.-B.; Sifre, L.; and Jumper, J. 2023. Accelerating Large Language Model Decoding with Speculative Sampling. arXiv:2302.01318

    arXiv 2023

  6. [6]

    Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; Pinto, H. P. D. O.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    arXiv 2021

  7. [7]

    C.-C.; and Huang, J

    Chen, Z.; Yang, X.; Lin, J.; Sun, C.; Chang, K. C.-C.; and Huang, J. 2024. Cascade Speculative Drafting for Even Faster LLM Inference. arXiv:2312.11462

    arXiv 2024

  8. [8]

    Cobbe, K.; Kosaraju, V.; Bavarian, M.; Chen, M.; Jun, H.; Kaiser, L.; Plappert, M.; Tworek, J.; Hilton, J.; Nakano, R.; et al. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    arXiv 2021

Show all 49 references
  1. [9]

    F.; Tao, D.; and Tu, Z

    Ding, L.; Wang, L.; Liu, X.; Wong, D. F.; Tao, D.; and Tu, Z. 2021. Understanding and Improving Lexical Choice in Non-Autoregressive Translation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net

    2021

  2. [10]

    Du, Z.; Qian, Y.; Liu, X.; Ding, M.; Qiu, J.; Yang, Z.; and Tang, J. 2022. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. arXiv:2103.10360

    2022 arXiv

  3. [11]

    Geng, X.; Feng, X.; and Qin, B. 2021. Learning to Rewrite for Non-Autoregressive Neural Machine Translation. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 3297--3308. Onl...

    2021

  4. [12]

    Ghazvininejad, M.; Levy, O.; Liu, Y.; and Zettlemoyer, L. 2019. Mask-Predict: Parallel Decoding of Conditional Masked Language Models. In Inui, K.; Jiang, J.; Ng, V.; and Wan, X., eds., Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and ...

    2019

  5. [13]

    Y.; Rozière, B.; Lopez-Paz, D.; and Synnaeve, G

    Gloeckle, F.; Idrissi, B. Y.; Rozière, B.; Lopez-Paz, D.; and Synnaeve, G. 2024. Better & Faster Large Language Models via Multi-token Prediction. arXiv:2404.19737

    2024 arXiv

  6. [14]

    Graves, A.; Fern\' a ndez, S.; Gomez, F.; and Schmidhuber, J. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, 369–376. New York, ...

    2006

  7. [15]

    Guo, J.; Tan, X.; He, D.; Qin, T.; Xu, L.; and Liu, T.-Y. 2019 a . Non-autoregressive neural machine translation with enhanced decoder input. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial In...

    2019

  8. [16]

    Guo, J.; Tan, X.; Xu, L.; Qin, T.; Chen, E.; and Liu, T.-Y. 2019 b . Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation. arXiv preprint arXiv:1911.08717

    2019 arXiv

  9. [17]

    Guo, J.; Tan, X.; Xu, L.; Qin, T.; Chen, E.; and Liu, T.-Y. 2020. Fine-Tuning by Curriculum Learning for Non-Autoregressive Neural Machine Translation. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05): 7839--7846

    2020

  10. [18]

    Guo, J.; Wang, M.; Wei, D.; Shang, H.; Wang, Y.; Li, Z.; Yu, Z.; Wu, Z.; Chen, Y.; Su, C.; Zhang, M.; Lei, L.; shimin tao; and Yang, H. 2021. Self-Distillation Mixup Training for Non-autoregressive Neural Machine Translation. arXiv:2112.11640

    2021 arXiv

  11. [19]

    Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the Knowledge in a Neural Network. arXiv:1503.02531

    2015 arXiv

  12. [20]

    Hochreiter, S.; and Schmidhuber, J. 1997. Long short-term memory. Neural computation, 9(8): 1735--1780

    1997

  13. [21]

    W.; Gholami, A.; and Keutzer, K

    Kim, S.; Mangalam, K.; Moon, S.; Malik, J.; Mahoney, M. W.; Gholami, A.; and Keutzer, K. 2023. Speculative Decoding with Big Little Decoder. In Oh, A.; Naumann, T.; Globerson, A.; Saenko, K.; Hardt, M.; and Levine, S., eds., Advances in Neural Information Processing Systems, v...

    2023

  14. [22]

    Leviathan, Y.; Kalman, M.; and Matias, Y. 2023. Fast Inference from Transformers via Speculative Decoding. arXiv:2211.17192

    2023 arXiv

  15. [23]

    Li, Y.; Wei, F.; Zhang, C.; and Zhang, H. 2024. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty. arXiv:2401.15077

    2024 arXiv

  16. [24]

    Miao, X.; Oliaro, G.; Zhang, Z.; Cheng, X.; Wang, Z.; Zhang, Z.; Wong, R. Y. Y.; Zhu, A.; Yang, L.; Shi, X.; Shi, C.; Chen, Z.; Arfeen, D.; Abhyankar, R.; and Jia, Z. 2024. SpecInfer: Accelerating Large Language Model Serving with Tree-based Speculative Inference and Verificat...

    2024

  17. [25]

    Monea, G.; Joulin, A.; and Grave, E. 2023. PaSS: Parallel Speculative Sampling. arXiv:2311.13581

    2023 arXiv

  18. [26]

    Olsson, C.; Elhage, N.; Nanda, N.; Joseph, N.; DasSarma, N.; Henighan, T.; Mann, B.; Askell, A.; Bai, Y.; Chen, A.; Conerly, T.; Drain, D.; Ganguli, D.; Hatfield-Dodds, Z.; Hernandez, D.; Johnston, S.; Jones, A.; Kernion, J.; Lovitt, L.; Ndousse, K.; Amodei, D.; Brown, T.; Cla...

    2022 arXiv

  19. [27]

    Qian, L.; Zhou, H.; Bao, Y.; Wang, M.; Qiu, L.; Zhang, W.; Yu, Y.; and Li, L. 2021. Glancing Transformer for Non-Autoregressive Neural Machine Translation. In Zong, C.; Xia, F.; Li, W.; and Navigli, R., eds., Proceedings of the 59th Annual Meeting of the Association for Comput...

    2021

  20. [28]

    Ran, Q.; Lin, Y.; Li, P.; and Zhou, J. 2020. Learning to Recover from Multi-Modality Errors for Non-Autoregressive Neural Machine Translation. In Jurafsky, D.; Chai, J.; Schluter, N.; and Tetreault, J., eds., Proceedings of the 58th Annual Meeting of the Association for Comput...

    2020

  21. [29]

    Roman, S. 1992. Coding and information theory, volume 134. Springer Science & Business Media

    1992

  22. [30]

    Santilli, A.; Severino, S.; Postolache, E.; Maiorca, V.; Mancusi, M.; Marin, R.; and Rodola, E. 2023. Accelerating Transformer Inference for Translation via Parallel Decoding. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meeting of the ...

    2023

  23. [31]

    Saxena, A. 2023. Prompt Lookup Decoding

    2023

  24. [32]

    Spector, B.; and Re, C. 2023. Accelerating LLM Inference with Staged Speculative Decoding. arXiv:2308.04623

    2023 arXiv

  25. [33]

    Stern, M.; Shazeer, N.; and Uszkoreit, J. 2018. Blockwise Parallel Decoding for Deep Autoregressive Models. In Bengio, S.; Wallach, H.; Larochelle, H.; Grauman, K.; Cesa-Bianchi, N.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volume 31. Curran As...

    2018

  26. [34]

    Sun, Z.; Li, Z.; Wang, H.; He, D.; Lin, Z.; and Deng, Z. 2019. Fast Structured Decoding for Sequence Models. In Wallach, H.; Larochelle, H.; Beygelzimer, A.; d Alch\' e -Buc, F.; Fox, E.; and Garnett, R., eds., Advances in Neural Information Processing Systems, volume 32. Curr...

    2019

  27. [35]

    van den Oord, A.; Dieleman, S.; Zen, H.; Simonyan, K.; Vinyals, O.; Graves, A.; Kalchbrenner, N.; Senior, A.; and Kavukcuoglu, K. 2016. WaveNet: A Generative Model for Raw Audio. arXiv:1609.03499

    2016 arXiv

  28. [36]

    van den Oord, A.; Li, Y.; Babuschkin, I.; Simonyan, K.; Vinyals, O.; Kavukcuoglu, K.; van den Driessche, G.; Lockhart, E.; Cobo, L.; Stimberg, F.; Casagrande, N.; Grewe, D.; Noury, S.; Dieleman, S.; Elsen, E.; Kalchbrenner, N.; Zen, H.; Graves, A.; King, H.; Walters, T.; Belov...

    2018

  29. [37]

    Wan, Z.; Wang, X.; Liu, C.; Alam, S.; Zheng, Y.; Liu, J.; Qu, Z.; Yan, S.; Zhu, Y.; Zhang, Q.; Chowdhury, M.; and Zhang, M. 2024. Efficient Large Language Models: A Survey. arXiv:2312.03863

    2024 arXiv

  30. [38]

    Wang, C.; Zhang, J.; and Chen, H. 2018. Semi-Autoregressive Neural Machine Translation. In Riloff, E.; Chiang, D.; Hockenmaier, J.; and Tsujii, J., eds., Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 479--488. Brussels, Belgium: Associ...

    2018

  31. [39]

    Wang, Y.; Tian, F.; He, D.; Qin, T.; Zhai, C.; and Liu, T.-Y. 2019. Non-Autoregressive Machine Translation with Auxiliary Regularization. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01): 5377--5384

    2019

  32. [40]

    Wertheimer, D.; Rosenkranz, J.; Parnell, T.; Suneja, S.; Ranganathan, P.; Ganti, R.; and Srivatsa, M. 2024. Accelerating Production LLMs with Combined Token/Embedding Speculators. arXiv:2404.19124

    2024 arXiv

  33. [41]

    Xia, H.; Ge, T.; Chen, S.-Q.; Wei, F.; and Sui, Z. 2023. Speculative Decoding: Lossless Speedup of Autoregressive Translation

    2023

  34. [42]

    Xia, H.; Yang, Z.; Dong, Q.; Wang, P.; Li, Y.; Ge, T.; Liu, T.; Li, W.; and Sui, Z. 2024. Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding. arXiv:2401.07851

    2024 arXiv

  35. [43]

    Xiao, Y.; Wu, L.; Guo, J.; Li, J.; Zhang, M.; Qin, T.; and yan Liu, T. 2023. A Survey on Non-Autoregressive Generation for Neural Machine Translation and Beyond. arXiv:2204.09269

    2023 arXiv

  36. [44]

    Xu, W.; Ma, S.; Zhang, D.; and Carpuat, M. 2021. How Does Distilled Data Complexity Impact the Quality and Confidence of Non-Autoregressive Machine Translation? In Zong, C.; Xia, F.; Li, W.; and Navigli, R., eds., Findings of the Association for Computational Linguistics: ACL-...

    2021

  37. [45]

    Zhang, J.; Wang, J.; Li, H.; Shou, L.; Chen, K.; Chen, G.; and Mehrotra, S. 2024. Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding. arXiv:2309.08168

    2024 arXiv

  38. [46]

    Zhao, Y.; Xie, Z.; Liang, C.; Zhuang, C.; and Gu, J. 2024. Lookahead: An Inference Acceleration Framework for Large Language Model with Lossless Generation Accuracy. arXiv:2312.12728

    2024 arXiv

  39. [47]

    Zheng, L.; Chiang, W.-L.; Sheng, Y.; Zhuang, S.; Wu, Z.; Zhuang, Y.; Lin, Z.; Li, Z.; Li, D.; Xing, E.; et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36

    2024

  40. [48]

    Zhu, X.; Li, J.; Liu, Y.; Ma, C.; and Wang, W. 2024. A Survey on Model Compression for Large Language Models. arXiv:2308.07633

    2024 arXiv

  41. [49]

    Łukasz Kaiser; Roy, A.; Vaswani, A.; Parmar, N.; Bengio, S.; Uszkoreit, J.; and Shazeer, N. 2018. Fast Decoding in Sequence Models using Discrete Latent Variables. arXiv:1803.03382

    2018 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.