Pith. sign in

REVIEW 3 major objections 4 minor 35 references

SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SpecASR speeds LLM-based speech recognition 3.04x-3.79x by exploiting audio-conditioned alignment between small draft and large target models, with no accuracy loss.

desk verdict ASR-specific speculative decoding ideas are plausible, but the headline speedups are simulated on a Whisper proxy, not measured on LLM-based ASR. read the letter →

arxiv 2507.18181 v2 pith:2B3RZPAK submitted 2025-07-24 eess.AS cs.SD

Linye Wei , Shuzhang Zhong , Songqiang Xu , Runsheng Wang , Ru Huang , Meng Li This is my paper
classification eess.AScs.SD
keywords speculativedecodingLLM-basedautomaticspeechrecognitionaudio-conditioneddraftsequencerecyclingsparsetokentreelatencyLibri
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SpecASR sets out to fix the main bottleneck of LLM-based automatic speech recognition: the autoregressive decoder's latency. The paper's central observation is that ASR decoding is audio-conditioned, so a small draft model and a large target model produce highly aligned token sequences even when they mismatch at intermediate steps. On that basis SpecASR builds speculative decoding with three features: adaptively truncating draft sequences when confidence drops, recycling rejected draft tokens instead of discarding them, and a two-pass sparse token tree for cases where the target model dominates latency. If the claims hold, LLM-based ASR can run several times faster at identical recognition accuracy. The paper reports 3.04x-3.79x and 1.25x-1.84x speedups over autoregressive and speculative decoding baselines respectively.

What carries the argument

The central mechanism is audio-conditioned alignment: because the draft and target models condition on the same audio embedding, their decoded token sequences stay close even when a particular token is wrong, which lets rejected tokens be merged back into future drafts. Three named techniques carry the argument: adaptive single-sequence prediction (truncate draft generation when normalized logits fall below a threshold, extending maximum prediction length to 24 tokens), draft sequence recycling (keep the unaccepted draft and the verified prefix as branches of a masked token tree, regenerating failed positions in parallel and merging branches when tokens match), and two-pass sparse-tree prediction (first greedily decode a main trunk, then extend only marked uncertain positions with the second-highest-probability token to balance draft and target latency). The attention mask from tree-based speculative decoding is used to verify all candidate sequences in one target forward pass.

What would settle it

Run SpecASR on an openly available LLM-based ASR system (for example, a Whisper encoder with a LoRA-tuned Llama decoder), measuring mean accepted tokens per verification round and end-to-end latency on LibriSpeech test-clean. If the acceptance rate or speedup falls materially below what the Whisper-trajectory simulation predicts, the transfer assumption is falsified; conversely, matching or exceeding the range 3.04x-3.79x would confirm it.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the alignment between small and large ASR models, induced by the shared audio conditioning, is strong enough to make speculative decoding much more effective for ASR than for text generation. Because both draft and target models hear the same audio, the target tends to continue along the draft's correct prefix even after a rejected token, so unaccepted draft tokens can be recycled rather than wasted. The framework turns this into a concrete recipe: keep the draft sequence long (up to 24 tokens), stop early when normalized logits drop below a threshold, reuse masked token trees to regenerate failed positions in parallel, and for large target models expand only the top-2 uncertain positions in a sparse tree. The result is claimed to be speedups of 3.04x-3.79x over autoregressive decoding and 1.25x-1.84x over prior speculative decoding on LibriSpeech, with no recognition accuracy loss.

Load-bearing premise

The whole speedup number rests on the assumption that acceptance and rejection trajectories measured with Whisper tiny.en/medium.en can be replayed with TinyLlama as the draft and Llama-7B/Vicuna-13B as the target, because the smaller WER gap between those models should mean at least as good alignment; this transfer is never validated against a real LLM-based ASR system.

Editorial extensions

If this is right

  • If the transfer assumption holds, SpecASR's techniques give LLM-based ASR systems a 3.04x-3.79x speedup over autoregressive decoding at identical word error rate, easing real-time constraints.
  • Adaptive truncation removes 74.1% of ineffective draft prediction steps and raises the per-round decoding-acceptance ratio to 94.4% in the Whisper-based configuration.
  • Two-pass sparse-tree prediction increases the average accepted length per verification round by 106.6%, making it the preferred mode when the target model dominates latency.
  • On noisy speech (test-other/dev-other) the speedup over autoregressive decoding is 3.04x-3.07x, about 19% lower than on clean speech, because the draft model's accuracy relative to the target degrades.
  • If actual Llama/Vicuna-based ASR models have a smaller WER gap between draft and target than Whisper tiny.en/medium.en, the paper's expectation is that SpecASR's speedups would be even larger than reported.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the alignment argument is not specific to transcription: any audio-conditioned sequence task with a small/large model pair, such as speech translation or audio captioning, should inherit the same draft-recycling benefit, though the acceptance statistics would need re-measurement.
  • Because the headline speedups were obtained by replaying Whisper trajectories with TinyLlama/Llama/Vicuna rather than running a real LLM-based ASR system, a direct end-to-end measurement on a public LLM-ASR model is the natural next test and could shift the numbers either way.
  • The logits-threshold truncation suggests that acceptance failures cluster at acoustically difficult segments; if true, acoustic confidence features could be used to set the truncation threshold per segment rather than globally, a testable extension.
  • The recycling strategy effectively treats the draft model as a denoiser that can repair failed tokens; this points toward using the same masked-tree mechanism in streaming ASR, where latency budgets are even tighter.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes SpecASR, a speculative decoding framework for LLM-based automatic speech recognition. It builds on the observation that ASR decoding is audio-conditioned, yielding high alignment between small draft models and large target models. The framework introduces three techniques: adaptive single-sequence prediction with logit-threshold truncation, draft sequence recycling via masked token trees, and two-pass sparse-tree prediction with top-k branch expansion. The experiments are conducted by first recording decoding trajectories with Whisper tiny.en/medium.en and then simulating those trajectories with TinyLlama as the draft model and Llama-7B/Vicuna-13B as target models using latency models. The paper reports 2.08x-2.60x (Llama-7B) and 3.04x-3.79x (Vicuna-13B) speedups over autoregressive decoding, and 1.21x-1.84x over baseline speculative decoding, claiming no loss in recognition accuracy.

Significance. If the results were directly measured on the target LLM-based ASR systems, SpecASR would be a useful contribution to ASR-specific speculative decoding: the observations on acceptance-rate distributions and draft-target alignment are well motivated, and the ablations show consistent latency reductions for the Whisper-based configuration. However, the central speedup and iso-accuracy claims are not tested on the systems that the paper is about. The transfer from Whisper trajectories to TinyLlama/Llama/Vicuna is an unvalidated assumption, and no WER table appears anywhere in the paper. These gaps are load-bearing: the headline numbers in the abstract and conclusion rest on a simulation whose proxy assumption is not verified.

major comments (3)
  1. [Section V.A and Section V.B] The headline speedups are not measured on the target LLM-based ASR systems. The authors state that public Llama/Vicuna ASR models are unavailable and instead 'record the decoding trajectories of Whisper-based SpecASR' and then 'simulate these trajectories using TinyLlama as the draft model and Llama-7B/Vicuna-13B as target models.' The acceptance/rejection decisions thus come from Whisper tiny.en/medium.en, while only the latency model comes from TinyLlama/Llama/Vicuna. The bridge that a smaller WER gap between TinyLlama and Llama/Vicuna 'suggests' better alignment is insufficient: token-level acceptance depends on per-step logits, tokenizer, decoding order, and audio encoder, none of which are captured by aggregate WER. The reported 3.04x-3.79x and 1.25x-1.84x speedups are therefore unsupported for the LLM-based ASR systems the paper claims to accelerate.
  2. [Section V, overall experimental evaluation] No recognition accuracy result is reported for any SpecASR configuration, yet the abstract and conclusion assert 'without any loss in recognition accuracy.' This is not a harmless omission: the standard speculative decoding losslessness guarantee holds when verification is performed by the actual target model on the actual draft output. In this paper, the verification decisions are replayed from Whisper medium.en, so the losslessness guarantee does not transfer to TinyLlama/Llama/Vicuna. At minimum, a WER comparison on the real system being evaluated is required; without it, the iso-accuracy claim is asserted rather than demonstrated.
  3. [Section IV.B and Table II] The draft sequence recycling strategy and the adaptive truncation threshold modify the standard draft-verify loop, but the paper provides neither a formal proof that the modified protocol preserves the target model's output distribution nor an empirical WER check. The ablation in Table II reports only latency (in milliseconds) and does not include any recognition accuracy metric, so it cannot establish that the merged and reused sequences are recognition-equivalent. This is a load-bearing correctness gap for the claim that SpecASR accelerates 'without compromising the recognition accuracy.'
minor comments (4)
  1. [Section V.B, Figure 11] The comparison uses a maximum prediction length of 24 for SpecASR but only 8 or 16 for the baseline speculative decoding configurations. If the baseline is not also evaluated with longer prediction lengths, part of the reported speedup may be attributable to the longer draft setting rather than to the proposed adaptive/recycling mechanisms.
  2. [Section V.C] The truncation threshold (0.4) and the choice of second-highest-probability token for tree expansion are data-dependent hyperparameters; the paper acknowledges that the optimal threshold 'may vary depending on the model.' This should be disclosed in the contributions or abstract so that the method is not presented as parameter-free.
  3. [Section II.A] There is a typo: 'audial embeddings' should read 'audio embeddings.' Minor grammatical issues also appear elsewhere, e.g., 'SpecASR achieves ... latency reduction' should be 'speedup' or 'latency reduction relative to baseline' for clarity.
  4. [Section V.A] No code or implementation artifacts are provided. Given that the main evaluation is a simulation over recorded Whisper trajectories, releasing the recording and replay scripts would be important for reproducibility and for verifying that the trajectory transfer is not sensitive to implementation details.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported speedups are measured from proxy-model trajectories and tuned hyperparameters, not derived from or equivalent to the paper's inputs by construction.

full rationale

The paper's derivation chain does not reduce to its own inputs at the equation or definitional level. The speedups in Section V.B are computed by recording Whisper tiny.en/medium.en acceptance and rejection trajectories and replaying them with TinyLlama and Llama-7B/Vicuna-13B latencies; this is a proxy evaluation, not a circular construction. The adaptive threshold (0.4) and the choice to expand with the second-highest logit are tuned hyperparameters selected using the same experimental data, but the paper presents them as design choices and does not rename a fitted parameter as an independent prediction. Reference [23] (ProPD) is authored by overlapping authors, but it is cited only as background on dynamic token trees and is not load-bearing for SpecASR's claimed speedups; the speedups are new measurements rather than consequences of that citation. The paper itself states in Section V.A that LLM-based ASR models are 'not yet publicly available' and that Whisper trajectories are used instead; this is an external-validity limitation, not an internal circularity. The claim of 'without any loss in recognition accuracy' is asserted without a directly measured WER for the simulated Llama systems, and the transfer from Whisper acceptance patterns to Llama/Vicuna is an unvalidated empirical assumption. These are correctness and validation concerns, not examples of the paper's outputs being equivalent to its inputs by definition or by self-citation. Therefore, no significant circularity is identified.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central performance claim rests on two tuned hyperparameters and an assumption that Whisper-based acceptance patterns transfer to Llama/Vicuna-based ASR systems. No new entities are introduced.

free parameters (3)
  • Normalized logits truncation threshold = 0.4
    Selected as optimal on LibriSpeech; controls when draft generation is truncated for early verification.
  • Tree expansion rank (top-k) = 2 (second-highest probability token)
    Chosen because over two-thirds of corrected tokens are rank 2; higher k adds overhead.
  • Maximum prediction length = 24 tokens
    Set to exploit high draft-target alignment; prior speculative decoding uses shorter lengths.
assumptions (4)
  • domain assumption ASR decoding is audio-conditioned, so small and large ASR models remain aligned even after intermediate output mismatches.
    Stated as a core observation in Section III and used to justify long draft lengths and token recycling; it is an empirical tendency, not proven.
  • ad hoc to paper Whisper tiny.en and medium.en acceptance trajectories are representative of TinyLlama and Llama-7B/Vicuna-13B acceptance behavior.
    Section V.A: the paper records Whisper trajectories and 'simulate[s] these trajectories using TinyLlama as the draft model and Llama-7B/Vicuna-13B as target models' without running the LLM-based ASR system.
  • ad hoc to paper Choosing the token with the second-highest draft probability is sufficient to capture two-thirds of target-model corrections.
    Section V.C: empirical analysis on the test set selects the top-2 expansion; this is tuned rather than derived.
  • domain assumption The verification procedure preserves the target model's decoding output exactly, so recognition accuracy is unchanged.
    Assumed throughout; no WER table is provided to confirm under the proposed changes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding." pith.science (2026). https://pith.science/paper/2B3RZPAK

@misc{pith2026250718181,
  author       = {Pith},
  title        = {Pith review of: SpecASR: Accelerating LLM-based Automatic Speech Recognition via Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2B3RZPAK}},
  note         = {Machine review of arXiv:2507.18181}
}
read the original abstract

Large language model (LLM)-based automatic speech recognition (ASR) has recently attracted a lot of attention due to its high recognition accuracy and enhanced multi-dialect support. However, the high decoding latency of LLMs challenges the real-time ASR requirements. Although speculative decoding has been explored for better decoding efficiency, they usually ignore the key characteristics of the ASR task and achieve limited speedup. To further reduce the real-time ASR latency, in this paper, we propose a novel speculative decoding framework specialized for ASR, dubbed SpecASR. SpecASR is developed based on our core observation that ASR decoding is audio-conditioned, which results in high output alignment between small and large ASR models, even given output mismatches in intermediate decoding steps. Therefore, SpecASR features an adaptive draft sequence generation process that dynamically modifies the draft sequence length to maximize the token acceptance length. SpecASR further proposes a draft sequence recycling strategy that reuses the previously generated draft sequence to reduce the draft ASR model latency. Moreover, a two-pass sparse token tree generation algorithm is also proposed to balance the latency of draft and target ASR models. With extensive experimental results, we demonstrate SpecASR achieves 3.04x-3.79x and 1.25x-1.84x speedup over the baseline autoregressive decoding and speculative decoding, respectively, without any loss in recognition accuracy.

Figures

Figures reproduced from arXiv: 2507.18181 by the authors.

Figure 1
Figure 1. (a) Parameter ratio and (b) relative latency of audio encoders [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Autoregressive (left) and speculative (right) decoding. [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. 2D attention mask for draft token tree. the “Draft-then-Verify” paradigm. As shown in [PITH_FULL_IMAGE:figures/full_fig_p002_4.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: (a) WERs of ASR models with multiple scales, (b) comparison [PITH_FULL_IMAGE:figures/full_fig_p003_5.png]
Figure 7
Figure 7. Figure 7: Decoding latency proportion on LibriSpeech clean-test. [PITH_FULL_IMAGE:figures/full_fig_p004_7.png]
Figure 8
Figure 8. Figure 8: The overview framework of SpecASR. prediction, dynamically adjusting the length of draft sequences in each prediction round to enhance verification efficiency. To mitigate the computational burden on the draft model, SpecASR incorporates a draft sequence recycling stra…
Figure 10
Figure 10. Figure 10: Two-pass sparse-tree prediction mechanism. [PITH_FULL_IMAGE:figures/full_fig_p005_10.png]
Figure 11
Figure 11. Figure 11: Speedup comparison with baseline methods, including [PITH_FULL_IMAGE:figures/full_fig_p005_11.png]
Figure 12
Figure 12. Figure 12: Comparison of speculative methods on test-clean: (a) the [PITH_FULL_IMAGE:figures/full_fig_p006_12.png]
Figure 13
Figure 13. Figure 13: (a) Step changes for draft prediction and target verification [PITH_FULL_IMAGE:figures/full_fig_p006_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 5 canonical work pages

  1. [1]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  2. [2]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  3. [3]

    Toolqa: A dataset for llm question answering with external tools,

    Y . Zhuang, Y . Yu, K. Wang, H. Sun, and C. Zhang, “Toolqa: A dataset for llm question answering with external tools,” Advances in Neural Information Processing Systems , vol. 36, pp. 50 117–50 143, 2023

    work page 2023

  4. [4]

    Rt-2: Vision-language- action models transfer web knowledge to robotic control,

    A. Brohan, N. Brown, J. Carbajal, Y . Chebotar, X. Chen, K. Choroman- ski, T. Ding, D. Driess, A. Dubey, C. Finn et al., “Rt-2: Vision-language- action models transfer web knowledge to robotic control,” arXiv preprint arXiv:2307.15818, 2023

    arXiv 2023

  5. [5]

    On decoder-only architecture for speech-to-text and large language model integration,

    J. Wu, Y . Gaur, Z. Chen, L. Zhou, Y . Zhu, T. Wang, J. Li, S. Liu, B. Ren, L. Liu et al. , “On decoder-only architecture for speech-to-text and large language model integration,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) . IEEE, 2023, pp. 1–8

    work page 2023

  6. [6]

    Speech reallm– real-time streaming speech recognition with multimodal llms by teaching the flow of time,

    F. Seide, M. Doulaty, Y . Shi, Y . Gaur, J. Jia, and C. Wu, “Speech reallm– real-time streaming speech recognition with multimodal llms by teaching the flow of time,” arXiv preprint arXiv:2406.09569 , 2024

    arXiv 2024

  7. [7]

    Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition,

    Y . Bai, J. Chen, J. Chen, W. Chen, Z. Chen, C. Ding, L. Dong, Q. Dong, Y . Du, K. Gao et al. , “Seed-asr: Understanding diverse speech and contexts with llm-based speech recognition,” arXiv preprint arXiv:2407.04675, 2024

    arXiv 2024

  8. [8]

    Llama-omni: Seamless speech interaction with large language models,

    Q. Fang, S. Guo, Y . Zhou, Z. Ma, S. Zhang, and Y . Feng, “Llama-omni: Seamless speech interaction with large language models,” arXiv preprint arXiv:2409.06666, 2024

    arXiv 2024

Show all 35 references
  1. [9]

    Prompting large language models with speech recognition abilities,

    Y . Fathullah, C. Wu, E. Lakomkin, J. Jia, Y . Shangguan, K. Li, J. Guo, W. Xiong, J. Mahadeokar, O. Kalinli et al., “Prompting large language models with speech recognition abilities,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processin...

    2024

  2. [10]

    Bestow: Efficient and streamable speech language model with the best of two worlds in gpt and t5,

    Z. Chen, H. Huang, O. Hrinchuk, K. C. Puvvada, N. R. Koluguri, P. ˙Zelasko, J. Balam, and B. Ginsburg, “Bestow: Efficient and streamable speech language model with the best of two worlds in gpt and t5,” arXiv preprint arXiv:2406.19954, 2024

    2024 arXiv

  3. [11]

    Accelerating large language model decoding with speculative sam- pling,

    C. Chen, S. Borgeaud, G. Irving, J.-B. Lespiau, L. Sifre, and J. Jumper, “Accelerating large language model decoding with speculative sam- pling,” arXiv preprint arXiv:2302.01318 , 2023

    2023 arXiv

  4. [12]

    Fast inference from transform- ers via speculative decoding,

    Y . Leviathan, M. Kalman, and Y . Matias, “Fast inference from transform- ers via speculative decoding,” in International Conference on Machine Learning. PMLR, 2023, pp. 19 274–19 286

    2023

  5. [13]

    Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification,

    X. Miao, G. Oliaro, Z. Zhang, X. Cheng, Z. Wang, Z. Zhang, R. Y . Y . Wong, A. Zhu, L. Yang, X. Shiet al., “Specinfer: Accelerating generative large language model serving with tree-based speculative inference and verification,” arXiv preprint arXiv:2305.09781 , 2023

    2023 arXiv

  6. [14]

    Medusa: Simple llm inference acceleration framework with multiple decoding heads,

    T. Cai, Y . Li, Z. Geng, H. Peng, J. D. Lee, D. Chen, and T. Dao, “Medusa: Simple llm inference acceleration framework with multiple decoding heads,” arXiv preprint arXiv:2401.10774 , 2024

    2024 arXiv

  7. [15]

    Eagle: Speculative sampling re- quires rethinking feature uncertainty,

    Y . Li, F. Wei, C. Zhang, and H. Zhang, “Eagle: Speculative sampling re- quires rethinking feature uncertainty,” arXiv preprint arXiv:2401.15077, 2024

    2024 arXiv

  8. [16]

    Dyspec: Faster speculative decoding with dynamic token tree structure,

    Y . Xiong, R. Zhang, Y . Li, T. Wu, and L. Zou, “Dyspec: Faster speculative decoding with dynamic token tree structure,” arXiv preprint arXiv:2410.11744, 2024

    2024 arXiv

  9. [17]

    Whisper in medusa’s ear: Multi-head efficient decoding for transformer- based asr,

    Y . Segal-Feldman, A. Shamsian, A. Navon, G. Hetz, and J. Keshet, “Whisper in medusa’s ear: Multi-head efficient decoding for transformer- based asr,” arXiv preprint arXiv:2409.15869 , 2024

    2024 arXiv

  10. [18]

    Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling,

    S. Gandhi, P. von Platen, and A. M. Rush, “Distil-whisper: Robust knowledge distillation via large-scale pseudo labelling,” arXiv preprint arXiv:2311.00430, 2023

    2023 arXiv

  11. [19]

    Conformer: Convolution-augmented transformer for speech recognition,

    A. Gulati, J. Qin, C.-C. Chiu, N. Parmar, Y . Zhang, J. Yu, W. Han, S. Wang, Z. Zhang, Y . Wu et al., “Conformer: Convolution-augmented transformer for speech recognition,” arXiv preprint arXiv:2005.08100 , 2020

    2005 arXiv

  12. [20]

    Robust speech recognition via large-scale weak supervi- sion,

    A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, “Robust speech recognition via large-scale weak supervi- sion,” in International conference on machine learning . PMLR, 2023, pp. 28 492–28 518

    2023

  13. [21]

    Qwen technical report,

    J. Bai, S. Bai, Y . Chu, Z. Cui, K. Dang, X. Deng, Y . Fan, W. Ge, Y . Han, F. Huang et al. , “Qwen technical report,” arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  14. [22]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685 , 2021

    2021 arXiv

  15. [23]

    Propd: Dynamic token tree pruning and generation for llm parallel decoding,

    S. Zhong, Z. Yang, M. Li, R. Gong, R. Wang, and R. Huang, “Propd: Dynamic token tree pruning and generation for llm parallel decoding,” arXiv preprint arXiv:2402.13485 , 2024

    2024 arXiv

  16. [24]

    Eagle-2: Faster infer- ence of language models with dynamic draft trees,

    Y . Li, F. Wei, C. Zhang, and H. Zhang, “Eagle-2: Faster infer- ence of language models with dynamic draft trees,” arXiv preprint arXiv:2406.16858, 2024

    2024 arXiv

  17. [25]

    Sequoia: Scalable, robust, and hardware-aware speculative decoding,

    Z. Chen, A. May, R. Svirschevski, Y . Huang, M. Ryabinin, Z. Jia, and B. Chen, “Sequoia: Scalable, robust, and hardware-aware speculative decoding,” arXiv preprint arXiv:2402.12374 , 2024

    2024 arXiv

  18. [26]

    Multi-candidate speculative decoding,

    S. Yang, S. Huang, X. Dai, and J. Chen, “Multi-candidate speculative decoding,” arXiv preprint arXiv:2401.06706 , 2024

    2024 arXiv

  19. [27]

    Librispeech: an asr corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: an asr corpus based on public domain audio books,” in2015 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  20. [28]

    Lrs3-ted: a large-scale dataset for visual speech recognition,

    T. Afouras, J. S. Chung, and A. Zisserman, “Lrs3-ted: a large-scale dataset for visual speech recognition,” arXiv preprint arXiv:1809.00496, 2018

    2018 arXiv

  21. [29]

    Parallel- spec: Parallel drafter for efficient speculative decoding,

    Z. Xiao, H. Zhang, T. Ge, S. Ouyang, V . Ordonez, and D. Yu, “Parallel- spec: Parallel drafter for efficient speculative decoding,” arXiv preprint arXiv:2410.05589, 2024

    2024 arXiv

  22. [30]

    Online speculative decoding,

    X. Liu, L. Hu, P. Bailis, A. Cheung, Z. Deng, I. Stoica, and H. Zhang, “Online speculative decoding,” arXiv preprint arXiv:2310.07177 , 2023

    2023 arXiv

  23. [31]

    Spectr: Fast speculative decoding via optimal transport,

    Z. Sun, A. T. Suresh, J. H. Ro, A. Beirami, H. Jain, and F. Yu, “Spectr: Fast speculative decoding via optimal transport,” Advances in Neural Information Processing Systems , vol. 36, 2024

    2024

  24. [32]

    Recursive speculative decoding: Accelerating llm inference via sampling without replacement,

    W. Jeon, M. Gagrani, R. Goel, J. Park, M. Lee, and C. Lott, “Recursive speculative decoding: Accelerating llm inference via sampling without replacement,” arXiv preprint arXiv:2402.14160 , 2024

    2024 arXiv

  25. [33]

    Recurrent drafter for fast speculative decoding in large language models,

    Y . Cheng, A. Zhang, X. Zhang, C. Wang, and Y . Wang, “Recurrent drafter for fast speculative decoding in large language models,” arXiv preprint arXiv:2403.09919, 2024

    2024 arXiv

  26. [34]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023,

    W.-L. Chiang, Z. Li, Z. Lin, Y . Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y . Zhuang, J. E. Gonzalez et al., “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023,” URL https://lmsys. org/blog/2023-03-30-vicuna, vol. 3, no. 5, 2023

    2023

  27. [35]

    An embarrassingly simple approach for llm with strong asr capacity,

    Z. Ma, G. Yang, Y . Yang, Z. Gao, J. Wang, Z. Du, F. Yu, Q. Chen, S. Zheng, S. Zhang et al., “An embarrassingly simple approach for llm with strong asr capacity,” arXiv preprint arXiv:2402.08846 , 2024

    2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.