Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Mamba Drafters for Speculative Decoding

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Using a 130M Mamba state-space model as an external drafter makes speculative decoding both faster and memory-light, matching self-speculation without retraining.

desk verdict Useful incremental work on Mamba drafters, but the cross-model claim is undercut by an unaddressed tokenizer mismatch and the evidence base is thinner than the abstract suggests. read the letter →

arxiv 2506.01206 v1 pith:Y52HCMJ7 submitted 2025-06-01 cs.CL cs.AI

Daewon Choi , Seunghyuk Oh , Saket Dingliwal , Jihoon Tack , Kyuyoung Kim , Woomin Song , Seojin Kim , Insu Han
show 4 more authors
Jinwoo Shin Aram Galstyan Shubham Katiyar Sravan Babu Bodapati
This is my paper
classification cs.CLcs.AI
keywords speculativedecodingMambadrafterstatespacemodelstree-structureddraftingmulti-armedbandittreesearchcross-modeladaptabilitylong-contextinferenceLLMacceleration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Speculative decoding speeds up a large language model by having a fast drafter propose tokens that the target model verifies in parallel, but existing drafters force a choice between retraining for each target model and slow Transformer drafting. The paper claims that a small Mamba state-space model resolves this trade-off: because Mamba computes tokens through a linear recurrence with a fixed-size state, a 130M Mamba drafter drafts at speeds comparable to self-speculation methods while using nearly constant memory as the context grows, and it can be paired with different Transformer targets without retraining. Across summarization, math, instruction-following, code, and long-context benchmarks, the drafter is reported to roughly double the throughput of same-size Transformer external drafters and to match the throughput of EAGLE, a retrained self-speculation drafter, while using up to about 20 GB less peak memory at 8k context. A further contribution is a batch tree-drafting scheme plus a test-time multi-armed bandit that adaptively picks the draft-tree shape, which the paper shows improves acceptance length and throughput. If these results hold, one small generic drafter could serve as a plug-in accelerator for many Transformer models.

What carries the argument

The central mechanism is the Mamba selective state-space layer, which updates a fixed-size hidden state through a linear recurrence, so the per-token cost of drafting does not scale with input length and no KV cache accumulates. The authors build tree drafting on this property: branching simply copies the current state and samples multiple continuations in one batch, with a graph cache reusing the compiled computation for each batch size, rather than re-running overlapping attention histories. The adaptive part is a multi-armed bandit over tree shapes (N1,...,Nγ), where the UCB policy balances each candidate's average accepted tokens against how often it has been tried, and the reward function r = (1 + λγ·γ)/Naccept is the inverse-speedup objective from the speculative decoding time formula.

What would settle it

Use the instruction-tuned Mamba-Mistral-130M drafter on several Transformer target models whose tokenizers differ from Mistral's, and record acceptance length and throughput; if acceptance length falls to the no-drafter baseline or throughput fails to beat a same-size Transformer drafter, the cross-model generality claim fails. Separately, measure peak memory at 16k and 64k context; if it grows linearly with input length rather than staying nearly flat, the constant-memory claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that Mamba-based drafters get the best of both existing speculative-decoding families. As external drafters, they do not require training on the target model's hidden states, yet their linear state-space structure makes drafting latency and peak memory comparable to self-speculation methods; the authors attribute this to Mamba's fixed-size recurrent state, which avoids both the quadratic attention cost and the growing KV cache of Transformer drafters. They also observe empirically that a smaller Mamba can outproduce a larger Transformer drafter because its drafting speed compensates for slightly lower acceptance length. To exploit that speed, the paper introduces batch tree-structured drafting for Mamba based on copying the recurrent state rather than the token history, and a UCB-based multi-armed bandit that selects the draft-tree configuration at test time to minimize the inverse of the speculative-decoding speedup. With these components, the paper reports throughput comparable to EAGLE on instruction-following tasks and superior memory behavior on long contexts, plus a cross-model transfer result in which a Mamba drafter trained with one tokenizer accelerates a target model with a different tokenizer.

Load-bearing premise

The paper's cross-model adaptability claim rests on a single experiment in which one Mamba drafter trained with one tokenizer is applied to one target model with a different tokenizer; the paper assumes this positive result extends to arbitrary target models and tokenizers, and if it does not, the claim that Mamba combines the best of both worlds weakens substantially.

Editorial extensions

If this is right

  • A 130M Mamba drafter can be deployed as a generic accelerator in front of Transformer models without per-target retraining, cutting deployment cost.
  • Long-context speculative decoding can run with much lower peak memory, up to about 20 GB less at 8k input, because drafter memory does not grow with the KV cache.
  • Tree drafting that duplicates states instead of attention histories lets Mamba drafters match or beat Transformer drafters that use tree attention.
  • Adaptively selecting tree shape at test time gives additional throughput, most visibly on code-generation benchmarks.
  • Advances in state-space models, such as Mamba-2, directly inherit the same drafting gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the paper's single cross-model transfer experiment leaves open whether the result generalizes; a plausible extension is that any Mamba drafter tokenized for one model family can accelerate any target sharing a compatible tokenizer, with the bandit tree search absorbing the mismatch.
  • Editorial inference: because the drafter's memory stays flat while the target model's memory still grows, the dominant memory cost of speculative decoding would shift to the target model, which could change batching and context-length decisions in serving systems; the paper does not test this.
  • Editorial inference: the bandit reward could be reweighted for latency or energy budgets without changing the algorithm, offering a serving knob the paper does not explore.
  • Editorial inference: the same fixed-state reasoning suggests a single Mamba model could eventually act as both drafter and target, eliminating the separate drafter entirely; the paper lists this as future work but does not evaluate it.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes using a small Mamba state-space model as an external drafter for speculative decoding of Transformer LLMs, arguing that Mamba's linear recurrence gives lower drafting latency and more constant memory than Transformer drafters, while remaining target-agnostic. The authors introduce a tree-structured drafting procedure for Mamba based on state copying and batched generation, and a test-time multi-armed bandit (MAB) algorithm that selects among tree configurations using a reward derived from the speculative-decoding speedup formula. Experiments compare Mamba-130M drafters with Pythia/Mistral Transformer drafters and with EAGLE on XSum, CNN-DailyMail, GSM-8K, MT-bench, Alpaca, HumanEval, and LongBench, reporting throughput, acceptance length, and peak memory. The main claims are that Mamba drafters outperform external Transformer drafters, are competitive with EAGLE, and maintain cross-model adaptability without retraining.

Significance. If the results hold, the contribution is practically significant: a single 130M Mamba drafter could in principle accelerate multiple Transformer targets with lower memory than EAGLE and without per-target retraining. The paper also provides a concrete engineering recipe for tree-structured drafting with recurrent models (state copying, pre-allocated state caches, graph caching) and a useful diagnostic showing better calibration of a small Mamba drafter relative to a same-size Transformer drafter (Figure 3). The measured throughput and memory numbers on standard benchmarks are valuable even where the cross-model claim needs more support. However, the support for the advertised cross-model adaptability currently rests on a single experiment with an unstated tokenizer-alignment mechanism, and several reproducibility details for the MAB tree search and the stochastic-sampling comparisons are missing.

major comments (4)
  1. [§5.4, Table 3] The cross-model adaptivity experiment uses a Mamba drafter trained with the Mistral-7B tokenizer on the Pythia-6.9B target, but Mistral and Pythia use different tokenizers (Mistral's SentencePiece-style vocabulary vs. Pythia's GPT-NeoX ByteLevel BPE vocabulary). The paper never describes any tokenizer alignment, mapping, or re-encoding between the drafter and the target. In speculative decoding, the acceptance criterion in Algorithm 1 compares target and drafter probabilities for the same token identity, which is undefined across non-identical vocabularies. As written, the reported acceptance length of 2.45 in Table 3 cannot be reproduced, and this experiment therefore does not support the abstract's claim of cross-model adaptability. Please either describe the exact tokenizer adaptation used, or re-run the experiment with a properly aligned vocabulary.
  2. [A.4, Eq. (2)–(6)] The MAB reward derivation assumes T_draft/T_target is a constant λγ, but the value of λγ is never reported, and λ_UCB in Eq. (1) and the tree configuration set S are also not specified. Since the test-time tree search is a central claimed contribution and is used in the main experiments, the missing hyperparameters make those experiments irreproducible. In addition, the statement that the reward 'directly originated' from the SD speedup objective overstates the derivation: Eq. (5) becomes Eq. (6) only by treating T_target(γ)/T_target ≈ 1 and by introducing λγ as an unreported tunable coefficient rather than a measured quantity.
  3. [A.1, Tables 1–3] The evaluation uses approximately 80 samples per dataset, and the sampling experiments (temperature = 1) report no seeds, variance, or confidence intervals. Several key comparisons are close enough that noise could change the qualitative conclusion; for example, on Mistral-7B/HumanEval in Table 1b, Ours (118.04) is about 11% slower than EAGLE (132.69), while on Alpaca Ours is faster (96.83 vs. 94.03). Without repeated runs or error bars, the claims of 'comparable' or 'outperform' against EAGLE and Transformer drafters are not supported at the reported precision.
  4. [§5.5, Tables 4–6] The test-time tree search ablation is internally inconsistent. Table 6 reports 124.99 for the no-search baseline on MT-bench and says the fixed configuration is (3,2,2,1), but Table 5 shows that 124.99 is the throughput of (3,3,2,1), and Table 4 reports a fixed configuration of (3,2,2,1,1) with throughput 127.37. This makes the claimed gain of the MAB search (128.21 vs. 124.99) ambiguous, because the best fixed tree configuration in the ablation set may already reach 127.37. Please correct the configuration labels and report the fixed-tree baseline consistently across Tables 4–6.
minor comments (5)
  1. [§3, Preliminaries] In the sentence 'candidate tokens along with xprefix are passed to the target model Mq', Mq should be Mp; the drafter and target model names are otherwise consistently distinguished.
  2. [§4.2, Eq. (2)] The reward r(t)_k uses Naccept without a round subscript and defines the indicator I, but the notation is not fully consistent with the UCB update; please clarify that the reward is only observed for the selected arm and that Naccept depends on round t.
  3. [Table 1a] The table formatting for the Ours/130M row is broken in the text (the throughput values and the method label are split across lines); please fix the layout so each row is clearly associated with one method.
  4. [Table 3] The 'N/A' entries for EAGLE cross-target transfer are unexplained; a sentence noting that EAGLE requires the same tokenizer/hidden-state interface as the target would help readers understand the comparison.
  5. [A.1] The appendix says datasets are subsampled 'following EAGLE', but the exact number of samples per task is given only as 'approximately 80'; please list the actual sample counts and any filtering steps for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's performance claims are direct empirical measurements, and its MAB reward is a reparameterized speedup objective rather than a fitted prediction.

full rationale

The central claims—throughput, acceptance length, memory usage—are reported from direct experimental measurements (Tables 1-5, Figures 1-3), so they do not reduce to fitted inputs. The only derivation-like step is the MAB reward in Section 4.2/Appendix A.4, where Eq. (6) is obtained from the speedup formula Eq. (4) by substituting lambda_gamma = T_draft/T_target and assuming T_target(gamma)/T_target is approximately 1. This is an explicit modeling approximation, and the paper acknowledges in Limitations that the tree-search performance 'depends on hyperparameter settings'; the unmeasured lambda_gamma is a hyperparameter, not a parameter fitted to the reported throughputs. Because the algorithm observes N_accept as a bandit reward and the final throughputs are measured after tree selection, the tree-search component does not generate a predicted quantity that is definitionally equal to its input. The cross-model claim (Section 5.4, Table 3) rests on a single Mamba-Mistral-to-Pythia transfer, which is a generalization/validity risk, especially given the tokenizer mismatch between the Mistral and Pythia vocabularies, but that is an empirical weakness, not circularity. No load-bearing self-citation chain appears: the bandit formulation cites Kim et al. (2024b), whose authors do not overlap with the present paper, and the Mamba/EAGLE comparisons are externally benchmarked. Therefore no step reduces the paper's conclusions to its assumptions by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central efficiency claim rests on Mamba's architectural properties (axiom 1). The MAB tree search introduces two unspecified hyperparameters, lambda_UCB and lambda_gamma, and a handpicked tree set, which are fitted rather than derived. No new physical or conceptual entities are postulated.

free parameters (3)
  • lambda_UCB
    UCB exploration coefficient in Eq (1), not specified in the paper; the tree-search results depend on its value.
  • lambda_gamma
    Weight in the MAB reward Eq (2), described in Appendix A.4 as the ratio T_draft/T_target but treated as a tunable hyperparameter; its value is not reported.
  • Tree configuration set S
    The handpicked set of three tree shapes used in the test-time search (Section 5.5, Table 6); the choice of this set influences the reported search gains.
assumptions (4)
  • domain assumption Mamba's linear recurrence keeps per-token latency and memory constant regardless of context length.
    Invoked throughout Section 3.1 and Figure 1; this is an architectural property of SSMs, asserted from prior work.
  • standard math Speculative decoding with rejection sampling preserves the target model's output distribution.
    Stated in Section 3 Preliminaries and assumed in the verification algorithm (Algorithm 1); standard result from Leviathan et al. 2023.
  • ad hoc to paper In the reward derivation, T_target(gamma)/T_target is approximately 1 and T_draft/T_target is a constant lambda_gamma.
    Appendix A.4, Eq (3-6). This approximation is used to justify the MAB reward, and the constant is left as an unspecified hyperparameter.
  • domain assumption An instruction-tuned external drafter transfers to target models it was not trained on.
    Central to the cross-model adaptability claim; supported only by the single transfer experiment in Table 3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mamba Drafters for Speculative Decoding." pith.science (2026). https://pith.science/paper/Y52HCMJ7

@misc{pith2026250601206,
  author       = {Pith},
  title        = {Pith review of: Mamba Drafters for Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y52HCMJ7}},
  note         = {Machine review of arXiv:2506.01206}
}
read the original abstract

Speculative decoding has emerged as a promising approach to accelerating large language model (LLM) generation using a fast drafter while maintaining alignment with the target model's distribution. However, existing approaches face a trade-off: external drafters offer flexibility but can suffer from slower drafting, while self-speculation methods use drafters tailored to the target model but require re-training. In this paper, we introduce novel drafters based on Mamba, a state-of-the-art state space model (SSM), as a solution that combines the best aspects of both approaches. By leveraging the linear structure of SSMs, our approach avoids the quadratic complexity inherent in traditional Transformer-based methods, enabling faster drafting and lower memory usage while maintaining the flexibility to work across different target models. We further enhance efficiency with a novel test-time tree search algorithm for generating high-quality draft candidates. Our empirical evaluation demonstrates that Mamba-based drafters not only outperform existing external drafting methods but are also comparable to state-of-the-art self-speculation approaches while using less memory and maintaining their cross-model adaptability.

Figures

Figures reproduced from arXiv: 2506.01206 by the authors.

Figure 1
Figure 1. Comparison of drafting time & peak memory usage during encoding and decoding. Mamba drafter maintains nearly constant decoding speed and memory usage, whereas both EAGLE, which employs self-speculation with a single-layer Transformer within the large target model, and the Mistral-based external drafter, which also utilizes a Transformer, exhibit substantially higher memory requirements as the context length increase… view at source ↗
Figure 3
Figure 3. Most interestingly, we found that smaller Mamba can be a stronger drafter than larger Mam￾bas, as smaller Mamba shows significant drafting speed compared to larger sizes, generating more [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. KnapSpec: Self-Speculative Decoding via Adaptive Layer Selection as a Knapsack Problem

    cs.LG 2026-02 conditional novelty 6.0 of 10

    By modeling layer skipping as a knapsack problem with context-dependent attention/MLP latencies, KnapSpec adaptively selects draft sub-networks that speed up self-speculative decoding by up to 1.47×.

Reference graph

Works this paper leans on

46 extracted references · 11 canonical work pages · cited by 1 Pith paper

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  4. [4]

    P Auer. 2002. Finite-time analysis of the multiarmed bandit problem

    2002

  5. [5]

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. https://arxiv.org/abs/2308.14508 Longbench: A bilingual, multitask benchmark for long context understanding . Preprint, arXiv:2308.14508

    arXiv 2023

  6. [6]

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432--7439

    2020

  7. [7]

    Tianle Cai, Yuhong Li, Zhengyang Geng, Hongwu Peng, Jason D Lee, Deming Chen, and Tri Dao. 2024. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774

    arXiv 2024

  8. [8]

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. 2023. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318

    arXiv 2023

Show all 46 references
  1. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374

    2021 arXiv

  2. [10]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1

    2018 arXiv

  3. [11]

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

    2021 arXiv

  4. [12]

    Tri Dao and Albert Gu. 2024. Transformers are SSM s: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML)

    2024

  5. [13]

    Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179--211

    1990

  6. [14]

    Yichao Fu, Peter Bailis, Ion Stoica, and Hao Zhang. 2024. Break the sequential dependency of llm inference using lookahead decoding. arXiv preprint arXiv:2402.02057

    2024 arXiv

  7. [15]

    Albert Gu and Tri Dao. 2023. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752

    2023 arXiv

  8. [16]

    Albert Gu, Karan Goel, Ankit Gupta, and Christopher R \'e . 2022 a . On the parameterization and initialization of diagonal state space models. In Advances in Neural Information Processing Systems

    2022

  9. [17]

    Albert Gu, Karan Goel, and Christopher R \'e . 2022 b . Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations

    2022

  10. [18]

    Zhenyu He, Zexuan Zhong, Tianle Cai, Jason D Lee, and Di He. 2023. Rest: Retrieval-based speculative decoding. arXiv preprint arXiv:2311.08252

    2023 arXiv

  11. [19]

    Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. http://papers.nips.cc/paper/5945-teaching-machines-to-read-and-comprehend Teaching machines to read and comprehend . In NIPS, pages 1693--1701

    2015

  12. [20]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

    2023 arXiv

  13. [21]

    Rudolph Emil Kalman. 1960. A new approach to linear filtering and prediction problems

    1960

  14. [22]

    Alex Kim, Maximilian Muhn, and Valeri Nikolaev. 2024 a . Financial statement analysis with large language models. arXiv preprint arXiv:2407.17866

    2024 arXiv

  15. [23]

    Taehyeon Kim, Hojung Jung, and Se-Young Yun. 2024 b . A unified framework for speculative decoding with multiple drafters as a bandit. In Proceedings of the Fourth Workshop on Efficient Natural Language and Speech Processing (ENLSP-IV): Highlighting New Architectures for Futur...

    2024

  16. [24]

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2023. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274--19286. PMLR

    2023

  17. [25]

    Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. 2024 a . Videomamba: State space model for efficient video understanding. arXiv preprint arXiv:2403.06977

    2024 arXiv

  18. [26]

    Shufan Li, Harkanwar Singh, and Aditya Grover. 2024 b . Mamba-nd: Selective state space modeling for multi-dimensional data. arXiv preprint arXiv:2402.05892

    2024 arXiv

  19. [27]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 c . Eagle-2: Faster inference of language models with dynamic draft trees. arXiv preprint arXiv:2406.16858

    2024 arXiv

  20. [28]

    Yuhui Li, Fangyun Wei, Chao Zhang, and Hongyang Zhang. 2024 d . Eagle: Speculative sampling requires rethinking feature uncertainty. arXiv preprint arXiv:2401.15077

    2024 arXiv

  21. [29]

    Harsh Mehta, Ankit Gupta, Ashok Cutkosky, and Behnam Neyshabur. 2023. Long range language modeling via gated state spaces. In International Conference on Learning Representations

    2023

  22. [30]

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Zhengxin Zhang, Rae Ying Yee Wong, Alan Zhu, Lijie Yang, Xiaoxiang Shi, et al. 2024. Specinfer: Accelerating large language model serving with tree-based speculative inference and verification. In Proceedings...

    2024

  23. [31]

    Cohen, and Mirella Lapata

    Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/1808.08745

    2018 arXiv

  24. [32]

    Vinh Nguyen, Michael Carilli, Sukru Burc Eryilmaz, Vartika Singh, Michelle Lin, Natalia Gimelshein, Alban Desmaison, and Edward Yang. 2021. https://pytorch.org/blog/accelerating-pytorch-with-cuda-graphs/ Accelerating pytorch with cuda graphs . Accessed: 2025-01-27

    2021

  25. [33]

    Guilherme Penedo, Hynek Kydl \' c ek, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, Thomas Wolf, et al. 2024. The fineweb datasets: Decanting the web for the finest text data at scale. arXiv preprint arXiv:2406.17557

    2024 arXiv

  26. [34]

    Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2023. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.00071

    2023 arXiv

  27. [35]

    Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. 2025. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1--8

    2025

  28. [36]

    Aleksandrs Slivkins et al. 2019. Introduction to multi-armed bandits. Foundations and Trends in Machine Learning , 12(1-2):1--286

    2019

  29. [37]

    Mitchell Stern, Noam Shazeer, and Jakob Uszkoreit. 2018. Blockwise parallel decoding for deep autoregressive models. Advances in Neural Information Processing Systems, 31

    2018

  30. [38]

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Stanford alpaca: An instruction-following llama model

    2023

  31. [39]

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021. Long range arena: A benchmark for efficient transformers. In International Conference on Learning Representations

    2021

  32. [40]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems

    2017

  33. [41]

    Junxiong Wang, Daniele Paliotta, Avner May, Alexander M Rush, and Tri Dao. 2024. The mamba in the llama: Distilling and accelerating hybrid models. arXiv preprint arXiv:2408.15237

    2024 arXiv

  34. [42]

    Heming Xia, Tao Ge, Peiyi Wang, Si-Qing Chen, Furu Wei, and Zhifang Sui. 2023. Speculative decoding: Exploiting speculative execution for accelerating seq2seq generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3909--3925

    2023

  35. [43]

    Sen Yang, Shujian Huang, Xinyu Dai, and Jiajun Chen. 2024. Multi-candidate speculative decoding. arXiv preprint arXiv:2401.06706

    2024 arXiv

  36. [44]

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830

    2019 arXiv

  37. [45]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. https://arxiv.org/abs/2306.05685 Judging llm-as-a-judge with mt-bench and chatbot arena . P...

    2023 arXiv

  38. [46]

    Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. 2024. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.