Pith. sign in

REVIEW 4 major objections 5 minor 51 references

RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A model's own decoding speed can reveal which attention layers are redundant, and replacing those with state-space blocks plus targeted distillation can beat the original teacher.

desk verdict RAD is a genuinely new recipe for building hybrid models, but its headline numbers are confounded by teacher size, initialization, and training recipe; only the opt-vs-worse ablation cleanly supports the layer-selection claim. read the letter →

arxiv 2505.22135 v1 pith:3MIJ3XGF submitted 2025-05-28 cs.CL cs.LG

Yuichiro Hoshino , Hideyuki Tachibana , Muneyoshi Inahara , Hiroto Takegawa This is my paper
classification cs.CLcs.LG
keywords hybridmodelsknowledgedistillationself-speculativedecodingattentionlayerredundancystatespaceMambaBayesianoptimizationselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a recipe for turning an existing Transformer into a smaller, faster hybrid model without giving up accuracy: first find which attention layers are functionally redundant, then replace exactly those layers with state-space-model (SSM) blocks, then distill. The redundancy test is self-speculative decoding: the model itself is used as its own draft model with certain attention layers skipped, and the layers whose skipping most speeds up token generation, while keeping proposals acceptable to the full model, are declared redundant. On the Llama-3.2-3B-Instruct base, replacing eight such layers with Mamba, Mamba2, or Longhorn and self-distilling only the new blocks raised GSM8K from 56.25 to 62.77-64.22 and CRUX from 26.12 to 26.62-27.62. In the standard distillation setting, a hybrid distilled from an 8B teacher using RAD's layer choice reached GSM8K 71.27 and CRUX 28.25, beating the equal-interval baseline distilled from a 70B teacher, which scored 46.17 and 22.75. The paper argues that treating layer selection as part of distillation, rather than as a fixed architectural heuristic, is what makes the gain possible.

What carries the argument

The load-bearing mechanism is the self-speculative-decoding throughput $E[\tau(M_p,\{x_l\})]$ used as a redundancy score for attention layers. A draft model is formed by skipping a chosen subset of attention layers of the teacher; the expected throughput obeys $E[\tau] \approx \frac{1}{K T(q) + T(p)} \sum_{k=0}^K \beta^k$, where $\beta = 1 - D_{\mathrm{TV}}(p,q)$ is the token acceptance rate, $T(q)$ is the draft's per-forward time, and $T(p)$ is the teacher's. Maximizing throughput therefore rewards skip sets whose removed layers least disturb the model's output distribution, which is the paper's operational definition of computational redundancy. Bayesian optimization over the binary skip vector $\{x_l\}$, with top-$k$ discretization when a fixed number of layers must be replaced, finds the configuration. The replacement step copies 'out_proj' weights and zero-initializes part of 'in_proj' in the SSM block so that the block starts as a no-op mirroring the skipped state, and distillation trains only the newly added SSM parameters with forward KL divergence.

What would settle it

Run the RAD layer-selection on several different proxy tasks (for example, GSM8K itself, CRUX, and a long-context task) and compare the selected skip configurations and downstream scores; if the 'opt' layers chosen on PG-19 do not consistently beat equal-interval or per-task selections on held-out benchmarks, or if the selected sets are unstable across Bayesian-optimization runs and seeds, the claim that throughput identifies intrinsic redundancy would be refuted. A second decisive check is to compute, for every candidate layer, the correlation between spec-decoding throughput gain and the actual task-accuracy drop when that layer is ablated across a diverse task suite: near-zero or negative correlation would falsify the proxy.

Watch

Extended reading notes

Core claim

RAD claims that the throughput of self-speculative decoding, optimized over which attention layers to skip, identifies the attention layers whose computation can be most profitably replaced by an SSM block. The objective is $E[\tau(M_p,\{x_l\})]$, the average tokens per second when a draft model built by skipping layers $\{x_l\}$ proposes tokens for the full model $M_p$ to verify; high throughput requires both that the draft be cheap and that its proposals be accepted, so a layer is 'redundant' if skipping it preserves enough agreement with the teacher. The identified layers are swapped for Mamba, Mamba2, or Longhorn blocks, initialized by copying the 'out_proj' weights and zeroing 'in_proj' so the new block initially behaves like a skipped layer, and only the new block parameters are trained under forward KL divergence from the teacher's logits. The central experimental claim is that this procedure produces students that outperform the teacher on GSM8K and CRUX in self-distillation, and that in standard distillation a RAD student distilled from Llama-3.1-8B-Instruct surpasses an equal-interval student distilled from Llama-3.1-70B-Instruct (GSM8K 71.27 vs 46.17; CRUX 28.25 vs 22.75 for Mamba2).

Load-bearing premise

The load-bearing assumption is that the measured self-speculative-decoding throughput on just ten PG-19 summarization texts is a valid, task-independent signal of which attention layers are functionally redundant, so that layers selected on that proxy are the right ones to replace for math, code, and long-context tasks.

Editorial extensions

If this is right

  • If the redundancy score is a valid proxy, hybridizing an existing pretrained Transformer becomes a per-model optimization problem rather than an ad hoc architectural choice: the same base model can be converted with different SSM families by re-running the decoding-speed diagnostic.
  • A student hybrid can outperform the teacher it was distilled from on reasoning tasks (the reported 'born-again' effect), so layer replacement plus targeted distillation can reallocate capacity toward math and code rather than merely preserve it.
  • In standard distillation, the quality of the teacher matters less than the match between architecture and layer choice: an 8B-teacher RAD model beat a 70B-teacher equal-interval model on GSM8K and CRUX, implying that where you replace layers can dominate teacher scale.
  • Training only the replaced SSM blocks converges faster (about 2x faster in the reported loss curves), so the framework lowers the compute cost of hybrid distillation.
  • The same pipeline appears to generalize across SSM variants (Mamba, Mamba2, Longhorn) with similar gains, suggesting the layer-selection signal, not the specific SSM, carries the effect.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the throughput diagnostic could be repurposed as a general layer-importance probe for any architecture where layers can be skipped at decoding time, potentially replacing human-designed pruning heuristics with a model-specific measurement.
  • The redundancy identification uses only 10 PG-19 summarization instances, far narrower than the evaluation suite; a natural extension is to test whether choosing layers on the target benchmarks or on a diverse task mixture changes the selected set and downstream gains, which would quantify how much the proxy's task specificity matters.
  • Because the zero-initialization trick makes the new block a no-op at initialization, the method resembles a warm start from a pruned model; one could test whether the same gains appear when the 'skipped state' is replicated by another mechanism, such as an identity residual or a gated bypass, which would separate the initialization effect from the layer-selection effect.
  • The reported gains on GSM8K and CRUX come with much longer generated answers (reasoning length ratio over 2 for the Mamba2 opt/8B model), so part of the improvement may be a length effect; a sharper test would evaluate accuracy under length-controlled decoding or fixed token budgets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces RAD, an algorithm that first uses self-speculative decoding to identify attention layers in a pretrained Transformer whose removal maximises measured decoding throughput, then replaces those layers with SSM blocks (Mamba, Mamba2, or Longhorn), initializes the new blocks to mimic skipped layers, and distills the original model into the hybrid. In self-distillation experiments on Llama-3.2-3B-Instruct, replacing 8 selected attention layers and training only the SSM parameters is reported to improve GSM8K and CRUX scores over the unmodified teacher. In standard distillation, an 8B-teacher RAD Mamba2 model is reported to outperform public equal-interval baselines distilled from a 70B teacher on GSM8K (71.27 vs 46.17) and CRUX (28.25 vs 22.75). The paper also includes an ablation contrasting 'opt' and 'worse' layer selections, with layer lists and throughput measurements in Appendix A.2.

Significance. If the causal role of layer selection is confirmed, RAD is a practical recipe: it gives a model-specific, task-agnostic criterion for converting a Transformer into a hybrid without an ad hoc equal-interval schedule, and it shows a converted student can outperform its teacher on reasoning tasks. The paper is transparent about its limitations, and the opt-vs-worse ablation (Tables 6 and 9; Figures 3-7) supplies credible internal evidence that the choice of replaced layers matters. The measured throughput differences in Table 8 are much larger than the reported standard errors, which is a real strength. However, the headline standard-distillation comparison and the Born-Again claim are not yet supported by controlled experiments, so the current evidence does not establish that the layer-selection objective, rather than initialization, teacher size, or continued training, drives the results.

major comments (4)
  1. [§4.2, Tables 3–5, Fig. 2] The standard-distillation headline comparison is uncontrolled. 'Mamba2 (opt / 8B)' differs from 'Mamba2 (eql / 70B)' simultaneously in teacher size (8B vs 70B), layer-selection policy (optimized vs equal intervals), initialization (zero in_proj vs qkv/out_proj copied from the original attention), and possibly data/recipe, since the baselines are public models from [43]. The 'eql / 8B' control appears only in Fig. 8 for passkey retrieval, not in the GSM8K/CRUX or LongBench tables. Without a same-teacher, same-initialization, same-data equal-interval (or random-layer) condition, the GSM8K gap 71.27 vs 46.17 cannot be attributed to redundancy-aware selection.
  2. [§4.2, Table 2] The self-distillation 'Born-Again' result is not separated from continued training on the 20B-token mixture. RAD students are compared only with the original teacher, which has not seen those tokens; the improvement on GSM8K/CRUX could arise from the distillation data and training budget rather than from redundancy-aware replacement. A control that fine-tunes the original teacher on the same mixture, or at least trains an equal-interval replacement with zero-init under an identical recipe, is needed to support the claim that the architectural change plus targeted distillation causes the gain.
  3. [§3.1, Eq. (1); §4.1] The redundancy proxy is estimated from only 10 PG-19 summarization instances at 2048-token context, and the same instances are used to drive 2000 BO iterations; there is no held-out estimate of E[τ] or of task transfer. The opt-vs-worse differences in Table 8 are statistically clear, but the selected configuration could still be overfit to those instances. The authors should validate the proxy on held-out PG-19 instances, on other long-context tasks, or against a direct layer-importance metric (e.g., accuracy drop when each layer is removed) to show that the BO objective is not task-specific.
  4. [§4.2, Fig. 2] The '~2x faster convergence' claim is not precisely defined. The figure shows KL loss versus steps for RAD versus the [43] baseline, but no convergence threshold, evaluation checkpoint, or matched-compute comparison is supplied. State the threshold and report downstream accuracy at matched steps/tokens to support the convergence claim.
minor comments (5)
  1. [Throughout] Model names are inconsistent: 'Llama3.2-3B-Instruct' in §4.1 and 'Llama-3.2-3B-Instruct' elsewhere; please standardize.
  2. [References] Reference [34] is incomplete: 'arXiv preprint arXiv:2406.XXXX' contains a placeholder volume number.
  3. [§6] The limitation paragraph says the authors 'primarily tested replacing 50% of attention layers,' but the main self-distillation experiments replace 8 of 28 layers (~29%) and only the standard-distillation experiments replace 50%; reconcile this statement.
  4. [Appendix D.2 vs Algorithm 1] The BO discretization is described inconsistently: D.2 says a 0.5 threshold is used to map z to {0,1}^L, while Algorithm 1 and D.3 use top-k selection; align the description.
  5. [Appendix C, Eq. (18)] The approximation D_KL ≈ 2 D_TV^2 is presented without qualification; Pinsker's inequality gives only an inequality in one direction, and the quadratic relation holds only in a local regime. Add a citation or state explicit conditions.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: RAD's layer selection, initialization, and distillation form a coherent, benchmark-independent pipeline.

full rationale

RAD's central claim—that throughput-optimized layer replacement plus distillation improves hybrid models—is not equivalent to its inputs. Equation (1) maximizes a measured self-speculative-decoding throughput on PG-19; no GSM8K, CRUX, LongBench, or passkey score is used to select layers. Equation (2) is forward KL divergence to the teacher, and the zero-initialization of in_proj recreates the skipped state assumed during selection, so the hybrid's initial state is exactly the draft model whose throughput was optimized. This is self-consistency, not circularity: whether that starting point leads to good downstream performance is an empirical question tested on held-out benchmarks. The opt-versus-worse ablations compare downstream performance against two extremes of the same throughput proxy and show large differences, which is a non-tautological empirical result. The standard-distillation comparison confounds teacher size, layer schedule, and initialization, and the PG-19 selection set is only 10 instances, but these are experimental attribution and generalization concerns, not circularity. Citations are to external prior work ([43], [49], etc.), with no load-bearing self-citations by the present authors. The paper is self-contained against external benchmarks, so the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new entities; its contributions are a diagnostic use of an existing mechanism and an initialization/distillation recipe. The load-bearing choices are hyperparameters and the redundancy-proxy hypothesis.

free parameters (4)
  • Bayesian Optimization iteration count = 2000
    Chosen by the authors as 'sufficiently large'; affects which layer configuration is found.
  • PG-19 sample count for redundancy identification = 10
    10 random instances used to estimate throughput; the proxy objective depends on this small sample.
  • Peak learning rates for distillation = 2e-4 (Mamba/Longhorn self-distillation), 2e-5 (Mamba2 self-distillation and standard distillation)
    AdamW cosine schedules with these peaks; different values per SSM variant and setup are hand-tuned.
  • Number of layers replaced / replacement fraction = 8 layers (self-distillation), 50% = 14 layers (standard distillation)
    Self-distillation uses the BO optimum count; standard distillation forces 50% to match baselines, which constrains the optimization.
assumptions (5)
  • ad hoc to paper Throughput of self-speculative decoding is a valid proxy for attention-layer redundancy.
    Central hypothesis in Section 3.1; no theoretical justification beyond the argument that skipped layers that keep acceptance high are redundant. The paper validates it only through the opt-vs-worse ablation.
  • standard math Speculative sampling preserves the target distribution.
    From [26], used in Appendix C to justify the throughput model.
  • standard math Pinsker's inequality linking KL and total variation distance.
    Used in Appendix C to argue that a TVD-minimizing initialization reduces initial KL quadratically.
  • domain assumption The SSM blocks (Mamba/Mamba2/Longhorn) can functionally replace attention layers after distillation.
    Assumed throughout; the paper provides empirical support but no capacity or expressivity analysis.
  • domain assumption Bayesian optimization over [0,1]^L with top-k discretization explores the constrained skip configuration space effectively.
    Algorithm 1 in Appendix D; no convergence guarantee, acknowledged in Section 6.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding." pith.science (2026). https://pith.science/paper/3MIJ3XGF

@misc{pith2026250522135,
  author       = {Pith},
  title        = {Pith review of: RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3MIJ3XGF}},
  note         = {Machine review of arXiv:2505.22135}
}
read the original abstract

Hybrid models combining Transformers and State Space Models (SSMs) are promising for balancing performance and efficiency. However, optimizing these hybrid models, particularly by addressing the potential redundancy inherent within the Transformer components, remains a significant challenge. In this paper, we propose RAD (Redundancy-Aware Distillation), a novel framework that uses self-speculative decoding as a diagnostic tool to identify redundant attention layers within the model. These identified layers are then selectively replaced with SSM components, followed by targeted (self-)distillation. Specifically, RAD focuses knowledge transfer on the components identified as redundant, considering architectural changes and specific weight initialization strategies. We experimentally demonstrate that self-distillation using RAD significantly surpasses the performance of the original base model on mathematical and coding tasks. Furthermore, RAD is also effective in standard knowledge distillation settings, achieving up to approximately 2x faster convergence compared to baseline methods. Notably, while a baseline model distilled from a Llama-3.1 70B teacher achieves scores of 46.17 on GSM8K and 22.75 on CRUX, RAD achieves significantly higher scores of 71.27 on GSM8K and 28.25 on CRUX, even when using a much smaller Llama-3.1 8B teacher. RAD offers a new pathway for efficient optimization and performance enhancement in the distillation of hybrid models.

Figures

Figures reproduced from arXiv: 2505.22135 by the authors.

Figure 1
Figure 1. Overview of our proposed RAD (Redundancy-Aware Distillation) framework. (a) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Comparison of training loss curves with baseline proposed in [43] for 50% Mamba2 distillation. 10 0 10 1 10 2 10 3 Training Step 10 2 10 3 10 4 KL Loss Mamba: 8 layers (worse) Mamba: 8 layers (opt) Mamba2: 14 layers (worse) Mamba2: 14 layers (opt) Longhorn: 8 layers (worse) Longhorn: 8 layers (opt) [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 4
Figure 4. Comparison of self-distillation (50% layers replaced with Mamba2) loss performance with [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figures from the paper (5 more)
Figure 5
Figure 5. Figure 5: Comparison of training loss curves in the self-distillation setting, with 8 layers replaced in all models [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]
Figure 6
Figure 6. Figure 6: Passkey retrieval performance up to 64K words (roughly 85K tokens) for Llama3.2 under [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]
Figure 7
Figure 7. Figure 7: Full ablation study on LongBench in the self-distillation setting. Eight layers (opt or worse) [PITH_FULL_IMAGE:figures/full_fig_p018_7.png]
Figure 8
Figure 8. Figure 8: Passkey retrieval up to 32K words (roughly 42K tokens) for Llama3.2-Mamba2-3B (50% [PITH_FULL_IMAGE:figures/full_fig_p019_8.png]
Figure 9
Figure 9. Figure 9: Loss spike during standard distillation for Llama3.2-Longhorn-3B (50% opt layers replaced) [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 22 canonical work pages

  1. [43]

    Rush, and Tri Dao

    Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, and Tri Dao. The Mamba in the Llama: Distilling and accelerating hybrid models. Advances in Neural Information Processing Systems , 37:62432–62457, 2024

    work page 2024

  2. [1]

    LongBench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (V olume 1: Long Papers), page...

    work page

  3. [2]

    Puzzle: Distillation-based NAS for inference-optimized LLMs

    Akhiad Bercovich, Tomer Ronen, Talor Abramovich, Nir Ailon, Nave Assaf, Mohammad Dabbah, Ido Galil, Amnon Geifman, Yonatan Geifman, Izhak Golan, Netanel Haber, Ehud Karpas, Roi Koren, Itay Levy, Pavlo Molchanov, Shahar Mor, Zach Moshe, Najeeb Nabwani, Omri Puny, Ran Rubin, Itamar Schen, Ido Shahaf, Oren Tropp, Omer Ullman Argov, Ran Zilberstein, and Ran E...

    arXiv 2024

  4. [3]

    On attention redundancy: A comprehensive study

    Yuchen Bian, Jiaji Huang, Xingyu Cai, Jiahong Yuan, and Kenneth Church. On attention redundancy: A comprehensive study. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors,Proceedings of the 2021 Conference of the North American Chapter of the ...

    work page 2021

  5. [4]

    Xing, J Zico Kolter, and Albert Gu

    Aviv Bick, Kevin Li, Eric P. Xing, J Zico Kolter, and Albert Gu. Transformers to SSMs: Distilling quadratic knowledge to subquadratic models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=FJlrSZBMCD

    work page 2024

  6. [5]

    Nemotron-H: A family of accurate and efficient hybrid Mamba-transformer models, 2025

    Aaron Blakeman, Aarti Basant, Abhinav Khattar, Adithya Renduchintala, Akhiad Bercovich, Aleksander Ficek, Alexis Bjorlin, Ali Taghibakhshi, Amala Sanjay Deshmukh, Ameya Sunil Mahabaleshwarkar, Andrew Tao, Anna Shors, Ashwath Aithal, Ashwin Poojary, Ayush Dattagupta, Balaram Buddharaju, Bobby Chen, Boris Ginsburg, Boxin Wang, Brandon Norick, Brian Butterfi...

    arXiv 2025

  7. [6]

    Accelerating large language model decoding with speculative sampling

    Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre, and John Jumper. Accelerating large language model decoding with speculative sampling. arXiv preprint arXiv:2302.01318, 2023

    arXiv 2023

  8. [7]

    GenQA: Generating millions of instructions from a handful of prompts

    Jiuhai Chen, Rifaa Qadri, Yuxin Wen, Neel Jain, John Kirchenbauer, Tianyi Zhou, and Tom Goldstein. GenQA: Generating millions of instructions from a handful of prompts. CoRR, abs/2406.10323, 2024. URL https://doi.org/10.48550/arXiv.2406.10323

    arXiv doi:10.48550/arxiv.2406.10323 2024

Show all 51 references
  1. [8]

    Streamlining redundant layers to compress large language models

    Xiaodong Chen, Yuxuan Hu, Jing Zhang, Yanling Wang, Cuiping Li, and Hong Chen. Streamlining redundant layers to compress large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=IC5RJvRoMp

    2025

  2. [9]

    Stuffed mamba: State collapse and state capacity of rnn-based long-context modeling, 2024

    Yingfa Chen, Xinrong Zhang, Shengding Hu, Xu Han, Zhiyuan Liu, and Maosong Sun. Stuffed mamba: State collapse and state capacity of rnn-based long-context modeling, 2024. URL https://arxiv.org/ abs/2410.07145

    2024

  3. [10]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  4. [11]

    Analyzing redundancy in pretrained transformer models

    Fahim Dalvi, Hassan Sajjad, Nadir Durrani, and Yonatan Belinkov. Analyzing redundancy in pretrained transformer models. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors,Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)...

    2020 doi

  5. [12]

    Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML) , 2024

    2024

  6. [13]

    Born again neural networks

    Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research, pages 1607–1616. PMLR, 2018

    2018

  7. [14]

    A framework for few-shot language model evaluation, September 2021

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2021 doi

  8. [15]

    Zamba: A compact 7B SSM hybrid model, 2024

    Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7B SSM hybrid model, 2024. URL https://arxiv.org/abs/ 2405.16712

    2024 arXiv

  9. [16]

    RADLADS: Rapid attention distillation to linear attention decoders at scale, 2025

    Daniel Goldstein, Eric Alcaide, Janna Lu, and Eugene Cheah. RADLADS: Rapid attention distillation to linear attention decoders at scale, 2025. URL https://arxiv.org/abs/2505.03005

    2025

  10. [17]

    Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A. Roberts. The unreasonable ineffectiveness of the deeper layers, 2024

    2024

  11. [18]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  12. [19]

    Combining recurrent, convolutional, and continuous-time models with linear state space layers

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in neural information processing systems, 34:572–585, 2021

    2021

  13. [20]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations , 2022

    2022

  14. [21]

    Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I. Wang. CRUXEval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065, 2024. 11

    2024 arXiv

  15. [22]

    What matters in transformers? not all attention is needed, 2024

    Shwai He, Guoheng Sun, Zheyu Shen, and Ang Li. What matters in transformers? not all attention is needed, 2024. URL https://arxiv.org/abs/2406.15786

    2024 arXiv

  16. [23]

    Query-key normal- ization for transformers

    Alex Henry, Prudhvi Raj Dachapally, Shubham Shantaram Pawar, and Yuxuan Chen. Query-key normal- ization for transformers. In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 4246–4253, 2020. URL https://aclanthology.org/2020.findings-emnlp.379/

    2020

  17. [24]

    Hinton, Oriol Vinyals, and Jeffrey Dean

    Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. CoRR, abs/1503.02531, 2015

    2015 arXiv

  18. [25]

    Kakade, and Eran Malach

    Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying. In Proceedings of the 41st International Conference on Machine Learning, 2024

    2024

  19. [26]

    Fast inference from transformers via speculative decoding

    Yaniv Leviathan, Matan Kalman, and Yossi Matias. Fast inference from transformers via speculative decoding. In In International Conference on Machine Learning , pages 19274–19286, 2023. PMLR

    2023

  20. [27]

    Jamba: A hybrid transformer-mamba language model, 2024

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Ere...

    2024 arXiv

  21. [28]

    ZeroEval: A unified framework for evaluating language models, 2024

    Bill Yuchen Lin. ZeroEval: A unified framework for evaluating language models, 2024. URL https: //github.com/WildEval/ZeroEval

    2024

  22. [29]

    Longhorn: State space models are amortized online learners

    Bo Liu, Rui Wang, Lemeng Wu, Yihao Feng, Peter Stone, and Qiang Liu. Longhorn: State space models are amortized online learners. arXiv preprint arXiv:2407.14207, 2024

    2024 arXiv

  23. [30]

    ShortGPT: Layers in large language models are more redundant than you expect, 2024

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. ShortGPT: Layers in large language models are more redundant than you expect, 2024

    2024

  24. [31]

    Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, volume 32, 2019

    Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, volume 32, 2019. URL https://proceedings.neurips.cc/ paper_files/paper/2019/file/2c601ad9d2ff9bc8b282670cdd54f69f-Paper.pdf

    2019

  25. [32]

    Compact language models via pruning and knowledge distillation

    Saurav Muralidharan, Sharath Turuvekere Sreenivas, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Compact language models via pruning and knowledge distillation. arXiv preprint arXiv:2407.14679, 2024

    2024 arXiv

  26. [33]

    Bayesian Optimization: Open source constrained global optimization tool for Python, 2014–

    Fernando Nogueira. Bayesian Optimization: Open source constrained global optimization tool for Python, 2014–. URL https://github.com/bayesian-optimization/BayesianOptimization

    2014

  27. [34]

    Infinity instruct

    Beijing Academy of Artificial Intelligence (BAAI). Infinity instruct. arXiv preprint arXiv:2406.XXXX, 2024

    2024

  28. [35]

    Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng

    Bo Peng, Ruichong Zhang, Daniel Goldstein, Eric Alcaide, Xingjian Du, Haowen Hou, Jiaju Lin, Jiaxing Liu, Janna Lu, William Merrill, Guangyu Song, Kaifeng Tan, Saiteja Utpala, Nathan Wilce, Johan S. Wind, Tianyi Wu, Daniel Wuttke, and Christian Zhou-Zheng. RWKV-7 "goose" with ...

    2025 arXiv

  29. [36]

    Compressive transformers for long-range sequence modelling

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, Chloe Hillier, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019

    1911 arXiv

  30. [37]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling

    Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint, 2024. URL https://arxiv.org/abs/2406.07522

    2024 arXiv

  31. [38]

    TAID: Temporally adaptive interpo- lated distillation for efficient knowledge transfer in language models

    Makoto Shing, Kou Misaki, Han Bao, Sho Yokoi, and Takuya Akiba. TAID: Temporally adaptive interpo- lated distillation for efficient knowledge transfer in language models. In The Thirteenth International Con- ference on Learning Representations, 2025. URL https://openreview.net...

    2025

  32. [39]

    Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding

    Hanshi Sun, Zhuoming Chen, Xinyu Yang, Yuandong Tian, and Beidi Chen. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.11912, 2024

    2024 arXiv

  33. [40]

    Transformer layers as painters

    Qi Sun, Marc Pickett, Aakash Kumar Nain, and Llion Jones. Transformer layers as painters. arXiv preprint arXiv:2407.09298, 2024. 12

    2024 arXiv

  34. [41]

    OpenHermes 2.5: An open dataset of synthetic data for generalist LLM assistants, 2023

    Teknium. OpenHermes 2.5: An open dataset of synthetic data for generalist LLM assistants, 2023

    2023

  35. [42]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017

    2017

  36. [44]

    Ke Alexander Wang, Jiaxin Shi, and Emily B. Fox. Test-time regression: a unifying framework for designing sequence models with associative memory, 2025. URL https://arxiv.org/abs/2501. 12352

    2025

  37. [45]

    Parallelizing linear transformers with the delta rule over sequence length

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. In Proceedings of NeurIPS, 2024

    2024

  38. [46]

    Gated delta networks: Improving Mamba2 with delta rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving Mamba2 with delta rule. In Proceedings of ICLR, 2025

    2025

  39. [47]

    KV cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches

    Jiayi Yuan, Hongyi Liu, Shaochen Zhong, Yu-Neng Chuang, Songchen Li, Guanchu Wang, Duy Le, Hongye Jin, Vipin Chaudhary, Zhaozhuo Xu, Zirui Liu, and Xia Hu. KV cache compression, but what must we give in return? a comprehensive benchmark of long context capable approaches. In T...

    2024

  40. [48]

    Root mean square layer normalization

    Biao Zhang and Rico Sennrich. Root mean square layer normalization. Advances in Neural Information Processing Systems, 32, 2019

    2019

  41. [49]

    w/ Mamba

    Jun Zhang, Jue Wang, Huan Li, Lidan Shou, Ke Chen, Gang Chen, and Sharad Mehrotra. Draft & verify: Lossless large language model acceleration via self-speculative decoding.arXiv preprint arXiv:2309.08168, 2023. 13 Table 7: Evaluation on LongBench and passkey retrieval for self...

    2023 arXiv

  42. [398]

    URL https://aclanthology.org/2020.emnlp-main.398/

    2020

  43. [2024]

    doi: 10.18653/v1/2024.acl-long.172

    Association for Computational Linguistics. doi: 10.18653/v1/2024.acl-long.172. URL https: //aclanthology.org/2024.acl-long.172

    2024 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.