Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DSLA-Serve converts Transformer layers to dual-state linear attention on the fly, and on an augmented Azure trace this cuts per-token latency by 2.29x while keeping accuracy near the teacher.

desk verdict A promising dual-state linear attention plus adaptive distillation framework, but the accuracy table contains an internal inconsistency that currently undermines the headline 'comparable performance' claim. read the letter →

arxiv 2506.09316 v3 pith:Q7O7QN5W submitted 2025-06-11 cs.LG

Yeonju Ro , Zhenyu Zhang , Souvik Kundu , Zhangyang Wang , Aditya Akella This is my paper · ORCID
classification cs.LG
keywords dual-statelinearattentionknowledgedistillationinferenceservinglong-contextLLMsKVcacheadaptiverecencybias
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DSLA-Serve claims that a 7B Transformer can be converted, layer by layer, into a linear-attention model during serving, without retraining the teacher, and that the result is both faster and nearly as accurate. The conversion uses dual-state linear attention (DSLA), a module with two gated hidden states: one initialized to favor distant history and one to favor recent tokens, plus a learned per-layer weight balancing them. On an augmented Azure trace, DSLA-Serve cuts per-token end-to-end latency by 2.29x versus Llama2-7B and 3.0x versus Zamba-7B, while retaining comparable scores on long-context QA, commonsense reasoning, and summarization. If correct, this gives serving systems a practical accuracy-memory knob that can be turned mid-request.

What carries the argument

The load-bearing object is the dual-state linear attention module and the serving policy around it. Each DSLA layer keeps two recurrent hidden states with data-dependent forget gates, one biased toward history and one toward recency, and a per-layer learnable coefficient $\gamma$ that blends their outputs, so a single layer can cover both far and near context at $O(T)$ cost. The other half of the machinery is DSLA-Serve: an offline phase ranks Transformer layers by attention entropy, treating low entropy as low sensitivity to linearization, then distills layers one at a time in that order with chained fine-tuning, committing each converted layer before the next; at runtime, the system converts layers from the least-sensitive end whenever memory pressure rises, stopping when quality is at risk.

What would settle it

Run DSLA-Serve at 50–75% conversion on a long-context retrieval benchmark with 16K+ tokens where evidence sits in the first quarter of the context, and compare accuracy against Llama2-7B. A sharp drop at these lengths would contradict the claim that the history state actually preserves old context, since the paper's evaluations stop near 8K tokens.

Watch

Extended reading notes

Core claim

The paper's central claim is that the reason distilled linear-attention models lose accuracy is a single compressed state that forgets older tokens, and that this can be fixed by giving each attention layer two specialized states. DSLA maintains $S^1_t = G^1_t \odot S^1_{t-1} + k_t^\top v_t$ and $S^2_t = G^2_t \odot S^2_{t-1} + k_t^\top v_t$, with the history gate $G^1$ initialized near the identity and the recency gate $G^2$ initialized randomly; the output blends them as $o_t = q_t(\gamma S^1_t + (1-\gamma)S^2_t)$, with $\gamma$ learned per layer. The paper claims that with a contrastive penalty that pushes the two gates apart and KL distillation that aligns outputs with self-attention, the two states specialize—one holding long-range context and one tracking recent tokens—so that partial or full replacement of Transformer layers preserves quality while removing the KV cache. On an augmented Azure serving trace the adaptive conversion yields a 2.29x per-token latency reduction against Llama2-7B and 3.0x against Zamba-7B, with downstream accuracy comparable to the Transformer teacher at 25% and 50% conversion.

Load-bearing premise

The claim rests on the empirical assumption that two hidden states—a history gate initialized near the identity and a recency gate initialized randomly—after contrastive regularization and 1.6B-token distillation can cover the attention patterns of all 32 layers well enough that converting up to half of them does not hurt downstream quality.

Editorial extensions

If this is right

  • With 25% of layers converted, DSLA matches or beats Llama2-7B on several long-context QA benchmarks while cutting KV cache memory by 1–2 GB per request at 4K–8K context.
  • Converting 50% of layers still outperforms single-state linear models such as GLA-7B and Mamba-7B on long-context QA, while keeping commonsense-reasoning averages close to the teacher.
  • Because converted layers drop their KV cache, decoding latency stops growing with generated length and memory-allocation stalls shrink, which the paper cites as the main source of the measured speedup.
  • The chained fine-tuning procedure means all intermediate conversion depths—25%, 50%, and beyond—share consistent weights, so the runtime can switch conversion levels without retraining.
  • The attention-entropy ranking gives a deployable ordering rule: in serviced workloads, the method can decide on the fly how far to convert before quality SLOs are threatened.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural generalization the paper leaves implicit is to apply the dual-state design to other single-state sequence models: Mamba, RetNet, or any gated recurrence could gain a history-explicit state initialized toward identity, with the same contrastive penalty.
  • The runtime conversion policy could be made adaptive at a finer grain than prompt length: instead of fixed conversion caps per length bucket, an online controller could estimate the accuracy-efficiency Pareto frontier from the live request mix and adjust conversion depth continuously.
  • Because the paper loads both the Transformer and DSLA weight copies simultaneously, the stated memory savings are partially offset by extra weight memory; offloading or prefetching the inactive layer weights would be the direct test of whether the net memory win survives in production.
  • The reported 2.29x speedup is measured on a trace where Transformer latency suffers intermittent cudaMalloc stalls; a system with preallocated memory pools might see a smaller end-to-end gain, so the architecture's true advantage is best isolated at long prefill lengths.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes dual-state linear attention (DSLA), an extension of gated linear attention with two hidden states intended to preserve historical and recency contexts, and DSLA-Serve, an inference-time framework that progressively converts Transformer self-attention layers to DSLA layers based on sensitivity ordering and chained fine-tuning. The main empirical claims are that DSLA-Serve achieves a 2.29x end-to-end speedup over Llama2-7B and a 3.0x speedup over Zamba-7B while retaining comparable accuracy on long-context QA, commonsense reasoning, and summarization. The paper also reports ablations on the number of states, sensitivity metrics, and applications to different teacher scales (7B, 8B, 1.5B) and includes a limitation discussion of the memory overhead of loading both architectures.

Significance. If the results hold, the central idea is valuable: it offers a dynamic accuracy-efficiency trade-off for LLM serving without retraining the whole model, and the dual-state design with contrastive regularization is simple and interpretable. The paper deserves credit for providing a code link, multiple ablations (number of states, sensitivity metrics, different teachers/scales), and a practical latency-fluctuation analysis (Section 5.3, Appendix E). The chained fine-tuning procedure is a sound approach to train-test consistency. However, the significance is conditional: the internal inconsistencies in the baseline accuracy tables and in the speedup arithmetic currently undermine the two pillars of the paper's central claim.

major comments (4)
  1. [Table 1, Table 6, Section 5.1] Table 1 reports Llama2-7B at HotpotQA 5.63, 2WikiMQA 10.24, and TriviaQA 6.19, whereas Table 6 reports Llama2-7B-chat at 29.79, 27.15, and 86.11 on the same tasks. The main-text DSLA[25%] model, distilled from the base model, scores 11.07, 14.20, and 87.46, exceeding its teacher by up to 81.27 accuracy points. No explanation is given for the 80-point gap between base and chat baselines or for a 1.6B-token distilled model surpassing its teacher on multiple QA benchmarks. Because the 'comparable performance' claim rests on these tables, the authors must provide exact evaluation details (prompt template, decoding hyperparameters, metric) and corrected baseline numbers; as reported, Table 1 is internally inconsistent.
  2. [Section 5.5, Introduction/Abstract, Table 4] The paper claims in the abstract and introduction that DSLA-Serve is 3.0x faster than Zamba-7B and 2.29x faster than Llama2-7B, while Section 5.5 states that Zamba-7B is 3.0x slower than Llama2-7B. With L as Llama2 latency, Z=3L, and D=Z/3=L, these two speedups are mutually exclusive; additionally, the Zamba measurement is on A100 while the 2.29x end-to-end result in Table 4 is on A6000. The authors should report raw latencies for Llama2-7B, Zamba-7B, and DSLA-Serve under identical hardware and decoding settings and reconcile the numbers.
  3. [Section 5.4, Table 4] The end-to-end trace experiment reports only latency before/after (93.64 ms to 40.83 ms) and the conversion rate schedule; it does not report any accuracy metric on the replayed requests or a link to offline benchmark accuracy at the applied conversion rates. The sentence 'while maintaining performance' is therefore unsupported by the data in that section.
  4. [Section 4.2, Figures 5-6] The paper interprets the learned gamma values and gate similarity as evidence that the two states specialize to history and recency. Because gamma, the gate initializations, and the contrastive penalty are all fitted during the same distillation, Figures 5-6 describe the fit rather than test the specialization claim. An out-of-sample test (e.g., evaluating attention behavior on unseen tasks, or ablating the contrastive term and the initialization while keeping the two-state architecture) would be needed to support the causal claim that the dual-state design is responsible for the observed separation.
minor comments (5)
  1. [Equation (11)] The expression Entropy(A) = sum A_{T,i} log A_{T,i} is written without a minus sign; as written it is negative entropy, so the statement 'lower entropy' should be clarified.
  2. [Table 7, Section 5.5] The text says 'We report the standard deviation of our measurements for reference,' but only the DSLA row in Table 7 includes error bars; the other baselines do not, so the sentence is misleading.
  3. [Appendix B] Generation hyperparameters (temperature, top-p, max new tokens) for Tables 1, 2, and 6 are not reported; without them the benchmark numbers are not reproducible.
  4. [Section 5.1] The phrase 'up to a 72.23% performance improvement compared to Mamba-7B on TriviaQA' appears to describe an absolute accuracy difference, not a relative percentage improvement; please state which one is meant.
  5. [Table 4] The conversion rates (12.5%, 25%, 37.5%, 50%) are presented without the rule that maps current load to a specific conversion rate; describe the threshold function used by DSLA-Serve.

Circularity Check

1 steps flagged · score 4.0 of 10

Secondary 'history vs recency specialization' claim is partly definitional (identity-initialized history gate plus contrastive loss); central accuracy and latency claims are external benchmark measurements and are not circular.

  1. self definitional [Section 4.1 (Eqs. 4-5), Section 4.2 (Eq. 9), Section 5.5 (Figs. 5-6)]
    ""By default, we initialize G1 t (the “history” gate) to be closer to the identity matrix, helping preserve older context, while G2 t (the “recency” gate) is randomly initialized (e.g., from N(0, σ2)) to have a broader forgetting effect." ... "Unlike the single-state GLA (Fig.1), the recency state and history state in DSLA are specialized to attend to different regions of the input.""

    The history/recency specialization is installed by construction rather than discovered. With G1 initialized near identity, S1_t = G1_t ⊙ S_{t-1} + k_t^T v_t accumulates past tokens with little decay, while a randomly initialized G2 forgets and therefore emphasizes recent tokens; L_cont = sim(G1,G2) then explicitly forces the gates apart. Observing in Figure 5 that S1 retains earlier tokens and S2 highlights recent context is reading back Eqs. (4)-(5), the initialization, and the contrastive penalty. The abstract's claim that 'DSLA's dual states capture both global and local dependencies' is thus substantially a restatement of the design, not an independent test. The circularity is partial: the layer-dependent gamma pattern (Fig.

full rationale

The main claimed results—2.29x/3.0x end-to-end speedups and 'comparable' accuracy (Tables 1-4)—are measurements on external benchmarks (LongBench-style QA, lm-eval, CNN/DailyMail, XSum, and the augmented Azure trace), not derived predictions that reduce to the model's fitted parameters. The sensitivity ordering is validated against measured perplexity impact (Fig. 7), and Table 5 is an external ablation on CoLA; neither is a fitted parameter renamed as a prediction. The only definitional element is the secondary specialization story: because G1 starts near identity and G2 is random, and the contrastive loss forces the gates apart, the observed history-vs-recency split in Figures 5-6 is substantially built into the construction. I therefore flag one self-definitional step but keep the score at 4 rather than 6+, since the central accuracy and latency claims are independent benchmark evidence. The self-citation (Ro et al., 2024, Appendix F) is not load-bearing. The serious internal inconsistencies noted by the skeptical reader—Table 1 vs Table 6 Llama2-7B scores and the Zamba speedup arithmetic—are correctness/validity concerns, not circularity under the rubric, and the Limitations section's admissions (loaded both architectures, degradation beyond 75% conversion) do not reveal a circular derivation chain.

Assumptions & free parameters 6 free parameters · 5 assumptions · 1 invented entities

The central system is empirical and rests on a mix of standard recurrence math, design choices fitted to validation (gamma, lambda, gate initializations, conversion thresholds), and workload assumptions (trace replay). No formal proof links the dual-state design to preserved accuracy; support comes only from the reported benchmarks.

free parameters (6)
  • per-layer blend coefficient gamma = learned per layer, roughly 0.3 to 0.7 (Figure 6)
    Determines the mix of history vs recency states in each layer; fitted to the teacher via distillation and then interpreted as evidence of layer specialization.
  • contrastive loss weight lambda = not stated explicitly
    Hyperparameter controlling how strongly the two gate matrices are pushed apart; affects whether the two states specialize and the final accuracy.
  • history gate initialization scale = close to identity, no numeric sigma given
    Initialization choice intended to preserve historical context; a design assumption that is not derived.
  • recency gate random initialization sigma = N(0, sigma^2), sigma not specified
    Initialization choice for the recency gate; influences specialization and final performance.
  • number of hidden states for DSLA = 2 (Table 5 ablation)
    Set to two based on diminishing returns observed on CoLA; the paper states that more than two states gives no sizable gain.
  • per-bucket maximum conversion rates = 12.5%, 25%, 37.5%, 50% by prompt length bucket (Table 4)
    Chosen for the replay trace; these rates directly set the reported 2.29x latency improvement and are not validated on a held-out trace.
assumptions (5)
  • standard math GLA recurrence equations (1)-(3) correctly model linear attention outputs.
    The DSLA update and attention score decomposition inherit this recurrence; no proof is given beyond prior GLA work.
  • domain assumption Per-layer distillation plus chained fine-tuning transfers to any partial layer subset at inference.
    Assumed in Section 4.4; the paper does not test all 2^32 subsets, only 25% and 50% conversions.
  • domain assumption Attention entropy is a valid sensitivity proxy across tasks and workloads.
    Used to rank layers in Section 4.3; validated only on WikiText-2 perplexity and limited tasks in Appendix F.
  • domain assumption The augmented Azure trace replay and session synthesis (Poisson lambda=3) faithfully reproduce production serving load.
    The 2.29x end-to-end speedup is measured on this augmented trace, not on a real deployment.
  • ad hoc to paper Two hidden states suffice to capture long-range and local attention.
    Justified empirically in Section 4.2 and Table 5 with diminishing returns; no theoretical argument is given.
invented entities (1)
  • Dual-state memory S1 (history) and S2 (recency)
    purpose: Provide fixed-size memory that preserves long-range context while also tracking recent tokens, replacing the KV cache in converted layers.
    The specialization of the two states is demonstrated only through the authors' own fitted model and internal ablations (Figures 5-6, Table 5); there is no external falsifiable prediction tied to the entity outside this paper.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention." pith.science (2026). https://pith.science/paper/Q7O7QN5W

@misc{pith2026250609316,
  author       = {Pith},
  title        = {Pith review of: On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q7O7QN5W}},
  note         = {Machine review of arXiv:2506.09316}
}
read the original abstract

Large language models (LLMs) excel at capturing global token dependencies via self-attention but face prohibitive compute and memory costs on lengthy inputs. While sub-quadratic methods (e.g., linear attention) can reduce these costs, they often degrade accuracy due to overemphasizing recent tokens. In this work, we first propose dual-state linear attention (DSLA), a novel design that maintains two specialized hidden states-one for preserving historical context and one for tracking recency-thereby mitigating the short-range bias typical of linear-attention architectures. To further balance efficiency and accuracy under dynamic workload conditions, we introduce DSLA-Serve, an online adaptive distillation framework that progressively replaces Transformer layers with DSLA layers at inference time, guided by a sensitivity-based layer ordering. DSLA-Serve uses a chained fine-tuning strategy to ensure that each newly converted DSLA layer remains consistent with previously replaced layers, preserving the overall quality. Extensive evaluations on commonsense reasoning, long-context QA, and text summarization demonstrate that DSLA-Serve yields 2.3x faster inference than Llama2-7B and 3.0x faster than the hybrid Zamba-7B, while retaining comparable performance across downstream tasks. Our ablation studies show that DSLA's dual states capture both global and local dependencies, addressing the historical-token underrepresentation seen in prior linear attentions. Codes are available at https://github.com/utnslab/DSLA-Serve.

Figures

Figures reproduced from arXiv: 2506.09316 by the authors.

Figure 2
Figure 2. Illustration of resource usage spikes (memory) over time and across nodes. Long-lived or bursty sessions can heavily strain certain GPUs. multiple nodes. In production-scale deployments, load fluc￾tuations arise due to: • Temporal variability: Sporadic bursts of lengthy re￾quests or multiple concurrent sessions can dramatically increase memory usage when each session accumulates a large KV cache. • Spatial imbalance… view at source ↗
Figure 3
Figure 3. Chained Fine-Tuning. We replace layers in ascending sensitivity order, freezing all other layers. Upon completion of each stage, we commit the newly trained DSLA layer so that subsequent stages see the updated architecture. layers have already been converted. To address this, we adopt a chained fine-tuning approach that ensures consis￾tency between training and deployment architectures. As illustrated in [PITH_FULL… view at source ↗
Figure 4
Figure 4. Prefill (left) and per-token decoding (right) latency. nism’s reliance on past token interactions, resulting in linear cost growth. Conversely, the converted model maintains a constant latency increase by updating only two states for his￾torical and recent information, significantly reducing overall latency. During decoding, we observed an interesting phenomenon: the latency fluctuation is significantly higher for t… view at source ↗
Figures from the paper (9 more)
Figure 5
Figure 5. Figure 5: Attention score analysis. History vs recency per layer. We visualize the γ value used as a weighting factor that determines how the two hidden states contribute to the output (ot = qt(γ·S 1 t +(1−γ)·S 2 t )) in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Hidden states weighting factor visualization. In the figure, the state with the higher value dominates each layer. Specifically, the recency state is predominant in the initial layers (0–3) and the final layers (30–31), whereas the history state takes precedence in the…
Figure 7
Figure 7. Figure 7: Attention entropy as a sensitivity metric. DSLA employs a dual-state mechanism to separately re￾tain historical and recent information. Additionally, DSLA￾Serve enables dynamic adaptation to accuracy-latency trade￾offs. On average, DSLA achieves superior performance. W…
Figure 8
Figure 8. Figure 8: Overview of DSLA-Serve. We provide a visual overview of DSLA-Serve in [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: (a) Number of turns per session follows Poisson distribution (λ = 3). (b) Varying load defined as the number of user requests in real world serving scenario. Across time, the load to the server fluctuates, indicating the need to adaptive de-stressing solution for a ser…
Figure 10
Figure 10. Figure 10: Prompt length distribution D. Serving of Mixed Batch Assume there are four requests waiting in the queue from R1 to R4 ( [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Serving of Mixed Batch [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Latency fluctuation pytorch profiler, we observed that this is due to the cudaMalloc operation, which takes 300ms-1000ms intermittently. This unexpected delay can be mitigated by converting more transformer layers into linear layers, which reduces the memory usage by …
Figure 13
Figure 13. Figure 13: Different possible sensitivity metrics: (a) attention entropy, (b) outlier percentage, and (c) downstream task accuracy. We also observed that attention entropy tends to perform reasonably well across a variety of downstream tasks [PITH_FULL_IMAGE:figures/full_fig_p0…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Key to Going Linear: Analysis-Driven Transformer Linearization

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Delta-rule linear attention faithfully approximates softmax attention through key-dependent rank-1 projections, enabling efficient post-hoc linearization of LLMs up to 32B parameters.

Reference graph

Works this paper leans on

73 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    https://huggingface.co/blog/bamba

    B amba: I nference- E fficient H ybrid M amba2 M odel --- huggingface.co. https://huggingface.co/blog/bamba. [Accessed 05-01-2025]

    work page 2025

  3. [3]

    nvidia.com

    A100, N. nvidia.com. https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf. [Accessed 26-01-2025]

    work page 2025

  4. [4]

    Y., Rajbhandari, S., Awan, A

    Aminabadi, R. Y., Rajbhandari, S., Awan, A. A., Li, C., Li, D., Zheng, E., Ruwase, O., Smith, S., Zhang, M., Rasley, J., et al. Deepspeed-inference: enabling efficient inference of transformer models at unprecedented scale. In SC22: International Conference for High Performance Computing, Networking, Storage and Analysis, pp.\ 1--15. IEEE, 2022

    work page 2022

  5. [5]

    E., and Pedram, M

    Azizi, S., Kundu, S., Sadeghi, M. E., and Pedram, M. Mambaextend: A training-free approach to improve long context extension of mamba. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=LgzRo1RpLS

    work page 2025

  6. [6]

    L ong B ench: A bilingual, multitask benchmark for long context understanding

    Bai, Y., Lv, X., Zhang, J., Lyu, H., Tang, J., Huang, Z., Du, Z., Liu, X., Zeng, A., Hou, L., Dong, Y., Tang, J., and Li, J. L ong B ench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 3119--3137, Bangkok, Thailand, A...

    doi:10.18653/v1/2024.acl-long.172 2024

  7. [7]

    E., and Cohan, A

    Beltagy, I., Peters, M. E., and Cohan, A. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  8. [8]

    Decimamba: Exploring the length extrapolation potential of mamba

    Ben-Kish, A., Zimerman, I., Abu-Hussein, S., Cohen, N., Globerson, A., Wolf, L., and Giryes, R. Decimamba: Exploring the length extrapolation potential of mamba. arXiv preprint arXiv:2406.14528, 2024

    arXiv 2024

Show all 73 references
  1. [9]

    Y., Xing, E

    Bick, A., Li, K. Y., Xing, E. P., Kolter, J. Z., and Gu, A. Transformers to ssms: Distilling quadratic knowledge to subquadratic models, 2024. URL https://arxiv.org/abs/2408.10189

    2024 arXiv

  2. [10]

    Read-me: Refactorizing llms as router-decoupled mixture of experts with system co-design

    Cai, R., Ro, Y., Kim, G.-W., Wang, P., Ehteshami Bejnordi, B., Akella, A., Wang, Z., et al. Read-me: Refactorizing llms as router-decoupled mixture of experts with system co-design. Advances in Neural Information Processing Systems, 37: 0 116126--116148, 2024 a

    2024

  3. [11]

    D., Chen, D., and Dao, T

    Cai, T., Li, Y., Geng, Z., Peng, H., Lee, J. D., Chen, D., and Dao, T. Medusa: Simple llm inference acceleration framework with multiple decoding heads. arXiv preprint arXiv:2401.10774, 2024 b

    2024 arXiv

  4. [12]

    Generating long sequences with sparse transformers

    Child, R., Gray, S., Radford, A., and Sutskever, I. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019

    1904 arXiv

  5. [13]

    Rethinking attention with performers

    Choromanski, K., Likhosherstov, V., Dohan, D., Song, X., Gane, A., Sarlos, T., Hawkins, P., Davis, J., Mohiuddin, A., Kaiser, L., et al. Rethinking attention with performers. arXiv preprint arXiv:2009.14794, 2020

    2009 arXiv

  6. [14]

    B., Bierbaum, M., O'Keeffe, K

    Clement, C. B., Bierbaum, M., O'Keeffe, K. P., and Alemi, A. A. On the use of arxiv as a dataset, 2019

    2019

  7. [15]

    and Gu, A

    Dao, T. and Gu, A. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. arXiv preprint arXiv:2405.21060, 2024

    2024 arXiv

  8. [16]

    Y., Ermon, S., Rudra, A., and Ré, C

    Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Ré, C. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022. URL https://arxiv.org/abs/2205.14135

    2022 arXiv

  9. [17]

    and Mazur, D

    Eliseev, A. and Mazur, D. Fast inference of mixture-of-experts language models with offloading. arXiv preprint arXiv:2312.17238, 2023

    2023 arXiv

  10. [18]

    Stack exchange data dump, 2024

    Excahnge, S. Stack exchange data dump, 2024. URL https://archive.org/details/stackexchange

    2024

  11. [19]

    Wikimedia downloads

    Foundation, W. Wikimedia downloads. URL https://dumps.wikimedia.org

  12. [20]

    and Alistarh, D

    Frantar, E. and Alistarh, D. Sparsegpt: Massive language models can be accurately pruned in one-shot. In International Conference on Machine Learning, pp.\ 10323--10337. PMLR, 2023

    2023

  13. [21]

    Gptq: Accurate post-training quantization for generative pre-trained transformers

    Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323, 2022

    2022 arXiv

  14. [22]

    The P ile: An 800gb dataset of diverse text for language modeling

    Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The P ile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020

    2020 arXiv

  15. [23]

    Zamba: A compact 7b ssm hybrid model

    Glorioso, P., Anthony, Q., Tokpanov, Y., Whittington, J., Pilault, J., Ibrahim, A., and Millidge, B. Zamba: A compact 7b ssm hybrid model. arXiv preprint arXiv:2405.16712, 2024

    2024 arXiv

  16. [24]

    and Dao, T

    Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  17. [25]

    N V I D I A H 100 T ensor C ore G P U D atasheet --- resources.nvidia.com

    H100, N. N V I D I A H 100 T ensor C ore G P U D atasheet --- resources.nvidia.com. https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet. [Accessed 26-01-2025]

    2025

  18. [26]

    M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P

    Hermann, K. M., Kocisky, T., Grefenstette, E., Espeholt, L., Kay, W., Suleyman, M., and Blunsom, P. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015

    2015

  19. [27]

    Distilling the knowledge in a neural network

    Hinton, G. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  20. [28]

    What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics, 2019

    Jawahar, G., Sagot, B., and Seddah, D. What does bert learn about the structure of language? In ACL 2019-57th Annual Meeting of the Association for Computational Linguistics, 2019

    2019

  21. [29]

    Kasai, J., Peng, H., Zhang, Y., Yogatama, D., Ilharco, G., Pappas, N., Mao, Y., Chen, W., and Smith, N. A. Finetuning pretrained transformers into rnns. arXiv preprint arXiv:2103.13076, 2021

    2021 arXiv

  22. [30]

    Transformers are rnns: Fast autoregressive transformers with linear attention

    Katharopoulos, A., Vyas, A., Pappas, N., and Fleuret, F. Transformers are rnns: Fast autoregressive transformers with linear attention. In International conference on machine learning, pp.\ 5156--5165. PMLR, 2020

    2020

  23. [31]

    Matformer: Nested transformer for elastic inference

    Kudugunta, S., Kusupati, A., Dettmers, T., Chen, K., Dhillon, I., Tsvetkov, Y., Hajishirzi, H., Kakade, S., Farhadi, A., Jain, P., et al. Matformer: Nested transformer for elastic inference. arXiv preprint arXiv:2310.07707, 2023

    arXiv

  24. [32]

    H., Gonzalez, J., Zhang, H., and Stoica, I

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pp.\ 611--626, 2023 a

    2023

  25. [33]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention, 2023 b . URL https://arxiv.org/abs/2309.06180

    2023 arXiv

  26. [34]

    Fast inference from transformers via speculative decoding

    Leviathan, Y., Kalman, M., and Matias, Y. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pp.\ 19274--19286. PMLR, 2023

    2023

  27. [35]

    Li, Y., Bubeck, S., Eldan, R., Del Giorno, A., Gunasekar, S., and Lee, Y. T. Textbooks are all you need ii: phi-1.5 technical report. arXiv preprint arXiv:2309.05463, 2023

    2023 arXiv

  28. [36]

    Snapkv: Llm knows what you are looking for before generation

    Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: Llm knows what you are looking for before generation. arXiv preprint arXiv:2404.14469, 2024

    2024 arXiv

  29. [37]

    Jamba: A hybrid transformer-mamba language model

    Lieber, O., Lenz, B., Bata, H., Cohen, G., Osin, J., Dalmedigos, I., Safahi, E., Meirom, S., Belinkov, Y., Shalev-Shwartz, S., et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024

    2024 arXiv

  30. [38]

    Rouge: A package for automatic evaluation of summaries

    Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.\ 74--81, 2004

    2004

  31. [39]

    Awq: Activation-aware weight quantization for on-device llm compression and acceleration

    Lin, J., Tang, J., Tang, H., Yang, S., Chen, W.-M., Wang, W.-C., Xiao, G., Dang, X., Gan, C., and Han, S. Awq: Activation-aware weight quantization for on-device llm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 0 87--100, 2024

    2024

  32. [40]

    Fantastic semantics and where to find them: Investigating which layers of generative llms reflect lexical semantics

    Liu, Z., Kong, C., Liu, Y., and Sun, M. Fantastic semantics and where to find them: Investigating which layers of generative llms reflect lexical semantics. arXiv preprint arXiv:2403.01509, 2024 a

    2024 arXiv

  33. [41]

    Mobilellm: Optimizing sub-billion parameter language models for on-device use cases

    Liu, Z., Zhao, C., Iandola, F., Lai, C., Tian, Y., Fedorov, I., Xiong, Y., Chang, E., Shi, Y., Krishnamoorthi, R., et al. Mobilellm: Optimizing sub-billion parameter language models for on-device use cases. arXiv preprint arXiv:2402.14905, 2024 b

    2024 arXiv

  34. [42]

    Decoupled weight decay regularization

    Loshchilov, I. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  35. [43]

    Llm-pruner: On the structural pruning of large language models

    Ma, X., Fang, G., and Wang, X. Llm-pruner: On the structural pruning of large language models. Advances in neural information processing systems, 36: 0 21702--21720, 2023

    2023

  36. [44]

    Linearizing large language models

    Mercat, J., Vasiljevic, I., Keh, S., Arora, K., Dave, A., Gaidon, A., and Kollar, T. Linearizing large language models. arXiv preprint arXiv:2405.06640, 2024

    2024 arXiv

  37. [45]

    B., and Lapata, M

    Narayan, S., Cohen, S. B., and Lapata, M. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.08745, 2018

    2018 arXiv

  38. [46]

    Splitwise: Efficient generative llm inference using phase splitting

    Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, \'I ., Maleki, S., and Bianchini, R. Splitwise: Efficient generative llm inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA), pp.\ 118--132. IEEE, 2024

    2024

  39. [47]

    Rwkv: Reinventing rnns for the transformer era

    Peng, B., Alcaide, E., Anthony, Q., Albalak, A., Arcadinho, S., Biderman, S., Cao, H., Cheng, X., Chung, M., Grella, M., et al. Rwkv: Reinventing rnns for the transformer era. arXiv preprint arXiv:2305.13048, 2023

    2023 arXiv

  40. [48]

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv e-prints, 2019

    2019

  41. [49]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling

    Ren, L., Liu, Y., Lu, Y., Shen, Y., Liang, C., and Chen, W. Samba: Simple hybrid state space models for efficient unlimited context language modeling. arXiv preprint arXiv:2406.07522, 2024

    2024 arXiv

  42. [50]

    Optimizing transformer inference with selective distillation: Layerwise conversion to linear attention

    Ro, Y., Zhang, Z., Chidambaram, V., and Akella, A. Optimizing transformer inference with selective distillation: Layerwise conversion to linear attention. 2nd Workshop on Hot Topics in System Infrastructure (HotInfra 2024), 2024

    2024

  43. [51]

    J., and Manning, C

    See, A., Liu, P. J., and Manning, C. D. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXiv:1704.04368, 2017

    2017 arXiv

  44. [52]

    Slimpajama-dc: Understanding data combinations for llm training, 2024

    Shen, Z., Tao, T., Ma, L., Neiswanger, W., Liu, Z., Wang, H., Tan, B., Hestness, J., Vassilieva, N., Soboleva, D., and Xing, E. Slimpajama-dc: Understanding data combinations for llm training, 2024. URL https://arxiv.org/abs/2309.10818

    2024 arXiv

  45. [53]

    Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding

    Sun, H., Chen, Z., Yang, X., Tian, Y., and Chen, B. Triforce: Lossless acceleration of long sequence generation with hierarchical speculative decoding. arXiv preprint arXiv:2404.11912, 2024

    2024 arXiv

  46. [54]

    Sun, M., Liu, Z., Bair, A., and Kolter, J. Z. A simple and effective pruning approach for large language models. arXiv preprint arXiv:2306.11695, 2023 a

    2023 arXiv

  47. [55]

    Retentive network: A successor to transformer for large language models

    Sun, Y., Dong, L., Huang, S., Ma, S., Xia, Y., Xue, J., Wang, J., and Wei, F. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023 b

    2023 arXiv

  48. [56]

    Quest: Query-aware sparsity for efficient long-context llm inference

    Tang, J., Zhao, Y., Zhu, K., Xiao, G., Kasikci, B., and Han, S. Quest: Query-aware sparsity for efficient long-context llm inference. arXiv preprint arXiv:2406.10774, 2024

    2024 arXiv

  49. [57]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  50. [58]

    Attention is all you need

    Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017

    2017

  51. [59]

    An empirical study of mamba-based language models, 2024

    Waleffe, R., Byeon, W., Riach, D., Norick, B., Korthikanti, V., Dao, T., Gu, A., Hatamizadeh, A., Singh, S., Narayanan, D., Kulshreshtha, G., Singh, V., Casper, J., Kautz, J., Shoeybi, M., and Catanzaro, B. An empirical study of mamba-based language models, 2024. URL https://a...

    2024 arXiv

  52. [60]

    Glue: A multi-task benchmark and analysis platform for natural language understanding

    Wang, A. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461, 2018

    2018 arXiv

  53. [61]

    M., and Dao, T

    Wang, J., Paliotta, D., May, A., Rush, A. M., and Dao, T. The mamba in the llama: Distilling and accelerating hybrid models, 2024 a . URL https://arxiv.org/abs/2408.15237

    2024 arXiv

  54. [62]

    Understanding and mitigating bottlenecks of state space models through the lens of recency and over-smoothing

    Wang, P., Cai, R., Wang, Y., Zhu, J., Srivastava, P., Wang, Z., and Li, P. Understanding and mitigating bottlenecks of state space models through the lens of recency and over-smoothing. arXiv preprint arXiv:2501.00658, 2024 b

    2024 arXiv

  55. [63]

    Z., Khabsa, M., Fang, H., and Ma, H

    Wang, S., Li, B. Z., Khabsa, M., Fang, H., and Ma, H. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  56. [64]

    Smoothquant: Accurate and efficient post-training quantization for large language models

    Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp.\ 38087--38099. PMLR, 2023 a

    2023

  57. [65]

    Efficient streaming language models with attention sinks

    Xiao, G., Tian, Y., Chen, B., Han, S., and Lewis, M. Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453, 2023 b

    2023 arXiv

  58. [66]

    Gated linear attention transformers with hardware-efficient training

    Yang, S., Wang, B., Shen, Y., Panda, R., and Kim, Y. Gated linear attention transformers with hardware-efficient training. arXiv preprint arXiv:2312.06635, 2023

    2023 arXiv

  59. [67]

    Flashinfer: Efficient and customizable attention engine for llm inference serving

    Ye, Z., Chen, L., Lai, R., Lin, W., Zhang, Y., Wang, S., Chen, T., Kasikci, B., Grover, V., Krishnamurthy, A., and Ceze, L. Flashinfer: Efficient and customizable attention engine for llm inference serving. arXiv preprint arXiv:2501.01005, 2025. URL https://arxiv.org/abs/2501.01005

    2025 arXiv

  60. [68]

    S., Kim, G.-W., Kim, S., and Chun, B.-G

    Yu, G.-I., Jeong, J. S., Kim, G.-W., Kim, S., and Chun, B.-G. Orca: A distributed serving system for \ Transformer-Based \ generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pp.\ 521--538, 2022

    2022

  61. [69]

    A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al

    Zaheer, M., Guruganesh, G., Dubey, K. A., Ainslie, J., Alberti, C., Ontanon, S., Pham, P., Ravula, A., Wang, Q., Yang, L., et al. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33: 0 17283--17297, 2020

    2020

  62. [70]

    Lolcats: On low-rank linearizing of large language models

    Zhang, M., Arora, S., Chalamala, R., Wu, A., Spector, B., Singhal, A., Ramesh, K., and R \'e , C. Lolcats: On low-rank linearizing of large language models. arXiv preprint arXiv:2410.10254, 2024 a

    2024 arXiv

  63. [71]

    The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry

    Zhang, M., Bhatia, K., Kumbong, H., and R \'e , C. The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry. arXiv preprint arXiv:2402.04347, 2024 b

    2024 arXiv

  64. [72]

    H2o: Heavy-hitter oracle for efficient generative inference of large language models

    Zhang, Z., Sheng, Y., Zhou, T., Chen, T., Zheng, L., Cai, R., Song, Z., Tian, Y., R \'e , C., Barrett, C., et al. H2o: Heavy-hitter oracle for efficient generative inference of large language models. Advances in Neural Information Processing Systems, 36: 0 34661--34710, 2023

    2023

  65. [73]

    P., Zhang, H., Gonzalez, J

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.