Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models

T0 review · 2 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A hybrid state space model can expand its eidetic memory span by reserving part of its attention context for tokens retrieved by relevance rather than recency.

desk verdict Useful efficient fine-tuning recipe for hybrid SSMs, but the eidetic-memory-at-inference claim is not supported by the experiments. read the letter →

arxiv 2412.13328 v2 pith:YKTIGY5M submitted 2024-12-17 cs.CL cs.LG

Elvis Nunez , Luca Zancato , Benjamin Bowman , Aditya Golatkar , Wei Xia , Stefano Soatto This is my paper · ORCID
classification cs.CLcs.LG
keywords hybridstatespacemodelsspan-expandedattentioneideticmemoryrelevance-basedretrievallong-contextadaptationLoRAfine-tuningRULERbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Hybrid state space models combine a recurrent 'fading' memory with finite-window attention, so tokens beyond the attention span are only accessible through lossy compression. This paper argues that memory allocation should follow relevance, not recency: reserve a fraction of the attention context for tokens retrieved from arbitrarily far back, and let the model decide which past blocks matter for the current query. The proposed mechanism, Span-Expanded Attention (SE-Attn), together with a hybrid-specific LoRA variant (HyLoRA), lets a pre-trained 2.7B hybrid model be adapted to sequences up to eight times its pre-training length with training cost comparable to sliding-window attention and better long-context accuracy than sliding-window and shifted-sparse attention baselines. A reader would care because it offers a cheap path to long-context recall in memory-efficient architectures without extra hardware or full-length attention during training.

What carries the argument

The load-bearing object is the 'expansion span': a reserved fraction of each attention chunk's context, populated not by the most recent tokens but by the top-k 'memory blocks' retrieved from the past. Each memory block is summarized by a single vector c_j, the average of a non-causal self-attention output over the block; a chunk's queries score all blocks by a cross-attention relevancy score, a causal mask blocks future blocks, and softmax plus top-k selects which blocks enter the expansion span. This keeps the per-chunk attention computation at a fixed size, a few hundred retrieved tokens plus the chunk itself, while giving the layer access to tokens from arbitrarily far back. It is trained end-to-end with HyLoRA, which adapts attention projections with low-rank adapters and also trains the 1D convolutions inside SSM layers.

What would settle it

Fine-tune the same hybrid model with SE-Attn, with SE-Attn-Random (random memory-block retrieval), and with SE-Attn-NoMem (no retrieval), keeping HyLoRA and all training details identical, then evaluate all three with full attention on RULER at 16k tokens: if relevance-based retrieval does not clearly beat random retrieval, the paper's claim that retrieval quality drives the expanded memory span is unsupported.

Watch

Extended reading notes

Core claim

The paper's central claim is that a hybrid SSM's eidetic (verbatim) memory span can be expanded by making a small part of each attention layer's context a 'reserve' populated through retrieval rather than recency. SE-Attn splits the input into chunks and, for each chunk, computes relevancy scores between the chunk's queries and compressed summaries of past memory blocks, retrieves the top-k blocks, and concatenates their keys and values into the chunk's attention context. Because the model is trained with this retrieval pattern, it learns to allocate memory where it matters; at inference the authors revert to full attention, which benefits from the retrieval-trained weights. The authors report that on RULER, LM Harness long-context tasks, and LongBench, models fine-tuned this way match full-attention fine-tuning and beat sliding-window and shifted-sparse attention, and that HyLoRA, which augments LoRA+ by training the small 1D convolutional layers inside SSM blocks, is what makes the adaptation work.

Load-bearing premise

All headline results are obtained by fine-tuning with SE-Attn but evaluating with full attention, so the load-bearing premise is that retrieval during training alone, not the fine-tuning protocol or the adapted convolutional layers, is what creates the expanded recall; if that transfer fails, the central eidetic-expansion claim is not established.

Editorial extensions

If this is right

  • Pre-trained hybrid SSMs with short 2k contexts can be adapted to sequences up to 8 times longer without expensive full-length fine-tuning.
  • SE-Attn offers a runtime-memory trade-off comparable to sliding-window attention while giving better long-context task accuracy, so relevance-based retrieval can replace recency-based windows without extra hardware.
  • The method transfers beyond the main test model: it improves long-context performance on Llama 1 7B and extends Zamba2-Hybrid's context from 4k to 12k tokens.
  • HyLoRA, which trains the 1D convolution layers in addition to LoRA+ on attention layers, is necessary for strong long-context adaptation of hybrid SSMs; LoRA or LoRA+ alone underperform.
  • Perplexity is not a faithful indicator of long-context recall; retrieval-style benchmarks like RULER reveal differences that perplexity hides.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper evaluates retrieval benefit under a full-attention decoder, a natural next step is to make the retrieval mechanism itself inference-friendly, such as caching retrieved block summaries, to close the gap shown in the appendix between SE-Attn evaluation and full-attention evaluation.
  • The relevance-scoring idea could combine with learned compression tokens or hierarchical summaries, allowing the expansion span to scale sublinearly with history while keeping the retrieval decision cheap.
  • Relevance-based memory allocation may also benefit pure recurrent models or linear-attention models, where the analogous failing is that the state is updated uniformly by recency rather than by importance.
  • Since random retrieval already helps over no retrieval, part of the gain may be a regularizing effect of variable-length chunked training; ablating retrieval quality while holding chunking constant would separate the two effects.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes Span-Expanded Attention (SE-Attn), a sparse attention mechanism that reserves part of the context for past memory blocks retrieved by relevance to the current query, and HyLoRA, a LoRA-based fine-tuning method that also adapts 1D convolutional layers in hybrid SSMs. The authors claim that SE-Attn expands the eidetic memory span of hybrid SSMs and allows efficient fine-tuning of pre-trained models on sequences up to 8 times longer than the pre-training context. The main experiments fine-tune Mamba-2-Hybrid 2.7B, Zamba2 1.2B, and Llama1 7B, evaluating on PG-19, RULER, LM Harness, LongBench, and in-context recall tasks, comparing against Full-Attn, SW-Attn, and S2-Attn.

Significance. If the central claim were established, SE-Attn would be a valuable drop-in mechanism for extending the effective context of hybrid SSMs with modest training cost, and HyLoRA would be a useful recipe for adapting the recurrent and convolutional components of such models. The manuscript has several strengths: it ships a clean ablation chain (retrieval vs. random vs. no retrieval; chunk size; block size/top-k), an empirical runtime-memory analysis, and evaluation across three model families. The internal comparisons are consistent, and the paper is honest about the limitations of perplexity as a long-context metric. However, the headline claim is not validated in the deployed configuration, because the retrieval mechanism is disabled at inference, and the one direct inference-time test shows substantial degradation.

major comments (2)
  1. [§4.1, Appendix A] The experimental protocol replaces SE-Attn with Full-Attn at evaluation: §4.1 states that SE-Attn is used for efficient fine-tuning and evaluation reverts to Full-Attn, and Appendix A repeats this. Consequently, all headline results in Table 1, Figure 2, and Tables 6–9 measure a model whose inference-time attention is standard full attention. The abstract's claim that 'our models can eidetically access tokens from beyond the Attention span' is therefore not supported by these experiments, since the retrieval mechanism—the relevancy scoring and top-k block selection—is not exercised at inference.
  2. [Appendix E.4, Figure 11] The only direct evaluation of SE-Attn at inference, Figure 11(b), shows a clear degradation on the NIAH-Single-1 RULER task compared to evaluating the same checkpoint with Full-Attn. This is load-bearing because it undermines the interpretation that the training-time gains are caused by an expanded eidetic span; the gains could instead come from the HyLoRA fine-tuning protocol or from retrieval acting as a regularizer during training. The authors should either provide SE-Attn-at-inference results across the full RULER suite (and ideally on the other benchmarks) showing competitive performance, or explicitly reframe the contribution as an efficient long-context fine-tuning method rather than an inference-time memory expansion.
minor comments (5)
  1. [Abstract/Introduction] The fact that evaluation reverts to Full-Attn is disclosed only in §4.1 and Appendix A; it should be stated prominently in the abstract or introduction, because the current abstract implies that SE-Attn is the deployed mechanism.
  2. [Appendix E.4, Figure 11] The SE-Attn versus Full-Attn curves in Figure 11 would be easier to interpret with numeric values or error bars, since the differences are small at some sequence lengths and the curves are read only from the plot.
  3. [Table 1 and Figure 2] No variance or statistical significance is reported for the RULER and LM Harness comparisons; given that several gaps between SE-Attn and Full-Attn are under one or two percentage points, reporting standard deviations across seeds would strengthen the claims.
  4. [Appendix J] The abbreviation list contains a typo: 'ScorllsNarrativeQA' should be 'ScrollsNarrativeQA'.
  5. [Appendix E.1] There is a typo in the text: 'perfromance' should be 'performance'.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central claims are empirical benchmark results; the train-with-SE-Attn/evaluate-with-Full-Attn protocol is a validity limitation, not a definitional reduction.

full rationale

I walked the paper's claimed derivation chain. The core proposals (SE-Attn retrieval and HyLoRA) are defined constructively in Section 3, and their benefits are established by benchmark comparisons in Section 4; no equation defines SE-Attn in terms of the RULER or LM-Harness results it is said to predict, and no fitted parameter is renamed as a prediction. The only self-citation of the authors' prior work, Zancato et al. (2024), is used to motivate adapting 1D convolution layers in HyLoRA ('building upon previous observations Zancato et al. (2024); Yang et al. (2024b), we also adapt the 1D convolutional layers'); that design choice is then verified empirically in Figure 2(b) and Appendix B, so the citation is not load-bearing. The main weakness is not circularity: the paper fine-tunes with SE-Attn but evaluates with Full-Attn (Section 4.1 and Appendix A), and Figure 11 shows that evaluating the SE-Attn-fine-tuned checkpoint with SE-Attn on NIAH-Single-1 is worse than evaluating it with Full-Attn. This means the headline 'eidetic access at inference' claim is not validated in the deployed configuration, but that is a scope/protocol limitation rather than the reduction of a derivation to its own inputs. Hyperparameters such as block size, top-k, and chunk sizes were selected on RULER, which is tuning rather than circular prediction. Accordingly, no circular step is exhibited and the circularity score is minimal.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on benchmark measurements, not derivations. There are no free physical parameters, but the method's hyperparameters (block size, top-k, chunking scheme, LoRA rank, learning rates) were tuned on the evaluation benchmark RULER. The method assumes block summaries retain information needed for retrieval, and that relevance scores computed via cross-attention select useful blocks. The evaluation protocol assumes training-time efficient attention transfers to full-attention inference.

free parameters (5)
  • memory block size S = 32
    Selected via RULER ablation (Fig. 3c-3d); larger blocks with smaller top-k perform better.
  • top-k K = 8
    Same ablation; expansion span of 256 tokens (8 blocks x 32) gave the strongest RULER average.
  • chunk size M = sampled uniformly from {2048, 4096}
    Random per-layer chunk sizes outperformed fixed sizes (Fig. 3b); this is a tuned design choice.
  • LoRA rank r and alpha = r=32, alpha=64
    Rank ablation (Fig. 6) shows higher rank helps, with saturation near 64; r=32 was used for Mamba-2-Hybrid.
  • learning rate = 2e-4 for SE-Attn/SW-Attn/Full-Attn; 2e-5 for S2-Attn
    Learning rate per method was tuned; S2 did not improve with 2e-4 (Fig. 13).
assumptions (4)
  • domain assumption Block summaries computed as the mean of non-causal attention outputs preserve enough information for effective retrieval.
    Section 3.3 defines c_j in Equation (4) as the compressed representation; no analysis quantifies information loss or retrieval accuracy.
  • domain assumption Cross-attention relevancy scores between chunk queries and block summaries identify the most useful past blocks.
    Section 3.3 uses Equation (4) to retrieve top-k blocks; the paper does not evaluate retrieval precision/recall or compare against other selection signals.
  • domain assumption Training with an efficient retrieval-based attention transfers to inference with full attention without loss.
    Appendix A and Section 4.1 specify the fine-tune-with-SE-Attn, evaluate-with-Full-Attn protocol; Fig. 11 shows the reverse switch (evaluate with SE-Attn) hurts RULER, so this assumption is load-bearing.
  • domain assumption RULER benchmark scores are a reliable proxy for long-context recall capabilities.
    Section 4.1 selects RULER as the main long-context metric; the authors themselves note RULER is synthetic, and Appendix B argues perplexity is not reliable, making RULER the primary evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models." pith.science (2026). https://pith.science/paper/YKTIGY5M

@misc{pith2026241213328,
  author       = {Pith},
  title        = {Pith review of: Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YKTIGY5M}},
  note         = {Machine review of arXiv:2412.13328}
}
read the original abstract

The "state" of State Space Models (SSMs) represents their memory, which fades exponentially over an unbounded span. By contrast, Attention-based models have "eidetic" (i.e., verbatim, or photographic) memory over a finite span (context size). Hybrid architectures combine State Space layers with Attention, but still cannot recall the distant past and can access only the most recent tokens eidetically. Unlike current methods of combining SSM and Attention layers, we allow the state to be allocated based on relevancy rather than recency. In this way, for every new set of query tokens, our models can "eidetically" access tokens from beyond the Attention span of current Hybrid SSMs without requiring extra hardware resources. We introduce a method to expand the memory span of the hybrid state by "reserving" a fraction of the Attention context for tokens retrieved from arbitrarily distant in the past, thus expanding the eidetic memory span of the overall state. We call this reserved fraction of tokens the "expansion span," and the mechanism to retrieve and aggregate it "Span-Expanded Attention" (SE-Attn). To adapt Hybrid models to using SE-Attn, we propose a novel fine-tuning method that extends LoRA to Hybrid models (HyLoRA) and allows efficient adaptation on long spans of tokens. We show that SE-Attn enables us to efficiently adapt pre-trained Hybrid models on sequences of tokens up to 8 times longer than the ones used for pre-training. We show that HyLoRA with SE-Attn is cheaper and more performant than alternatives like LongLoRA when applied to Hybrid models on natural language benchmarks with long-range dependencies, such as PG-19, RULER, and other common natural language downstream tasks.

Figures

Figures reproduced from arXiv: 2412.13328 by the authors.

Figure 1
Figure 1. Span-Expanded Attention (SE-Attn) overview. SE-Attn is a Sparse Attention mechanism used to expand the memory span of Hybrid SSMs. Left: SE-Attn works by reserving a fraction of the Attention context for tokens retrieved arbitrarily far back in the past. We call this reserve the “expansion span,” and we populate it with blocks of previous tokens (“memory blocks”). When new tokens arrive, a similarity-based search co… view at source ↗
Figure 2
Figure 2. Fine-tuning with SE-Attn outperforms SW-Attn and S 2 -Attn on the RULER bench￾mark. HyLoRA outperforms LoRA and LoRA+ on Hybrid models. (a): We fine-tune Mamba-2- Hybrid with a context size of 8192 and evaluate on eleven RULER tasks, as explained in Appendix I.4. Fine-tuning with SE-Attn consistently outperforms SW-Attn and S 2 -Attn even when evaluating on context sizes beyond the fine-tuning size. (b): We fine-tun… view at source ↗
Figure 3
Figure 3. SE-Attn ablations on Mamba-2-Hybrid. (a): Attention-based memory retrieval (SE-Attn) improves upon no retrieval and random retrieval. (b): Using SE-Attn with a chunk size chosen randomly from {2048, 4096} acts as a regularizer and outperforms SE-Attn with fixed chunk sizes of 2048 and 4096. (c): SE-Attn with larger memory blocks (i.e., more tokens per block) with a smaller top-k tends to do better than smaller block… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: SE-Attn offers a greater runtime-memory trade-off than other Attention variants. We profile various attention layers during a single training step. SW-Attn uses a window size of 4096 and SE-Attn alternates between a chunk size of 2048 and 4096. (a): Runtime of a single…
Figure 5
Figure 5. Figure 5: HyLoRA outperforms LoRA and LoRA+ on Hybrid models. We fine-tune Mamba-2- Hybrid with Full-Attn using LoRA, LoRA+, and HyLoRA. We find that LoRA and LoRA+ perform sub-optimally compared to HyLoRA which also trains 1D convolution layers. In [PITH_FULL_IMAGE:figures/ful…
Figure 6
Figure 6. Figure 6: Fine-tuning with a larger LoRA rank using HyLoRA improves performance on the RULER benchmark. We fine-tune Mamba-2-Hybrid using HyLoRA with different LoRA ranks (we maintain a LoRA rank to alpha ratio of 2). We observe that fine-tuning with a larger rank produces stron…
Figure 7
Figure 7. Figure 7: Fine-tuning with SE-Attn outperforms SW-Attn and S 2 -Attn on the RULER bench￾mark when applied to Llama1. We fine-tune Llama1 with a context size of 16384 using various Attention variants. We average over eleven RULER tasks, as explained in Appendix I.4. Fine-tuning w…
Figure 8
Figure 8. Figure 8: Mamba-2-Hybrid RULER benchmark fine-tuned on natural language. We fine-tune Mamba-2-Hybrid with a context size of 8192 using multiple Attention variants and evaluate on the RULER benchmark (see Appendix I.4 for definitions of each metric). On average, fine-tuning with …
Figure 9
Figure 9. Figure 9: Mamba-2-Hybrid RULER benchmark fine-tuned on natural language + code. We fine-tune Mamba-2-Hybrid with different Attention layers on a dataset that consists of 70% natural language and 30% code and evaluate on RULER. Compared to fine-tuning on only natural language (as…
Figure 10
Figure 10. Figure 10: Mamba-2-Hybrid RULER benchmark fine-tuned on PG-19. We fine-tune Mamba￾2-Hybrid with different Attention layers on the PG-19 Rae et al. (2019b) dataset and then evaluate on RULER. Compared to fine-tuning on other natural language datasets with a greater variety of tex…
Figure 11
Figure 11. Figure 11: Evaluating with efficient Attention mechanisms on RULER does not do as well as evaluating with standard full Attention. We fine-tune Mamba-2-Hybrid with SW-Attn (a) and SE-Attn (b) and then evaluate on the NIAH-Single-1 RULER task using either the same Attention layer…
Figure 12
Figure 12. Figure 12: Summarizing memory blocks via average pooling of attention yields stronger performance than summarizing them using “landmark” tokens. We fine-tune Mamba-2-Hybrid using our SE-Attn, and SE-Attn-LM, which summarizes memory blocks with a landmark token (similar to Mohtas…
Figure 13
Figure 13. Figure 13: A larger learning rate does not improve the performance of S 2 . Here we fine-tune a Mamba-2-Hybrid model using S 2 -Attn with two different learning rates: 2 × 10−4 and 2 × 10−5 . The learning rate used in Chen et al. (2024) is 2 × 10−5 , which we found to work well.…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Capability Convergence Hypothesis: Capability from Access Structure, Not Scale

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Long-context capability converges to hybrid architectures pairing O(1) state with a scalable index; pure scaling of SSMs or Transformers alone hits information, horizon, or circuit walls.

Reference graph

Works this paper leans on

40 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Just read twice: closing the recall gap for recurrent language models

    Simran Arora, Aman Timalsina, Aaryan Singhal, Benjamin Spector, Sabri Eyuboglu, Xinyi Zhao, Ashish Rao, Atri Rudra, and Christopher R \'e . Just read twice: closing the recall gap for recurrent language models. arXiv preprint arXiv:2407.05483, 2024

    arXiv 2024

  2. [2]

    L ong B ench: A bilingual, multitask benchmark for long context understanding

    Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. L ong B ench: A bilingual, multitask benchmark for long context understanding. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association f...

    2024

  3. [3]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  4. [4]

    Extending context window of large language models via positional interpolation

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595, 2023

    arXiv 2023

  5. [5]

    Longlo RA : Efficient fine-tuning of long-context large language models

    Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlo RA : Efficient fine-tuning of long-context large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=6PmJoRfdaK

    work page 2024

  6. [6]

    Transformer-xl: Attentive language models beyond a fixed-length context

    Zihang Dai. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860, 2019

    arXiv 1901

  7. [7]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=mZn2Xyh9Ec

    2024

  8. [8]

    Transformers are ssms: generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are ssms: generalized models and efficient algorithms through structured state space duality. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    work page 2024

Show all 40 references
  1. [9]

    Griffin: Mixing gated linear recurrences with local attention for efficient language models

    Soham De, Samuel L Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models. arXiv preprint ...

    2024 arXiv

  2. [10]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  3. [11]

    Parameter-efficient fine-tuning of state space models

    Kevin Galim, Wonjun Kang, Yuchen Zeng, Hyung Il Koo, and Kangwook Lee. Parameter-efficient fine-tuning of state space models. arXiv preprint arXiv:2410.09016, 2024

    2024 arXiv

  4. [12]

    A framework for few-shot language model evaluation, 07 2024

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024

  5. [13]

    The zamba2 suite: Technical report

    Paolo Glorioso, Quentin Anthony, Yury Tokpanov, Anna Golubeva, Vasudev Shyam, James Whittington, Jonathan Pilault, and Beren Millidge. The zamba2 suite: Technical report. arXiv preprint arXiv:2411.15242, 2024 a

    2024 arXiv

  6. [14]

    Zamba: A compact 7b ssm hybrid model

    Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b ssm hybrid model. arXiv preprint arXiv:2405.16712, 2024 b

    2024 arXiv

  7. [15]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  8. [16]

    Combining recurrent, convolutional, and continuous-time models with linear state space layers

    Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher R\' e . Combining recurrent, convolutional, and continuous-time models with linear state space layers. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors,...

    2021

  9. [17]

    Efficiently modeling long sequences with structured state spaces

    Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=uYLFoz1vlAC

    2022

  10. [18]

    Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? arXiv preprint arXiv:2404.06654, 2024

    2024 arXiv

  11. [19]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  12. [20]

    Kakade, and Eran Malach

    Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: transformers are better than state space models at copying. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  13. [21]

    A new approach to linear filtering and prediction problems

    Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. 1960

    1960

  14. [22]

    Needle in a haystack - pressure testing llms., 2023

    Gregory Kamradt. Needle in a haystack - pressure testing llms., 2023. URL https://github.com/gkamradt/LLMTest_NeedleInAHaystack/tree/main

    2023

  15. [23]

    Reformer: The efficient transformer

    Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In The International Conference on Learning Representations (ICLR), 2020

    2020

  16. [24]

    Jamba: A hybrid transformer-mamba language model

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, et al. Jamba: A hybrid transformer-mamba language model. arXiv preprint arXiv:2403.19887, 2024

    2024 arXiv

  17. [25]

    Random-access infinite context length for transformers

    Amirkeivan Mohtashami and Martin Jaggi. Random-access infinite context length for transformers. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc

    2023

  18. [26]

    Leave no context behind: Efficient infinite context transformers with infini-attention

    Tsendsuren Munkhdalai, Manaal Faruqui, and Siddharth Gopal. Leave no context behind: Efficient infinite context transformers with infini-attention. arXiv preprint arXiv:2404.07143, 2024

    2024 arXiv

  19. [27]

    Resurrecting recurrent neural networks for long sequences

    Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, and Soham De. Resurrecting recurrent neural networks for long sequences. In International Conference on Machine Learning, pages 26670--26698. PMLR, 2023

    2023

  20. [29]

    Compressive transformers for long-range sequence modelling

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling. arXiv preprint arXiv:1911.05507, 2019 b

    1911 arXiv

  21. [30]

    Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3...

    2020

  22. [31]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568: 0 127063, 2024

    2024

  23. [32]

    Retentive network: A successor to transformer for large language models

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models. arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  24. [33]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023 a

    2023 arXiv

  25. [34]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023 b

    2023 arXiv

  26. [35]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural In...

    2017

  27. [36]

    An empirical study of mamba-based language models

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, et al. An empirical study of mamba-based language models. arXiv preprint arXiv:2406.07887, 2024

    2024 arXiv

  28. [37]

    Gated linear attention transformers with hardware-efficient training

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024 a

    2024

  29. [38]

    Parallelizing linear transformers with the delta rule over sequence length

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing ...

    2024

  30. [39]

    Gated delta networks: Improving mamba2 with delta rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=r8H7xhYPwz

    2025

  31. [40]

    B mojo: Hybrid state space realizations of foundation models with eidetic and fading memory

    Luca Zancato, Arjun Seshadri, Yonatan Dukler, Aditya Golatkar, Yantao Shen, Benjamin Bowman, Matthew Trager, Alessandro Achille, and Stefano Soatto. B mojo: Hybrid state space realizations of foundation models with eidetic and fading memory. In A. Globerson, L. Mackey, D. Belg...

    2024

  32. [41]

    Lots of code, 2017

    Vladislav Zavadskyy. Lots of code, 2017. URL https://www.kaggle.com/datasets/zavadskyy/lots-of-code. Accessed: 2024-10-27

    2017

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.