Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The Transformer-SSM retrieval gap comes down to a few specialized Gather-and-Aggregate heads, not the whole model.

desk verdict A credible and well-triangulated mechanistic story that localizes the Transformer–SSM retrieval gap to a few Gather-and-Aggregate heads, but the headline single-head cliff is an artifact of heavy pruning and the broad claim needs reining in. read the letter →

arxiv 2504.18574 v2 pith:WQHNPUIJ submitted 2025-04-22 cs.LG cs.AI

Aviv Bick , Eric Xing , Albert Gu This is my paper · ORCID
classification cs.LGcs.AI
keywords in-contextretrievalstate-spacemodelsattentionheadsGather-and-AggregatemechanismmechanisticinterpretabilityMMLUhybridlanguageMamba
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the performance gap between Transformer and state-space-model (SSM) language models on retrieval-heavy tasks is not a broad architectural deficit but a localized one: in both architectures, in-context retrieval is carried out by a small number of specialized heads that implement a two-stage Gather-and-Aggregate mechanism. A Gather Head condenses relevant segments of context into single summary tokens, and an Aggregate Head later combines those summaries to answer the query. The authors show that removing a single such head from a layer-pruned Llama-3.1-8B collapses MMLU accuracy from 66% to 25% (random guessing), while knowledge benchmarks hardly move. In SSMs the same mechanism exists but is implemented more diffusely and with smoother attention patterns, so more heads are needed to approach Transformer-level retrieval. This reframes MMLU as largely a retrieval benchmark and gives a concrete target for hybrid designs: attention layers succeed precisely because they take over the Aggregate Head role.

What carries the argument

The central object is the Gather-and-Aggregate (G&A) mechanism, a two-stage circuit for in-context retrieval. The first stage, a Gather Head, attends over a segment of input and compresses it into a summary token, typically the segment-final newline; the second stage, an Aggregate Head, attends from the query position to those summary tokens and effectively selects the correct one. The paper isolates this pair with layer pruning, head ablations, attention masking, and layer-replacement experiments, and shows that the Transformer-SSM gap in retrieval-heavy benchmarks tracks how sharply each architecture's G&A heads can implement these two operations.

What would settle it

A direct falsifier would be to use activation patching on a natural retrieval-heavy benchmark to identify the heads that carry the correct-answer information and check whether they coincide with the G&A heads found by the synthetic diagnostic. If patching points to different heads, or if removing the identified G&A heads leaves some model's MMLU unaffected while removing a different head collapses it, the localization claim would be refuted. A simpler check is to run the same head-ablation ranking on the KV-retrieval task with a trailing space in the prompt and see whether the top-ranked heads change.

Watch

Extended reading notes

Core claim

Both Transformer- and SSM-based language models retrieve information from context through the same two-stage Gather-and-Aggregate mechanism, and in both families the mechanism is concentrated in a very small number of heads. A Gather Head condenses each relevant segment of context into a single representative token; an Aggregate Head then attends to those summaries from the query position and selects the relevant one. In a minimal Llama-3.1-8B, two heads, L16H22 and L17H24, are jointly responsible for MMLU letter selection: removing either drops accuracy from 66% to about 25%, while retaining just these two heads in the final layers sustains performance. Falcon-Mamba-7B and Llamba-8B show the same pattern, with a few Mamba channels or heads recovering most of the full model's MMLU score. The paper further shows that SSM heads implement the mechanism with smoother, less sharply localized attention, so they need more heads to approach Transformer-level retrieval, and that replacing one SSM Aggregate Head layer with attention raises MMLU from 33% to 50%.

Load-bearing premise

The cross-task conclusion rests on the assumption that the synthetic KV-retrieval diagnostic, run with answer scoring and deliberately without a trailing space, activates the same heads and the same retrieval process that drive performance on natural benchmarks such as MMLU, GSM8K, BBH, and dialogue comprehension.

Editorial extensions

If this is right

  • MMLU scores on these models should be read partly as retrieval ability: a model can score 66% on MMLU while its knowledge-task scores are already degraded, and can fall to 25% when one head is removed.
  • Hybrid models work because attention takes over aggregation: ablating six attention heads in Zamba2-7B drops MMLU from 64.3% to 34.9% while knowledge-task accuracy remains at 70%.
  • For hybrid design, attention is most valuable in middle layers where G&A heads emerge; when distilling a Transformer into an SSM, preserving attention at layers containing Aggregate Heads recovers most of the lost MMLU performance.
  • SSMs can compensate for weak per-head aggregation by using more heads, but this redundancy only partially closes the gap; replacing a single Aggregate Head layer with attention gave the largest measured gain.
  • The localized-retrieval account extends beyond MMLU to GSM8K, BBH, SWDE, and chat-formatted ARC, so the bottleneck is not specific to multiple-choice letter selection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If this diagnosis is right, then adding parameters or training data to pure SSMs will not by itself fix retrieval; the cheaper lever is architectural, inserting a small number of attention slots at the layers where G&A heads emerge.
  • The diagnostic format matters: the paper deliberately uses answer scoring without a trailing space because that format concentrates retrieval in a few heads. A natural extension is to test whether the same head identities are recovered under more natural token-boundary formats or longer contexts, since the head ranking could shift with prompt format.
  • Because MMLU's letter-selection format is what makes retrieval decisive, leaderboard comparisons between architectures may partly be comparisons of retrieval-format handling rather than knowledge; reformatting benchmarks to free-form answers could change relative rankings.
  • Another testable extension is to run the same head-ablation protocol across models of different scale, pretraining objective, or depth to see whether G&A heads always settle in middle layers or whether their preferred depth shifts with architecture and data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a unified mechanistic account of in-context retrieval in Transformer- and SSM-based language models. It identifies a two-stage Gather-and-Aggregate (G&A) mechanism: a Gather Head compresses each answer segment into a summary token, and an Aggregate Head later combines these summaries to select the correct label. The authors argue that this mechanism is concentrated in a small number of heads in both architecture families, that SSMs implement it less sharply (smoother attention patterns, more redundant heads), and that the resulting retrieval bottleneck can explain much of the Transformer–SSM performance gap on tasks like MMLU, GSM8K, BBH, and dialogue comprehension. Evidence comes from layer/head pruning and zeroing in Llama-3.1-8B, Falcon-Mamba-7B, Llamba-8B, and Zamba2, a synthetic KV-retrieval diagnostic, masking interventions that restrict heads to the hypothesized pattern, and hybrid layer-replacement experiments on a distilled SSM. The central claim is that the retrieval gap 'exists in just a few heads' rather than reflecting a broad architectural deficiency, with implications for hybrid model design.

Significance. If the central claim holds, the paper makes a valuable contribution: it extends the 'retrieval heads' literature from Transformers to SSMs, refines the single-head view by emphasizing a coordinated two-stage mechanism, and offers a concrete design rule for hybrid models (place or retain attention near the layers where G&A heads emerge). The paper's strengths include multiple independent interventions (layer pruning, head zeroing, runtime masking, hybrid replacement), a synthetic diagnostic intended to isolate retrieval, and honest caveats in Section 3.6 about cases where the two layers are non-adjacent. The findings are falsifiable and the experiments are publicly released. However, as detailed below, the strongest quantitative claim (a single-head MMLU cliff) is demonstrated only in a heavily pruned model, and the cross-task generalization from the synthetic diagnostic to natural benchmarks is not fully validated. These issues are load-bearing for the paper's headline conclusion, so the manuscript needs substantial revision rather than acceptance in its current form.

major comments (3)
  1. [§3.4, Table 1 and §6.2, Table 2]
  2. [§6.1, Appendix A.5, Table 5]
  3. [§5.4, Hybrid Replacements]
minor comments (5)
  1. [Abstract and §1]
  2. [§2]
  3. [Appendix A.4]
  4. [§3.6]
  5. [Throughout]

Circularity Check

0 steps flagged · score 0.0 of 10

No material circularity: the G&A head identifications are validated by synthetic KV-retrieval and masking interventions, and the only self-referential elements are model artifacts that are not load-bearing.

full rationale

The paper's derivation chain is self-contained and externally checked. Critical heads are first located by ablating the minimal Llama-3.1-8B on MMLU (Section 3.4), then the same heads are tested on a synthetic KV-retrieval task that has no factual overlap with MMLU (Section 3.5), and later the KV-retrieval head rankings in Table 5 are used to ablate heads on GSM8K, BBH, and ARC-Chat (Section 6). These are independent measurements rather than re-statements of the selection criterion. The masking experiment in Section 4.3 is a sufficiency test: restricting the preserved heads to the hypothesized Gather-and-Aggregate attention pattern and observing the same MMLU accuracy does not define the result in terms of the conclusion; it tests whether the proposed pattern can reproduce the behavior. The paper's self-citations to MOHAWK and Llamba are model artifacts; the observation that Llamba places G&A heads in the teacher layers could be inherited from layer-wise distillation, but that observation is not load-bearing because from-scratch Falcon-Mamba shows the same mechanism, and the direct masking and KV-retrieval tests on Llamba stand independently. The choice of the no-trailing-space answer-scoring probe in Appendix A.5 is a probe-design choice that may bias which heads are flagged, but the natural-benchmark ablations are out-of-sample with respect to the probe and are not derived from it. No equation, fitted parameter, or uniqueness claim is reused as a prediction, so I find no circular step to report.

Assumptions & free parameters 0 free parameters · 5 assumptions · 2 invented entities

The paper introduces two interpretive entities, Gather and Aggregate Heads, but gives them independent empirical handles via ablation, masking, and KV-retrieval. The central claims rest on domain assumptions about the validity of head-zeroing interventions and the transfer of the tuned KV diagnostic to natural benchmarks. No numerical free parameters are fitted.

assumptions (5)
  • domain assumption Zeroing a head's output projection is a clean causal intervention that removes only that head's contribution to the residual stream.
    Used in Sections 3.4, 5.4, and 6 to attribute MMLU and benchmark changes to specific heads; nonlinear interactions in the residual stream could make this an approximation, but it is the standard ablation assumption in the interpretability literature.
  • domain assumption The synthetic KV-retrieval task with answer scoring and no trailing space measures the same in-context retrieval mechanism used by MMLU, GSM8K, BBH, and dialogue benchmarks.
    Appendix A.5 deliberately selects a prompt format that maximizes head-level signal and notes that the natural format requires much longer contexts; the paper shows overlap with MMLU-critical heads but does not independently verify equivalence on GSM8K or BBH.
  • domain assumption Knowledge-focused benchmarks such as ARC, PIQA, Winogrande, OpenBookQA, and HellaSwag have minimal in-context retrieval demands.
    Sections 3.1 to 3.3 use the contrast with these benchmarks to argue MMLU is retrieval-dominated; these tasks still require reading and some context use, so the separation is approximate.
  • domain assumption The hidden-attention approximation used to visualize SSM token mixing faithfully represents the smoothness of SSM G&A heads.
    SSMs have no explicit attention; Figure 5 and Section 5.1 compare attention smoothness across architectures, relying on a surrogate representation that is not derived in the paper.
  • domain assumption The pruned minimal model preserves the same retrieval machinery as the full model, so head criticality transfers to the unpruned model.
    Sections 3.2 to 3.4 identify critical heads after heavy layer pruning; if pruning changes head roles, the identified G&A heads could be artifacts of the minimal configuration rather than the full model's retrieval circuit.
invented entities (2)
  • Gather Head independent evidence
    purpose: Condenses each segment of context into a single representative token, such as the newline ending an answer choice.
    Supported by ablation sensitivity and by the masking experiment in Section 4.3 where constraining the head to within-segment attention preserves MMLU accuracy, giving a falsifiable behavioral handle.
  • Aggregate Head independent evidence
    purpose: Reads the compressed segment summaries and weights the relevant one most strongly to output the correct label or retrieved value.
    Supported by head ablation on MMLU, sensitivity to KV-retrieval dictionary size in Figure 4, hybrid layer replacement in Section 5.4, and masking to summary tokens; these are concrete empirical handles, not merely an interpretive label.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism." pith.science (2026). https://pith.science/paper/WQHNPUIJ

@misc{pith2026250418574,
  author       = {Pith},
  title        = {Pith review of: Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WQHNPUIJ}},
  note         = {Machine review of arXiv:2504.18574}
}
read the original abstract

State-space models (SSMs) offer efficient alternatives to Transformers for long sequences, but their fixed-size recurrent state limits capability on algorithmic tasks, such as retrieving past context. In this work, we examine how in-context retrieval operates in Transformer- and SSM-based language models and find that both rely on a similar Gather-and-Aggregate (G&A) mechanism: a Gather Head extracts relevant information pieces from context, which an Aggregate Head integrates into a single representation. In both architectures, G&A concentrates in a few heads, forming critical bottlenecks even for simple retrieval. For example, we show that disabling a single Gather or Aggregate Head in a pruned Llama-3.1-8B impairs retrieving the correct answer letter in MMLU, reducing its accuracy from 66% to 25% (random guessing). Moreover, this retrieval bottleneck can obscure limited knowledge demands of tasks as the pruned model succeeds on MMLU with functioning G&A heads yet fails on other knowledge benchmarks. The bottleneck similarly extends to tasks where SSMs typically underperform, such as GSM8K, BBH, and dialogue comprehension. We show that SSMs' retrieval challenges manifest in these heads, creating smoother attention patterns instead of the sharp token transitions effective G&A requires. Thus, the Transformer-SSM retrieval gap exists in just a few heads, rather than the entire language model. This suggests a unified explanation for Transformer vs. SSM performance gap while showing how to merge their strengths. We find that pretrained hybrid models, where SSMs are combined with a few attention layers, delegate the role of Aggregate Heads to attention. Similarly, replacing a single G&A head in a pretrained SSM with an attention variant boosts retrieval and benchmark scores.

Figures

Figures reproduced from arXiv: 2504.18574 by the authors.

Figure 1
Figure 1. An illustration of Gather Head (left matrix) at L16H22 and Aggregate Head (right matrix) at L17H24 of Llama-3.1-8B for a multiple-choice question. The Gather Head identifies critical segments and summarizes each segment to its last token “\n” as shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Performance of Llama-3.1-8B on knowledge tasks and MMLU as layers are gradually pruned, starting from the last. The leftmost bar shows the full 32-layer model, while subsequent bars represent pruned versions. Key observations: (1) MMLU scores stay stable despite declining knowledge tasks performance; (2) a sharp MMLU drop identifies a critical skill-encoding layer. (3) 30% of the model’s original score on knowledge … view at source ↗
Figure 3
Figure 3. Performance of Llama-3.1-8B-Minimal with layers removed and restored. We remove one layer at a time, evaluate performance on MMLU and knowledge tasks’ score, and then restore the layer before proceeding to the next. The results show that both L16 and L17 are critical for MMLU performance. Pruning either of these final layers causes a large drop in MMLU accuracy, while scores on knowledge tasks decline only slightly.… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Evaluation of Llama-3.1-8B on the KV-Retrieval task under three settings: unmodified, with only L16H22 removed, or with only L17H24 removed (the Gather and Aggregate Heads identified in Section 3.4, respectively). Performance is reported across 55 configurations with i…
Figure 5
Figure 5. Figure 5: Illustration of Gather Heads (top row) and Aggregate Heads (bottom row) across different model types. The left column shows attention patterns in Llama-3.1-8B (Transformer-based), while the center and right columns correspond to Llamba-8B (Mamba-2-based) and Falcon-Mam…
Figure 6
Figure 6. Figure 6: Visualization of the masking applied to the Gather Head (first row) and the Aggregate Head (second row) of Llamba-8B. The Gather Heads are restricted to interactions with the representative token and its associated answer, while the Aggregate Heads are limited to proce…
Figure 7
Figure 7. Figure 7: Results of the hybrid replacement experiment. Each Llamba-8B layer was replaced with its Llama-3.1-8B counterpart and tested on MMLU. Most swaps had little or negative impact on the 33% baseline, but replacing layer 17—where a key Aggregate Head is ( [PITH_FULL_IMAGE:…
Figure 8
Figure 8. Figure 8: Residual performance trends for knowledge tasks and the MMLU benchmark as layers are progressively [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Residual performance trends for knowledge tasks and the MMLU benchmark as layers are progressively [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Residual performance trends for the MMLU benchmark and knowledge tasks metrics in the minimal model [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Mechanistic Attention Guidance for Agent Memory Refinement

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Attention patterns reveal how an AI agent uses memory, and using these patterns to rewrite memory improves task performance and memory efficiency.

Reference graph

Works this paper leans on

59 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    The hidden attention of mamba models, 2024

    Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models, 2024. URL https://arxiv.org/abs/2403.01590

    arXiv 2024

  2. [2]

    When benchmarks are targets: Revealing the sensitivity of large language model leaderboards, 2024

    Norah Alzahrani, Hisham Abdullah Alyahya, Yazeed Alnumay, Sultan Alrashed, Shaykhah Alsubaie, Yusef Almushaykeh, Faisal Mirza, Nouf Alotaibi, Nora Altwairesh, Areeb Alowisheq, M Saiful Bari, and Haidar Khan. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards, 2024. URL https://arxiv.org/abs/2402.01781

    arXiv 2024

  3. [3]

    Mechanistic evaluation of transformers and state space models, 2025 a

    Aryaman Arora, Neil Rathi, Nikil Roashan Selvam, Róbert Csórdas, Dan Jurafsky, and Christopher Potts. Mechanistic evaluation of transformers and state space models, 2025 a . URL https://arxiv.org/abs/2505.15105

    arXiv 2025

  4. [4]

    Zoology: Measuring and improving recall in efficient language models, 2023

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models, 2023. URL https://arxiv.org/abs/2312.04927

    arXiv 2023

  5. [6]

    Simple linear attention language models balance the recall-throughput tradeoff, 2025 b

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, Dylan Zinsley, James Zou, Atri Rudra, and Christopher Ré. Simple linear attention language models balance the recall-throughput tradeoff, 2025 b . URL https://arxiv.org/abs/2402.18668

    arXiv 2025

  6. [7]

    xlstm: Extended long short-term memory, 2024

    Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory, 2024. URL https://arxiv.org/abs/2405.04517

    arXiv 2024

  7. [8]

    Li, Eric P

    Aviv Bick, Kevin Y. Li, Eric P. Xing, J. Zico Kolter, and Albert Gu. Transformers to ssms: Distilling quadratic knowledge to subquadratic models, 2024. URL https://arxiv.org/abs/2408.10189

    arXiv 2024

  8. [9]

    Llamba: Scaling distilled recurrent models for efficient language processing, 2025

    Aviv Bick, Tobias Katsch, Nimit Sohoni, Arjun Desai, and Albert Gu. Llamba: Scaling distilled recurrent models for efficient language processing, 2025. URL https://arxiv.org/abs/2502.14458

    arXiv 2025

Show all 59 references
  1. [10]

    Birth of a transformer: A memory viewpoint, 2023

    Alberto Bietti, Vivien Cabannes, Diane Bouchacourt, Herve Jegou, and Leon Bottou. Birth of a transformer: A memory viewpoint, 2023. URL https://arxiv.org/abs/2306.00802

    2023 arXiv

  2. [11]

    Piqa: Reasoning about physical commonsense in natural language, 2019

    Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language, 2019. URL https://arxiv.org/abs/1911.11641

    2019 arXiv

  3. [12]

    Sam Blouir, Jimmy T. H. Smith, Antonios Anastasopoulos, and Amarda Shehu. Birdie: Advancing state space models with reward-driven objectives and curricula, 2025. URL https://arxiv.org/abs/2411.01030

    2025 arXiv

  4. [13]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2020

  5. [14]

    Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URL https://arxiv.org/abs/1803.05457

    2018 arXiv

  6. [15]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168

    2021 arXiv

  7. [16]

    Transformers are SSM s: Generalized models and efficient algorithms through structured state space duality

    Tri Dao and Albert Gu. Transformers are SSM s: Generalized models and efficient algorithms through structured state space duality. In International Conference on Machine Learning (ICML), 2024

    2024

  8. [17]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness, 2022

    2022

  9. [18]

    A mathematical framework for transformer circuits

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dari...

    2021

  10. [19]

    Zamba: A compact 7b ssm hybrid model, 2024

    Paolo Glorioso, Quentin Anthony, Yury Tokpanov, James Whittington, Jonathan Pilault, Adam Ibrahim, and Beren Millidge. Zamba: A compact 7b ssm hybrid model, 2024. URL https://arxiv.org/abs/2405.16712

    2024 arXiv

  11. [20]

    Mamba: Linear-time sequence modeling with selective state spaces, 2023

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023

    2023

  12. [21]

    Efficiently modeling long sequences with structured state spaces, 2022

    Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2022

    2022

  13. [22]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300

    2021 arXiv

  14. [23]

    Kakade, and Eran Malach

    Samy Jelassi, David Brandfonbrener, Sham M. Kakade, and Eran Malach. Repeat after me: Transformers are better than state space models at copying, 2024. URL https://arxiv.org/abs/2402.01032

    2024 arXiv

  15. [24]

    Transformers are rnns: Fast autoregressive transformers with linear attention, 2020

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rnns: Fast autoregressive transformers with linear attention, 2020

    2020

  16. [25]

    Jamba: A hybrid transformer-mamba language model, 2024

    Opher Lieber, Barak Lenz, Hofit Bata, Gal Cohen, Jhonathan Osin, Itay Dalmedigos, Erez Safahi, Shaked Meirom, Yonatan Belinkov, Shai Shalev-Shwartz, Omri Abend, Raz Alon, Tomer Asida, Amir Bergman, Roman Glozman, Michael Gokhman, Avashalom Manevich, Nir Ratner, Noam Rozen, Ere...

    2024 arXiv

  17. [26]

    Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla, 2023

    Tom Lieberum, Matthew Rahtz, János Kramár, Neel Nanda, Geoffrey Irving, Rohin Shah, and Vladimir Mikulik. Does circuit analysis interpretability scale? evidence from multiple choice capabilities in chinchilla, 2023. URL https://arxiv.org/abs/2307.09458

    2023 arXiv

  18. [27]

    Locating and editing factual associations in gpt, 2023

    Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt, 2023. URL https://arxiv.org/abs/2202.05262

    2023 arXiv

  19. [28]

    Circuit component reuse across tasks in transformer language models, 2024

    Jack Merullo, Carsten Eickhoff, and Ellie Pavlick. Circuit component reuse across tasks in transformer language models, 2024. URL https://arxiv.org/abs/2310.08744

    2024 arXiv

  20. [29]

    Can a suit of armor conduct electricity? a new dataset for open book question answering, 2018

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering, 2018. URL https://arxiv.org/abs/1809.02789

    2018 arXiv

  21. [30]

    In-context learning and induction heads

    Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kam...

    2022

  22. [31]

    Li, Aviv Bick, J

    Daniele Paliotta, Junxiong Wang, Matteo Pagliardini, Kevin Y. Li, Aviv Bick, J. Zico Kolter, Albert Gu, François Fleuret, and Tri Dao. Thinking slow, fast: Scaling inference compute with distilled reasoners, 2025. URL https://arxiv.org/abs/2502.20339

    2025 arXiv

  23. [32]

    The lambada dataset: Word prediction requiring a broad discourse context, 2016

    Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context, 2016. URL https://arxiv.org/abs/1606.06031

    2016 arXiv

  24. [33]

    Can mamba learn how to learn? a comparative study on in-context learning tasks, 2024

    Jongho Park, Jaeseung Park, Zheyang Xiong, Nayoung Lee, Jaewoong Cho, Samet Oymak, Kangwook Lee, and Dimitris Papailiopoulos. Can mamba learn how to learn? a comparative study on in-context learning tasks, 2024. URL https://arxiv.org/abs/2402.04248

    2024 arXiv

  25. [34]

    Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence, 2024

    Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Xingjian Du, Teddy Ferdinan, Haowen Hou, Przemysław Kazienko, Kranthi Kiran GV, Jan Kocoń, Bartłomiej Koptyra, Satyapriya Krishna, Ronald McClelland Jr., Jiaju Lin, Niklas Mu...

    2024 arXiv

  26. [35]

    Hgrn2: Gated linear rnns with state expansion, 2024

    Zhen Qin, Songlin Yang, Weixuan Sun, Xuyang Shen, Dong Li, Weigao Sun, and Yiran Zhong. Hgrn2: Gated linear rnns with state expansion, 2024. URL https://arxiv.org/abs/2404.07904

    2024 arXiv

  27. [36]

    A practical review of mechanistic interpretability for transformer-based language models, 2024

    Daking Rai, Yilun Zhou, Shi Feng, Abulhair Saparov, and Ziyu Yao. A practical review of mechanistic interpretability for transformer-based language models, 2024. URL https://arxiv.org/abs/2407.02646

    2024

  28. [37]

    Samba: Simple hybrid state space models for efficient unlimited context language modeling, 2024

    Liliang Ren, Yang Liu, Yadong Lu, Yelong Shen, Chen Liang, and Weizhu Chen. Samba: Simple hybrid state space models for efficient unlimited context language modeling, 2024. URL https://arxiv.org/abs/2406.07522

    2024 arXiv

  29. [38]

    Winogrande: An adversarial winograd schema challenge at scale, 2019

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https://arxiv.org/abs/1907.10641

    2019 arXiv

  30. [39]

    Transformers, parallel computation, and logarithmic depth, 2024

    Clayton Sanford, Daniel Hsu, and Matus Telgarsky. Transformers, parallel computation, and logarithmic depth, 2024. URL https://arxiv.org/abs/2402.09268

    2024 arXiv

  31. [40]

    Retentive network: A successor to transformer for large language models, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models, 2023

    2023

  32. [41]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them, 2022. URL https://arxiv.org/abs/2210.09261

    2022 arXiv

  33. [42]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023

  34. [43]

    Listening to the wise few: Select-and-copy attention heads for multiple-choice qa, 2024

    Eduard Tulchinskii, Laida Kushnareva, Kristian Kuznetsov, Anastasia Voznyuk, Andrei Andriiainen, Irina Piontkovskaya, Evgeny Burnaev, and Serguei Barannikov. Listening to the wise few: Select-and-copy attention heads for multiple-choice qa, 2024. URL https://arxiv.org/abs/2410.02343

    2024 arXiv

  35. [44]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  36. [45]

    An empirical study of mamba-based language models, 2024

    Roger Waleffe, Wonmin Byeon, Duncan Riach, Brandon Norick, Vijay Korthikanti, Tri Dao, Albert Gu, Ali Hatamizadeh, Sudhakar Singh, Deepak Narayanan, Garvit Kulshreshtha, Vartika Singh, Jared Casper, Jan Kautz, Mohammad Shoeybi, and Bryan Catanzaro. An empirical study of mamba-...

    2024 arXiv

  37. [46]

    Rush, and Tri Dao

    Junxiong Wang, Daniele Paliotta, Avner May, Alexander M. Rush, and Tri Dao. The mamba in the llama: Distilling and accelerating hybrid models, 2024 a . URL https://arxiv.org/abs/2408.15237

    2024 arXiv

  38. [47]

    Mmlu-sr: A benchmark for stress-testing reasoning capability of large language models, 2024 b

    Wentian Wang, Sarthak Jain, Paul Kantor, Jacob Feldman, Lazaros Gallos, and Hao Wang. Mmlu-sr: A benchmark for stress-testing reasoning capability of large language models, 2024 b . URL https://arxiv.org/abs/2406.15468

    2024 arXiv

  39. [48]

    Rnns are not transformers (yet): The key bottleneck on in-context retrieval, 2024

    Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. Rnns are not transformers (yet): The key bottleneck on in-context retrieval, 2024. URL https://arxiv.org/abs/2402.18510

    2024 arXiv

  40. [49]

    Answer, assemble, ace: Understanding how lms answer multiple choice questions, 2025

    Sarah Wiegreffe, Oyvind Tafjord, Yonatan Belinkov, Hannaneh Hajishirzi, and Ashish Sabharwal. Answer, assemble, ace: Understanding how lms answer multiple choice questions, 2025. URL https://arxiv.org/abs/2407.15018

    2025 arXiv

  41. [50]

    Retrieval head mechanistically explains long-context factuality, 2024

    Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. Retrieval head mechanistically explains long-context factuality, 2024. URL https://arxiv.org/abs/2404.15574

    2024 arXiv

  42. [51]

    Efficient streaming language models with attention sinks, 2024

    Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks, 2024. URL https://arxiv.org/abs/2309.17453

    2024 arXiv

  43. [52]

    Gated delta networks: Improving mamba2 with delta rule, 2024 a

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule, 2024 a . URL https://arxiv.org/abs/2412.06464

    2024 arXiv

  44. [53]

    Gated linear attention transformers with hardware-efficient training, 2024 b

    Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention transformers with hardware-efficient training, 2024 b . URL https://arxiv.org/abs/2312.06635

    2024 arXiv

  45. [54]

    Gwon, and Sungroh Yoon

    Sangwon Yu, Jongyoon Song, Bongkyu Hwang, Hoyoung Kang, Sooah Cho, Junhwa Choi, Seongho Joe, Taehee Lee, Youngjune L. Gwon, and Sungroh Yoon. Correcting negative bias in large language models through negative attention score alignment, 2024. URL https://arxiv.org/abs/2408.00137

    2024 arXiv

  46. [55]

    Hellaswag: Can a machine really finish your sentence?, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?, 2019. URL https://arxiv.org/abs/1905.07830

    2019 arXiv

  47. [56]

    Interpreting and improving large language models in arithmetic calculation, 2024 a

    Wei Zhang, Chaoqun Wan, Yonggang Zhang, Yiu ming Cheung, Xinmei Tian, Xu Shen, and Jieping Ye. Interpreting and improving large language models in arithmetic calculation, 2024 a . URL https://arxiv.org/abs/2409.01659

    2024 arXiv

  48. [57]

    Gated slot attention for efficient linear-time sequence modeling, 2024 b

    Yu Zhang, Songlin Yang, Ruijie Zhu, Yue Zhang, Leyang Cui, Yiqiao Wang, Bolun Wang, Freda Shi, Bailin Wang, Wei Bi, Peng Zhou, and Guohong Fu. Gated slot attention for efficient linear-time sequence modeling, 2024 b . URL https://arxiv.org/abs/2409.07146

    2024 arXiv

  49. [58]

    Mmlu-cf: A contamination-free multi-task language understanding benchmark, 2024

    Qihao Zhao, Yangyu Huang, Tengchao Lv, Lei Cui, Qinzheng Sun, Shaoguang Mao, Xin Zhang, Ying Xin, Qiufeng Yin, Scarlett Li, and Furu Wei. Mmlu-cf: A contamination-free multi-task language understanding benchmark, 2024. URL https://arxiv.org/abs/2412.15194

    2024 arXiv

  50. [59]

    Attention heads of large language models: A survey, 2024

    Zifan Zheng, Yezhaohui Wang, Yuxin Huang, Shichao Song, Mingchuan Yang, Bo Tang, Feiyu Xiong, and Zhiyu Li. Attention heads of large language models: A survey, 2024. URL https://arxiv.org/abs/2409.03752

    2024 arXiv

  51. [60]

    Falcon mamba: The first competitive attention-free 7b language model, 2024

    Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, and Hakim Hacid. Falcon mamba: The first competitive attention-free 7b language model, 2024. URL https://arxiv.org/abs/2410.05355

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.