REVIEW 4 major objections 6 minor 12 references
Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Mamba's better recall of early sequence items is carried by a sparse subset of internal channels; disrupting those channels erases early-token memory while preserving recent memory.
desk verdict U-shaped recall in Mamba is probably real, but the paper's causal 'long-term memory channel' claim rests on an underspecified ablation. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the selective state-space recurrence $h_t^{(i)} = A_t^{(i)} h_{t-1}^{(i)} + B_t^{(i)} x_t^{(i)}$, with its unrolled form expressing each hidden state as a sum of past input contributions scaled by products of the recurrence matrices $A_k$. The paper defines a long-term memory coefficient $M^{(i)} = \mathrm{diag}(\prod_{t=2}^T A_t^{(i)})$ to score how much of each channel's hidden state survives to the final context timestep, and uses the learned per-token discretization $\Delta_t$ as the forgetting gate that couples input injection and state decay. Together these give a quantitative account of why early tokens persist (high $A$ products in a few channels), why recent tokens dominate (fewer decay steps), and why periodic or repeated inputs forget faster ($\Delta_t$ grows when input frequency is low).
What would settle it
Run the model with the targeted intervention while recording the contribution of the first token $x_1$ to the final hidden state via the unrolled sum; if the contribution is unchanged yet first-position accuracy falls, the ablation is not blocking early-token flow. Comparing zeroing $A$ at the first triplet's timestep against zeroing all later $A$ products would settle whether the identified channels are the actual carriers of primacy.
Extended reading notes
Core claim
On structured recall tasks with subject-relation-object triples, Falcon Mamba 7B and Mamba 1.4B produce a consistent U-shaped accuracy profile across sequence lengths 8 through 128. The central discovery is that primacy in Mamba is causally supported by a sparse subset of channels, concentrated at particular layers (notably layer 17 in Falcon Mamba 7B), whose cumulative recurrence product $M^{(i)} = \mathrm{diag}(\prod_{t=2}^T A_t^{(i)})$ stays above threshold; ablating the recurrence matrices of these channels at the first triplet's timestep drops first-position accuracy sharply, while ablating random channels does not. Recency is explained by the unrolled dynamics $y_t^{(i)} = C_t \sum_{j=1}^t (\prod_{k=j+1}^t A_k^{(i)}) B_j^{(i)} x_j^{(i)}$, where recent inputs undergo fewer decay steps; inserting distractor tokens erodes this advantage. Finally, repeating the same relation token across the sequence shifts the learned discretization $\Delta_t$ toward faster forgetting and stronger input injection, worsening the lost-in-the-middle effect, whereas randomizing relations partially restores middle recall.
Load-bearing premise
The causal claim assumes that zeroing the recurrence matrix $A_t$ at the timestep of the first triplet blocks the first token's influence; under the stated update rule $h_t = A_t h_{t-1} + B_t x_t$ the intervention leaves $B_1 x_1$ in the state, so the observed drop in early recall may not isolate long-term memory channels.
Editorial extensions
If this is right
- Long-term recall in Mamba can be degraded surgically: ablating the recurrence matrices of high-memory channels selectively damages first-position recall, while random ablations of equal size leave it intact.
- Recency in Mamba is a dynamic, saturating property rather than a fixed architectural bias; inserting 192 to 384 distractor tokens degrades recall across all positions, with the largest losses near the end of the sequence.
- Input regularity changes memory allocation: repeated relations push $\Delta_t$ to larger values, producing faster forgetting of middle items, while randomizing relations improves middle recall.
- The localization pattern generalizes across scale: Mamba 1.4B also shows the effect, though its long-term memory channels are more widely distributed and require ablating more layers to disrupt.
- Because the same $\Delta_t$ modulates both the recurrence matrix $A_t$ and the input gate $B_t$, the architecture couples forgetting with input integration; decoupling these could improve retrieval from arbitrary positions.
Reading between the lines
- A testable extension the paper does not run: if long-term memory is genuinely confined to a sparse channel set, then weight-pruning or low-bit quantization targeted at those channels should degrade long-context retrieval far more than short-context retrieval, with a random-pruning control separating channel identity from mere parameter count.
- The $\Delta_t$ periodicity result implies a simple prompt-level lever: diversifying syntactic and semantic templates in long contexts should flatten the lost-in-the-middle curve even without fine-tuning; that is a direct, untested prediction of the paper's mechanism.
- The paper's localization claim concerns Mamba specifically; whether sparse long-term memory channels are a general property of state-space architectures or a learned byproduct of this architecture's initialization would require the same intervention in other state-space model families.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies primacy and recency effects in Mamba-style state space models using a structured subject-relation-object recall task on Falcon Mamba 7B and Mamba 1.4B. It reports a U-shaped recall accuracy curve and proposes three mechanisms: long-term memory localized in a sparse subset of selective SSM channels (causal claim), recency from delta-modulated exponential decay, and semantic regularity modulating delta gating to increase forgetting of intermediate items. The central causal claim is based on ablating the recurrence matrices A_t in channels selected by a memory coefficient.
Significance. If the causal claims were established, the paper would be a valuable mechanistic account of memory biases in attention-free architectures and could inform SSM design. The U-shaped behavioral result is clearly presented and replicated across two model families and several context lengths, and the task design (single-token vocabulary, 50 sequences per position) is thoughtful. However, the central causal claim about localized long-term memory is not supported as written because the intervention is underspecified relative to the recurrence equations, the selection/evaluation protocol is ambiguous, and quantitative evidence for the required selectivity is missing. The third mechanism also rests on an indirect synthetic experiment rather than direct measurement on the recall task.
major comments (4)
- [4.2, Eq. (2)] The causal intervention is underspecified and, on the natural reading, does not block the first token's influence. The paper states that zeroing A_t 'at the timestep of the first triplet' blocks early information from entering long-memory pathways. From Eq. (2), the first token's contribution at time t is (Π_{k=2}^t A_k) B_1 x_1; zeroing A_1 leaves B_1 x_1 in the state, while zeroing A_3 (the object token of the first triplet) resets the state to B_3 x_3 and drops the subject/relation contributions but still leaves the object token in the state. Neither operation is an ablation of 'early information entering' these pathways. The text does not say which timestep is zeroed, nor does it provide control ablations (e.g., zeroing B_1 or zeroing A at a later triplet) or the exact code. As written, the central claim of localized long-term memory is not supported.
- [4.2, Algorithm 1 and Figure 3b] The channel-selection and evaluation protocol is ambiguous about whether the same input sample is used to select channels and to measure the intervention effect. Algorithm 1 says a single forward pass on an input sample is used to extract A_t and B_t and to select channels; if the intervention is then evaluated on the same sample(s), the reported drop may be specific to those inputs rather than a general property of the channels. The paper should report the stability of selected channels across the 50 sequences and evaluate interventions on held-out sequences.
- [Figures 2, 3b, 6, 11-13] No error bars, confidence intervals, or significance tests are reported for the recall curves or the intervention effects. Claims such as 'only the targeted intervention leads to a significant drop' are unsupported; with 50 examples per position, the reader cannot assess whether the observed differences are within sampling noise. Bootstrap or permutation tests, at least for the first-position intervention comparison, are needed.
- [4.4, Figures 7-9] The synthetic periodic-token experiment does not directly test the repeated-relation condition of the recall task. The recall task repeats an entire relation token within an s-r-o structure, while the synthetic inputs repeat a single token every k positions. The paper asserts that the resulting delta behavior 'could explain' the repeated-relation effect, but it does not measure delta on the actual repeated-relation versus random-relation contexts. Without that measurement, the third mechanism remains a hypothesis rather than a validated finding.
minor comments (6)
- [3.1] There is a typo in the description of Eq. (1): 'dimesion' should be 'dimension'.
- [4.2, Eq. (4) and Definition box] Using the phrase 'probability p' for the deterministic cutoff p is confusing; a term such as 'proportion cutoff' would be clearer.
- [4.3, Eq. (5)] The statement that the j-th contribution is 'proportional to A^{t-j}' assumes a time-invariant A; in Mamba, A_t is input-dependent, so the product Π A_k should be used unless a constant-A simplification is explicitly stated.
- [4.4] The period values are inconsistent: the text lists k∈{2,4,8,16,32,64}, but Figures 7-9 use 'Period 1' and legends with 1, 4, 16, 64. Please make the notation uniform.
- [Appendix B, Algorithm 1] The line 'M←Q T t=2 At' is missing a product symbol and has unclear indexing; the phrase 'do Perform a forward pass' is ungrammatical. Also specify which input sequence (length and relation type) is used for channel selection.
- [Figure 5 caption] The caption ends mid-sentence: 'Effect of initializing the recurrent state at Layer 31 using uniform values on repeated relation' should be completed.
Circularity Check
No significant circularity: the primacy mechanism is an empirical ablation finding, not a fitted or self-referential derivation.
full rationale
The paper's central derivation chain is not circular. The U-shaped recall profile is an empirical observation on frozen pretrained models, and the long-term memory channel identification in Eq. (3) uses the model's own A matrices rather than fitting any parameter to the behavioral recall curves. The causal intervention in Section 4.2 ablates A_t on channels selected by their cumulative A product, so selection and intervention share the same mechanism; this is a legitimate validity/selection-bias concern, but it is not a derivation-level circularity because the reported drop in first-position recall is an experimental outcome tested against random-channel ablation rather than a quantity computed from the identification criterion. The exponential-decay intuition in Eq. (5) is a rough constant-A approximation of the time-varying recurrence, not a circular reuse of the conclusion. No load-bearing self-citation chain appears: prior work is cited for architectural context and external phenomena, and the paper's claims are evaluated against frozen model behavior with targeted ablations, perturbations, and parameter-sensitivity checks. The intervention description is underspecified about which timestep is zeroed, but that is a reproducibility/correctness issue, not circularity.
Assumptions & free parameters
free parameters (2)
- tau =
0.7
- p =
0.7
assumptions (3)
- domain assumption The recurrence equations (Equation 1) accurately describe Mamba's forward dynamics.
- ad hoc to paper The synthetic periodic-token experiment captures the repeated-relation condition of the recall task.
- domain assumption Top-1 accuracy on proper-noun recall reflects the model's memory rather than lexical shortcuts.
Cite this review
Pith. "Pith review of Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View." pith.science (2026). https://pith.science/paper/2UP6XXR2
@misc{pith2026250615156,
author = {Pith},
title = {Pith review of: Emergence of Primacy and Recency Effect in Mamba: A Mechanistic Point of View},
year = {2026},
howpublished = {\url{https://pith.science/paper/2UP6XXR2}},
note = {Machine review of arXiv:2506.15156}
}
read the original abstract
We study memory in state-space language models using primacy and recency effects as behavioral tools to uncover how information is retained and forgotten over time. Applying structured recall tasks to the Mamba architecture, we observe a consistent U-shaped accuracy profile, indicating strong performance at the beginning and end of input sequences. We identify three mechanisms that give rise to this pattern. First, long-term memory is supported by a sparse subset of channels within the model's selective state space block, which persistently encode early input tokens and are causally linked to primacy effects. Second, short-term memory is governed by delta-modulated recurrence: recent inputs receive more weight due to exponential decay, but this recency advantage collapses when distractor items are introduced, revealing a clear limit to memory depth. Third, we find that memory allocation is dynamically modulated by semantic regularity: repeated relations in the input sequence shift the delta gating behavior, increasing the tendency to forget intermediate items. We validate these findings via targeted ablations and input perturbations on two large-scale Mamba-based language models: one with 1.4B and another with 7B parameters.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[5]
arXiv preprint arXiv:2311.03839
Aspects of human mem- ory and large language models. arXiv preprint arXiv:2311.03839. Samy Jelassi, David Brandfonbrener, Sham M Kakade, and Eran Malach
-
[7]
arXiv preprint arXiv:2411.19455
Autocorrela- tion matters: Understanding the role of initializa- tion schemes for state space models. arXiv preprint arXiv:2411.19455. Takashi Morita
-
[8]
Emergence of the Primacy Effect in Structured State-Space Models
Emergence of the primacy effect in structured state-space models. Preprint, arXiv:2502.13729. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, and 1 oth- ers
-
[10]
Understanding and mitigating bottlenecks of state space models through the lens of recency and over-smoothing. Preprint, arXiv:2501.00658. Xinyi Wu, Yifei Wang, Stefanie Jegelka, and Ali Jad- babaie
-
[11]
arXiv preprint arXiv:2502.01951
On the emergence of position bias in transformers. arXiv preprint arXiv:2502.01951. Jingwei Zuo, Maksim Velikanov, Dhia Eddine Rhaiem, Ilyas Chahed, Younes Belkada, Guillaume Kunsch, and Hakim Hacid
-
[12]
Falcon mamba: The first competitive attention-free 7b language model. Preprint, arXiv:2410.05355. 9 Appendix A Recall Task Construction and Experimental Setup Task Construction: To evaluate memory be- havior in Mamba-based models, we designed a structured recall task consisting of subject-relation- object triplets. Each input sequence contains L unique tr...
-
[1913]
New York: Teachers College, Columbia University
Memory: A contribu- tion to experimental psychology (1913). New York: Teachers College, Columbia University. Murray Glanzer and Anita R Cunitz
work page 1913
-
[2021]
arXiv preprint arXiv:2111.00396
Effi- ciently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00396. Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin
Show all 12 references
-
[2022]
arXiv preprint arXiv:2209.11895
In-context learning and induction heads. arXiv preprint arXiv:2209.11895. Peihao Wang, Ruisi Cai, Yuehao Wang, Jiajun Zhu, Pragya Srivastava, Zhangyang Wang, and Pan Li
-
[2023]
arXiv preprint arXiv:2312.00752
Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré
-
[2024]
arXiv preprint arXiv:2402.01032
Repeat after me: Trans- formers are better than state space models at copying. arXiv preprint arXiv:2402.01032. Fusheng Liu and Qianxiao Li
-
[2025]
Preprint, arXiv:2410.10781
When attention sink emerges in language models: An empirical view. Preprint, arXiv:2410.10781. John C Jahnke
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.