{"id":"73872383-85e1-4a0c-a3a1-b040548cefdb","arxiv_id":"2506.15156","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Mamba's primacy and recency effects trace to sparse long-memory channels, recurrence-based decay, and relation-repetition-dependent delta gating.","lead":"State-space language models like Mamba show a U-shaped memory curve in structured recall: the first and last items are recalled best, and the middle is worst. The paper locates this pattern in sparse long-term-memory channels, exponential decay in the recurrence, and a delta gate that reacts to repeated relations.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The causal ablation in §4.2 is underspecified: zeroing A at the first token's timestep cannot remove B_1 x_1 from the recurrence, so the reported primacy drop requires either a corrected intervention or another mechanism.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the causal intervention in §4.2, as described, does not follow from the recurrence equations. My analysis confirms this under the literal interpretation of zeroing A at the first token's timestep, and shows that even under the more charitable object-token interpretation the intervention is a state reset rather than an entry-blocking ablation of early tokens. Because the central claim depends on this intervention, and no code or precise timestep specification is provided to resolve the ambiguity, the paper as written does not support its causal conclusion. I therefore do not recommend changing the reader's REJECT verdict. At the same time, the behavioral U-shape and the other empirical observations may be sound; the flaw is in the mechanistic attribution, and a corrected description or code could make the claim testable.","tokens_in":11978,"tokens_out":7692,"duration_ms":84704,"concrete_test":"Obtain or reconstruct the intervention code and run two ablations on Falcon Mamba 7B (L=16, repeated-relation condition) using channels selected by M^(i)>0.7: (i) set A_1=0 at the first token's timestep; (ii) set A_3=0 at the first triplet's object token. Record accuracy at position 1 and also compute the first-token contribution to the query-time state (via Eq. 2) before and after each intervention. If (i) leaves accuracy and first-token contribution unchanged while (ii) drops them, the paper's paragraph describes a state reset rather than a selective early-token ablation. If neither matches, the selection criterion or intervention is acting through a different, unreported mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that first-position recall in Mamba is causally supported by a sparse set of selective-SSM channels, established by the targeted ablation in §4.2. The described intervention zeroes out A_t 'at the timestep of the first triplet.' From the unrolled recurrence in Eq. 2, h_t = Σ_j (Π_{k=j+1}^t A_k) B_j x_j, setting A_1 = 0 leaves the first token contribution B_1 x_1 in the state, and that contribution propagates through A_2...A_T unchanged. Thus, under the literal reading, the intervention cannot block early-token influence. If instead 'timestep of the first triplet' means the object-token timestep (t=3), zeroing A_3 discards the pre-object state containing s_1 and r_1 but leaves B_3 x_{o1} in the state; this is a one-step state reset, not an ablation of 'early information entering' the long-memory pathways. The paper does not specify which timestep is zeroed, does not provide code, and does not report controls such as zeroing B_1 or zeroing A at a later timestep. Moreover, the selection criterion M^(i) measures only the cumulative product of A_t; it does not verify that B_1 x_1 is nonzero for the selected channels. The behavioral U-shape may be reproducible, but the causal link between 'long-term memory channels' and early-token content is not established by the text as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies primacy and recency effects in Mamba-style state space models using a structured subject-relation-object recall task on Falcon Mamba 7B and Mamba 1.4B. It reports a U-shaped recall accuracy curve and proposes three mechanisms: long-term memory localized in a sparse subset of selective SSM channels (causal claim), recency from delta-modulated exponential decay, and semantic regularity modulating delta gating to increase forgetting of intermediate items. The central causal claim is based on ablating the recurrence matrices A_t in channels selected by a memory coefficient.","tokens_in":12212,"tokens_out":8137,"duration_ms":80754,"significance":"If the causal claims were established, the paper would be a valuable mechanistic account of memory biases in attention-free architectures and could inform SSM design. The U-shaped behavioral result is clearly presented and replicated across two model families and several context lengths, and the task design (single-token vocabulary, 50 sequences per position) is thoughtful. However, the central causal claim about localized long-term memory is not supported as written because the intervention is underspecified relative to the recurrence equations, the selection/evaluation protocol is ambiguous, and quantitative evidence for the required selectivity is missing. The third mechanism also rests on an indirect synthetic experiment rather than direct measurement on the recall task.","major_comments":[{"comment":"The causal intervention is underspecified and, on the natural reading, does not block the first token's influence. The paper states that zeroing A_t 'at the timestep of the first triplet' blocks early information from entering long-memory pathways. From Eq. (2), the first token's contribution at time t is (Π_{k=2}^t A_k) B_1 x_1; zeroing A_1 leaves B_1 x_1 in the state, while zeroing A_3 (the object token of the first triplet) resets the state to B_3 x_3 and drops the subject/relation contributions but still leaves the object token in the state. Neither operation is an ablation of 'early information entering' these pathways. The text does not say which timestep is zeroed, nor does it provide control ablations (e.g., zeroing B_1 or zeroing A at a later triplet) or the exact code. As written, the central claim of localized long-term memory is not supported.","section":"4.2, Eq. (2)"},{"comment":"The channel-selection and evaluation protocol is ambiguous about whether the same input sample is used to select channels and to measure the intervention effect. Algorithm 1 says a single forward pass on an input sample is used to extract A_t and B_t and to select channels; if the intervention is then evaluated on the same sample(s), the reported drop may be specific to those inputs rather than a general property of the channels. The paper should report the stability of selected channels across the 50 sequences and evaluate interventions on held-out sequences.","section":"4.2, Algorithm 1 and Figure 3b"},{"comment":"No error bars, confidence intervals, or significance tests are reported for the recall curves or the intervention effects. Claims such as 'only the targeted intervention leads to a significant drop' are unsupported; with 50 examples per position, the reader cannot assess whether the observed differences are within sampling noise. Bootstrap or permutation tests, at least for the first-position intervention comparison, are needed.","section":"Figures 2, 3b, 6, 11-13"},{"comment":"The synthetic periodic-token experiment does not directly test the repeated-relation condition of the recall task. The recall task repeats an entire relation token within an s-r-o structure, while the synthetic inputs repeat a single token every k positions. The paper asserts that the resulting delta behavior 'could explain' the repeated-relation effect, but it does not measure delta on the actual repeated-relation versus random-relation contexts. Without that measurement, the third mechanism remains a hypothesis rather than a validated finding.","section":"4.4, Figures 7-9"}],"minor_comments":[{"comment":"There is a typo in the description of Eq. (1): 'dimesion' should be 'dimension'.","section":"3.1"},{"comment":"Using the phrase 'probability p' for the deterministic cutoff p is confusing; a term such as 'proportion cutoff' would be clearer.","section":"4.2, Eq. (4) and Definition box"},{"comment":"The statement that the j-th contribution is 'proportional to A^{t-j}' assumes a time-invariant A; in Mamba, A_t is input-dependent, so the product Π A_k should be used unless a constant-A simplification is explicitly stated.","section":"4.3, Eq. (5)"},{"comment":"The period values are inconsistent: the text lists k∈{2,4,8,16,32,64}, but Figures 7-9 use 'Period 1' and legends with 1, 4, 16, 64. Please make the notation uniform.","section":"4.4"},{"comment":"The line 'M←Q T t=2 At' is missing a product symbol and has unclear indexing; the phrase 'do Perform a forward pass' is ungrammatical. Also specify which input sequence (length and relation type) is used for channel selection.","section":"Appendix B, Algorithm 1"},{"comment":"The caption ends mid-sentence: 'Effect of initializing the recurrent state at Layer 31 using uniform values on repeated relation' should be completed.","section":"Figure 5 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid behavioral core and a clearly presented U-shaped recall result, but the causal mechanism section needs substantial additional work: an unambiguous description of the intervention, control ablations, and stability/statistical validation. These are fixable within the manuscript's scope, which is why I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the U-shaped recall profile in Mamba is likely real and worth knowing about, but the paper's central causal claim about sparse 'long-term memory channels' rests on an intervention that is not specified tightly enough to support it.\n\nWhat's new: unlike prior work that looked at primacy in S4 or recency in SSMs separately, this paper documents both effects in Falcon Mamba 7B and Mamba 1.4B on a controlled recall task, and it adds a channel-level localization attempt plus a delta-gating analysis for repeated relations. The random-intervention control is a good idea, and the observation that low-frequency inputs increase average delta is a concrete, testable handle on the architecture.\n\nSoft spots: the intervention in §4.2 is the load-bearing piece. The text says zeroing the recurrence matrix A_t at the timestep of the first triplet blocks early input from entering the long-memory pathways. Under the literal reading (zeroing A_1), B_1 x_1 remains in the state and propagates through later A's, so the intervention does not isolate early tokens. If the authors meant zeroing A at the final token of the first triplet, that is a state reset, and the paper does not say. No code is provided, and no control like zeroing B_1 or zeroing a later A is reported. The channel-selection metric M(i) is the cumulative product of A alone; it does not check whether B_1 x_1 is nonzero for the selected channels, so a channel could score high while carrying no early information. Meanwhile, channels are identified and tested on the same sequences, which invites selection bias. The error bars are also missing. The delta-periodicity analysis is interesting but only indirectly tied to the recall-task finding.\n\nNone of this kills the empirical U-shape: the figures are consistent across two model sizes and multiple lengths. But as written, the mechanistic story is not established. This is fixable with a corrected or clarified intervention, code release, and held-out channel selection.\n\nWho it's for: people working on SSM interpretability and memory in attention-free models. It deserves a serious referee, but the version in front of us would need major revision, not quick acceptance. I'd send it out, with a clear instruction to the authors to clarify the ablation.","headline":"U-shaped recall in Mamba is probably real, but the paper's causal 'long-term memory channel' claim rests on an underspecified ablation.","tokens_in":12827,"tokens_out":4174,"would_cite":true,"duration_ms":41916,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mamba's better recall of early sequence items is carried by a sparse subset of internal channels; disrupting those channels erases early-token memory while preserving recent memory.","keywords":["Mamba","state space models","primacy effect","recency effect","long-term memory","mechanistic interpretability","structured recall","selective state space block"],"falsifier":"Run the model with the targeted intervention while recording the contribution of the first token $x_1$ to the final hidden state via the unrolled sum; if the contribution is unchanged yet first-position accuracy falls, the ablation is not blocking early-token flow. Comparing zeroing $A$ at the first triplet's timestep against zeroing all later $A$ products would settle whether the identified channels are the actual carriers of primacy.","tokens_in":11711,"feed_emoji":"🧠","tokens_out":7203,"duration_ms":64825,"temperature":0.7,"pith_summary":"The paper tries to establish that Mamba's U-shaped recall curve—strong memory for the first and last items, poor memory for the middle—is not a superficial artifact but the product of three identifiable mechanisms: a sparse set of channels that act as a long-term store, exponential decay that favors recent tokens, and a learned discretization gate that speeds forgetting when input structure repeats. It matters because it challenges the assumption that attention is required for structured memory and because it shows where, mechanistically, long-range memory lives in an attention-free architecture. A sympathetic reader would take away that early-token retention is localized, recent-token retention is fragile under distraction, and semantic regularity actively reshapes the model's forgetting schedule.","feed_headline":"Mamba stores long-term memory in a sparse channel set","feed_subtitle":"Ablating those channels erases recall of early inputs while leaving recent recall intact, says a causal study of two large Mamba models.","key_machinery":"The load-bearing object is the selective state-space recurrence $h_t^{(i)} = A_t^{(i)} h_{t-1}^{(i)} + B_t^{(i)} x_t^{(i)}$, with its unrolled form expressing each hidden state as a sum of past input contributions scaled by products of the recurrence matrices $A_k$. The paper defines a long-term memory coefficient $M^{(i)} = \\mathrm{diag}(\\prod_{t=2}^T A_t^{(i)})$ to score how much of each channel's hidden state survives to the final context timestep, and uses the learned per-token discretization $\\Delta_t$ as the forgetting gate that couples input injection and state decay. Together these give a quantitative account of why early tokens persist (high $A$ products in a few channels), why recent tokens dominate (fewer decay steps), and why periodic or repeated inputs forget faster ($\\Delta_t$ grows when input frequency is low).","core_discovery":"On structured recall tasks with subject-relation-object triples, Falcon Mamba 7B and Mamba 1.4B produce a consistent U-shaped accuracy profile across sequence lengths 8 through 128. The central discovery is that primacy in Mamba is causally supported by a sparse subset of channels, concentrated at particular layers (notably layer 17 in Falcon Mamba 7B), whose cumulative recurrence product $M^{(i)} = \\mathrm{diag}(\\prod_{t=2}^T A_t^{(i)})$ stays above threshold; ablating the recurrence matrices of these channels at the first triplet's timestep drops first-position accuracy sharply, while ablating random channels does not. Recency is explained by the unrolled dynamics $y_t^{(i)} = C_t \\sum_{j=1}^t (\\prod_{k=j+1}^t A_k^{(i)}) B_j^{(i)} x_j^{(i)}$, where recent inputs undergo fewer decay steps; inserting distractor tokens erodes this advantage. Finally, repeating the same relation token across the sequence shifts the learned discretization $\\Delta_t$ toward faster forgetting and stronger input injection, worsening the lost-in-the-middle effect, whereas randomizing relations partially restores middle recall.","pith_inferences":["A testable extension the paper does not run: if long-term memory is genuinely confined to a sparse channel set, then weight-pruning or low-bit quantization targeted at those channels should degrade long-context retrieval far more than short-context retrieval, with a random-pruning control separating channel identity from mere parameter count.","The $\\Delta_t$ periodicity result implies a simple prompt-level lever: diversifying syntactic and semantic templates in long contexts should flatten the lost-in-the-middle curve even without fine-tuning; that is a direct, untested prediction of the paper's mechanism.","The paper's localization claim concerns Mamba specifically; whether sparse long-term memory channels are a general property of state-space architectures or a learned byproduct of this architecture's initialization would require the same intervention in other state-space model families."],"forward_implications":["Long-term recall in Mamba can be degraded surgically: ablating the recurrence matrices of high-memory channels selectively damages first-position recall, while random ablations of equal size leave it intact.","Recency in Mamba is a dynamic, saturating property rather than a fixed architectural bias; inserting 192 to 384 distractor tokens degrades recall across all positions, with the largest losses near the end of the sequence.","Input regularity changes memory allocation: repeated relations push $\\Delta_t$ to larger values, producing faster forgetting of middle items, while randomizing relations improves middle recall.","The localization pattern generalizes across scale: Mamba 1.4B also shows the effect, though its long-term memory channels are more widely distributed and require ablating more layers to disrupt.","Because the same $\\Delta_t$ modulates both the recurrence matrix $A_t$ and the input gate $B_t$, the architecture couples forgetting with input integration; decoupling these could improve retrieval from arbitrary positions."],"supporting_citations":[{"why":"Defines the Mamba selective state-space architecture and the S6 recurrence that the paper's memory-coefficient and delta-gating analyses operate on.","marker":"Gu and Dao, 2023"},{"why":"Provides Falcon Mamba 7B, the primary model on which the U-shaped recall curve and channel ablations are measured.","marker":"Zuo et al., 2024"},{"why":"Supplies the two-store memory model (long-term versus short-term) that motivates separating primacy from recency and the channel analysis.","marker":"Glanzer and Cunitz, 1966"},{"why":"Prior observation of primacy in structured state-space models that this work extends by causally localizing the effect in Mamba.","marker":"Morita, 2025"},{"why":"Prior formal analysis of recency and over-smoothing in state space models that the paper uses for the exponential-decay interpretation of recency.","marker":"Wang et al., 2025"},{"why":"Names the Ranschburg effect, the repetition-induced forgetting pattern the paper adapts to explain repeated-relations accuracy loss.","marker":"Jahnke, 1969"},{"why":"Documents position bias and attention-sink primacy in transformers, the contrast that motivates showing attention is not necessary for structured memory.","marker":"Wu et al., 2025"}],"fun_headline_variants":["Sparse channels and delta-modulated recurrence shape Mamba recall","Primacy and recency in Mamba traced to three mechanisms","Mamba's memory: sparse channels for early, decay for recent","U-shaped accuracy in Mamba: how memory fades over time"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The causal claim assumes that zeroing the recurrence matrix $A_t$ at the timestep of the first triplet blocks the first token's influence; under the stated update rule $h_t = A_t h_{t-1} + B_t x_t$ the intervention leaves $B_1 x_1$ in the state, so the observed drop in early recall may not isolate long-term memory channels.","fun_headline_variants_meta":{"raw":{"variants":["Sparse channels and delta-modulated recurrence shape Mamba recall","Primacy and recency in Mamba traced to three mechanisms","Mamba's memory: sparse channels for early, decay for recent","U-shaped accuracy in Mamba: how memory fades over time"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1512,"prompt_tokens":1004,"completion_tokens":508,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":435}},"tokens_in":620,"tokens_out":508,"duration_ms":5284,"temperature":1.0,"reasoning_tokens":435,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:42:30.734097+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the model with the targeted intervention while recording the contribution of the first token $x_1$ to the final hidden state via the unrolled sum; if the contribution is unchanged yet first-position accuracy falls, the ablation is not blocking early-token flow. Comparing zeroing $A$ at the first triplet's timestep against zeroing all later $A$ products would settle whether the identified channels are the actual carriers of primacy.","supporting_citations":[],"review_version":2}