{"id":"317accbc-f16e-4156-b74b-ad060537d950","arxiv_id":"2412.13328","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SE-Attn and HyLoRA fine-tune hybrid SSMs on sequences up to 8x the pre-training length, approaching full-attention performance at lower cost.","lead":"This paper introduces a training-time attention variant, SE-Attn, that retrieves the most relevant past blocks of tokens for each chunk, plus a LoRA extension, HyLoRA, for fine-tuning hybrid state space models on much longer contexts than their pre-training length. The authors report this adaptation is faster than full attention and beats other sparse attention baselines on long-context benchmarks like RULER.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim treats SE-Attn as an inference-time expansion of eidetic memory, but all headline evaluations revert to Full-Attn; the one SE-Attn-at-inference result (Fig. 11) shows substantial degradation, so the claimed capability is unsupported as deployed.","rationale":"I agree with the reader's weakest-assumption analysis. The load-bearing issue is that the evaluation protocol never tests the claimed mechanism at inference: SE-Attn is used only during fine-tuning, and all headline results use Full-Attn during evaluation. Since the retrieval head is absent from the inference-time computation, any performance gain in the reported comparisons cannot be attributed to SE-Attn's eidetic expansion. Fig. 11 directly shows that deploying SE-Attn at inference degrades a recall-heavy task, which is exactly the setting where the expansion should help. This is a protocol/scope mismatch rather than a hidden internal inconsistency, and it is addressable: either provide a deployment protocol that actually uses SE-Attn and matches Full-Attn performance, or reframe the contribution as an efficient fine-tuning objective. A conditional verdict is therefore appropriate, matching the reader's position. I do not see a separate concern that would warrant outright rejection, because the paper's weaker claim—that SE-Attn fine-tuning plus HyLoRA is a cheaper way to obtain strong full-attention long-context performance—has empirical support; the open question is whether the stronger architectural claim about eidetic memory expansion is real.","tokens_in":24143,"tokens_out":5211,"duration_ms":46531,"concrete_test":"Evaluate the SE-Attn fine-tuned Mamba-2-Hybrid using SE-Attn at inference on the full 11-task RULER suite (not just NIAH-Single-1), using the same hyperparameters as training: chunk sizes drawn from {2048, 4096}, memory block size S=32, top-k=8, and the same RoPE scaling, across context lengths 8192–32768. Compare (i) SE-Attn-at-inference, (ii) the same checkpoint with Full-Attn, and (iii) SW-Attn and Full-Attn baselines. If SE-Attn-at-inference lags Full-Attn by more than 5 average RULER points at 16k/32k, the eidetic-memory-expansion claim fails for deployment; if it matches, Fig. 11's degradation is task-specific and the mechanism is real.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that SE-Attn expands the eidetic memory span of hybrid SSMs by retrieving relevant past blocks. But the experimental protocol in §4.1 and Appendix A explicitly reverts to Full-Attn during evaluation: 'we utilize SE-Attn for efficient fine-tuning and revert to using Full-Attn during evaluation' and 'Our models are efficiently fine-tuned using SE-Attn and use Full-Attn during evaluation.' Consequently, all headline RULER and LM-Harness numbers (Fig. 2; Tables 1, 4, 6–9) measure a model whose inference-time attention is standard full attention over the whole context. The retrieval head—the relevancy scoring and top-k block selection that defines SE-Attn—is not exercised at inference. The only direct test of SE-Attn at inference, Fig. 11(b), shows that evaluating the SE-Attn-fine-tuned checkpoint with SE-Attn on NIAH-Single-1 is substantially worse than evaluating the same checkpoint with Full-Attn. Thus the 'eidetic expansion' is not validated in the deployed configuration; the observed gains could come from the HyLoRA training of 1D convolution layers, the fine-tuning protocol, or the chunked retrieval objective as a regularizer. This is a scope/protocol mismatch: the paper demonstrates an efficient fine-tuning recipe, not an inference-time memory expansion. The abstract's 'our models can eidetically access tokens from beyond the Attention span' is therefore unsupported as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Span-Expanded Attention (SE-Attn), a sparse attention mechanism that reserves part of the context for past memory blocks retrieved by relevance to the current query, and HyLoRA, a LoRA-based fine-tuning method that also adapts 1D convolutional layers in hybrid SSMs. The authors claim that SE-Attn expands the eidetic memory span of hybrid SSMs and allows efficient fine-tuning of pre-trained models on sequences up to 8 times longer than the pre-training context. The main experiments fine-tune Mamba-2-Hybrid 2.7B, Zamba2 1.2B, and Llama1 7B, evaluating on PG-19, RULER, LM Harness, LongBench, and in-context recall tasks, comparing against Full-Attn, SW-Attn, and S2-Attn.","tokens_in":24395,"tokens_out":4555,"duration_ms":38406,"significance":"If the central claim were established, SE-Attn would be a valuable drop-in mechanism for extending the effective context of hybrid SSMs with modest training cost, and HyLoRA would be a useful recipe for adapting the recurrent and convolutional components of such models. The manuscript has several strengths: it ships a clean ablation chain (retrieval vs. random vs. no retrieval; chunk size; block size/top-k), an empirical runtime-memory analysis, and evaluation across three model families. The internal comparisons are consistent, and the paper is honest about the limitations of perplexity as a long-context metric. However, the headline claim is not validated in the deployed configuration, because the retrieval mechanism is disabled at inference, and the one direct inference-time test shows substantial degradation.","major_comments":[{"comment":"The experimental protocol replaces SE-Attn with Full-Attn at evaluation: §4.1 states that SE-Attn is used for efficient fine-tuning and evaluation reverts to Full-Attn, and Appendix A repeats this. Consequently, all headline results in Table 1, Figure 2, and Tables 6–9 measure a model whose inference-time attention is standard full attention. The abstract's claim that 'our models can eidetically access tokens from beyond the Attention span' is therefore not supported by these experiments, since the retrieval mechanism—the relevancy scoring and top-k block selection—is not exercised at inference.","section":"§4.1, Appendix A"},{"comment":"The only direct evaluation of SE-Attn at inference, Figure 11(b), shows a clear degradation on the NIAH-Single-1 RULER task compared to evaluating the same checkpoint with Full-Attn. This is load-bearing because it undermines the interpretation that the training-time gains are caused by an expanded eidetic span; the gains could instead come from the HyLoRA fine-tuning protocol or from retrieval acting as a regularizer during training. The authors should either provide SE-Attn-at-inference results across the full RULER suite (and ideally on the other benchmarks) showing competitive performance, or explicitly reframe the contribution as an efficient long-context fine-tuning method rather than an inference-time memory expansion.","section":"Appendix E.4, Figure 11"}],"minor_comments":[{"comment":"The fact that evaluation reverts to Full-Attn is disclosed only in §4.1 and Appendix A; it should be stated prominently in the abstract or introduction, because the current abstract implies that SE-Attn is the deployed mechanism.","section":"Abstract/Introduction"},{"comment":"The SE-Attn versus Full-Attn curves in Figure 11 would be easier to interpret with numeric values or error bars, since the differences are small at some sequence lengths and the curves are read only from the plot.","section":"Appendix E.4, Figure 11"},{"comment":"No variance or statistical significance is reported for the RULER and LM Harness comparisons; given that several gaps between SE-Attn and Full-Attn are under one or two percentage points, reporting standard deviations across seeds would strengthen the claims.","section":"Table 1 and Figure 2"},{"comment":"The abbreviation list contains a typo: 'ScorllsNarrativeQA' should be 'ScrollsNarrativeQA'.","section":"Appendix J"},{"comment":"There is a typo in the text: 'perfromance' should be 'performance'.","section":"Appendix E.1"}],"recommendation":"major_revision","confidential_remarks":"The central issue is a protocol mismatch rather than an internally inconsistent derivation. I believe the paper can be made suitable by either adding inference-time evaluations of SE-Attn on the full benchmark suite or by recalibrating the claims to describe an efficient fine-tuning method. The existing ablation and runtime analyses are solid and should be retained. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper reports a useful empirical result: fine-tuning hybrid SSMs with retrieval-based sparse attention plus a LoRA+ variant gives better downstream performance than sliding-window or shifted-sparse baselines. But the central claim about inference-time eidetic memory is not supported. Every headline number comes from evaluating with full attention, and the one experiment that deploys SE-Attn at inference (Fig. 11) shows it underperforms full attention substantially.\n\nThe SE-Attn design—reserving part of the attention context for retrieved blocks—is a reasonable combination of Landmark Attention and B'MOJO, and HyLoRA (adding 1D conv training to LoRA+) is a simple, effective trick. The ablation study is thorough: retrieval beats no retrieval and random retrieval, and the block size/top-k tradeoff is explored. The runtime analysis is honest and useful. The transfer to Llama and Zamba2 strengthens the empirical case.\n\nThe soft spot is not minor. The paper says the models can 'eidetically access tokens from beyond the Attention span' but never tests that at inference. Fig. 11 shows that evaluating with SE-Attn sharply drops RULER NIAH accuracy. So what's demonstrated is a training-time efficiency tool, not an inference-time memory expansion. The authors are transparent about the full-attention evaluation in Appendices A and E.4, but the abstract and intro overstate the result. Also: the main fine-tuning dataset is never named ('a common language dataset'), no code or data is released, there are no error bars, and the closest prior methods (Landmark Attention, B'MOJO) are not used as baselines. Hyperparameters tuned on RULER (variable chunk sizes, learning rate) are a minor concern given the benchmark-specific gains.\n\nBottom line: a solid empirical paper with a flawed framing. For practitioners fine-tuning Mamba-2 or Zamba models, the recipe is likely useful; the internal comparisons are consistent and the ablations support the training-time benefit. I'd accept this for peer review with major revisions: either provide a deployment protocol for SE-Attn at inference or reframe the contribution as efficient fine-tuning. A serious referee should engage with the method and the eval gap.","headline":"Useful efficient fine-tuning recipe for hybrid SSMs, but the eidetic-memory-at-inference claim is not supported by the experiments.","tokens_in":25038,"tokens_out":2524,"would_cite":true,"duration_ms":23954,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hybrid state space model can expand its eidetic memory span by reserving part of its attention context for tokens retrieved by relevance rather than recency.","keywords":["hybrid state space models","span-expanded attention","eidetic memory","relevance-based retrieval","long-context adaptation","LoRA fine-tuning","RULER benchmark"],"falsifier":"Fine-tune the same hybrid model with SE-Attn, with SE-Attn-Random (random memory-block retrieval), and with SE-Attn-NoMem (no retrieval), keeping HyLoRA and all training details identical, then evaluate all three with full attention on RULER at 16k tokens: if relevance-based retrieval does not clearly beat random retrieval, the paper's claim that retrieval quality drives the expanded memory span is unsupported.","tokens_in":23874,"feed_emoji":"🧠","tokens_out":5517,"duration_ms":44757,"temperature":0.7,"pith_summary":"Hybrid state space models combine a recurrent 'fading' memory with finite-window attention, so tokens beyond the attention span are only accessible through lossy compression. This paper argues that memory allocation should follow relevance, not recency: reserve a fraction of the attention context for tokens retrieved from arbitrarily far back, and let the model decide which past blocks matter for the current query. The proposed mechanism, Span-Expanded Attention (SE-Attn), together with a hybrid-specific LoRA variant (HyLoRA), lets a pre-trained 2.7B hybrid model be adapted to sequences up to eight times its pre-training length with training cost comparable to sliding-window attention and better long-context accuracy than sliding-window and shifted-sparse attention baselines. A reader would care because it offers a cheap path to long-context recall in memory-efficient architectures without extra hardware or full-length attention during training.","feed_headline":"Attention that retrieves by relevance extends hybrid SSM memory 8x","feed_subtitle":"A 2.7B hybrid model fine-tuned with a reserved retrieval span matches full-attention long-context recall at sliding-window cost.","key_machinery":"The load-bearing object is the 'expansion span': a reserved fraction of each attention chunk's context, populated not by the most recent tokens but by the top-k 'memory blocks' retrieved from the past. Each memory block is summarized by a single vector c_j, the average of a non-causal self-attention output over the block; a chunk's queries score all blocks by a cross-attention relevancy score, a causal mask blocks future blocks, and softmax plus top-k selects which blocks enter the expansion span. This keeps the per-chunk attention computation at a fixed size, a few hundred retrieved tokens plus the chunk itself, while giving the layer access to tokens from arbitrarily far back. It is trained end-to-end with HyLoRA, which adapts attention projections with low-rank adapters and also trains the 1D convolutions inside SSM layers.","core_discovery":"The paper's central claim is that a hybrid SSM's eidetic (verbatim) memory span can be expanded by making a small part of each attention layer's context a 'reserve' populated through retrieval rather than recency. SE-Attn splits the input into chunks and, for each chunk, computes relevancy scores between the chunk's queries and compressed summaries of past memory blocks, retrieves the top-k blocks, and concatenates their keys and values into the chunk's attention context. Because the model is trained with this retrieval pattern, it learns to allocate memory where it matters; at inference the authors revert to full attention, which benefits from the retrieval-trained weights. The authors report that on RULER, LM Harness long-context tasks, and LongBench, models fine-tuned this way match full-attention fine-tuning and beat sliding-window and shifted-sparse attention, and that HyLoRA, which augments LoRA+ by training the small 1D convolutional layers inside SSM blocks, is what makes the adaptation work.","pith_inferences":["Because the paper evaluates retrieval benefit under a full-attention decoder, a natural next step is to make the retrieval mechanism itself inference-friendly, such as caching retrieved block summaries, to close the gap shown in the appendix between SE-Attn evaluation and full-attention evaluation.","The relevance-scoring idea could combine with learned compression tokens or hierarchical summaries, allowing the expansion span to scale sublinearly with history while keeping the retrieval decision cheap.","Relevance-based memory allocation may also benefit pure recurrent models or linear-attention models, where the analogous failing is that the state is updated uniformly by recency rather than by importance.","Since random retrieval already helps over no retrieval, part of the gain may be a regularizing effect of variable-length chunked training; ablating retrieval quality while holding chunking constant would separate the two effects."],"forward_implications":["Pre-trained hybrid SSMs with short 2k contexts can be adapted to sequences up to 8 times longer without expensive full-length fine-tuning.","SE-Attn offers a runtime-memory trade-off comparable to sliding-window attention while giving better long-context task accuracy, so relevance-based retrieval can replace recency-based windows without extra hardware.","The method transfers beyond the main test model: it improves long-context performance on Llama 1 7B and extends Zamba2-Hybrid's context from 4k to 12k tokens.","HyLoRA, which trains the 1D convolution layers in addition to LoRA+ on attention layers, is necessary for strong long-context adaptation of hybrid SSMs; LoRA or LoRA+ alone underperform.","Perplexity is not a faithful indicator of long-context recall; retrieval-style benchmarks like RULER reveal differences that perplexity hides."],"supporting_citations":[{"why":"Supplies the S2-Attn baseline, the LoRA+ fine-tuning recipe, and the training/evaluation protocol that SE-Attn and HyLoRA build on.","marker":"Chen et al. (2024)"},{"why":"Landmark Attention is the closest retrieval-based method; SE-Attn positions itself as native retrieval without landmark tokens.","marker":"Mohtashami and Jaggi (2023)"},{"why":"B'MOJO introduces hybrid SSMs with eidetic plus fading memory and the idea of attending to hard-to-predict past tokens that SE-Attn generalizes.","marker":"Zancato et al. (2024)"},{"why":"Provides Mamba-2 and the structured state space duality underlying the Mamba-2-Hybrid architecture used in the main experiments.","marker":"Dao and Gu (2024)"},{"why":"Introduces the Mamba-2-Hybrid 2.7B model that serves as the main evaluation subject.","marker":"Waleffe et al. (2024)"},{"why":"RULER benchmark supplies the long-context recall tasks on which SE-Attn is compared.","marker":"Hsieh et al. (2024)"},{"why":"LoRA is the parameter-efficient fine-tuning base that HyLoRA extends by training SSM convolutions.","marker":"Hu et al. (2022)"},{"why":"Sliding-window attention is one of the two primary baselines and a cost reference.","marker":"Beltagy et al. (2020)"},{"why":"Full attention is the paragon baseline and the core mechanism that SE-Attn modifies.","marker":"Vaswani et al. (2017)"}],"fun_headline_variants":["Retrieval-based attention gives hybrid SSMs 8x longer eidetic memory","Relevance beats recency: new attention expands hybrid SSM memory span","Hybrid SSMs get eidetic recall of distant past with SE-Attn and HyLoRA","SE-Attn retrieves relevant past tokens to extend hybrid SSM context 8x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All headline results are obtained by fine-tuning with SE-Attn but evaluating with full attention, so the load-bearing premise is that retrieval during training alone, not the fine-tuning protocol or the adapted convolutional layers, is what creates the expanded recall; if that transfer fails, the central eidetic-expansion claim is not established.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval-based attention gives hybrid SSMs 8x longer eidetic memory","Relevance beats recency: new attention expands hybrid SSM memory span","Hybrid SSMs get eidetic recall of distant past with SE-Attn and HyLoRA","SE-Attn retrieves relevant past tokens to extend hybrid SSM context 8x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000928,"raw_usage":{"total_tokens":4052,"prompt_tokens":1097,"completion_tokens":2955,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":713,"completion_tokens_details":{"reasoning_tokens":2862}},"tokens_in":713,"tokens_out":2955,"duration_ms":18772,"temperature":1.0,"reasoning_tokens":2862,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:14:30.638254+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fine-tune the same hybrid model with SE-Attn, with SE-Attn-Random (random memory-block retrieval), and with SE-Attn-NoMem (no retrieval), keeping HyLoRA and all training details identical, then evaluate all three with full attention on RULER at 16k tokens: if relevance-based retrieval does not clearly beat random retrieval, the paper's claim that retrieval quality drives the expanded memory span is unsupported.","supporting_citations":[{"cited_title":"Random-access infinite context length for transformers","cited_arxiv_id":null,"evidence_quote":"Landmark Attention is the closest retrieval-based method; SE-Attn positions itself as native retrieval without landmark tokens."},{"cited_title":"B mojo: Hybrid state space realizations of foundation models with eidetic and fading memory","cited_arxiv_id":null,"evidence_quote":"B'MOJO introduces hybrid SSMs with eidetic plus fading memory and the idea of attending to hard-to-predict past tokens that SE-Attn generalizes."}],"review_version":1}