{"id":"68a0a775-6502-49eb-a5f6-7b2e64a2136b","arxiv_id":"2607.25357","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Raven is a linear-time sequence model whose sparse, input-dependent routing writes tokens into dedicated memory slots, preserving long-context recall and extrapolating 16x beyond training length.","lead":"Raven is a new linear-time language model that chooses which memory slots to overwrite for each token, instead of updating all memory densely. It retrieves specific past tokens far longer than state-space models or window attention, keeping near-perfect passkey recall at 16x training length.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 400M results appear to use TopK=128 while Section 6.4 recommends TopK=32; the central configuration is internally inconsistent.","rationale":"The paper's central empirical claim is that Raven, at 400M scale, retains >91% NIAH-1 accuracy at 32K tokens, 16× its 2K training length, due to sparse input-dependent routing. For that claim to be valid, the evaluated model must be the same model described and ablated in the paper. The evidence shows a concrete discrepancy: the headline 400M Raven row in Table 2 matches the M=256/TopK=128 row in Table 7 almost exactly, whereas the default configuration used in the ablations and recommended in Section 6.4 is M=256/TopK=32. This is not a matter of missing error bars or code alone; it is an internal inconsistency that makes the central result unreproducible from the text as written. The reader's weakest assumption concerned router collapse and specialization at scale; my concern is adjacent but distinct—it concerns which router configuration actually produced the headline numbers. This does not overturn the paper's potential value, but it sharpens the condition under which the central claim can be accepted: the authors must specify and verify the exact TopK setting. The proposed concrete test—releasing the config and rerunning both K values—would settle the issue definitively. I therefore keep the reader's CONDITIONAL verdict unchanged.","tokens_in":29504,"tokens_out":9137,"duration_ms":81428,"concrete_test":"Release the exact training config and checkpoints for the 400M model used in Table 2 (or the config file from the linked code) and verify the topk value. Then rerun the full Table 2 evaluation for both TopK=32 and TopK=128 with M=256 under the same training recipe. If Table 2 reproduces only with TopK=128, the paper must either adopt that as the stated default and correct Section 6.4, or the headline claim must be re-evaluated at the recommended K=32. If Table 2 reproduces with K=32, Section 6.4 is fine but Table 2's provenance still needs a one-line clarification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.4 concludes: \"We find that K=32 (out of M=256 slots, i.e., 12.5% occupancy) strikes the best balance,\" and the default used in the architectural ablations (Table 5) and router ablations (Table 6) is 256 slots / Top-32. However, the headline 400M row in Table 2 is numerically the M=256 / TopK=128 row from Table 7: SWDE 34.1, FDA 22.7, SQuAD 35.4, NIAH-1 32K 91.4, and the NIAH-2/NIAH-3 columns also match; the only difference is one NIAH-1 4K cell (99.8 vs 99.4). The paper never states that the final model uses TopK=128, and it does not report the TopK=32 configuration on the headline benchmarks. This matters because K is not a nuisance parameter: Table 7 shows NIAH-2 at 32K varies from 2.4 (K=32) to 23.0 (K=128), and NIAH-3 at 8K from 11.8 to 1.0. The central claim—that sparse routing with ~12.5% writes preserves tokens to 16× training length—is attached to an underspecified or internally inconsistent configuration. The concern is about reproducibility of the empirical result, not about author intent.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Routing Slot Memories (RSMs), a framework that unifies linear sequence models by making the write location explicit through a routing vector, and introduces Raven, an instantiation with sparse input-dependent top-K routing and slot-wise decay. The authors argue that this combination sits between dense SSMs and sliding-window attention and preserves individual token memories far beyond the training context. They evaluate Raven at 400M and 800M scales on NIAH passkey retrieval, real-world extraction tasks (SWDE, FDA, SQuAD), zero-shot language modeling, and hybrid architectures. The paper also includes ablations of router design, memory shape, and top-K sparsity, plus analyses of slot specialization via effective sequence length and memory visualizations.","tokens_in":29814,"tokens_out":5326,"duration_ms":51803,"significance":"If the empirical claims hold, Raven is a valuable architecture: it is simple, linear-time, needs no convolutions, and reportedly retains high recall at 16x the training length, a regime where dense SSMs and SWA both fail. The RSM framework is a clean unifying perspective and the formal slot-separability proposition is a nice contribution. The paper is also candid about the router-collapse risk and provides a reasonable set of ablations. However, the strength of the paper rests on a small number of benchmark numbers, and the current manuscript has an internal inconsistency about the exact configuration used for the headline results, plus a lack of any variance information. These issues are fixable but need to be addressed before the empirical claims can be fully trusted.","major_comments":[{"comment":"The headline 400M Raven row in Table 2 (SWDE 34.1, FDA 22.7, SQuAD 35.4, NIAH-1 32K 91.4, NIAH-2 32K 23.0, NIAH-3 8K 13.4) is numerically identical to the M=256/K=128 row of Table 7, up to one NIAH-1 4K cell (99.8 vs. 99.4). Yet §6.4 concludes that K=32 \"strikes the best balance,\" and Tables 5 and 6 use K=32. The paper never states which K is used in Table 2. This is load-bearing for reproducibility: K is not a nuisance parameter, since Table 7 shows NIAH-2 at 32K varies from 2.4 (K=32) to 23.0 (K=128). In addition, the sentence in §6.4 that \"Smaller K ... improving recall on SWDE\" is contradicted by Table 7 for M=256, where SWDE is 31.5 at K=32 and 34.1 at K=128. Please state the exact configuration of Table 2, reconcile it with §6.4, and if K=128 is the final choice, report the K=32 configuration on the headline benchmarks.","section":"§6.4 vs. Table 2 and Table 7"},{"comment":"The central empirical claim is partly circular because the same benchmarks used as evidence in Table 2 appear in the model-selection ablations (Tables 5–7). The paper selects router type, Gumbel noise, M, K, and α using NIAH/SWDE/FDA/SQuAD performance, and then reports the best configuration on the same tasks as a headline result. The text in §6.2.3 says \"we select the best-performing configuration per Table 6,\" but no held-out split or selection protocol is described. To support the claim that sparse routing itself (not the particular hyperparameter choice) provides the reported benefits, the authors should either pre-register a fixed configuration and report all configurations, or use a held-out validation split for model selection and then evaluate the final model only once.","section":"§5.1 and Tables 5–7"},{"comment":"All benchmark numbers are point estimates with no standard deviations, confidence intervals, or seed counts. NIAH is known to be noisy, and several of the claimed advantages are small (e.g., Raven 400M vs. GLA on SWDE, 34.1 vs. 29.0; Raven 400M vs. GDN on SWDE, 34.1 vs. 29.5). Without multiple seeds or at least a stability analysis, these differences may not be significant. Please report the number of runs, the aggregation method, and ideally error bars or at least the variance across seeds for the main tables.","section":"Tables 2–4, 7, 9"},{"comment":"The router normalization α is set to 1 for the 400M model and 4 for the 800M model, but no ablation or sensitivity analysis is provided for this choice. Since α directly scales the routing weights and interacts with the top-K threshold and the decay rate, a scale-dependent heuristic is not sufficient justification. Please provide an ablation over α (at least for one scale) or a principled motivation for why α=1 and α=4 are correct.","section":"§4.1, Eq. (20)"}],"minor_comments":[{"comment":"The 400M Raven hybrid row in Table 4 reports NIAH-3 at 1K as 90.0, while Table 9 reports 89.6 for the same configuration. Please verify and unify these numbers.","section":"Table 4 vs. Table 9"},{"comment":"The caption mentions \"Rec. mem. and Conv. mem.\" but the table body only shows a single memory column; clarify the column layout and what \"12.5 / 0.0\" means.","section":"Table 2 caption"},{"comment":"The \"Raven Code\" and \"Raven Blog\" items are not hyperlinked; if code is meant to be released, provide a working URL or repository identifier.","section":"Paper front matter"},{"comment":"The phrase \"K=32 ... strikes the best balance\" is undefined. Is it based on a single task average, a qualitative judgment, or a computational constraint? Please state the criterion used.","section":"§6.4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a novel architecture and a useful unifying framework, and the core idea is likely sound. However, the internal inconsistency between the headline configuration and the recommended configuration, the lack of variance reporting, and the model-selection circularity need to be resolved before the claims can be accepted. None of these issues appear unfixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Raven is a genuinely interesting design point for recall in linear-time sequence models. The RSM framework gives a clean taxonomy, and the sparse input-dependent routing plus decay combination appears to be new. But before you spend time on the empirical claims, know this: the paper's headline 400M results are internally inconsistent. The Raven row in Table 2 matches the M=256/TopK=128 row of Table 7 almost exactly, while Section 6.4 concludes that K=32 is the best balance and the ablations in Tables 5 and 6 use TopK=32. The paper never states that the final model uses K=128. That is a real flaw, not a nitpick: the K choice changes results substantially. On NIAH-2 at 32K, K=32 gives 2.4 while K=128 gives 8.8; on NIAH-3 at 8K, K=32 gives 11.8 while K=128 gives 1.0. So the 16× extrapolation claim, and much of the head-to-head comparison, rests on a configuration that the paper's own analysis does not recommend and does not clearly document.\n\nOn the positive side, the conceptual contribution is real. Casting SSMs and SWA as endpoints on a routing spectrum is useful, and the ablation study is more thorough than typical: they vary router type, Gumbel noise, projection, memory shape, and TopK. The slot-separability proposition is straightforward but correct. The block drops short convolutions, which is a nice simplification if it holds up. The paper is also honest about Raven trailing GDN at 800M scale on average accuracy. The learned-allocation analysis (retrieval slots) is a nice piece of evidence that routing is doing something content-aware, though it is qualitative.\n\nBeyond the config inconsistency, the usual concerns apply: no code, no commit hash, no variance or seed counts, and NIAH is known to be noisy. The router normalization α and other hyperparameters were selected on the same recall benchmarks used in the headline tables, which makes the numbers look a bit too clean. The router collapse risk is only mitigated by Gumbel noise, and they provide no load-balancing or formal guarantee; the empirical specialization is suggestive but limited to one synthetic task. None of these are fatal on their own, but combined with the TopK mix-up they mean the empirical section needs a serious revision.\n\nThis paper deserves a serious referee, not a desk reject. The architecture is plausible, the framework is valuable, and the central flaw is addressable: fix the configuration reporting, release code, add variance estimates, and re-run the headline comparisons for the config actually used. I would not cite the empirical claims until that is done. The paper is most useful for researchers designing hybrid or linear sequence models who want the RSM taxonomy and the memory-allocation analysis. Recommend conditional acceptance: engage with the work, but make the authors pin down what exactly they evaluated.","headline":"Raven is a genuinely interesting design point for recall in linear-time models, but the paper's headline configuration is internally inconsistent — the 400M results match a TopK=128 setup while the text recommends TopK=32.","tokens_in":30330,"tokens_out":4120,"would_cite":true,"duration_ms":39235,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A linear-time sequence model preserves token memories over long contexts by routing writes to a sparse, input-dependent subset of memory slots, and shows near-perfect recall at 16x its training length.","keywords":["routing slot memories","sparse routing","linear-time sequence models","long-context recall","state-space models","sliding-window attention","length generalization","memory slots"],"falsifier":"Train a Raven-scale model on a distribution where many retrieval-critical tokens compete for the same slots and measure whether recall degrades; or track the fraction of slots ever selected in a long run and check whether collapse drives most of them to zero usage.","tokens_in":29368,"feed_emoji":"🧠","tokens_out":5984,"duration_ms":50641,"temperature":0.7,"pith_summary":"The paper introduces Raven, a linear-time sequence model that keeps a fixed set of memory slots but updates only a sparse, input-dependent subset at each step, with decay applied only to those slots. The authors argue this combination fills a gap between dense state-space models, which overwrite all memory at every step and suffer interference, and sliding-window attention, which preserves tokens but evicts them by position. They report that Raven maintains near-perfect accuracy on needle-in-a-haystack retrieval at context lengths up to 16 times its training length, where both dense SSMs and windowed attention collapse. If correct, this would give fixed-size recurrent models a practical route to long-context recall without quadratic attention or short-range convolutions.","feed_headline":"16x recall at long context via sparse slot routing","feed_subtitle":"Raven updates only a few memory slots per step, so its recall survives haystacks 16x longer than training.","key_machinery":"The central object is the Routing Slot Memory, a slot-separable linear recurrence St = (1 - rt) ⊙ S(t-1) + rt ⊙ (Dt S(t-1) At + Ut), where rt is a sparse input-dependent router and decay applies only to updated slots. In Raven, rt is a sigmoid-scored Top-K vector normalized to sum to one, and the write content is the stacked key/value vector; this lets the model protect unselected slots from overwrite while gradually forgetting selected ones, generalizing both sliding-window attention and dense state-space models.","core_discovery":"The central claim is that sparse, input-dependent routing to memory slots, paired with explicit decay on only the selected slots, lets a linear-time model preserve individual token memories over long contexts. The authors formalize a class of 'routing slot memories' in which every slot is updated independently and a router vector decides write intensity per slot; sliding-window attention and state-space models appear as extremes on a sparsity axis. Raven uses a sigmoid-scored Top-K router and a per-head scalar decay, and the paper reports that this design achieves ≥99% retrieval accuracy up to 16K tokens at 400M scale, and >91% at 32K, 16x its training length, while remaining competitive on","pith_inferences":["One could apply the same sparse-slot principle to cache management in Transformers, routing important tokens to protected slots to improve long-context QA; the paper does not test this.","Monitoring slot-selection entropy during training could serve as a practical early warning for router collapse, and the Gumbel-noise safeguard may need re-tuning at larger scale.","The effective-sequence-length analysis suggests a testable prediction: training with shorter contexts may already expose the model to longer effective horizons, so extrapolation should scale smoothly with the number of slots.","Combining Raven's slot-level routing with head-level routing could produce even sparser writes, a direction the paper flags as future work."],"forward_implications":["If the central claim holds, linear-time recurrent models can achieve long-context recall without short-range convolutions, simplifying the architecture.","Models trained at one context length can extrapolate to many multiples of it without a length curriculum, because each slot naturally sees a different effective sequence length.","In hybrid architectures, replacing dense recurrent components with Raven's sparse-routed memory preserves long-range retrieval while retaining linear-time inference.","The routing/forgetting decoupling gives future designs a separate axis to control: where to write and how long to keep, rather than conflating both in a decay matrix."],"fun_headline_variants":["Raven: sparse memory routing boosts long-context recall","Linear-time recall: update few slots, remember far back","Selective slot writes preserve memories 16x beyond training","Raven's routed memory beats dense and windowed models at recall","Sparse slot routing: recall past tokens without full-state updates"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The learned router must distribute tokens across slots rather than collapsing onto a few; the paper's only safeguard is Gumbel noise during training, with no formal guarantee that specialization holds at larger scales or on very different data.","fun_headline_variants_meta":{"raw":{"variants":["Raven: sparse memory routing boosts long-context recall","Linear-time recall: update few slots, remember far back","Selective slot writes preserve memories 16x beyond training","Raven's routed memory beats dense and windowed models at recall","Sparse slot routing: recall past tokens without full-state updates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000493,"raw_usage":{"total_tokens":2262,"prompt_tokens":753,"completion_tokens":1509,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":1425}},"tokens_in":497,"tokens_out":1509,"duration_ms":11696,"temperature":1.0,"reasoning_tokens":1425,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:38:58.671178+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a Raven-scale model on a distribution where many retrieval-critical tokens compete for the same slots and measure whether recall degrades; or track the fraction of slots ever selected in a long run and check whether collapse drives most of them to zero usage.","supporting_citations":[],"review_version":1}