{"id":"38c4631a-61b1-4acd-b675-5338100518ae","arxiv_id":"2608.12435","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"MARCH stores checkpoints of a recurrent state and uses learned content keys to route queries to the right checkpoint, improving long-context recall over fixed-state baselines.","lead":"MARCH adds periodically saved snapshots of a recurrent model's memory state and lets each token search those snapshots by content instead of relying only on one fixed-size state. The result is a recurrent model that recalls old information better and keeps working at context lengths beyond its training data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Anchor keys may not carry enough state-content signal for routing; the central retrieval gain depends on an unverified property of Eq. (11).","rationale":"The reader's weakest assumption identifies the same load-bearing premise: the compact anchor keys must encode enough state content for routing to be meaningful. I agree with that diagnosis and have sharpened it. The paper's own construction shows the keys cannot be content-dependent at layer 1, and only become content-dependent through a single shared-query readout of the checkpoint; whether that low-dimensional summary is sufficient for targeted retrieval is empirically unverified. This is not an internal contradiction, and the paper deserves credit for consistent gains across heterogeneous tasks, clean ablations of chunk size and routing design, and a plausible length-extrapolation story. But the central mechanism is the content-conditioned router, and the evidence for it is indirect: better NIAH numbers and degradation under Top-4 are compatible with a router that exploits temporal position or checkpoint density rather than content. A direct probe of routing mass against ground-truth anchor identity would settle the question. Since this is an untested assumption rather than a demonstrated flaw, the existing CONDITIONAL verdict remains appropriate; the condition should include providing such a routing-quality analysis or releasing code so the intervention can be run.","tokens_in":19262,"tokens_out":10872,"duration_ms":98101,"concrete_test":"Run a controlled intervention on the trained 21-layer checkpoint using synthetic NIAH-style sequences in which each needle is placed in a distinct anchor interval, with some needles subsequently overwritten by a conflicting write. For each query, compute the routing distribution π_t over visible anchors and compare the mass assigned to the unique ground-truth anchor against the chance level 1/M. Then swap the checkpoint matrices A_m and A_n at two intervals (which changes later-layer keys through Eq. (10)) and re-measure routing mass. If the mass follows the content to the swapped position, the keys are genuinely content-conditioned; if it stays at the original temporal positions, the router is using temporal or other non-content cues. Require above-chance mass on the ground-truth anchor in the first condition and re-routing in the second for the central mechanism to be validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that content-conditioned router keys let MARCH retrieve the state checkpoint containing a target association. This requires the compact keys κ_m^{(ℓ)} in Eq. (11) to encode enough content of A_m that the softmax in Eq. (13) can select the correct anchor. The construction does not guarantee this. At layer 1, every anchor position receives the same shared learned embedding ξ, so κ_m^{(1)} is identical for all m; content dependence first appears only through Eq. (10), o_m^{(1)} = A_m W_q^{(1)} ξ, a single projection of the full d_v×d_k state matrix onto one shared query direction. The resulting u_m^{(2)} and hence κ_m^{(2)} could be dominated by coarse properties of the checkpoint such as norm or recency rather than by the key–value associations a future query needs. If this is the case, the historical readout in Eq. (14) becomes a soft average over many cumulative states, and the improvement over Log-Linear Attention collapses to checkpoint density rather than content routing. Table 5 is consistent with this failure mode: Top-4 routing reduces NIAH Avg from 51.33 to 44.85, showing routing precision matters, but it does not demonstrate that the keys are content-selective. The sentence in Section 3.1 claiming that the routing key 'becomes conditioned on the content retained by the aligned state anchor' is a necessary condition, not a demonstrated property.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MARCH (Memory-Anchor Routing across Context History), an architecture that augments Gated DeltaNet by periodically checkpointing the cumulative recurrent state into a bank of state anchors. Each anchor is paired with a compact learned routing key; at each text token, a routing query scores all causally visible anchors plus a learned null option, and the resulting weighted average of anchor states is read and added as a residual to the current-state readout. The paper claims that after 50B-token pretraining, MARCH consistently outperforms vanilla Gated DeltaNet and Log-Linear Gated DeltaNet on commonsense reasoning, LongBench, RULER/NIAH, and in-context retrieval, while extrapolating to 32K beyond the 16K training length. The authors also provide ablations on checkpoint interval, routing dimension, Top-K sparsity, null option, and an I/O-aware fused implementation with throughput measurements.","tokens_in":19513,"tokens_out":4494,"duration_ms":42877,"significance":"If the empirical claims hold, MARCH is a useful architectural contribution: it decouples the total available recurrent memory from the fixed state dimension without modifying the underlying recurrence, and it brings query-dependent content routing to historical state checkpoints. The design is clearly specified, end-to-end trainable, and the ablations in Tables 4 and 5 isolate the main design choices. The efficiency discussion in Figure 4 is a strength, showing that dense and Top-K variants have a practical implementation path. The paper does not provide formal guarantees, and its significance rests entirely on the empirical comparisons, so the robustness of those comparisons is what determines whether the central claims are supported.","major_comments":[{"comment":"The central retrieval claim requires the compact anchor keys κ_m^{(ℓ)} to encode the content of their aligned state checkpoints A_m, but this property is not demonstrated. As the paper itself notes, at layer 1 all anchor positions receive the same shared embedding ξ, so κ_m^{(1)} is identical for all m; content dependence first enters only through the single projection o_m^{(1)} = A_m W_q^{(1)} ξ. This one shared query direction could encode coarse properties such as state norm or recency rather than the key–value associations a future query needs. Table 5 shows that routing precision matters (Top-4 drops NIAH average from 51.33 to 44.85), but it does not establish that the selected anchors are content-selective. I ask for a direct diagnostic: compare learned routing against recency-only or random-key routing, measure routing entropy as a function of anchor age, or inspect selected anchors on a controlled insertion/retrieval task.","section":"§3.1, Eqs. (10)-(11); Table 5"},{"comment":"The two free hyperparameters C=512 and d_r=64 are selected after inspecting the same benchmark suites that are used for the main results. Table 4 chooses C by comparing NIAH and in-context retrieval scores, and Table 5 chooses d_r by comparing aggregated benchmark averages. This selection-on-the-evaluation-set procedure can inflate apparent gains and makes the reported improvements harder to interpret. The authors should either use a separate validation split for hyperparameter choice or explicitly report the selection procedure and show that the main conclusions are stable across a small grid around the chosen values.","section":"§4.1, Tables 4-5"},{"comment":"Every reported number in the main comparison tables is a single point estimate with no error bars, no multiple seeds, and no significance testing. Some of the headline differences are small, for example the commonsense average of 41.5 for MARCH versus 41.3 for the 21-layer Transformer and 41.4 for the 24-layer Transformer in Table 1, and a 14.9 versus 15.4 difference in the LongBench average in Table 2. Without variance information, the claim of 'consistent' improvement is not fully supported. I request at least two or three seeds for the main configurations, or an explicit statement of which reported numbers are single runs and which are averaged.","section":"Tables 1-3"},{"comment":"The 32K extrapolation result is presented as a strong piece of evidence, but the figure and text do not report confidence intervals or repeated evaluations for the 32K condition. Since the authors state that all baselines score zero on the 32K multi-needle and several single-needle tasks, it is important to exclude the possibility of evaluation artifacts at long context (e.g., truncation or metric decoding issues) and to report exact per-task scores rather than only the plotted curves. If the 32K numbers are confirmed, this is a valuable result; the manuscript should make the evaluation protocol at 32K fully explicit.","section":"§4.2, Figure 3"}],"minor_comments":[{"comment":"Table 4 is difficult to read because several entries are run together, e.g., the chunk-256 row shows '36.76 49.2344.83 58.17 49.2544.83', with missing column separators. The table should be reformatted so each column is unambiguous.","section":"Table 4"},{"comment":"The text says 'six single-neddle and multi-needle tasks'; 'single-neddle' should be 'single-needle'.","section":"§4.1"},{"comment":"The Fenwick tree row is described as reporting performance 'close to [Guo et al., 2026]', but no comparison numbers from that paper are given in the table or the text. Please add the reference numbers so the reader can verify the claim.","section":"Table 4, Fenwick row"},{"comment":"The figure caption contains spacing artifacts such as 'S t a t e   S p a c e   U p d a t e' and the right panel labels are hard to parse; please redraw the figure and caption.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript's core idea is plausible and the writing is generally clear, but the empirical support has three interconnected weaknesses: no error bars, hyperparameters chosen on the evaluation set, and no direct evidence for the content-selectivity of the routing keys. The first two are fixable with additional runs and an explicit validation procedure. The third is the most important: if the routing keys are not actually content-selective, the method degrades to a recency-weighted state cache, which would substantially weaken the claimed contribution. I did not see code or checkpoints in the submission, which is worth requesting if the journal has a reproducibility policy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, short version: MARCH does something genuinely useful—it takes Gated DeltaNet and adds periodic cumulative state checkpoints plus learned content-conditioned routing, so a token can read both the current state and selected historical states. On its own benchmarks it consistently beats vanilla GDN and the Log-Linear variant, and the ablation structure is good. If I were refereeing, I would send it out rather than reject it, but I would ask for reproducibility polish and one targeted experiment before accepting.\n\nWhat is actually new: previous state-expansion work either grows the state at every step or keeps a hierarchy of states with fixed or learned index structure. MARCH's twist is routing over checkpoints with keys that are themselves derived from each checkpoint's content, plus a learned null route to bypass history. The cumulative checkpointing is simple, and the residual fusion preserves the original recurrence, which is clean. The efficiency plots make the cost structure concrete; Top-4 routing is a reasonable efficiency knob.\n\nCredit where due: the paper tests its own design choices—chunk size, routing dimension, null option, sparse routing—and the ablations tell a consistent story. The Fenwick-tree inference experiment is a nice bonus: it suggests the learned router generalizes across bank organizations. The gains are not huge on commonsense (+1.4 points over GDN) but they are consistent, and the NIAH/LongBench improvements are larger, especially multi-doc QA. That pattern is what you want from a memory mechanism.\n\nSoft spots, in order of seriousness. First, every number is a single point estimate. No seeds, no error bars. On NIAH differences of several points, that is a real problem, since retrieval tasks are noisy. Second, the key hyperparameters (chunk 512, routing dim 64) are picked using the same task families later reported in the main tables. The ablation shows C=512 is balanced, but selection on the same benchmarks can inflate apparent gains. Third—and this is the one I would want addressed—the whole mechanism depends on the claim that anchor keys encode checkpoint content. The construction does not guarantee it. At layer 1 all anchor keys are identical because every anchor position receives the same learned embedding; content dependence only appears through Eq. (10), one projection of a d_v-by-d_k matrix onto a shared query direction. That could easily be dominated by coarse properties like norm or recency. Table 5's Top-4 degradation shows routing matters, but it does not prove the keys are content-selective. A direct probe—correlating key similarity with shared underlying associations—would settle it. That said, the stress-test concern is not fatal: the empirical improvements could still come from checkpoint density plus a partially content-sensitive router, and the architecture is coherent either way.\n\nBottom line: this is a paper worth taking seriously. It deserves a real referee, and it is probably a conditional accept after seeds, code/data release, and one key-content probe. I would cite it if I were working on state expansion.","headline":"MARCH is a solid, well-ablated extension of Gated DeltaNet with checkpointed states and content routing; the retrieval gains look real, but the key-content assumption is not directly tested and all headline numbers are single runs.","tokens_in":20057,"tokens_out":2317,"would_cite":true,"duration_ms":21307,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MARCH claims that a recurrent sequence model can escape its fixed-size memory bottleneck by periodically saving cumulative state snapshots and routing each token to the snapshot whose content matches what it needs to recall.","keywords":["MARCH","state anchors","content-based routing","recurrent memory","long-context retrieval","linear attention","gated delta rule","in-context retrieval"],"falsifier":"Retrain or fine-tune MARCH with the anchor-key gradient detached from the state-conditioned representation, for example by driving keys only from a learned per-position embedding, keeping the routing pathway otherwise intact, and compare 8K NIAH accuracy; if the gap over vanilla Gated DeltaNet survives, content-conditioned routing is not load-bearing, and if it collapses, the claim is confirmed.","tokens_in":19052,"feed_emoji":"🧠","tokens_out":6676,"duration_ms":59819,"temperature":0.7,"pith_summary":"Recurrent language models compress the whole history into one fixed-size state, so old details get overwritten; Transformers avoid this by keeping every token, at quadratic cost. This paper claims that a recurrent model can get Transformer-like long-range recall without giving up its efficient recurrence: checkpoint the evolving state every so often, attach each checkpoint to a compact key that describes its content, and let each token route to the checkpoints whose content matches what it needs. The resulting architecture, MARCH, is trained from scratch on 50B tokens and consistently beats its Gated DeltaNet backbone and a log-linear variant on commonsense reasoning, LongBench, needle-in-a-haystack retrieval, and six in-context retrieval benchmarks, and it keeps working at 32K despite being trained at 16K. If the claim holds, recurrent models can scale memory with context length while keeping a linear computation path.","feed_headline":"Recurrent model beats Transformers at long-context recall","feed_subtitle":"MARCH checkpoints the evolving memory state and routes each token to the snapshot that holds the answer.","key_machinery":"The central object is the state anchor: a checkpoint $A_m = S_{b_m}$ of the cumulative recurrent state taken at periodic text boundaries, paired with an occurrence of a shared anchor token whose hidden representation becomes a compact routing key $\\boldsymbol{\\kappa}_m = W_k \\mathbf{u}_m$. Because the anchor token is aligned with its checkpoint and reads it before the next layer, the key at deeper layers is conditioned on what the checkpoint actually contains. At every text token a routing query $\\boldsymbol{\\rho}_t$ scores all causally visible anchor keys plus a learned null candidate through a softmax, and the resulting probabilities weight the anchor states, which are read with the token's normal state-read query and added to the current-state readout. This turns the otherwise transient state trajectory into a persistent, content-addressable memory bank whose size grows with context length.","core_discovery":"On the paper's own terms, the discovery is that the fixed-state bottleneck of linear recurrences is not the recurrence itself but the loss of earlier state versions. MARCH shows that periodically saving cumulative snapshots $A_m$ of the Gated DeltaNet state, giving each snapshot a learned content-conditioned key $\\boldsymbol{\\kappa}_m$, and aggregating the relevant snapshots with a softmax over visible anchor keys plus a null option restores direct access to earlier memory. The current-state readout $S_t q_t$ is preserved, and the historical readout $\\sum_j \\pi_{t,j} A_j q_t$ is added as a residual branch, so the native recurrent path is untouched. Across single-needle and multi-needle tasks this raises retrieval sharply, with single-needle NIAH at 8K climbing from 55.3 to 81.7, and the gains also appear on general-purpose benchmarks.","pith_inferences":["If the anchor keys are truly content-addressable, the same routing layer should transfer to other recurrent backbones, such as Mamba-style selective states, without retraining from scratch; the paper's Fenwick-tree ablation already hints that the router is not tied to positional indexing.","The routing probabilities and null-route mass are a ready-made interpretability signal: they could identify which tokens depend on old context and which checkpoint supplies a recalled fact, something this paper does not analyze.","The growing anchor bank is effectively an external, content-addressed memory written by the recurrence, which suggests uses the paper does not explore, such as editing or consolidating memory at test time for continual learning.","Fixed-interval checkpointing is a convenience, not a requirement; replacing the constant chunk size with novelty-triggered checkpointing, where rapid state change generates more anchors, is a direct testable extension that could help fast-changing contexts."],"forward_implications":["Needle-in-a-haystack recall jumps from 55.3 to 81.7 on single-needle NIAH at 8K, and across all 24 NIAH task-length settings MARCH wins 19 and ties the remaining five against the stronger recurrent baseline; at 32K it keeps nonzero accuracy on all six tasks where the Transformers and the log-linear baseline score zero.","On LongBench, MARCH lifts the average from 11.9 for vanilla Gated DeltaNet and 12.5 for the log-linear variant to 14.9, a 25% relative gain over the backbone, concentrated in multi-document QA and summarization.","On six in-context retrieval benchmarks, MARCH raises the average from 20.5 to 23.3, with relative gains between 8% and 23% over the stronger Gated DeltaNet baseline on every task.","The checkpoint interval sets a controllable accuracy-memory trade-off: chunk size 512 is the best default, and Top-4 sparse routing keeps most of the benefit while more than doubling training throughput at 128K and exceeding FlashAttention-2 at that length.","Because routing is content-based and the bank grows with context, MARCH extrapolates to 32K from a 16K training length without introducing anchor-specific parameters for the new anchors."],"supporting_citations":[{"why":"Supplies the Gated DeltaNet backbone, including its delta-rule state update and chunkwise implementation, that MARCH checkpoints and augments.","marker":"[Yang et al., 2025]"},{"why":"Provides the Log-Linear Attention baseline with its temporally organized state hierarchy, and the 50B-token pretraining protocol MARCH follows.","marker":"[Guo et al., 2026]"},{"why":"Provides the RULER needle-in-a-haystack suite that carries the paper's core long-context retrieval evidence.","marker":"[Hsieh et al., 2024]"},{"why":"Defines the six-benchmark in-context retrieval evaluation and the recall-throughput perspective motivating the method.","marker":"[Arora et al., 2024b]"},{"why":"Represents the temporal state-preservation line of work MARCH extends, and supplies the evaluation protocol for in-context retrieval.","marker":"[Wang et al., 2026]"},{"why":"The closest prior growing-memory approach; MARCH positions itself against it by adding content-conditioned routing over checkpoints.","marker":"[Behrouz et al., 2026]"},{"why":"Provides the token-level memory view of Transformer attention that frames the fixed-state bottleneck MARCH addresses.","marker":"[Vaswani et al., 2017]"},{"why":"Contributes the I/O-aware tiling pattern that MARCH's historical reader borrows to avoid materializing dense routing scores.","marker":"[Dao et al., 2022]"}],"fun_headline_variants":["Content-routed state anchors lift recurrent models' recall","MARCH caches state snapshots to outperform linear attention on long context","Recurrent memory scales via content-addressed state checkpoints","MARCH: Snapshots of recurrent state enable efficient long-range recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole retrieval gain rides on the compact anchor keys truly encoding what their checkpoints contain, so that the routing query can separate useful snapshots from irrelevant ones; if the keys barely vary with state content, the softmax averages all checkpoints and the benefit disappears.","fun_headline_variants_meta":{"raw":{"variants":["Content-routed state anchors lift recurrent models' recall","MARCH caches state snapshots to outperform linear attention on long context","Recurrent memory scales via content-addressed state checkpoints","MARCH: Snapshots of recurrent state enable efficient long-range recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000903,"raw_usage":{"total_tokens":3908,"prompt_tokens":988,"completion_tokens":2920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":2849}},"tokens_in":604,"tokens_out":2920,"duration_ms":24490,"temperature":1.0,"reasoning_tokens":2849,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:17:33.693936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain or fine-tune MARCH with the anchor-key gradient detached from the state-conditioned representation, for example by driving keys only from a learned per-position embedding, keeping the routing pathway otherwise intact, and compare 8K NIAH accuracy; if the gap over vanilla Gated DeltaNet survives, content-conditioned routing is not load-bearing, and if it collapses, the claim is confirmed.","supporting_citations":[{"cited_title":"and Kaiser, Lukasz and Polosukhin, Illia , EDITOR =","cited_arxiv_id":null,"evidence_quote":"Provides the token-level memory view of Transformer attention that frames the fixed-state bottleneck MARCH addresses."},{"cited_title":"and Ermon, Stefano and Rudra, Atri and R","cited_arxiv_id":null,"evidence_quote":"Contributes the I/O-aware tiling pattern that MARCH's historical reader borrows to avoid materializing dense routing scores."}],"review_version":1}