{"id":"fa0355ff-a5f7-41bd-8e69-296f15b8c977","arxiv_id":"2606.26560","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":1,"one_line_summary":"EDA decouples erase and write addresses in delta-rule linear attention by adding a targeted erase step along a learned direction before the corrective write, yielding best results on 2.5B dense and 25B MoE models in pretraining and long-context tasks.","lead":"The paper proposes Erase-then-Delta Attention (EDA), a memory update rule that adds a separate learned erase step before the standard delta-rule write in linear attention. Smart generalists might read it because better memory cleanup in efficient recurrent models could help scale long-context language models without full quadratic attention costs.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Independence of learned erase direction from write address may fail to hold without interfering with delta-rule correction","rationale":"The reader's weakest assumption matches the load-bearing point exactly; the experimental superiority is only credible if that assumption is verified. The update analysis is presented as suggestive rather than exhaustive, leaving the independence claim under-supported relative to the scale of the reported gains.","tokens_in":1810,"tokens_out":316,"duration_ms":18313,"concrete_test":"After training, extract the erase-direction and write-direction vectors per layer/head on a held-out checkpoint; compute average cosine similarity across 100 random sequences. If mean similarity exceeds 0.4 or if ablating the erase step (setting its coefficient to zero) changes the delta-rule update norm by >5% on the same sequences, the independence assumption does not hold.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The performance claims rest on the erase step cleanly suppressing stale memory at an address chosen independently of the current write address. The abstract states that a compact update analysis 'suggests' the delta-rule corrective write remains intact, yet this analysis is described only at high level and does not address whether the jointly learned erase direction can remain orthogonal (or at least non-interfering) to the write direction throughout optimization, nor whether the additional gradient path destabilizes training. If the learned erase vector correlates strongly with the write vector or modulates the same memory slots, the claimed decoupling benefit collapses and observed gains could arise from other unablated factors.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes Erase-then-Delta Attention (EDA), a memory update rule for delta-rule linear attention that first applies a targeted erase along a learned erase direction before performing the standard delta-style corrective write. This is intended to decouple erase and write addresses so that stale information at an independent address can be suppressed. Pretraining experiments on dense 2.5B and MoE 25B-A2.8B models report that EDA performs best in both settings; the advantage persists after 80B-token long-context midtraining, where EDA also leads on evaluations from 4k to 128k contexts. A compact update analysis and memory-state probes are offered to explain the behavior.","tokens_in":1941,"tokens_out":455,"duration_ms":17600,"significance":"If the decoupling holds without destabilizing the delta-rule correction, the method would expand the memory-management capacity of recurrent linear attention beyond address-tied correction. The cross-scale experiments (2.5B dense and 25B MoE) and persistence through long-context midtraining constitute concrete empirical support; the memory-state probes add mechanistic insight. These elements would be strengths if the independence of the learned erase direction is rigorously verified.","major_comments":[{"comment":"Abstract and update-analysis paragraph: the claim that the delta-rule corrective write 'remains intact' rests on a high-level analysis that does not quantify correlation between the jointly learned erase vector and the write vector, nor bound potential interference with the corrective update; this is load-bearing for the decoupling assertion.","section":"Abstract"},{"comment":"Experiments section (performance claims): no quantitative effect sizes, ablation isolating the erase-direction contribution, or controls for training instability are reported in the abstract, and the full text must supply these to substantiate that observed gains arise from address decoupling rather than other unablated factors.","section":"Experiments"}],"minor_comments":[{"comment":"Notation for the learned erase direction should be introduced with an explicit equation and distinguished from the write direction to avoid ambiguity in the update rule.","section":"Method"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our work. We address the two major comments point by point below, offering clarifications from the manuscript and committing to targeted revisions that strengthen the evidence for decoupling without overstating current results.","responses":[{"response":"We agree that the current update analysis is high-level and that explicit quantification would make the independence claim more rigorous. The manuscript's compact analysis shows the erase step acts along a learned direction prior to the write, and the memory-state probes indicate differential cleanup behavior when passive decay is weak. To directly address the concern, we will add measurements of the correlation between the jointly learned erase and write vectors plus a bound on interference in the revised version.","revision_made":"yes","referee_comment":"[Abstract] Abstract and update-analysis paragraph: the claim that the delta-rule corrective write 'remains intact' rests on a high-level analysis that does not quantify correlation between the jointly learned erase vector and the write vector, nor bound potential interference with the corrective update; this is load-bearing for the decoupling assertion."},{"response":"The full manuscript already reports consistent gains on 2.5B dense and 25B MoE models that persist after long-context midtraining. We acknowledge, however, that explicit effect sizes, an ablation isolating the erase direction, and training-stability controls are needed to rule out confounding factors. In revision we will insert quantitative effect sizes (e.g., perplexity deltas), a dedicated ablation on the erase-direction term, and stability metrics across runs.","revision_made":"yes","referee_comment":"[Experiments] Experiments section (performance claims): no quantitative effect sizes, ablation isolating the erase-direction contribution, or controls for training instability are reported in the abstract, and the full text must supply these to substantiate that observed gains arise from address decoupling rather than other unablated factors."}],"tokens_in":1470,"tokens_out":410,"duration_ms":22843,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that Erase-then-Delta Attention adds an independent erase operation before the delta write in linear attention, allowing cleanup at a different memory address. This is presented as fixing a limitation where stale info at non-write addresses couldn't be actively removed.\n\nThe paper introduces the EDA rule with a learned erase direction followed by the standard delta correction. It reports better performance than baselines in pretraining both a 2.5B dense model and a 25B-A2.8B MoE model. Those gains hold after additional 80B token long-context training on the MoE version, with strong results on contexts from 4k to 128k. A brief analysis and memory probes are included to explain when the erase path is most active.\n\nThis approach is a direct extension of delta-rule ideas and the experiments cover relevant scales and settings for the subfield.\n\nThe main concern is whether the learned erase direction truly stays decoupled from the write direction without causing interference or instability. The provided analysis is high-level and doesn't fully address training dynamics or potential correlation between the two vectors. The lack of detailed ablations or quantitative breakdowns in the reported results makes it difficult to isolate the contribution of the erase step specifically. If the erase and write end up aligned, the claimed benefit reduces.\n\nThis paper is for people focused on improving recurrent memory in transformers for long sequences. It has a clear technical proposal and consistent empirical findings, so it should go through peer review rather than a desk reject. The idea is worth testing and refining even if the current evidence leaves some questions open.","headline":"EDA adds an independent erase step before delta-rule writes in linear attention and reports gains on 2.5B and 25B-scale models, but the decoupling may not survive joint training without further checks.","tokens_in":2479,"tokens_out":407,"would_cite":false,"duration_ms":20128,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Decoupling erase and write addresses lets delta-rule linear attention actively remove stale memory before writing new content.","keywords":["Erase-then-Delta Attention","delta-rule linear attention","recurrent memory","language model pretraining","long-context modeling","memory management","linear attention"],"falsifier":"If side-by-side pretraining runs of identical 2.5B or 25B models show that EDA does not reduce validation perplexity relative to standard delta-rule attention after the same token budget, the performance claim would be falsified.","tokens_in":2714,"feed_emoji":"🧠","tokens_out":676,"duration_ms":16409,"temperature":0.7,"pith_summary":"Delta-rule linear attention corrects stored values at the current write address but cannot actively clear outdated information stored elsewhere. Erase-then-Delta Attention (EDA) adds a separate learned erase step that targets a different address before the standard corrective write occurs. This preserves the delta-rule correction while expanding the model's ability to manage recurrent memory. Pretraining runs on dense 2.5B and MoE 25B models show EDA outperforming prior delta-rule variants, with the advantage holding after 80B tokens of long-context midtraining and across 4k-to-128k evaluations.","feed_headline":"Decoupling erase from write improves delta-rule attention","feed_subtitle":"EDA adds a learned erase step at a separate address before the delta correction, yielding best results on 2.5B dense and 25B MoE pretraining","key_machinery":"Erase-then-Delta Attention (EDA) update rule that performs a learned-direction erase step before the standard delta-rule corrective write.","core_discovery":"The central claim is that recurrent memory models benefit when the erase address is chosen independently of the write address: EDA first applies a targeted erase along a learned direction, then performs the usual delta-style correction at the write address, thereby keeping the corrective behavior intact while adding an explicit cleanup path that is most active when passive decay is weak.","pith_inferences":["The same decoupling idea could be tested in other recurrent state-space or linear-attention variants that currently rely only on decay or overwriting.","Explicit erase directions might reduce reliance on learned decay rates or attention sinks in long-context settings.","If the erase direction can be made input-dependent at inference time, the method could support on-the-fly memory editing without retraining."],"forward_implications":["EDA achieves the lowest perplexity in both dense 2.5B and MoE 25B-A2.8B pretraining.","The advantage remains after 80B-token long-context midtraining of the MoE models.","EDA records the best results on long-context benchmarks spanning 4k to 128k contexts.","Memory-state probes indicate the extra erase path is used most when passive decay is weak while the delta correction stays unchanged."],"fun_headline_variants":["Decoupling erase from write in delta-rule linear attention","EDA performs targeted erase at separate address then delta write","Independent erase path added to delta-rule attention updates","Erase address chosen separately from write in recurrent memory"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"A learned erase direction can be trained to target stale memory at an address independent of the write address without interfering with the delta-rule corrective update or introducing training instability.","fun_headline_variants_meta":{"raw":{"variants":["Decoupling erase from write in delta-rule linear attention","EDA performs targeted erase at separate address then delta write","Independent erase path added to delta-rule attention updates","Erase address chosen separately from write in recurrent memory"]},"model":"grok-4.3","cost_usd":0.004141,"raw_usage":{"total_tokens":2124,"prompt_tokens":720,"num_sources_used":0,"completion_tokens":59,"cost_in_usd_ticks":41412000,"prompt_tokens_details":{"text_tokens":720,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1345,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":720,"tokens_out":59,"duration_ms":11432,"temperature":1.0,"reasoning_tokens":1345,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T05:27:45.021577+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"If side-by-side pretraining runs of identical 2.5B or 25B models show that EDA does not reduce validation perplexity relative to standard delta-rule attention after the same token budget, the performance claim would be falsified.","supporting_citations":[],"review_version":1}