{"id":"2d572cb7-4f57-48e0-8bde-5329cf83c732","arxiv_id":"2506.08184","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLM retrieval accuracy decays log-linearly with the number of interfering prior updates, and 'forget' instructions barely improve it.","lead":"This paper shows that large language models lose the ability to recall the newest value of a key when the prompt contains many older, similar values for the same key. The authors argue this interference limit is separate from context window length, and that simple prompts telling the model to forget old data barely help.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fixed-length experiment confounds interference with the number of required outputs; the independence claim needs an output-length control.","rationale":"The reader's verdict is CONDITIONAL with high confidence, and I agree. The central phenomenon—retrieval accuracy declines as interfering updates accumulate—is well supported by Experiment 1 and the error-distribution analyses (Figures 3, 5, 19): errors are predominantly earlier values of the same key, and the decline is robust across prompts, model families, and parameter scales. What is not yet established is the stronger, headline claim that this decline is a pure interference effect independent of context length and output requirements. The paper's own Section 3.2.3 rests the independence claim on the similarity of Experiments A and B, but both experiments vary the number of keys the model must report, so the measured accuracy conflates retrieval failure with generation/formatting failure. This is the weakest load-bearing link. I am not raising this as a disagreement with consensus or as an ad hominem concern; it is a specific experimental confound with a concrete fix. The paper deserves credit for releasing code and data, for the step-like sequential-order contrast, and for the per-key error clustering analyses that support the interference interpretation at the mechanism level. But those supports do not by themselves remove the output-length confound in the length-independence argument. A control with constant output length and variable interference would directly test whether the log-linear decline persists; if it does, the paper's strongest claim is substantially validated. Until then, the verdict should remain CONDITIONAL rather than ACCEPT, and not REJECT, because the core phenomenon is real and the requested control is feasible and well-specified.","tokens_in":24920,"tokens_out":2564,"duration_ms":24333,"concrete_test":"Run Experiment B with a fixed number of queried outputs (e.g., always ask for the same 5 target keys plus K filler keys that are updated but never queried). Vary the number of interfering updates per key (or the total number of updated keys) while keeping the answer format identical. If accuracy still declines log-linearly with interference under constant output length, the confounding is resolved. Additionally, report the same control for Experiment A: fix the queried set at 5 keys while varying the number of updated (interfering) keys from 1 to 46; if the decline persists, the length-independence claim survives. A second cheap check is to compute accuracy on the first K outputs only for the original T-key condition; if accuracy on the first K outputs is roughly independent of T, output length is the active confound.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's core contribution is the claim that interference independently limits retrieval, separate from context length, supported by the near-identical log-linear declines in Experiments A and B (Section 3.2.3). The reader correctly identifies that Experiment B (Section 3.1.2) varies the number of tracked keys T while holding input length fixed, but T also equals the number of key-value pairs the model must generate in its answer. Accuracy is measured as the fraction of correctly retrieved values across all tracked keys (Section 2.1), so as T grows, both the interference load and the output length grow. The same confounding applies to Experiment A, where the number of updated keys equals the number of queried keys. Thus the parallel declines do not isolate interference from generation load: a model could fail more with more outputs due to format drift, response truncation, or attention dilution at decoding time, not due to retrieval interference. The paper explicitly frames these curves as evidence that interference is independent of input length (Section 3.2.3), and this claim is load-bearing for the 'working memory bottleneck beyond context access' conclusion. The mitigation experiments do not resolve this: the mock QA reset improves accuracy but does not manipulate output length. A control that holds the number of required outputs constant while varying interference would settle whether the log-linear decline is a retrieval-interference effect or partially an output-generation effect.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PI-LLM, a synthetic key-value update task in which models must report the most recent value for each tracked key after a stream of repeated updates. The central empirical finding is a roughly log-linear decline in retrieval accuracy as the number of interfering updates, tracked keys, or value tokens grows, across 35 models. The authors argue that the decline is not explainable by input length because a fixed-input-length variant shows the same pattern, and they report that larger models have higher Interference Endurance Scores (IES) while nominal context length does not predict IES. Prompt-based interventions such as directed forgetting are mostly ineffective, while a mock QA reset partially restores accuracy. The paper concludes that LLM retrieval is limited by a working-memory-like anti-interference capacity distinct from context length.","tokens_in":25103,"tokens_out":8935,"duration_ms":110432,"significance":"The task design is simple, reproducible, and well suited to isolating proactive interference in modern LLMs, and the coverage of 35 dense, MoE, and CoT models gives the benchmark broad descriptive value. The main update-count experiment controls the number of outputs (46 keys) and the output format, reports bootstrapped confidence intervals, and includes an error-location analysis showing that mistakes are often prior values; these are genuine strengths. Public code and data are promised, which supports reproducibility. If the independence-from-context-length claim survives an output-length control, the paper would establish a practically important evaluation axis. As it stands, the evidence supports an interference effect but not yet the stronger 'beyond context length' conclusion.","major_comments":[{"comment":"Experiment B varies the number of tracked keys T while holding input length fixed, but accuracy is measured as the fraction of correctly retrieved values across all T keys (Section 2.1), so output length and decoding load grow with T. Experiment A has the same property because the number of updated keys equals the number of queried keys. The nearly identical declines in Figure 9 therefore only rule out input length as the sole cause; they do not rule out generation-load effects such as format drift, response truncation, or attention dilution at decoding time. No control is reported that holds the number of required outputs constant while varying interference, and the error-location analysis in Figures 5 and 19 is performed on Experiment 1, where output length is fixed at 46 keys. Please add such a control or otherwise quantify the fraction of failures attributable to incomplete or malformed outputs in Experiments A and B.","section":"3.1.2 and 3.2.3"},{"comment":"The 'log-linear decline' is a central quantitative claim, but no regression is reported for any of the curves in Figures 1, 3, or 9: there are no slopes, intercepts, R-squared values, or comparisons against alternative functional forms. Visual inspection of log-scaled axes is not sufficient to establish a universal log-linear law, especially since Figure 4 shows a step-like pattern for sequential ordering. Please fit per-model models such as accuracy ~ a + b log(update_count), and the equivalent for tracked keys and value length, and report goodness of fit and model comparisons.","section":"2.2 and 3.2.3"},{"comment":"The regression of IES on parameter size class and context length is descriptive and may be underpowered: IES is the AUC of the same accuracy curves, the analysis uses N=30 after excluding reasoning and proprietary models, and context lengths cluster at 128k-131k. The current reporting of t and p for two predictors with R-squared = 0.261 does not address collinearity or the limited range of context lengths. I ask for a continuous-size regression where parameter counts are public, a check of variance inflation, and an explicit statement that this is a descriptive fit rather than a claim that context length is causally irrelevant.","section":"2.2.2"}],"minor_comments":[{"comment":"The text says IES is computed from the accuracy-versus-update-count function 'see Figure 5', but the IES curves are shown in Figure 16; the figure citation should be corrected.","section":"2.2.2"},{"comment":"The caption of Figure 19 ends with 'See Figure 19 for additional models', which is circular and likely intended to refer to Figure 5 or Figure 18.","section":"Appendix A"},{"comment":"The phrase 'key-value pair length of 6' is ambiguous; it should specify whether this is the value length in words or in tokens.","section":"Figure 12"},{"comment":"There is a typo in the Introduction and reference list: 'V odrahalli' should be 'Vodrahalli'.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The output-length confound in Experiments A and B is the key issue for the paper's headline claim. The core update-count experiment is well controlled and the benchmark is useful, so I would not recommend rejection if the authors can add a clean control that holds output length fixed while varying interference, or clearly report failure modes such as truncation. The title and abstract should be softened or explicitly qualified until that control is provided."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The core phenomenon is real and cleanly demonstrated: stream key–value updates, query only the final values, and accuracy falls log-linearly with update count across 35 models, with bootstrapped CIs and an error distribution showing the errors really are previously overwritten values. That is a solid, useful result. The second thing is that the headline claim—interference limits retrieval independently of context length—rests on an experiment with a confound, and the strongest version of that claim is not supported.\n\nWhat's genuinely new: the A-B/A-C proactive-interference probe applied to LLM retrieval, with the target always the last value so search difficulty is minimized, is a real design improvement over Lost-in-the-Middle and the long-context benchmarks. The IES metric is a simple AUC summary and it does discriminate between models. The error-distribution analysis (localized to dispersed to hallucinatory) is descriptive but nice. The 'forget' prompts largely failing, and the mock-QA reset partially working, are useful empirical findings. Code and data are public.\n\nNow the soft spots. Experiment B (Section 3.1.2) is the control meant to show the decline is independent of input length, and it varies T, the number of tracked keys, while holding the update stream fixed. But T is also the number of outputs the model must generate, and accuracy is measured across all T outputs. So the decline could be generation load—format drift, truncation, output-token pressure—rather than retrieval interference. The near-identical declines in Experiments A and B are consistent with a shared output-length confound. There is no control that holds output length constant while varying the number of tracked keys. The value-length experiment (Section 4) does hold the number of outputs fixed, so it shows at least one load dimension degrades retrieval without that confound, but it does not rescue the tracked-keys interpretation. That is a real gap, and it is fixable. The 'log-linear' law is never fitted—no slopes, R², or goodness-of-fit are reported—so it is an eye-fit. The 'single unified bottleneck' is an interpretation: the three manipulations give similar shapes but very different slopes, so 'one resource' is not established. The IES regression on model size and context length is a descriptive fit on the same curves that define IES, so it is a summary, not a predictive test.\n\nWho this is for: people building long-context or retrieval evals. The benchmark is worth having, the update-count result is solid, and the weaknesses are addressable. I would send it to peer review—it deserves referee time—but a serious referee should ask for an output-length control, reported fits, and language that matches the evidence.","headline":"Solid, well-controlled benchmark for proactive interference in LLM retrieval, but the headline 'independent of context length' claim rests on a confounded experiment and needs an output-length control before it can stand.","tokens_in":25655,"tokens_out":6616,"would_cite":true,"duration_ms":71515,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM retrieval accuracy declines log-linearly toward zero as interfering prior updates accumulate, even when the target is the most recent value and is explicitly cued—a limit independent of context length.","keywords":["proactive interference","working memory","LLM retrieval","key-value tracking","interference endurance score","long-context evaluation","log-linear decay","prompt engineering"],"falsifier":"Run the fixed-length setup while always asking the model to report a single key's final value, varying only the number of prior updates or updated keys; if accuracy no longer falls log-linearly with interference, the claimed isolation of interference from output load is falsified.","tokens_in":24693,"feed_emoji":"🧠","tokens_out":8356,"duration_ms":95558,"temperature":0.7,"pith_summary":"This paper tries to establish that LLMs possess a working-memory-like capacity limit, distinct from context length, which is exposed by proactive interference: when a stream of semantically similar key–value updates precedes a retrieval target, accuracy falls log-linearly toward zero even though the target is always the most recent value and is explicitly cued. It introduces PI-LLM, a synthetic key-update tracking task adapted from the paired-associate forgetting paradigm of human memory research, and reports the same log-linear decay across 35 models spanning 0.6B to over 600B parameters. The decline persists when input length is held constant, and error analysis shows models retrieve earlier overwritten values rather than the target. The paper concludes that resistance to interference is an independent model capability—correlated with parameter count, not context-window size—and that natural-language instructions to forget or focus, and chain-of-thought reasoning, do not remove the bottleneck.","feed_headline":"LLMs lose the latest value log-linearly as old values pile up","feed_subtitle":"Even an explicitly cued, most-recent answer gets buried by similar earlier updates—context length doesn't rescue it.","key_machinery":"The load-bearing object is the proactive-interference (PI) retrieval task: an interleaved sequence of key–value updates over a fixed set of keys, with the instruction to return only the current, last value of each key, so every earlier value of the same key is a semantically similar distractor. The paper's quantitative index is the Interference Endurance Score (IES), defined as the area under the curve of retrieval accuracy against log-scaled update count, which ranks models by how slowly they degrade. The argument is carried by the error distribution: erroneous answers are dominated by previously overwritten values of the same key, shifting from adjacent positions to distant ones and eventually to unseen values as interference grows, which is presented as evidence that LLMs lack an unbinding or gating mechanism rather than a search failure.","core_discovery":"On the paper's own terms, the discovery is that LLM retrieval is constrained by a single, resource-like anti-interference capacity: accuracy declines approximately log-linearly toward zero as the number of updates per key, the number of tracked keys, or the length of each value grows, and this decline is not explained by input length. Because the task always asks for the last-presented value of a repeated key and states the target keys before and after the stream, search difficulty is deliberately minimized, so the failures measure interference directly. Error patterns evolve in three stages—confusion with nearby earlier values, then with distant earlier values, then hallucinated values never present in the input—which the paper interprets as the exhaustion of a working-memory buffer. Parameter size, not advertised context length, predicts the Interference Endurance Score, the area under the accuracy-versus-update-count curve. Explicit prompts to forget outdated updates or focus on recent ones give only marginal gains, chain-of-thought variants do not recover the lost accuracy, and only a mock-QA reset that inserts a hard context boundary substantially helps, though accuracy still declines with interference.","pith_inferences":["A control not run in the paper would query one key at a time while varying interference in a constant-length context; that would separate retrieval interference from the load of generating many answers.","The success of the mock-QA reset hints that any mechanism imposing a hard context boundary, such as a learned forget gate or a memory reset, might reproduce the gain, but it is unknown whether such gating can be selective rather than discarding all history.","The three-stage error shift from nearby values to hallucinations is a behavioral signature that could be tracked mechanistically, for example by probing attention or induction-circuit weights, to test whether old associations remain bound rather than being unbound.","Because the benchmark uses synthetic word categories, an extension to naturalistic streams such as patient vitals, versioned code state, or mutable ledgers would show whether the log-linear law survives realistic semantic overlap."],"forward_implications":["Length-based long-context benchmarks are measuring a different axis: extending context appears not to give a model more anti-interference capacity.","Model-size comparisons should account for interference load, since larger parameter counts predict slower log-linear decline while context-window length does not.","Prompt-level fixes such as 'ignore earlier updates' or 'focus on recent text' are insufficient, so improving interference tolerance likely needs architectural changes, for instance an explicit gating or reset mechanism.","Reasoning or chain-of-thought modes do not cure interference-affected retrieval, so allocating more inference compute will not by itself close this gap.","A single interference budget is taxed by three different manipulations—update count, tracked-key count, and value length—so retrieval reliability under distraction can be expected to degrade on any of these axes independently."],"supporting_citations":[{"why":"It provides the Lost-in-the-Middle retrieval benchmark whose position-based task the paper contrasts with its interference-controlled key-value design.","marker":"Liu et al., 2024"},{"why":"It introduces Michelangelo, a long-context retrieval benchmark with similar distractors; the paper cites it as the baseline that does not isolate interference.","marker":"Vodrahalli et al., 2024"},{"why":"It supplies the proactive-interference paradigm and the human unbinding mechanism whose absence the paper claims for LLMs.","marker":"Oberauer & Vockenber, 2009"},{"why":"It provides the word-length effect in human working memory that motivates the paper's value-length interference manipulation.","marker":"Baddeley et al., 1975"},{"why":"It shows directed forgetting reduces proactive interference in humans, motivating the natural-language forget instructions that the paper finds ineffective.","marker":"Festini & Reuter-Lorenz, 2014"},{"why":"It demonstrates instruction-hijacking techniques that inspire the mock-QA-reset prompt, the paper's one partially successful intervention.","marker":"Kuo et al., 2025"},{"why":"It offers induction heads as a mechanism for in-context association, which the paper frames its behavioral PI findings as constraining.","marker":"Anthropic, 2022"}],"fun_headline_variants":["Working memory bottleneck: LLMs forget latest updates as old values pile up","Proactive interference plagues LLM retrieval: accuracy drops log-linearly","Even clear targets fail: LLMs lose recent values to interference","Context length isn't the fix: LLM memory bottleneck from interference","LLMs show a working memory limit: proactive interference beats context"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"That in the fixed-input-length experiment, varying the number of tracked keys changes only interference load; in fact it also changes how many values the model must output, so the observed decline could partly reflect generation load rather than retrieval interference.","fun_headline_variants_meta":{"raw":{"variants":["Working memory bottleneck: LLMs forget latest updates as old values pile up","Proactive interference plagues LLM retrieval: accuracy drops log-linearly","Even clear targets fail: LLMs lose recent values to interference","Context length isn't the fix: LLM memory bottleneck from interference","LLMs show a working memory limit: proactive interference beats context"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000579,"raw_usage":{"total_tokens":2740,"prompt_tokens":967,"completion_tokens":1773,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":583,"completion_tokens_details":{"reasoning_tokens":1692}},"tokens_in":583,"tokens_out":1773,"duration_ms":14923,"temperature":1.0,"reasoning_tokens":1692,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:17:22.935784+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the fixed-length setup while always asking the model to report a single key's final value, varying only the number of prior updates or updated keys; if accuracy no longer falls log-linearly with interference, the claimed isolation of interference from output load is falsified.","supporting_citations":[{"cited_title":"D., Thomson, N., and Buchanan, M","cited_arxiv_id":null,"evidence_quote":"It provides the word-length effect in human working memory that motivates the paper's value-length interference manipulation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It shows directed forgetting reduces proactive interference in humans, motivating the natural-language forget instructions that the paper finds ineffective."},{"cited_title":"In-context learning and induction heads, March 2022","cited_arxiv_id":null,"evidence_quote":"It offers induction heads as a mechanism for in-context association, which the paper frames its behavioral PI findings as constraining."}],"review_version":1}