{"id":"1a2f21c5-a49c-47f0-a1df-56dedb783bd3","arxiv_id":"2412.15113","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A residual connection that copies previous-layer attention values into the current layer improves in-context learning in transformers up to 1B parameters.","lead":"This paper builds a hand-designed associative-memory attention model that solves a synthetic in-context learning task, then uses it to motivate passing attention values from one layer to the next in Transformers. Small language models with this 'residual value stream' show faster and stronger in-context learning on synthetic tasks, an 8M toy model, and a 1B model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 1B-scale ICL gains rest on a single run per architecture, and the paired checkpoint t-test does not control for seed or run variance.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the 1B-scale conclusion is underpowered because it uses one run per architecture and an invalid paired t-test over correlated checkpoints. The small-scale evidence is stronger: the two-layer Transformer experiments use four seeds and report threshold-crossing times with t-tests across independent runs, and the 8M toy-language-model experiments use three matched seeds with large reported differences. The paper's own limitation section acknowledges that generalizability remains uncertain and that gains are modest and task-specific, which is consistent with a CONDITIONAL verdict. My stress-test does not change the reader's verdict because the concern already motivates CONDITIONAL; however, I would sharpen the condition: the required revision is not merely a better p-value on the existing data, but either multiple independent runs at 1B scale or a clearly reframed claim that the 1B results are a single-run indication rather than a demonstrated improvement. The proposed concrete test, additional matched seeds with final-checkpoint comparisons, directly settles whether the observed gap is reproducible or run-specific.","tokens_in":32949,"tokens_out":4208,"duration_ms":41483,"concrete_test":"Retrain the 1B classic and residual-value-stream models with at least three matched random seeds each, using identical data ordering per seed pair, and report final-checkpoint accuracies on ARC, OpenBookQA, and IOI-Hard separately. Run a paired t-test over seeds on final-checkpoint performance; if the residual-value advantage is not consistently positive with a bootstrap confidence interval excluding zero, the 1B-scale improvement claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central scale-up claim, stated in the abstract as improved in-context learning 'at these larger and more naturalistic scales,' depends on the 1B-parameter experiments in Section 5.2. For this claim to hold, the observed advantage of the residual value stream over the classic architecture must reflect a reproducible architectural benefit rather than run-specific noise. That condition is not secured: each 1B architecture was trained once, so there is no estimate of seed or initialization variance. The reported significance test is a paired two-sample t-test over evaluation checkpoints across training time. These checkpoints are not independent samples; they are highly autocorrelated observations from the same run, so the effective sample size is far smaller than the number of checkpoints and the p-values are not valid. The authors themselves report a 0.993 correlation between the two models' accuracy trajectories, which underscores the non-independence of the paired series. Additionally, the test is applied to the average accuracy across tasks and shots over training steps, conflating learning speed with final performance; if the gap narrows or vanishes at the final checkpoint, the claim of improved performance is weaker than the average-over-training suggests. The IOI-Hard result in Figure 6 is visually striking, but it is also from the same single pair of runs. Because the 1B comparison is the only evidence that the architectural modification scales, this concern is load-bearing: if the gap is seed-specific, the headline conclusion is unsupported, even though the two-layer and 8M experiments with multiple seeds remain valid.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AMICL, a one-layer associative memory model that performs an in-context classification task through an attention-like update, and uses it as inspiration for a residual values stream: in each Transformer layer after the first, the values of each attention head are added to the values of the same-index head in the next layer. The authors report that in a two-layer Transformer on the synthetic ICL task, the residual values stream reaches 0.95 accuracy on held-out in-context probes roughly 24% faster than the classic architecture, with t-tests over four seeds. They then apply the same modification to 8M-parameter and 1B-parameter language models; the 8M models show higher accuracy on an IOI probe, and the single pair of 1B models shows modest average gains on ARC/OpenBookQA and a large gain on a constructed IOI-Hard task. The central claim is that this simple, parameter-neutral architectural change improves in-context learning and that the improvement scales to naturalistic language-model settings.","tokens_in":33242,"tokens_out":6445,"duration_ms":55025,"significance":"If the 1B-scale result were secure, this would be a practically interesting result: a one-line residual connection between same-index attention head values, requiring no additional parameters, that improves ICL at scale. The paper has real strengths: the code is released, the two-layer experiments use four seeds with standard deviations and t-tests on held-out probes, and the 8M experiments use three matched seeds. The AMICL construction is a useful conceptual bridge between associative memory and attention, although it is a constructed model rather than a derivation, and the smoothing parameter a is hand-set after a grid scan; this does not create circularity for the architecture experiments because those are evaluated on tasks not used to tune a. The main weakness is that the scale-up claim rests on a single pair of 1B training runs and an invalid paired t-test over correlated checkpoints, which is the load-bearing evidence for the abstract's claim of improved performance at larger scales.","major_comments":[{"comment":"The 1B-scale conclusion rests on a single training run per architecture, so there is no estimate of seed or initialization variance. The paired two-sample t-test is applied to evaluation checkpoints from these same two runs; the checkpoints are highly autocorrelated, and the authors themselves report a 0.993 correlation between the two accuracy trajectories. The effective sample size is therefore far smaller than the number of checkpoints, and the reported p-values do not provide valid evidence that the improvement would replicate across runs. Because this is the only evidence at the 1B scale, the abstract's claim of improved performance 'at these larger and more naturalistic scales' is not secured. Please provide multiple independent runs per architecture, or analyze final-checkpoint differences with appropriate uncertainty, and calibrate the language of the claims.","section":"§5.2, Figures 5 and 6, t-test paragraph"},{"comment":"The reported significance test is applied to average accuracy over training steps, which conflates learning speed with final performance. If the residual model learns faster but converges to the same final accuracy, the average-over-training would show a significant difference even though the final deployed model is no better. Please report final-checkpoint accuracies (with uncertainty if multiple runs become available) separately from the learning-speed comparison, and state whether the observed gap persists at the end of training.","section":"§5.2, Figure 5 and Appendix Figure 14"},{"comment":"The two natural-language ICL evaluations lack adequate inferential support for the strength of the claims. Table 2 reports large per-sentence standard deviations (for example, classic Sentence 1 correct probability 11.73 ± 15.62) but no significance test across the three seeds, and Figure 6 shows IOI-Hard curves with no error bars because each curve is a single run. Since these evaluations carry the scaling narrative from the synthetic task to language models, please add per-seed breakdowns or appropriate tests, or explicitly label the comparisons as descriptive.","section":"§4.2, Table 2, and §5.2, Figure 6"}],"minor_comments":[{"comment":"Please report the actual t-statistics, degrees of freedom, and p-values rather than 'much smaller than 0.05', and state whether the test was one-tailed or two-tailed in the main text.","section":"§5.2"},{"comment":"Please state explicitly what is averaged in the 'average accuracy': four tasks, two shot settings, and which checkpoints, and whether the average is unweighted.","section":"Figure 5 caption"},{"comment":"There are minor typographical issues: 'langauge models' and the extra space in 'V ariables'; these should be corrected.","section":"Appendix E, notation table"},{"comment":"The limitations paragraph appropriately acknowledges that the gains are modest and task-specific, but this sits in tension with the abstract and Section 5.2's concluding sentence; the strength of the claims should be aligned across the abstract, the results, and the limitations.","section":"§6.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is already published in TMLR according to the header, so this report is retrospective. If the manuscript were under review, the single-run 1B experiment would be a serious barrier to acceptance of the scale-up claim; however, the core architectural idea and the smaller-scale evidence are sound enough that the issue is fixable with additional runs or with a more restricted claim, rather than a fundamental error."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this for the two-layer and 8M experiments; the 1B claim is shakier than the abstract suggests.\n\nWhat's new: a residual connection that carries attention values from each head to the same-index head in the next layer. It's a simple, parameter-free change, and the paper tests it properly at small scale. In the two-layer synthetic ICL task, values-stream networks hit 0.95 accuracy roughly 24% faster than the classic network, across four seeds, with t-tests that look fine. At 8M parameters, three seeds show the residual network solving a basic IOI task at ~41% versus ~7% for the classic. That is a genuine, if modest, effect, and the code is public. The AMICL construction is a nice didactic link between associative memory and ICL, but it's hand-built and isn't the main contribution.\n\nThe soft spot is the 1B experiment in Section 5.2. One training run per architecture, and the 'statistically significant' claim comes from a paired t-test over evaluation checkpoints of those two runs. The checkpoints are highly autocorrelated—the authors themselves report a 0.993 correlation between the two accuracy trajectories—so the effective sample size is nowhere near the number of checkpoints, and the p-values don't mean what they claim. This is load-bearing because the abstract explicitly says the improvement holds at 'larger and more naturalistic scales.' If that gap is seed luck, the headline conclusion falls. The IOI-Hard result is visually striking but comes from the same single pair of runs. The authors' own limitation section helps: they admit gains are modest and task-specific, with PIQA and HellaSwag barely moving. That honesty doesn't fix the statistical reasoning.\n\nMinor: the AMICL parameter a is hand-set after a grid scan, but the architecture experiments use held-out tasks, so circularity is low. Not a problem.\n\nWho this is for: people working on ICL, transformer architectures, or associative-memory links who want a simple, easily testable idea. It deserves a serious referee—the small-scale evidence is solid and the idea is cheap to check—but the 1B claims need independent runs or a more cautious framing before they can be believed.","headline":"Useful small-scale result; the 1B scale-up claim rests on a single run per architecture and a statistically invalid paired t-test, so treat the headline as unproven.","tokens_in":33764,"tokens_out":2578,"would_cite":true,"duration_ms":23950,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a residual connection that passes attention values from each head to the same-index head in the next layer improves in-context learning in Transformers across scales from a two-layer synthetic task to 1-billion-parameter language…","keywords":["in-context learning","associative memory","residual values stream","attention","transformer","language models","indirect object identification","neuroscience-inspired architecture"],"falsifier":"Train the 1B classic and residual-values-stream architectures from several fresh random seeds, holding data order and compute budget fixed, and compare the distributions of final accuracy on ARC, OpenBookQA, and IOI-Hard; if the residual stream's advantage does not exceed run-to-run variance, the scaling claim fails. As a mechanism check at small scale, scramble the head-index mapping of the residual stream: if the IOI gain persists under scrambling, the improvement is not about preserving specific head values.","tokens_in":32767,"feed_emoji":"🧠","tokens_out":11027,"duration_ms":83201,"temperature":0.7,"pith_summary":"Large language models can adapt to tasks and patterns that appear only in their input context, an ability called in-context learning (ICL). This paper argues that ICL can be understood as associative memory, and constructs a one-layer associative memory model, AMICL, that performs an object–label ICL task in a single forward pass by letting attention values directly stand for the remembered tokens. From that model the paper derives an architectural prescription: let the attention values of each head in one layer flow directly into the same-index head of the next layer through a residual values stream. In a two-layer Transformer this modification reaches 0.95 accuracy on in-context probes about 24 percent faster than the classic network, and it also improves ICL-related evaluations in language models with 8 million and 1 billion parameters. If the claim is right, a small wiring change that adds no parameters could make Transformers acquire few-shot adaptation earlier in training and carry it to larger, more naturalistic settings.","feed_headline":"Value shortcuts speed in-context learning in Transformers","feed_subtitle":"A residual values stream reaches the accuracy target ~24% faster and lifts an 8M-model IOI probe from 7% to 41%.","key_machinery":"The load-bearing object is the residual values stream: a residual connection, present in every layer but the first, that feeds the value vectors $V_{n-1}$ of each attention head into the values $V_n$ of the same-index head in the next layer, so that $V_n = W_v^n X_n + V_{n-1}$. It adds no trainable parameters. In AMICL, the values are the input tokens themselves, so the stream is the Transformer analogue of retaining the remembered pattern; in a trained network it preserves 'look-back' information in the answers that attention produces, acting as a memory pathway across layers. The argument is framed through the universal associative memory operation $\\mathrm{projection}(\\mathrm{separation}(\\mathrm{similarity}(K,Q)),V)$, with the value stream carrying the projection term that determines what is retrieved.","core_discovery":"The paper's central claim is that residual attention value streams improve in-context learning. It first defines AMICL, an associative memory model in which each token's key and query are a weighted average of the previous and current token, the value is the token itself, and next-token prediction is the universal associative memory operation $\\mathrm{projection}(\\mathrm{separation}(\\mathrm{similarity}(K,Q)),V)$. Because values equal inputs, retrieval returns the original stored label, which the paper identifies as auto-association. Training experiments show that deleting value projections outright destabilizes learning, so the paper retains learned value matrices and instead adds the previous layer's values to the current layer's values, $V_2 = W_v^2 X + V_1$, with multi-head models adding same-index heads. In the two-layer synthetic Transformer, residual values streams cross the 0.95-accuracy threshold on the in-context probes IC and IC2 about 24 percent sooner than the classic architecture and beat residual streams applied to queries or keys. In 8M-parameter language models trained on TinyStories, an indirect-object-identification probe improves from about 7 percent to about 41 percent correct; in 1B-parameter Llama-style models, average single- and five-shot accuracy on ARC, PIQA, OpenBookQA, and HellaSwag is moderately higher, with the largest gains on ARC, OpenBookQA, and a 1,000-prompt IOI-Hard probe.","pith_inferences":["If the mechanism is preservation of input-identity information, then weighting the added value stream (analogous to AMICL's parameter $a$ or a successor-representation discount factor) could tune how far across layers a contextual association persists; the paper leaves that weight fixed at 1.","The same-index-head coupling is a convenience choice; a learned or random permutation of head indices between layers would test whether the benefit comes from preserving particular head values or from any preserved value information.","The 1B result is one training run per architecture, so a natural next experiment is to train both architectures from several random seeds and compare final accuracies; the paper's paired t-test over checkpoints does not capture seed variance.","If value streams make intermediate activations more stable, they may also aid interpretability by creating a linear path through layers that can be inspected to see where in-context associations are formed."],"forward_implications":["In the two-layer synthetic setup, residual values streams make in-context probes IC and IC2 cross the 0.95-accuracy threshold about 24 percent sooner than the classic network, and the ordering holds at lower thresholds of 0.5 and 0.9.","The benefit is tied to values rather than to any skip connection: residual streams on queries or keys also speed learning relative to classic, but residual values streams are significantly faster on IC and IC2.","At 8M parameters, the residual values stream lifts correct next-token probability on an indirect-object-identification probe from about 7 percent to about 41 percent while reducing confusion between the subject and the indirect object.","At 1B parameters, the modification yields modest but sustained average accuracy gains across ARC, PIQA, OpenBookQA, and HellaSwag, with a large gain on the IOI-Hard probe, suggesting the effect transfers to more naturalistic data and larger scale.","Because the modification adds no parameters, it is a pure wiring change that can be layered onto existing Transformer stacks without altering parameter count."],"supporting_citations":[{"why":"Defines the Transformer and attention mechanism that the residual values stream modifies.","marker":"Vaswani et al., 2017"},{"why":"Establishes the view of attention as a modern Hopfield associative memory update that motivates AMICL.","marker":"Ramsauer et al., 2021"},{"why":"Supplies the universal associative memory framework projection-separation-similarity in which AMICL is formulated.","marker":"Millidge et al., 2022"},{"why":"Provides the synthetic ICL classification task, the token embedding statistics, and the IW/IC/IC2 probe tasks used in the two-layer experiments.","marker":"Reddy, 2024"},{"why":"Documents induction heads and in-context learning in Transformers, used to contrast that ICL is not typically seen in single-layer Transformers.","marker":"Olsson et al., 2022"},{"why":"Identifies the indirect-object-identification circuit in GPT-2 Small and supplies the IOI evaluation used for the language models.","marker":"Wang et al., 2023"},{"why":"Provides the TinyStories training corpus for the 8M-parameter toy language models.","marker":"Eldan & Li, 2023"},{"why":"Specifies the Llama 3 style architecture and tokenizer used for the 1B-parameter models.","marker":"Grattafiori et al., 2024"}],"fun_headline_variants":["Residual value streams accelerate in-context learning in Transformers","Associative memory inspires faster in-context learning via value stream","Adding residual attention values speeds up Transformer in-context learning","New architecture boosts in-context learning with residual value flow","Residual values stream improves Transformer in-context learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 1-billion-parameter comparison rests on a single training run for each architecture, and the reported significance test compares evaluation checkpoints of those same two runs, so the observed advantage could shrink or vanish when the models are retrained from different random seeds.","fun_headline_variants_meta":{"raw":{"variants":["Residual value streams accelerate in-context learning in Transformers","Associative memory inspires faster in-context learning via value stream","Adding residual attention values speeds up Transformer in-context learning","New architecture boosts in-context learning with residual value flow","Residual values stream improves Transformer in-context learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000415,"raw_usage":{"total_tokens":2189,"prompt_tokens":1035,"completion_tokens":1154,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":1072}},"tokens_in":651,"tokens_out":1154,"duration_ms":6692,"temperature":1.0,"reasoning_tokens":1072,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:36:53.007846+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the 1B classic and residual-values-stream architectures from several fresh random seeds, holding data order and compute budget fixed, and compare the distributions of final accuracy on ARC, OpenBookQA, and IOI-Hard; if the residual stream's advantage does not exceed run-to-run variance, the scaling claim fails. As a mechanism check at small scale, scramble the head-index mapping of the residual stream: if the IOI gain persists under scrambling, the improvement is not about preserving specific head values.","supporting_citations":[{"cited_title":"Attention is all you need","cited_arxiv_id":null,"evidence_quote":"Defines the Transformer and attention mechanism that the residual values stream modifies."},{"cited_title":"a fl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Thomas Adler, David Kreil, Michael K Kopp, G \\","cited_arxiv_id":null,"evidence_quote":"Establishes the view of attention as a modern Hopfield associative memory update that motivates AMICL."},{"cited_title":"Universal hopfield networks: A general framework for single-shot associative memory models","cited_arxiv_id":null,"evidence_quote":"Supplies the universal associative memory framework projection-separation-similarity in which AMICL is formulated."}],"review_version":1}