{"id":"943d5df3-8a23-43a0-83f3-6f907edd756c","arxiv_id":"2501.05078","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Bypassing attention in the deepest transformer layers reduces extractable memorization with little loss on standard benchmarks, while early-layer bypass collapses the model.","lead":"Researchers replaced the attention operation in each transformer layer of large language models with an identity operation, then measured how memorized training text extraction and standard benchmark scores change. They find that disabling attention in the deepest layers cuts verbatim memorization sharply while benchmark accuracy stays roughly flat, and they conclude that deep attention is the main driver of memorization.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Identity-attention ablation lacks a perturbation control, so the deep-layer-specific memorization drop may reflect generic sensitivity of greedy exact-match decoding rather than a memorization mechanism.","rationale":"The reader's weakest_assumption identifies precisely the load-bearing gap: the identity-attention intervention is assumed to be a neutral removal of attention's contribution, but no baseline perturbation is tested. My reading of Sec. 3.2 and the results confirms that all evidence for the central claim comes from comparing identity-attention ablation depth-wise, without a control for generic perturbation. The exact-match metric makes this especially problematic, because a small distributional shift at any late layer can break exact greedy continuations, while benchmark accuracies are averaged over many samples and therefore tolerant to small shifts. The theoretical section does not rescue the argument: Theorems 1-2 bound differences in hidden representations, not memorization, and the proof treats attention as a convex combination of input vectors, omitting the learned value/output projections that the actual identity-attention replacement still applies. These issues do not invalidate the empirical pattern, but they do mean the paper overstates the attribution. A matched-perturbation control would settle the concern. Since the reader's conditional verdict already requires exactly this added evidence, my stress-test does not change the verdict: it remains CONDITIONAL pending the control experiment.","tokens_in":25441,"tokens_out":5448,"duration_ms":58807,"concrete_test":"On GPTNeo-1.3B and Pythia-2.8B, for each layer in the last quartile, construct three edited variants: (i) the paper's identity attention; (ii) attention weights replaced by a random doubly stochastic matrix with row entropy matched to the empirical attention distribution; (iii) additive Gaussian noise injected into the post-projection attention output, with variance tuned so the mean Frobenius change in the layer output equals that of identity attention. Evaluate exact match and token accuracy on the same memorized samples and accuracy/perplexity on ARC-Easy, PIQA, HellaSwag, LAMBADA, and Wikitext. If (ii) and (iii) also drive exact match near zero while preserving benchmark scores, the deep-layer attribution is a generic perturbation artifact; if they do not, the identity intervention is specific to the attention computation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that deep-layer attention is 'primarily responsible' for memorization rests on the assumption in Sec. 3.2 that replacing attention weights with the identity matrix cleanly isolates the attention contribution. The paper never tests whether a generic perturbation of deep layers produces the same effect. Exact Match (Eq. 3) is a brittle, thresholded metric: any intervention that slightly shifts deep-layer logits can push greedy decoding off a memorized suffix, while aggregate benchmark accuracies (ARC-Easy, PIQA, etc.) are far less sensitive to small per-sample distribution shifts. Thus the observed memorization drop could occur for any perturbation of deep layers, not specifically for removing the attention mechanism. The theoretical analysis (Theorems 1-2) does not close this gap: it bounds hidden-state norm differences, not memorization, and its proof assumes value vectors equal inputs and ignores learned value/output projections, so it does not describe the actual identity-attention operation on real models. Without comparing against noise injection or randomly fixed attention weights of matched perturbation magnitude, the paper cannot distinguish 'deep attention encodes memorization' from 'deep layers are close to the output head, so perturbing them breaks exact continuation.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an architectural attribution method for memorization in decoder-only LLMs. It \"short-circuits\" the attention module at a selected transformer block by replacing the attention weight matrix with the identity, leaving layer normalization and MLP blocks intact, and measures the effect on extractable memorization (exact match, token accuracy, completion entropy) and on five downstream benchmarks (ARC-Easy, HellaSwag, LAMBADA, PIQA, Wikitext). The authors report that short-circuiting attention in later blocks sharply reduces memorization while preserving downstream accuracy, whereas short-circuiting earlier blocks causes model collapse. They also present two theorems bounding the norm of the hidden-state difference under their intervention and claim these bounds, together with the experiments, show that deep-layer attention is primarily responsible for memorization and earlier layers are crucial for generalization and reasoning. The code and data are released.","tokens_in":25645,"tokens_out":4910,"duration_ms":52007,"significance":"If the central claim held, the paper would offer a practical, architecture-level mitigation for verbatim memorization in open-weight LLMs, with a clear disentanglement of memorization and general capabilities. The empirical sweep is a genuine strength: six models (Pythia 1.4B/2.8B/6.9B/12B, GPT-Neo 1.3B/2.7B), five benchmarks, three memorization metrics, and public code and datasets. The paper also honestly lists limitations (single-block interventions, greedy sampling only, no closed-source models). However, the significance is presently capped by two issues: the intervention is not compared to generic perturbations, so the attribution to \"attention\" rather than \"any deep-layer perturbation\" is not established, and the theoretical analysis does not actually connect its bounds to memorization. Both are addressable in revision, which is why I recommend major revision rather than rejection.","major_comments":[{"comment":"The central attribution claim—that deep-layer attention is \"primarily responsible\" for memorization—rests on the assumption that replacing the attention weight matrix with the identity isolates the attention mechanism's contribution. The paper never tests this assumption against a generic-perturbation control. For example, adding isotropic Gaussian noise to the deep-layer hidden states, replacing the attention weights with random doubly-stochastic matrices of matched norm, or short-circuiting the deep-layer FFN instead of attention would show whether the drop in exact-match and token accuracy is specific to removing attention or simply a consequence of perturbing any computation near the output head. This control is essential because exact match is a brittle thresholded metric: even a small per-sample logit shift can push greedy decoding off a memorized suffix, while aggregate benchmark accuracies are far less sensitive to such shifts. Without a perturbation control, the observed pattern is consistent with 'deep layers are close to the output head, so perturbing them breaks exact continuation' rather than 'deep attention encodes memorization.' This is the load-bearing bridge from the ablation to the attribution conclusion and must be addressed.","section":"§3.2, §4.2, Eq. (3)"},{"comment":"The theorems bound the norm of the difference between hidden-state vectors with and without identity attention, but memorization is defined and measured through greedy decoding (Eq. 3), token accuracy (Eq. 4), and completion entropy (Eq. 5). A hidden-state norm bound does not, by itself, imply anything about the probability of generating a memorized suffix. The abstract and Sections 1 and 5 state that the theoretical analysis 'reveals' or 'indicates' the memorization result, but the paper provides no argument linking the norm bounds to decoding behavior. To make the theory load-bearing, the authors would need to add a Lipschitz-style bound from hidden-state differences to token-logit differences and then to greedy-decoding outcomes, or explicitly reframe the theory as a heuristic motivation for the experiments rather than a proof of the memorization claim.","section":"§3.3, Theorems 1 and 2"},{"comment":"The proof assumes that the value vectors of the attention mechanism are the input token vectors x_i and that the output projection W_O is ignored. In the actual models, multi-head attention computes V = XW_V and applies Concat(head_1,...,head_h)W_O. The implemented short-circuit operation (replacing the attention weight matrix with the identity in all heads) therefore produces I·XW_V followed by W_O, not simply the input token vector. The theorem's object does not match the intervention that is run in the experiments, so the theoretical bounds cannot be claimed to describe the experimental setup. The authors should either prove the bounds for the full MHA computation (with W_V, W_O, and head concatenation) or clearly state that the theorems apply to a simplified proxy and justify why the proxy is informative.","section":"Appendix A.1, Theorem 1 proof"},{"comment":"The claim that replacing attention at an earlier layer L leads to larger output differences at layer L+1 than replacing attention at layer L+1 does not follow from the stated inequalities. The two bounds involve different quantities—M^L(1−α^L_n) versus M^{L+1}(1−α^{L+1}_n)—which are not ordered by layer index in any general way. Without an additional argument (e.g., an empirical measurement showing that M^L(1−α^L_n) is typically larger than M^{L+1}(1−α^{L+1}_n) in the tested models, or a lemma that bounds one in terms of the other), the theorem is consistent with either depth ordering. The paper's conclusion that the bounds 'suggest' the depth-dependent effect is therefore not supported by the mathematics as written.","section":"§3.3, Theorem 2"}],"minor_comments":[{"comment":"The notation in the short-circuit definition is ambiguous: 'SHORT CIRCUIT ATTENTION (Q, K, V) = I · V' does not state whether I is the identity matrix over sequence positions, over heads, or over the feature dimension, and it does not show how the result is concatenated and passed through W_O. Please specify the exact tensor operation used in the implementation.","section":"§3.2"},{"comment":"In the completion entropy definition, the notation p_j^θ(x_{i+1} | x_{1:i}) is confusing because j indexes the vocabulary while x_{i+1} is a token. It should be written as p_θ(x_{i+1} = v_j | x_{1:i}) or similar.","section":"Eq. (5)"},{"comment":"The memorized-sample collection is described as '15k samples ... highly memorized (more than 90% samples show extractable memorization) by all model scales.' Please clarify whether the same 15k samples are used for each model and whether the 90% figure is verified per model or is an aggregate statement. The differing prefix/suffix lengths for GPT-Neo (150/50) and Pythia (32/32) may also affect the comparability of the two families.","section":"§4.1"},{"comment":"The figures show 'Original Model' as a single point or line, but the text reports that some short-circuited variants outperform the original model (e.g., LAMBADA). Please add error bars or confidence intervals for the memorization and benchmark metrics, since the reported differences across layers are often small and may be within noise.","section":"Figures 1 and A7–A11"},{"comment":"The qualitative generation samples would benefit from a clearer statement of whether they are random draws or typical outputs, and how the 'Block 16' and 'Block 24' examples were selected. Without sampling details, the claim that later-block short-circuiting 'still results in coherent generations' is anecdotal.","section":"§4.2.3 and Figure 4"},{"comment":"The notation for the last token index is inconsistent: Theorem 1 and its proof use n, while Theorem 2 uses both n and l. Please unify the notation. Also, the proof states 'Using the linearity of the FFN' before the linear approximation is explicitly introduced; please move the approximation statement before its first use.","section":"§3.3 and Appendix A.1"},{"comment":"There are several typos and formatting issues in figures and captions: 'consistenly', 'GTPNeo' in Figures A12 and A13, 'T asks' and 'Language T asks' in Figures 6 and A22, and inconsistent spacing in 'T oken Accuracy'. These should be corrected before final submission.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper currently overstates the theoretical support for its central claim: the theorems do not address memorization and do not describe the actual MHA computation. The empirical pattern is interesting and the resource investment (six models, five benchmarks, public code) is substantial. I believe the manuscript is salvageable if the authors add perturbation controls, explicitly downgrade or repair the theory, and rephrase the abstract and conclusion to match what is actually shown. If the controls fail to distinguish identity attention from generic noise, the central claim would need to be substantially weakened; that is why I placed the recommendation at major revision rather than minor."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful empirical paper with an overstrong headline. What's new is the layer-resolved ablation: short-circuiting attention (replacing it with identity) in the last few blocks of Pythia and GPT-Neo consistently kills most greedy-decoding extractable memorization, while leaving ARC-Easy, PIQA, etc. roughly intact. That directly challenges Stoehr et al.'s claim that memorization can't be localized, and the breadth (6 models, 5 benchmarks) is solid. Code and data are released, and the intervention is trivial to reproduce. That alone warrants a serious look.\n\nThe paper's main problem is the attribution claim. The observation is that a specific deep-layer perturbation reduces exact-match continuation. What we don't know is whether it's the attention removal that matters or just any perturbation of deep layers. The deep layers sit right before the LM head; a small logit shift can push greedy decoding off a memorized suffix, while multiple-choice benchmarks are less brittle to per-sample shifts. The paper never tests noise injection or randomly fixed attention weights as controls, so 'primarily responsible for memorization' outruns the evidence. The reader's stress-test note lands; I read Sec. 3.2 carefully and the 'effectively removing the contribution' language is where the gap sits.\n\nThe theory doesn't rescue it. Theorems 1-2 bound norm differences in hidden states, not memorization, and the linear-FFN approximation carries an uncontrolled error term that makes the bounds vacuous in practice. The proof also assumes value vectors equal inputs and skips learned output projections, so it doesn't describe the actual identity-attention operation on real models. This section can stay, but it should be labeled as intuition, not as an explanation of memorization.\n\nThe memorized-sample selection is outsourced to prior work, which is fine; the metrics are standard; the greedy-sampling-only scope is disclosed in the limitations. No fit parameters, no circularity.\n\nBottom line: for a privacy-motivated reader who wants an inference-time lever to reduce verbatim extraction, this is a genuinely useful stepping stone. For an interpretability reader who wants a causal mechanism, it needs the perturbation control and softer language. I'd send it to review and ask for: (1) noise/random-attention baselines at matched perturbation magnitude, (2) a softened abstract, (3) either a repaired proof or an explicit statement that the theory is heuristic.","headline":"A clean, broad ablation suggests deep-layer attention drives extractable memorization, but the missing perturbation control leaves the causal claim underdetermined.","tokens_in":26149,"tokens_out":2172,"would_cite":true,"duration_ms":21763,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that attention modules in the deeper transformer blocks are the primary carriers of verbatim memorization in decoder-only LLMs, while earlier blocks carry generalization and reasoning, and that bypassing deep-layer…","keywords":["memorization","model attribution","attention short-circuiting","transformer interpretability","privacy leakage","generalization","Pythia","GPT-Neo"],"falsifier":"Run the identity-attention short-circuit in the same deep layers and compare it to control edits of comparable magnitude that do not specifically remove attention's token-mixing function, for example adding Gaussian noise to the attention output or fixing the attention weights to a random pattern, and measure extractable exact-match memorization and benchmark accuracy; if the controls erase memorization as effectively as the identity does, the conclusion that deep attention is the memorization mechanism fails.","tokens_in":25210,"feed_emoji":"🧠","tokens_out":9826,"duration_ms":87502,"temperature":0.7,"pith_summary":"Large language models can reproduce verbatim chunks of their training data, raising privacy and copyright concerns, but it has not been clear which parts of the architecture actually hold that memorized text. This paper argues that the attention modules in the deeper transformer blocks are the primary carriers of memorization, while the earlier blocks provide the model's general language and reasoning abilities. The evidence comes from \"short-circuiting\" attention in individual blocks, replacing the attention weight matrix with the identity so the layer's attention contributes nothing new while layer normalization and the MLP remain untouched. Short-circuiting deep blocks collapses exact-match and token-accuracy memorization scores while leaving benchmark accuracy largely intact, whereas short-circuiting early blocks destroys coherent generation. If the attribution is right, it gives an inference-time lever for suppressing verbatim training-data leakage in decoder-only LLMs without retraining.","feed_headline":"Deep attention layers are what make LLMs memorize","feed_subtitle":"Replacing deep-layer attention with identity cuts exact training-data recall while keeping benchmark scores.","key_machinery":"The load-bearing object is the attention short-circuit: replacing the multi-head self-attention weight matrix $\\text{softmax}(QK^T/\\sqrt{d_k})$ in a chosen layer with the identity matrix $I$, so that $\\text{ShortCircuit}(Q,K,V)=I\\cdot V=V$, removing token mixing while leaving the residual stream, layer normalization, and feed-forward MLP intact. This isolates the attention module's contribution at a given depth. Two theorems support the interpretation: Theorem 1 bounds the normed difference between the output vectors of a single block with standard versus identity attention by $(1+\\|W\\|)M(1-\\alpha_l)+\\|\\epsilon_{IA}-\\epsilon\\|$, and Theorem 2 shows that the difference from an early-layer short-circuit propagates and can amplify into layer $L+1$ through the FFN and residual terms, whereas a late-layer short-circuit produces a localized difference. This mechanism explains the empirical asymmetry between early and deep blocks.","core_discovery":"The paper's central claim is that memorization in decoder-only transformers is not spread uniformly through the network: the attention sublayer in the final layers is where extractable verbatim recall lives, whereas the early layers maintain the representations needed for generalization and reasoning. Replacing deep-layer attention with identity attention means forcing the attention output to equal the value vectors, so no token mixing occurs, and this reduces extractable memorization—exact match and token accuracy drop sharply—while downstream benchmark scores hold and in some cases even improve. The same intervention in early layers drives the model into gibberish, and the paper's theoretical bounds on representation differences explain why: perturbations from an early-layer edit propagate and amplify through all subsequent layers, while a late-layer edit stays localized. Across Pythia models from 1.4B to 12B and GPT-Neo models of 1.3B and 2.7B, tested on ARC-Easy, HellaSwag, LAMBADA, PIQA, and Wikitext, the pattern is consistent, with reasoning tasks more resistant than language-understanding tasks. The paper reads this as evidence that memorization and generalization are separable at the level of individual attention blocks.","pith_inferences":["A direct test of the paper's causal story would compare the identity short-circuit against other deep-layer perturbations of matched magnitude, such as adding noise to attention outputs or fixing attention to random patterns; if those also erase memorization, the effect may be generic disruption rather than removal of the memorization mechanism.","Because the paper evaluates only greedy sampling, an open question is whether deep-layer bypass also blunts stronger extraction attacks like beam search or sampling-based reconstruction; if deep attention is truly the memorization substrate, those attacks should weaken too.","The scale-dependence the paper reports invites a further hypothesis: in larger models, memorization may be spread across more layers, so a constant-depth bypass should become less effective as scale grows, predicting that a depth-proportional bypass restores the effect at larger sizes."],"forward_implications":["Bypassing attention in the last quartile of blocks reduces extractable exact-match memorization sharply, and bypassing an entire quartile drives it near zero, while benchmark accuracy is largely preserved.","The short-circuit can be applied per layer at inference time, offering a way to suppress verbatim training-data generation without retraining or weight modification.","Larger models resist the intervention: the memorization reduction shrinks as model scale grows from 1.4B to 12B, suggesting that capacity changes how memorization is stored.","Reasoning-oriented benchmarks such as PIQA and ARC-Easy show smaller relative accuracy drops than language-understanding benchmarks such as HellaSwag and LAMBADA under the same edits, so some late blocks appear to contribute almost exclusively to memorization.","Short-circuiting early blocks collapses the model, indicating that early attention layers are necessary for general language abilities."],"supporting_citations":[{"why":"Supplies the extractable-memorization definition and the memorized training samples used to build the evaluation set.","marker":"(Carlini et al., 2020)"},{"why":"Provides the memorized-sample datasets for Pythia and GPT-Neo and the greedy-sampling evaluation protocol.","marker":"(Prashanth et al., 2024)"},{"why":"Introduces the Pythia model family whose layers are short-circuited across four scales.","marker":"(Biderman et al., 2023)"},{"why":"Introduces GPT-Neo, the second model family tested.","marker":"(Black et al., 2021)"},{"why":"The Pile is the public training corpus these models were trained on, so memorized strings are verified as substrings of the training data.","marker":"(Gao et al., 2020)"},{"why":"Defines the quantitative memorization metrics such as exact match and token accuracy that the paper measures.","marker":"(Carlini et al., 2022)"},{"why":"ARC-Easy benchmark used to measure reasoning performance after short-circuiting.","marker":"(Clark et al., 2018)"},{"why":"HellaSwag benchmark used to measure commonsense and language understanding after short-circuiting.","marker":"(Zellers et al., 2019)"},{"why":"LAMBADA benchmark used to measure contextual word-prediction performance.","marker":"(Paperno et al., 2016)"},{"why":"PIQA benchmark used to measure physical commonsense reasoning.","marker":"(Bisk et al., 2020)"}],"fun_headline_variants":["Memorization lives in deep attention layers","LLM memorization traced to deep attention blocks","Deep layers memorize, early layers generalize","Swapping deep attention cuts memorization, not reasoning","Bypass deep attention to curb LLM memorization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that replacing a deep layer's attention weights with the identity is a neutral removal of that attention module's contribution, so any resulting drop in memorization is evidence that the module itself stored the memorized text, rather than an artifact of a generic perturbation that makes exact continuation less likely.","fun_headline_variants_meta":{"raw":{"variants":["Memorization lives in deep attention layers","LLM memorization traced to deep attention blocks","Deep layers memorize, early layers generalize","Swapping deep attention cuts memorization, not reasoning","Bypass deep attention to curb LLM memorization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1405,"prompt_tokens":986,"completion_tokens":419,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":358}},"tokens_in":602,"tokens_out":419,"duration_ms":3736,"temperature":1.0,"reasoning_tokens":358,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:19:17.665052+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the identity-attention short-circuit in the same deep layers and compare it to control edits of comparable magnitude that do not specifically remove attention's token-mixing function, for example adding Gaussian noise to the attention output or fixing the attention weights to a random pattern, and measure extractable exact-match memorization and benchmark accuracy; if the controls erase memorization as effectively as the identity does, the conclusion that deep attention is the memorization mechanism fails.","supporting_citations":[{"cited_title":"Brown, Dawn Xiaodong Song, \\'U lfar Erlingsson, Alina Oprea, and Colin Raffel","cited_arxiv_id":null,"evidence_quote":"Supplies the extractable-memorization definition and the memorized training samples used to build the evaluation set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the Pythia model family whose layers are short-circuited across four scales."}],"review_version":1}