{"id":"ba760101-9ae2-4ebe-885f-d9125dba6667","arxiv_id":"2412.17019","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A GPT's backward pass through attention computes a \"Reversed Attention\" matrix, which can rank attention heads and, when injected into the forward pass, improve task accuracy without weight updates.","lead":"This paper studies the gradients that flow backward through GPT attention layers and shows they form an implicit \"Reversed Attention\" matrix, the softmax derivative. The authors use this matrix to rank attention heads and to patch forward attention, steering model predictions without changing weights.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 17's batched RA matrix is not the softmax derivative: it subtracts a diagonal term rather than the row-wise weighted mean, so the paper's explicit definition of RA does not compute the claimed quantity.","rationale":"I read the paper in good faith. The scalar derivation (Eq. 15), the query/key VJP derivations (Eqs. 18–19), and the qualitative examples are coherent; the idea of the softmax derivative as an implicit signed attention-like map is legitimate, and the perturbation/patching results would be meaningful if the object were computed correctly. However, the paper's central formal definition is internally inconsistent: Eq. 17 is the only explicit batched definition of R, and it does not reproduce Eq. 15. This matters because Section 5 and all experiments rest on R being the softmax derivative; a wrong batched formula changes every RA map, head norm, and patching injection. The issue is fixable (Eq. 17 should use row-wise broadcast centering), and it is possible the authors actually implemented the scalar/autograd form, but with no code at submission the paper as written defines the wrong object. The reader's weakest assumption about template transferability is a separate empirical concern; I agree it is real but secondary. My concrete check settles whether the mathematical core is as claimed. The verdict remains CONDITIONAL: the paper needs a corrected Eq. 17 and a verification that the experiments use the corrected RA.","tokens_in":34908,"tokens_out":13664,"duration_ms":128632,"concrete_test":"Take a 2×2 causal attention matrix A with row1 = [0.5,0.5], row2 = [0,1], and arbitrary G = [[a,b],[c,d]]. Compute Eq. 15 row-wise and Eq. 17 as printed; the row-1 entries agree only if b = −a, i.e., not generally. Independently re-derive the batched softmax derivative as R = A⊙(G − c 1^T), where c = (A⊙G) 1_n, or compute it with autograd on a GPT attention layer. Then check whether the released code implements Eq. 17 or the correct form; if it implements Eq. 17, rerun the head-ranking and patching experiments with the corrected R to see whether the empirical conclusions survive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mathematical claim is that R, the attention-softmax derivative, is an implicit attention matrix. Eq. 15 gives the correct scalar row: r_j = A_j ⊙ (tilde e_j − (A_j·tilde e_j) 1). Eq. 17 purports to batch this as R = A ⊙ (tilde E^T − diag(A tilde E^T))^T. But diag(A tilde E^T) is a diagonal matrix with entry c_j = sum_k A_{j,k} tilde E_{j,k}; after transposition, row j of (tilde E^T − diag(...))^T is tilde E_j − c_j e_j, not tilde E_j − c_j 1. Hence Eq. 17 subtracts the centering term only on the diagonal, whereas softmax differentiation requires subtracting it from every entry of the row. For the minimal causal case A = [[0.5,0.5],[0,1]], G = [[a,b],[c,d]], Eq. 15 gives row1 = [0.25(a−b), −0.25(a−b)] while Eq. 17 gives [0.25(a−b), 0.5b]. Thus the paper's stated matrix definition does not compute the RA it defines, and any reader or code that follows Eq. 17 will obtain a different object. Because this is the formal core of the paper's contribution (contribution ii and Section 5), it must be corrected or the experiments must be shown to use the scalar/autograd form.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the backward pass of attention layers in GPT models. It derives the vector-Jacobian products for the attention weight matrices and identifies the derivative of the attention softmax, denoted R and termed \"Reversed Attention\" (RA), as an implicit attention map. The authors argue that RA encodes how gradient descent would edit attention, and they use RA norms to rank attention heads in a perturbation test and to perform \"attention patching,\" where averaged RA maps are injected into forward attention to steer predictions without weight updates. Experiments are conducted on GPT2-xl, OPT, GPT-j, and Llama2-7B across in-context learning and fact-relation tasks.","tokens_in":35210,"tokens_out":10322,"duration_ms":93006,"significance":"If the central derivation is corrected, the paper offers a cheap, single-backward-pass signal for attention-head localization and a no-weight-update intervention mechanism, which are attractive properties for interpretability research. The scalar softmax-derivative derivation (Eqs. 12-15 and 18-19) is correct, and the perturbation test provides an external benchmark that does not rely on the RA definition itself. The paper also compares across multiple models and tasks, which strengthens the empirical picture. However, the batched definition of R in Eq. 17 is mathematically inconsistent with the scalar formula, so the paper as written does not actually define the object it claims to study. The relationship between attention patching and a single gradient descent step on attention logits also deserves explicit recognition, since it affects how novel and how explanatory the intervention is.","major_comments":[{"comment":"Equation (17) is not equivalent to the row-wise formula in Eq. (15), and the statement \"hence R_j = r_j\" is false. In Eq. (17), the term diag(A tilde E^T) subtracts the scalar c_j = sum_k A_{j,k} tilde E_{j,k} only from the diagonal entry of row j after the transposition, yielding row j of R as A_j ⊙ tilde E_j - c_j A_{j,j} e_j. The correct softmax derivative requires subtracting c_j from every entry of the row, i.e., A_j ⊙ (tilde E_j - c_j 1). To make this concrete, for A = [[0.5,0.5],[0,1]] and tilde E = [[a,b],[c,d]], Eq. (15) gives row 1 = [0.25(a-b), -0.25(a-b)], while Eq. (17) gives [0.25(a-b), 0.5b]. This is a load-bearing error: any reader implementing RA from the paper's formal definition will compute a different object from the softmax derivative. The authors must correct the batched formula (e.g., by using a row-wise subtraction of the vector of row sums of A ⊙ tilde E) or explicitly state that all experiments use the scalar/autograd form.","section":"Section 4, Eq. (17)"},{"comment":"The attention-patching method injects a multiple of the RA map into the forward attention scores. Since R is the derivative of the loss with respect to the attention logits (up to the scaling factor), adding λ R with negative λ is exactly a gradient descent step on the attention logits. The paper should state this equivalence and discuss its consequences. In particular, the success of attention patching should be framed as evidence that a first-order approximation of a loss-minimizing edit can steer the model, rather than as independent evidence that RA reflects the model's \"desired attention\" in a broader sense. This does not invalidate the results, but it is central to the interpretation and novelty of contribution (v).","section":"Section 5.3"},{"comment":"The empirical validations assume that RA maps averaged over 25 examples from a single task template are stable and transferable to held-out examples. The paper does not report variance across example subsets or across random seeds, and the patching results depend on a fixed learning rate of -30 without a sensitivity analysis. Since the head-ranking and patching claims rest on this transferability, the authors should provide error bars or multiple-split results for the AUC values in Tables 1-12 and a learning-rate sensitivity study for the patching results in Table 13.","section":"Sections 5.2-5.3 and Appendices B-C"}],"minor_comments":[{"comment":"The outer-product order in the update equation for W_o is inconsistent with the surrounding notation: Eq. (9) writes δ_o × x_o^T, but the derivation of Eq. (11) requires x_o^T × δ_o. Please correct the notation.","section":"Section 4, Eqs. (8) and (10)"},{"comment":"The sentence \"In Equation 16 we defined R\" is inaccurate; Eq. (16) defines tilde E, while R is defined in Eq. (17).","section":"Section 4, text after Eq. (16)"},{"comment":"The method name is misspelled as \"Casual Mediation\" in several table headers; it should be \"Causal Mediation.\"","section":"Tables 1 and 2"},{"comment":"Using the term \"learning rate\" for the scalar that scales the injected RA map is potentially confusing, since no parameter update is performed; consider renaming it \"injection strength\" or \"patching coefficient.\"","section":"Section 5.3 and Appendix C"},{"comment":"The paper describes the analysis as \"assumption-free full GPT models,\" but it omits LayerNorm and, in later experiments, uses models with RoPE. The Limitations section acknowledges some of this, but the earlier claim should be softened to avoid overstatement.","section":"Section 3 and Limitations"},{"comment":"The perturbation test uses AUC as the summary statistic, but the tables do not report standard deviations or significance tests. Given that some differences between methods appear small, this makes it hard to judge whether RA's advantage over CM is reliable.","section":"Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The primary issue is the incorrect batched formula in Eq. (17), which is central to the paper's formal contribution. Once corrected, the scalar derivation and the empirical validation are plausible and the paper could be acceptable. The comparison to gradient descent on attention logits should be addressed explicitly, as it affects the novelty framing. I cannot verify the experiments because the code is not yet available despite the abstract promising a repository URL."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper identifies something real: in GPT attention backprop, the softmax derivative is an attention-like matrix, and they show two uses for it—ranking heads by RA norm and steering a frozen model by injecting averaged RA maps into the forward pass. The scalar derivation (Eqs. 12–15) is correct, and the qualitative examples genuinely illustrate how the backward pass moves queries and keys. The head-ranking perturbation test is an honest external benchmark, and RA is competitive with causal mediation across four models and 21 simple tasks. That is a real contribution, not just a rename of textbook backprop.\n\nThe novelty is modest but real. Grad-SAM and activation patching don't do this; treating the softmax Jacobian as an attention map and using it for no-weight-update steering is new in the cited literature.\n\nThe soft spot is the batched formula. Eq. 17 is wrong as written. The row-wise rule is r_j = A_j ⊙ (tilde e_j − c_j 1), with c_j = A_j · tilde e_j. Eq. 17 subtracts c_j only from the diagonal entry of each row after the transpose, so the closed-form matrix R is not the softmax derivative. The stress-test example shows the mismatch concretely. If the experiments used autograd, the results may be unaffected, but the paper's definition of RA is not what Eq. 17 computes, and this sits at the formal core of contribution (ii). It has to be fixed or the experiments shown to use the scalar/autograd form.\n\nTwo smaller issues. Attention patching is close to a single gradient step on attention logits, so its success is partly built into the setup; the paper should say that instead of presenting it as a surprise. And the empirical base is thin in one respect: rankings and patching use 25 same-template examples, with no variance estimates. Across tasks and models the pattern is consistent, so I don't think it's an artifact, but robustness on heterogeneous inputs is untested.\n\nWho is this for? Interpretability and in-context-learning people who want a cheap gradient-based head localizer. It deserves a serious referee; the central idea survives the Eq. 17 bug, but the authors should be pushed to fix the math and release code. I'd accept it for review and expect a revise-and-resubmit path.","headline":"RA is a real and useful object with two new applications, but the batched formula in Eq. 17 does not compute what it claims and the empirical support needs robustness work before the central claims can be trusted.","tokens_in":35768,"tokens_out":5246,"would_cite":true,"duration_ms":46021,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that the softmax derivative in GPT attention, called Reversed Attention, is an implicit attention map that reveals how gradient descent would edit attention, and that its norms rank head importance while averaged maps can…","keywords":["reversed attention","attention gradients","backward pass","interpretability","causal mediation","attention patching","GPT","softmax derivative"],"falsifier":"Run attention patching on a test set whose prompt lengths or templates differ from the 25 training examples: if the accuracy gains vanish (or degrade to the no-patch baseline), the premise that averaged RA maps are task-level editing instructions fails. Alternatively, compare the signed entries of $R$ for a single head against the actual change in forward attention after one genuine gradient step on that head; if positive RA entries do not correspond to increased attention under a negative learning rate, RA is not what gradient descent would do.","tokens_in":34662,"feed_emoji":"🧠","tokens_out":6626,"duration_ms":51250,"temperature":0.7,"pith_summary":"The paper claims that the softmax derivative computed during the backward pass of GPT attention is not just an engineering artifact but an implicit attention-like matrix, which they call Reversed Attention (RA). RA shows, for a chosen loss target, which token pairs a head would amplify or suppress if gradient descent updated its weights. The paper argues that this makes RA a cheap interpretability signal: RA norms rank attention heads by importance nearly as well as causal mediation, and injecting averaged RA maps into the forward pass ('attention patching') nudges a frozen model toward task behavior without any weight change. A sympathetic reading is that RA offers a single-backward-pass window into how GPTs would edit their own attention, with applications to localization and intervention.","feed_headline":"Attention's gradient is a map that can edit GPT predictions","feed_subtitle":"Averaged reversed-attention maps rank heads and nudge frozen GPT predictions toward a task, rivaling causal mediation at lower cost.","key_machinery":"The central object is the Reversed Attention (RA) matrix $R$, the derivative of the softmax attention scores with respect to their logits, assembled from per-token VJPs. Concretely, $R = A \\odot (\\tilde{E}^\\top - \\mathrm{diag}(A\\tilde{E}^\\top))^\\top \\sqrt{h/d}$ with $\\tilde{E} = \\Delta \\hat{W}_o^\\top V^\\top$, where $A$ is the forward attention, $\\Delta$ holds the output-projection VJPs, $\\hat{W}_o$ is the output projection, and $V$ holds the value vectors. Because $A$ is lower-triangular (causal masking), $R$ is lower-triangular too, and $R$ plays the role of an attention matrix in the backward pass: it multiplies forward keys $K$ to form query VJPs and transposed rows multiply forward queries $Q$ to form key VJPs. The paper's argument is that this matrix is what gradient descent would edit if it took a step: positive entries push the corresponding query and key together under a negative learning rate, negative entries push them apart.","core_discovery":"On the paper's own terms, the central discovery is that the derivative of the softmax operation inside each GPT attention head, $R = A \\odot (\\tilde{E}^\\top - \\mathrm{diag}(A\\tilde{E}^\\top))^\\top \\sqrt{h/d}$, is a triangular, attention-like matrix whose rows are the VJPs that gradient descent would use to update the query and key projections. Positive entries of $R$ mark query–key pairs whose attention score GD would increase (for a negative learning rate) and negative entries mark pairs it would decrease. The paper demonstrates two consequences: the per-head norm of $R$ ranks the importance of attention heads for a given prediction, competitive with causal mediation in perturbation tests; and the average of $R$ over 25 same-format examples, injected additively into the forward attention with a negative learning rate, improves task accuracy on several ICL and fact-relation tasks without touching weights. The authors present RA as what the backward pass is 'really doing' when a GPT is asked to move its prediction toward a target token.","pith_inferences":["One implicit extension is that RA could serve as a general-purpose attribution method for any differentiable attention-based model, not just decoder-only GPT; the derivation only relies on the structure of softmax attention, so encoder-decoder or bidirectional attention would produce analogous matrices.","RA patching's reliance on fixed-length templates is a constraint the paper acknowledges; a natural extension is to align RA maps across variable-length inputs (e.g., by position or by attention heads' token roles), which would let the method apply to open-ended generation.","A direct test the paper does not run is whether the signed pattern of $R$ predicts the actual change in forward attention after one real gradient step on a head; if the two signs systematically disagree, RA would be an interpretability heuristic rather than an exact editing plan.","Because RA is computed from a loss that selects a target token, it is essentially a gradient-based saliency map for the attention sublayer; combining RA norms with gradient×input-style scaling could yield even sharper head rankings than the plain norm."],"forward_implications":["RA norm per head provides an ordering of attention-head importance that performs competitively with causal mediation in perturbation tests, at the cost of one forward and one backward pass instead of one forward pass per head.","Averaging RA maps over a small set of same-length, same-format examples and adding them into the forward attention of a frozen model improves accuracy on several ICL tasks, sometimes matching few-shot prompting without showing any examples.","RA maps are sparse and target-specific: changing the editing target changes which token pair a head amplifies, suggesting the backward pass localizes task-relevant information per inference.","Because close-to-zero RA scores yield near-zero VJPs and gradients, focusing on high-RA-normal heads captures most of the gradient-descent update."],"supporting_citations":[{"why":"Defines the Transformer and multi-head attention whose backward pass the paper differentiates.","marker":"Vaswani et al., 2017"},{"why":"The GPT-2 family used for the qualitative and perturbation experiments.","marker":"Radford et al., 2019"},{"why":"The result that forward attention weights are not explanations, motivating RA as a backward-pass alternative.","marker":"Jain and Wallace, 2019"},{"why":"Introduces causal mediation analysis for attention heads, the main baseline RA is compared against.","marker":"Vig et al., 2020"},{"why":"Causal mediation for locating and editing factual associations in GPT, another baseline in the perturbation test.","marker":"Meng et al., 2022"},{"why":"Supplies the ICL task suite, the CM2 causal-mediation implementation, and the activation-patching context that attention patching extends.","marker":"Todd et al., 2023"},{"why":"Provides the natural-language fact-relation tasks and the linearity framing used to interpret relation decoding.","marker":"Hernandez et al., 2024"},{"why":"Shows that VJPs in the backward pass can be read as token-level embeddings, the direct precursor to interpreting RA as an editing signal.","marker":"Katz et al., 2024"}],"fun_headline_variants":["Attention's backward pass builds a map to tweak GPT predictions","Reversed attention: the hidden matrix behind GPT gradient edits","How GPT's attention backward pass can nudge predictions directly","New attention map from backprop edits GPT answers without weights","The gradient of attention is a reversible edit tool for GPT"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"RA maps averaged over 25 labeled examples of one fixed template (same length and format) are assumed to generalize to held-out examples of that task; the paper tests this only on simple, templated ICL and fact-relation tasks, without variance estimates or heterogeneous inputs.","fun_headline_variants_meta":{"raw":{"variants":["Attention's backward pass builds a map to tweak GPT predictions","Reversed attention: the hidden matrix behind GPT gradient edits","How GPT's attention backward pass can nudge predictions directly","New attention map from backprop edits GPT answers without weights","The gradient of attention is a reversible edit tool for GPT"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000863,"raw_usage":{"total_tokens":3730,"prompt_tokens":917,"completion_tokens":2813,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":2731}},"tokens_in":533,"tokens_out":2813,"duration_ms":17708,"temperature":1.0,"reasoning_tokens":2731,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:52:47.515013+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run attention patching on a test set whose prompt lengths or templates differ from the 25 training examples: if the accuracy gains vanish (or degrade to the no-patch baseline), the premise that averaged RA maps are task-level editing instructions fails. Alternatively, compare the signed entries of $R$ for a single head against the actual change in forward attention after one genuine gradient step on that head; if positive RA entries do not correspond to increased attention under a negative learning rate, RA is not what gradient descent would do.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The GPT-2 family used for the qualitative and perturbation experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The result that forward attention weights are not explanations, motivating RA as a backward-pass alternative."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces causal mediation analysis for attention heads, the main baseline RA is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Causal mediation for locating and editing factual associations in GPT, another baseline in the perturbation test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the natural-language fact-relation tasks and the linearity framing used to interpret relation decoding."}],"review_version":1}