{"id":"9cbc9d4d-6226-41a7-b0d8-05040d9e5739","arxiv_id":"1908.04211","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Self-attention weights in Transformers are non-identifiable for long sequences, and the paper offers effective attention and Hidden Token Attribution as diagnostic tools.","lead":"This paper proves that self-attention weights in Transformers are not uniquely determined by the model output when sequences are longer than the head dimension, and proposes 'effective attention' as a cleaner diagnostic. It also measures how much of a token's identity survives through BERT's layers and introduces a gradient-based method to quantify context mixing.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem in §3.3 is unproven for actual attention weights: constructed A+à satisfies simplex constraints but is not shown to be realizable as softmax of rank-≤dq logits.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern: Section 3.3 constructs alternative attention matrices within the probability simplex but never verifies that they are realizable as softmax of rank-≤dq logits. I agree that this is the central gap. The paper's theoretical contribution—the non-identifiability theorem—is what justifies the interpretability critique, and the proof's linear perturbation argument bypasses the defining softmax constraint. Without a realizability argument, the theorem applies only to a broader class of row-stochastic weight matrices, not to actual Transformer attention heads. This is not a purely cosmetic omission: the softmax image is a proper submanifold of the simplex when ds>dq, and the constructed output-equivalent affine subspace may or may not intersect it. Dimension counting suggests that for large ds and dq≈dv the intersection should be nonempty, so the gap is likely repairable by constructing perturbations in Q,K space or by adding a rank-constrained realizability lemma; for small ds near dv, however, the claim may be false. The separate issue that the simplex proof requires ds−dv>1 rather than ds>dv further weakens the stated threshold and should be corrected in any revision. Because the identified concern is a serious but plausibly fixable proof gap, and because the paper's empirical tools (effective attention, token identifiability, hidden token attribution) remain valuable independent contributions, the reader's conditional acceptance verdict is appropriate. No change to the verdict is needed, though the conditions should explicitly require the authors to either prove softmax realizability or restrict the theorem's scope.","tokens_in":24877,"tokens_out":17270,"duration_ms":182848,"concrete_test":"On a pretrained BERT-Base model, select a sentence with ds=128 and a head in layer 4. Compute Q, K, E, W_V, H, T=EW_VH, and A=softmax(QK^T/√dq). Compute a basis for LN([T,1]) and sample a nonzero à in this null space, scaled so that A+λà has strictly positive entries. Compute the row-mean-centered logits S' = log(A+λÃ) and check rank(S'). Repeat over many heads and layers. If rank(S')>dq for all nonzero λ, the constructed perturbation is not a valid transformer attention matrix. A stronger test: solve min_{Q',K'} ||softmax(Q'K'^T/√dq)−A|| subject to (softmax(Q'K'^T/√dq)−A)T=0 and A'≠A; if the minimum is zero for some head, the theorem's construction is repairable, and if not, the non-identifiability claim fails for that configuration.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract, §3.2–3.3) is that for ds>dv, attention weights A are not identifiable from the head output A T. The proof in §3.3 shows there exist infinitely many row-stochastic matrices A+à with ÃT=0, Ã1=0, and Ã≥−A. However, in a Transformer, attention weights are not arbitrary row-stochastic matrices: they must equal softmax(Q'K'^T/√dq) for some Q',K' ∈ R^{ds×dq}, so the logit matrix has rank at most dq. The paper never proves that any constructed à can be realized as softmax of a low-rank logit matrix; the linear construction operates in probability space and ignores the nonlinear, lower-dimensional softmax image. This gap matters: the set of softmax-realizable attention matrices has dimension at most 2·ds·dq − dq², while the affine space of output-equivalent simplex matrices has dimension ds·(ds−dv−1). For typical BERT (ds=128, dv=64, dq=64) these dimensions are about 12288 and 8064 respectively, which sum to more than the ambient simplex dimension 16256, so an intersection is plausible but not guaranteed; for smaller margins (e.g., ds=66, dv=64) the null-space dimension is small and the claimed non-identifiability may fail. The theorem as stated is therefore unsupported. A separate issue: the simplex argument in §3.3 requires ds−dv>1, not merely ds>dv, so the abstract's threshold is too strong.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper analyzes identifiability in Transformer self-attention. The main theoretical result (Sections 3.2–3.3) claims that when the input sequence length ds exceeds the attention head dimension dv, the attention weights A of a head are not identifiable from the head output: infinitely many row-stochastic matrices A+à yield the same output because the rows of à lie in the left null space of T=E W_V H. The paper proposes \"effective attention\" as the projection of A onto the orthogonal complement of this null space and argues that it is a better basis for attention-based explanations. The remaining sections report empirical studies on BERT: token identity can be recovered from contextual embeddings via learned nearest-neighbor maps, identity information is largely encoded in embedding angle, and a gradient-based \"Hidden Token Attribution\" method shows that context mixing is strong but predominantly local. The paper concludes that raw attention distributions are not directly interpretable and that the proposed tools improve interpretability.","tokens_in":25159,"tokens_out":20727,"duration_ms":225275,"significance":"If the non-identifiability theorem held for actual softmax attention weights, it would provide a clean theoretical counterpart to the empirical \"attention is not explanation\" literature and would justify the effective-attention tool. The rank-nullity calculation is correct, and the effective-attention idea is a useful diagnostic that does not depend on the theorem. The token-identifiability and attribution experiments are carefully designed, use multiple datasets, and give concrete, interpretable results; they are valuable even if the theorem is weakened. The paper also connects its findings to prior work in a balanced way. The main weakness is that the central theorem, as stated and proved, does not establish non-identifiability for the softmax-constrained attention weights actually used in Transformers, and the threshold in the abstract is too strong.","major_comments":[{"comment":"The proof of non-identifiability in §3.3 verifies only the simplex conditions (7a)–(7c) for A+Ã. Attention matrices in a Transformer are not arbitrary row-stochastic matrices: by Eq. (1), A = softmax(QK^T/√dq) for some Q,K ∈ R^{ds×dq}, so the logit matrix has rank at most dq. The constructed perturbation à is an element of LN([T,1]) scaled to satisfy à ≥ −A, but the paper never shows that A+à is realizable as softmax of a rank-≤dq logit matrix. This is a load-bearing gap: the softmax-realizable attention matrices form a lower-dimensional curved submanifold of the simplex (of dimension at most 2 ds dq − dq², up to row-constant invariances), while the constructed alternative attention matrices form an affine space of dimension ds(ds−dv−1); for small ds−dv the intersection with the softmax image can be empty. The central claim \"attention weights are not identifiable\" is therefore unproven for the actual attention weights defined in the paper. The authors should either construct Q',K' explicitly for their perturbations, or supply a transversality/dimension argument under explicit conditions, or restrict the claim to arbitrary row-stochastic matrices.","section":"§3.3, Eq. (7)"},{"comment":"The abstract and §3.2 state that ds>dv is enough for non-identifiability, but the proof in §3.3 requires dim LN([T,1]) > 0, i.e., ds−dv>1. When ds=dv+1, the left null space of T is one-dimensional and its generator x generically satisfies x^T 1 ≠ 0; since any row perturbation must be a scalar multiple of x and must also have zero row sum, all scalars are forced to zero. Thus for generic T there is no non-identifiability in the simplex-constrained problem. A concrete instance is ds=3, dv=2, rank(T)=2: the only row-stochastic A' with A'T=AT is A'=A. The theorem's stated threshold is therefore false; the correct statement (still subject to the softmax realizability issue in the previous comment) requires ds−dv>1, not merely ds>dv. This correction affects the abstract, the introduction, and the conclusion.","section":"Abstract and §3.2 vs. §3.3"}],"minor_comments":[{"comment":"The formal definition of token identifiability as the existence of a classifier c with c(e_i^l)=x_i is vacuous without a restriction on c: for any finite set of labeled embeddings one can define a function (e.g., a partition of R^d) that achieves perfect recovery. The operational definition used in the experiments (nearest neighbor after a learned linear/MLP map) is meaningful; the formal definition should be stated in terms of that restricted class.","section":"§4"},{"comment":"The text contains a dangling cross-reference: \"The attribution method proposed in Section??\" should refer to Section 5.","section":"Appendix C.1"},{"comment":"In the Related Work section, \"a theoretical proof of the identifiability of attention weights\" should read \"non-identifiability\".","section":"§6"},{"comment":"The sum in Eq. (11) runs from k=0, while tokens are indexed from 1 in Section 2; clarify whether the index 0 corresponds to the [CLS] token.","section":"§5, Eq. (11)"},{"comment":"Typographical errors: \"indentifiability\" in the introduction and \"identifiablity\" in Section 4.2.","section":"§1 and §4.2"},{"comment":"Typo: \"datsets\" in the opening paragraph of Appendix D.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be the ICLR 2020 version. If this is a journal submission, the editor should confirm whether substantial new material is expected beyond the conference version. The empirical sections are solid, but the headline theorem needs a genuine repair or a clear weakening; I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real contribution to the attention-interpretability debate, but the headline theorem is not proven for actual Transformer attention weights. The paper deserves a serious referee, and I would take the empirical diagnostics seriously, but the proof needs repair.\n\nWhat's new: the rank-nullity argument is a clean way to see why raw attention can contain large output-invariant components, and effective attention is a practical way to remove those components. The token-identifiability study and the Hidden Token Attribution analysis are original, carefully executed, and generalize to CoLA and MNLI. The paper is honest about its limitations; the authors explicitly note effective attention is not a probability distribution and that alternative formulations are open.\n\nThe soft spot is the main theorem. The authors show that for ds-dv>1 there are infinitely many row-stochastic matrices A+à with the same head output, because à lies in the left null space of [T,1]. That is correct. But A in a Transformer is not any row-stochastic matrix; it is softmax of logits with rank at most dq. The paper never shows that any constructed à can be realized as a softmax of low-rank logits. Without that, the theorem proves non-identifiability within the class of arbitrary simplex weights, not within the class of actual attention weights. The abstract also states ds>dv, while the simplex argument requires ds-dv>1; that is a minor but real inconsistency. None of this kills the paper's intuition, but the central claim as stated is unsupported.\n\nThe token-identifiability experiments have a smaller issue: they don't ablate positional embeddings. Since BERT's input is token plus segment plus position, a nearest-neighbour classifier could be recovering position rather than word identity. The paper discusses position but doesn't control for it, so the 'angle encodes identity' conclusion is a bit shakier than it looks.\n\nNet: the paper is worth engaging. Effective attention is likely to be a useful diagnostic, and the attribution method is a legitimate tracking tool. I would not cite the non-identifiability theorem as proven, but I would cite the effective-attention construction and the empirical phenomenon. For review: conditional accept, not desk reject. The proof gap is addressable, and the empirical parts are solid enough to stand on their own.","headline":"A genuinely useful set of empirical diagnostics for Transformer interpretability, but the headline non-identifiability theorem is not proven for actual softmax attention weights.","tokens_in":25715,"tokens_out":3916,"would_cite":true,"duration_ms":44510,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Attention weights cannot be uniquely determined from a transformer head's output.","keywords":["attention identifiability","effective attention","token identifiability","hidden token attribution","transformer interpretability","self-attention","contextual embeddings","gradient attribution"],"falsifier":"Take a trained transformer head with $d_s > d_v+1$, construct a nonzero $\\tilde A$ in the left null space of $[T,\\mathbf{1}]$ as the paper does, and test whether $A+\\tilde A$ equals $\\mathrm{softmax}(Q'K'^T/\\sqrt{d_q})$ for some $Q',K'$ whose product has rank at most $d_q$; if no such realisation exists, the non-identifiability claim does not hold for actual softmax attention weights.","tokens_in":24678,"feed_emoji":"🔍","tokens_out":8309,"duration_ms":77206,"temperature":0.7,"pith_summary":"Attention weights in transformer self-attention cannot in general be recovered from the head's output: once the sequence length exceeds the head dimension, infinitely many different attention distributions produce exactly the same output. The paper proves this by showing that the left null space of the value-projection matrix $T = E W_V H$ is non-trivial, and extends the argument to probability distributions, so the non-uniqueness persists among valid softmax outputs. This undercuts the common practice of treating attention maps as direct explanations, and the paper proposes effective attention, the component that actually affects the output, as a corrective diagnostic. The paper also shows that input token identity survives through the layers, mostly encoded in the angle of embeddings, even while self-attention strongly mixes context into hidden embeddings. A gradient-attribution method quantifies this mixing and shows that context is mostly local, with long-range dependencies emerging mainly in later layers.","feed_headline":"Attention weights aren't identifiable in long sequences","feed_subtitle":"When sequence length exceeds head dimension, infinitely many attention patterns yield identical outputs.","key_machinery":"The load-bearing object is the left null space $LN(T)$ of the matrix $T = E W_V H$, which maps attention rows to the head output; its dimension, bounded below by $d_s - d_v$ through rank-nullity, is exactly the family of perturbations $\\tilde A$ that change attention weights without changing outputs. For probability constraints the argument passes to the augmented matrix $[T, \\mathbf{1}]$, whose left null space contains the row-stochastic perturbations, and uses the strict positivity of softmax rows to allow small null-space steps. Effective attention is the orthogonal projection of raw attention onto the orthogonal complement of $LN(T)$. The empirical machinery for the remaining claims is a learned linear map with cosine nearest-neighbour lookup for token identity, and the L2 norm of the gradient of a hidden embedding with respect to each input token for measuring context mixing.","core_discovery":"The paper's central claim is that for a self-attention head with sequence length $d_s$ and value dimension $d_v$, whenever $d_s > d_v$ the attention matrix $A$ is not identifiable from the head's output. By rank-nullity, the left null space of $T = E W_V H$ has dimension at least $d_s - d_v$, so there exist infinitely many row-stochastic matrices $A + \\tilde A$ with $\\tilde A T = 0$ and $\\tilde A \\mathbf{1} = 0$ that yield exactly the same output. The construction works because softmax rows are strictly positive, so small perturbations in the null space of the augmented matrix $[T, \\mathbf{1}]$ stay inside the probability simplex. Consequently, raw attention weights are not directly interpretable, and the paper defines effective attention $A_\\perp = A - \\mathrm{Proj}_{LN(T)} A$ as the part of attention that actually influences the output.","pith_inferences":["A direct test of the theorem's reach: the paper only proves the alternative attention matrices stay in the probability simplex, so checking whether they are actually realizable as softmax of a low-rank query-key product would show how far the non-identifiability claim extends to real transformers.","The finding that context is mostly local suggests that explicit locality biases, such as distance masks, may be redundant for language transformers because the model learns local mixing on its own; comparing attribution profiles with and without such masks would test this.","Effective attention could serve as a head-pruning criterion, since heads whose effective attention is near zero contribute little to the output; this would connect the paper's diagnostic to model compression.","The angle-versus-magnitude result implies that interventions that preserve embedding direction, such as certain fine-tuning or adversarial perturbations, may preserve token identity better than those that change magnitude; this is a testable design principle."],"forward_implications":["When the sequence length exceeds the attention head dimension, a growing part of the raw attention matrix has no effect on the output, so attention-based explanations become less reliable as inputs get longer.","Effective attention can change interpretive conclusions, for example removing attention peaks on structural tokens like [SEP] and exposing meaningful long-range links such as coreference between distant pronouns.","Tokens remain largely identifiable from their contextual embeddings through all layers, with a linear projection and cosine similarity recovering about 93% of tokens in the last layer, so treating hidden embeddings as standing for their input words is a reasonable first approximation.","Context mixing is strong but orderly: the original token's contribution decreases monotonically with depth, the original token usually remains the largest contributor, and context is mostly local, with long-range contributions appearing predominantly in late layers.","The proposed gradient-based attribution method can quantify token mixing in any transformer architecture, making the analysis independent of a particular model or task."],"supporting_citations":[{"why":"Defines the multi-head self-attention mechanism and the softmax attention equation that the non-identifiability analysis starts from.","marker":"Vaswani et al. (2017)"},{"why":"Supplies the structural-identifiability notion used to frame attention identifiability as uniqueness from output.","marker":"Bellman & Åström (1970)"},{"why":"Shows empirically that attention weights are not unique for LSTM encoder-decoders, motivating the theoretical proof for transformers.","marker":"Jain & Wallace (2019)"},{"why":"Provides the counterpoint that alternative attention weights perform worse on a diagnostic task, framing the interpretability debate the paper addresses.","marker":"Wiegreffe & Pinter (2019)"},{"why":"Provides the BERT attention analysis and data used to demonstrate effective attention and to compare raw vs effective attention patterns.","marker":"Clark et al. (2019)"},{"why":"Introduces BERT, the pretrained model used for all token identifiability and mixing experiments.","marker":"Devlin et al. (2019)"},{"why":"Provides the gradient-attribution formulation that Hidden Token Attribution adapts to hidden embeddings.","marker":"Simonyan et al. (2014)"},{"why":"Supplies the MRPC development set on which the main token identifiability and attribution experiments are run.","marker":"Dolan & Brockett (2005)"}],"fun_headline_variants":["Attention weights unidentifiable when sequence exceeds head dimension","Long sequences yield infinitely many attention patterns with same output","Sequence length > head dimension makes attention weights ambiguous","For long sequences, attention weights are not directly interpretable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof assumes that any row-stochastic perturbation in the null space of $[T,\\mathbf{1}]$ is a valid attention matrix a real head can produce; in particular it never checks that $A+\\tilde A$ can be written as $\\mathrm{softmax}(Q'K'^T/\\sqrt{d_q})$ with $Q'K'^T$ of rank at most $d_q$, which is required for actual transformer attention weights.","fun_headline_variants_meta":{"raw":{"variants":["Attention weights unidentifiable when sequence exceeds head dimension","Long sequences yield infinitely many attention patterns with same output","Sequence length > head dimension makes attention weights ambiguous","For long sequences, attention weights are not directly interpretable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001049,"raw_usage":{"total_tokens":4374,"prompt_tokens":882,"completion_tokens":3492,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":3429}},"tokens_in":498,"tokens_out":3492,"duration_ms":25373,"temperature":1.0,"reasoning_tokens":3429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:48:24.342360+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a trained transformer head with $d_s > d_v+1$, construct a nonzero $\\tilde A$ in the left null space of $[T,\\mathbf{1}]$ as the paper does, and test whether $A+\\tilde A$ equals $\\mathrm{softmax}(Q'K'^T/\\sqrt{d_q})$ for some $Q',K'$ whose product has rank at most $d_q$; if no such realisation exists, the non-identifiability claim does not hold for actual softmax attention weights.","supporting_citations":[{"cited_title":"Gomez, Lukasz Kaiser, and Illia Polosukhin","cited_arxiv_id":null,"evidence_quote":"Defines the multi-head self-attention mechanism and the softmax attention equation that the non-identifiability analysis starts from."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows empirically that attention weights are not unique for LSTM encoder-decoders, motivating the theoretical proof for transformers."},{"cited_title":"BERT: pre-training of deep bidirectional transformers for language understanding","cited_arxiv_id":null,"evidence_quote":"Introduces BERT, the pretrained model used for all token identifiability and mixing experiments."},{"cited_title":"Deep inside convolutional networks: Visualising image classification models and saliency maps","cited_arxiv_id":null,"evidence_quote":"Provides the gradient-attribution formulation that Hidden Token Attribution adapts to hidden embeddings."}],"review_version":1}