Pith. sign in

REVIEW 2 major objections 6 minor 4 cited by

On Identifiability in Transformers

T0 review · 2 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Attention weights cannot be uniquely determined from a transformer head's output.

desk verdict A genuinely useful set of empirical diagnostics for Transformer interpretability, but the headline non-identifiability theorem is not proven for actual softmax attention weights. read the letter →

arxiv 1908.04211 v4 pith:MXKNR2CR submitted 2019-08-12 cs.CL cs.LG

Gino Brunner , Yang Liu , Damián Pascual , Oliver Richter , Massimiliano Ciaramita , Roger Wattenhofer This is my paper · ORCID
classification cs.CLcs.LG
keywords attentionidentifiabilityeffectivetokenhiddenattributiontransformerinterpretabilityself-attentioncontextualembeddingsgradient
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Attention weights in transformer self-attention cannot in general be recovered from the head's output: once the sequence length exceeds the head dimension, infinitely many different attention distributions produce exactly the same output. The paper proves this by showing that the left null space of the value-projection matrix $T = E W_V H$ is non-trivial, and extends the argument to probability distributions, so the non-uniqueness persists among valid softmax outputs. This undercuts the common practice of treating attention maps as direct explanations, and the paper proposes effective attention, the component that actually affects the output, as a corrective diagnostic. The paper also shows that input token identity survives through the layers, mostly encoded in the angle of embeddings, even while self-attention strongly mixes context into hidden embeddings. A gradient-attribution method quantifies this mixing and shows that context is mostly local, with long-range dependencies emerging mainly in later layers.

What carries the argument

The load-bearing object is the left null space $LN(T)$ of the matrix $T = E W_V H$, which maps attention rows to the head output; its dimension, bounded below by $d_s - d_v$ through rank-nullity, is exactly the family of perturbations $\tilde A$ that change attention weights without changing outputs. For probability constraints the argument passes to the augmented matrix $[T, \mathbf{1}]$, whose left null space contains the row-stochastic perturbations, and uses the strict positivity of softmax rows to allow small null-space steps. Effective attention is the orthogonal projection of raw attention onto the orthogonal complement of $LN(T)$. The empirical machinery for the remaining claims is a learned linear map with cosine nearest-neighbour lookup for token identity, and the L2 norm of the gradient of a hidden embedding with respect to each input token for measuring context mixing.

What would settle it

Take a trained transformer head with $d_s > d_v+1$, construct a nonzero $\tilde A$ in the left null space of $[T,\mathbf{1}]$ as the paper does, and test whether $A+\tilde A$ equals $\mathrm{softmax}(Q'K'^T/\sqrt{d_q})$ for some $Q',K'$ whose product has rank at most $d_q$; if no such realisation exists, the non-identifiability claim does not hold for actual softmax attention weights.

Watch

Extended reading notes

Core claim

The paper's central claim is that for a self-attention head with sequence length $d_s$ and value dimension $d_v$, whenever $d_s > d_v$ the attention matrix $A$ is not identifiable from the head's output. By rank-nullity, the left null space of $T = E W_V H$ has dimension at least $d_s - d_v$, so there exist infinitely many row-stochastic matrices $A + \tilde A$ with $\tilde A T = 0$ and $\tilde A \mathbf{1} = 0$ that yield exactly the same output. The construction works because softmax rows are strictly positive, so small perturbations in the null space of the augmented matrix $[T, \mathbf{1}]$ stay inside the probability simplex. Consequently, raw attention weights are not directly interpretable, and the paper defines effective attention $A_\perp = A - \mathrm{Proj}_{LN(T)} A$ as the part of attention that actually influences the output.

Load-bearing premise

The proof assumes that any row-stochastic perturbation in the null space of $[T,\mathbf{1}]$ is a valid attention matrix a real head can produce; in particular it never checks that $A+\tilde A$ can be written as $\mathrm{softmax}(Q'K'^T/\sqrt{d_q})$ with $Q'K'^T$ of rank at most $d_q$, which is required for actual transformer attention weights.

Editorial extensions

If this is right

  • When the sequence length exceeds the attention head dimension, a growing part of the raw attention matrix has no effect on the output, so attention-based explanations become less reliable as inputs get longer.
  • Effective attention can change interpretive conclusions, for example removing attention peaks on structural tokens like [SEP] and exposing meaningful long-range links such as coreference between distant pronouns.
  • Tokens remain largely identifiable from their contextual embeddings through all layers, with a linear projection and cosine similarity recovering about 93% of tokens in the last layer, so treating hidden embeddings as standing for their input words is a reasonable first approximation.
  • Context mixing is strong but orderly: the original token's contribution decreases monotonically with depth, the original token usually remains the largest contributor, and context is mostly local, with long-range contributions appearing predominantly in late layers.
  • The proposed gradient-based attribution method can quantify token mixing in any transformer architecture, making the analysis independent of a particular model or task.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the theorem's reach: the paper only proves the alternative attention matrices stay in the probability simplex, so checking whether they are actually realizable as softmax of a low-rank query-key product would show how far the non-identifiability claim extends to real transformers.
  • The finding that context is mostly local suggests that explicit locality biases, such as distance masks, may be redundant for language transformers because the model learns local mixing on its own; comparing attribution profiles with and without such masks would test this.
  • Effective attention could serve as a head-pruning criterion, since heads whose effective attention is near zero contribute little to the output; this would connect the paper's diagnostic to model compression.
  • The angle-versus-magnitude result implies that interventions that preserve embedding direction, such as certain fine-tuning or adversarial perturbations, may preserve token identity better than those that change magnitude; this is a testable design principle.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper analyzes identifiability in Transformer self-attention. The main theoretical result (Sections 3.2–3.3) claims that when the input sequence length ds exceeds the attention head dimension dv, the attention weights A of a head are not identifiable from the head output: infinitely many row-stochastic matrices A+à yield the same output because the rows of à lie in the left null space of T=E W_V H. The paper proposes "effective attention" as the projection of A onto the orthogonal complement of this null space and argues that it is a better basis for attention-based explanations. The remaining sections report empirical studies on BERT: token identity can be recovered from contextual embeddings via learned nearest-neighbor maps, identity information is largely encoded in embedding angle, and a gradient-based "Hidden Token Attribution" method shows that context mixing is strong but predominantly local. The paper concludes that raw attention distributions are not directly interpretable and that the proposed tools improve interpretability.

Significance. If the non-identifiability theorem held for actual softmax attention weights, it would provide a clean theoretical counterpart to the empirical "attention is not explanation" literature and would justify the effective-attention tool. The rank-nullity calculation is correct, and the effective-attention idea is a useful diagnostic that does not depend on the theorem. The token-identifiability and attribution experiments are carefully designed, use multiple datasets, and give concrete, interpretable results; they are valuable even if the theorem is weakened. The paper also connects its findings to prior work in a balanced way. The main weakness is that the central theorem, as stated and proved, does not establish non-identifiability for the softmax-constrained attention weights actually used in Transformers, and the threshold in the abstract is too strong.

major comments (2)
  1. [§3.3, Eq. (7)] The proof of non-identifiability in §3.3 verifies only the simplex conditions (7a)–(7c) for A+Ã. Attention matrices in a Transformer are not arbitrary row-stochastic matrices: by Eq. (1), A = softmax(QK^T/√dq) for some Q,K ∈ R^{ds×dq}, so the logit matrix has rank at most dq. The constructed perturbation à is an element of LN([T,1]) scaled to satisfy à ≥ −A, but the paper never shows that A+à is realizable as softmax of a rank-≤dq logit matrix. This is a load-bearing gap: the softmax-realizable attention matrices form a lower-dimensional curved submanifold of the simplex (of dimension at most 2 ds dq − dq², up to row-constant invariances), while the constructed alternative attention matrices form an affine space of dimension ds(ds−dv−1); for small ds−dv the intersection with the softmax image can be empty. The central claim "attention weights are not identifiable" is therefore unproven for the actual attention weights defined in the paper. The authors should either construct Q',K' explicitly for their perturbations, or supply a transversality/dimension argument under explicit conditions, or restrict the claim to arbitrary row-stochastic matrices.
  2. [Abstract and §3.2 vs. §3.3] The abstract and §3.2 state that ds>dv is enough for non-identifiability, but the proof in §3.3 requires dim LN([T,1]) > 0, i.e., ds−dv>1. When ds=dv+1, the left null space of T is one-dimensional and its generator x generically satisfies x^T 1 ≠ 0; since any row perturbation must be a scalar multiple of x and must also have zero row sum, all scalars are forced to zero. Thus for generic T there is no non-identifiability in the simplex-constrained problem. A concrete instance is ds=3, dv=2, rank(T)=2: the only row-stochastic A' with A'T=AT is A'=A. The theorem's stated threshold is therefore false; the correct statement (still subject to the softmax realizability issue in the previous comment) requires ds−dv>1, not merely ds>dv. This correction affects the abstract, the introduction, and the conclusion.
minor comments (6)
  1. [§4] The formal definition of token identifiability as the existence of a classifier c with c(e_i^l)=x_i is vacuous without a restriction on c: for any finite set of labeled embeddings one can define a function (e.g., a partition of R^d) that achieves perfect recovery. The operational definition used in the experiments (nearest neighbor after a learned linear/MLP map) is meaningful; the formal definition should be stated in terms of that restricted class.
  2. [Appendix C.1] The text contains a dangling cross-reference: "The attribution method proposed in Section??" should refer to Section 5.
  3. [§6] In the Related Work section, "a theoretical proof of the identifiability of attention weights" should read "non-identifiability".
  4. [§5, Eq. (11)] The sum in Eq. (11) runs from k=0, while tokens are indexed from 1 in Section 2; clarify whether the index 0 corresponds to the [CLS] token.
  5. [§1 and §4.2] Typographical errors: "indentifiability" in the introduction and "identifiablity" in Section 4.2.
  6. [Appendix D] Typo: "datsets" in the opening paragraph of Appendix D.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the identifiability theorem is a self-contained linear-algebra proof; effective attention is an explicit projection, not a fitted prediction.

full rationale

The paper's central claim (Sections 3.2-3.3) is a self-contained mathematical derivation. Given T = E W_V H, the output of a head is A T; the paper studies the left null space of [T, 1] and shows that any row-stochastic perturbation in that null space yields the same output. No parameter is fitted and no external conclusion is assumed: the cited works (Bellman & Åström, Vaswani et al., Jain & Wallace) provide only background and related work, not load-bearing premises. The 'effective attention' construction in Section 3.4 is explicitly defined as A minus its projection onto LN(T), with Eq. 8 showing AT = A_perp T by construction; it is an openly stated diagnostic projection rather than a hidden prediction derived from itself. The empirical observation that, e.g., the [SEP] peak lies in the null-space component is a property of BERT's learned T and attention weights, not an input to the definition. The possible objection that the constructed perturbations are not shown to be softmax-realizable as low-rank logits is a correctness gap in the theorem's scope, not a circularity. The paper's derivation chain is therefore self-contained and exhibits no circular step.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central proof relies on standard linear algebra (rank-nullity, rank submultiplicativity) and the fact that softmax outputs are strictly positive, which is standard. The load-bearing domain assumption is that alternative attention weights only need to satisfy simplex constraints, not the low-rank softmax realizability implicit in QK^T. The token identifiability experiments also assume that recovering the input embedding x_i implies recovering word identity, even though x_i includes positional and segment components.

assumptions (4)
  • standard math Rank-nullity theorem and rank inequalities rank(ABC) <= min(rank(A), rank(B), rank(C))
    Used in Sections 3.1-3.2 to bound rank(T) and derive the null space dimension.
  • standard math Softmax outputs are strictly positive elementwise
    Used in Section 3.3 to argue a small ball around zero in the null space satisfies condition (7c).
  • domain assumption Alternative attention matrices need only satisfy non-negativity and row-sum constraints; low-rank realizability as softmax(Q'K'^T) is not required
    Section 3.3 conditions (7a)-(7c) define validity without the rank-dq constraint on logits; this is load-bearing for the non-identifiability theorem.
  • domain assumption Gradient norm reflects causal contribution of input tokens to hidden embeddings
    Hidden Token Attribution in Section 5 and Eqs. (10)-(11) assumes the linear Taylor approximation of the network is adequate for quantifying information mixing; no ground truth validation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Identifiability in Transformers." pith.science (2026). https://pith.science/paper/MXKNR2CR

@misc{pith2026190804211,
  author       = {Pith},
  title        = {Pith review of: On Identifiability in Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MXKNR2CR}},
  note         = {Machine review of arXiv:1908.04211}
}
read the original abstract

In this paper we delve deep in the Transformer architecture by investigating two of its core components: self-attention and contextual embeddings. In particular, we study the identifiability of attention weights and token embeddings, and the aggregation of context into hidden tokens. We show that, for sequences longer than the attention head dimension, attention weights are not identifiable. We propose effective attention as a complementary tool for improving explanatory interpretations based on attention. Furthermore, we show that input tokens retain to a large degree their identity across the model. We also find evidence suggesting that identity information is mainly encoded in the angle of the embeddings and gradually decreases with depth. Finally, we demonstrate strong mixing of input information in the generation of contextual embeddings by means of a novel quantification method based on gradient attribution. Overall, we show that self-attention distributions are not directly interpretable and present tools to better understand and further investigate Transformer models.

Figures

Figures reproduced from arXiv: 1908.04211 by the authors.

Figure 1
Figure 1. (a) Each point represents the Pearson correlation coefficient of effective attention and raw [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. (a) Identifiability of contextual word embeddings at different layers. Here, [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. (a) Contribution of the input token to the embedding at the same position. The orange line [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (39 more)
Figure 4
Figure 4. Figure 4: (a) Relative contribution per layer of neighbours at different positions. (b) Total contribu [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Effective attention (a) vs. raw attention (b). (a) Each point represents the average effec [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 7
Figure 7. Figure 7: Train and test token identifiability rates for the linear perceptron and MLP. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]
Figure 8
Figure 8. Figure 8: Linear Perceptron trained to minimize L2 distance generalizing to all layers. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: MLP trained to minimize L2 distance generalizing to all layers. [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: MLP trained to minimize cosine distance generalizing to all layers. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Token identifiability across single layers. These results are for non fine-tuned BERT on [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]
Figure 12
Figure 12. Figure 12: Token identifiability across single layers, comparing non fine-tuned (dashed) BERT [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]
Figure 13
Figure 13. Figure 13: Token identifiability across single layers, comparing non fine-tuned BERT (dashed) [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Recovering neighbouring input tokens using [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Recovering neighbouring input tokens using [PITH_FULL_IMAGE:figures/full_fig_p022_15.png]
Figure 16
Figure 16. Figure 16: Recovering neighbouring input tokens using [PITH_FULL_IMAGE:figures/full_fig_p023_16.png]
Figure 17
Figure 17. Figure 17: Recovering neighbouring input tokens using [PITH_FULL_IMAGE:figures/full_fig_p023_17.png]
Figure 18
Figure 18. Figure 18: Normalized total contribution to the [CLS] token (a) centered around [CLS] at position [PITH_FULL_IMAGE:figures/full_fig_p025_18.png]
Figure 19
Figure 19. Figure 19: [CLS]: Aggregates context from all tokens but more strongly from those around the first [PITH_FULL_IMAGE:figures/full_fig_p025_19.png]
Figure 20
Figure 20. Figure 20: he: Aggregates most context from the main verb of the sentence, ”said”. [PITH_FULL_IMAGE:figures/full_fig_p026_20.png]
Figure 21
Figure 21. Figure 21: said: Aggregates context mainly from its neighborhood, the main verb of the subordinate [PITH_FULL_IMAGE:figures/full_fig_p026_21.png]
Figure 22
Figure 22. Figure 22: fit: In the first layers it aggregates most context from its neighborhood and towards the [PITH_FULL_IMAGE:figures/full_fig_p026_22.png]
Figure 23
Figure 23. Figure 23: long: It is part of a composed adjective (long-term) and aggregates most of its context [PITH_FULL_IMAGE:figures/full_fig_p027_23.png]
Figure 24
Figure 24. Figure 24: strategy: Aggregates context from the word growth, which is the first one of the noun [PITH_FULL_IMAGE:figures/full_fig_p027_24.png]
Figure 25
Figure 25. Figure 25: [SEP]: This token that has no semantic meaning aggregates context mostly from [CLS] [PITH_FULL_IMAGE:figures/full_fig_p027_25.png]
Figure 26
Figure 26. Figure 26: Layer 1: Most token types are equally mixed and have already less than 35% median [PITH_FULL_IMAGE:figures/full_fig_p028_26.png]
Figure 27
Figure 27. Figure 27: Layer 2: Similar to the previous layer with less contribution over all and [SEP] behaving [PITH_FULL_IMAGE:figures/full_fig_p028_27.png]
Figure 28
Figure 28. Figure 28: Layer 3: Similar to layer 2 with decreasing contribution overall. [PITH_FULL_IMAGE:figures/full_fig_p028_28.png]
Figure 29
Figure 29. Figure 29: Layer 4: The original input contribution to [CLS] and [SEP] falls significantly. The trend [PITH_FULL_IMAGE:figures/full_fig_p029_29.png]
Figure 30
Figure 30. Figure 30: Layer 5: The trend started in the previous layer continues, with a reduction of internal [PITH_FULL_IMAGE:figures/full_fig_p029_30.png]
Figure 31
Figure 31. Figure 31: Layer 6: Similar behavior as in the previous layer with minor evolution. [PITH_FULL_IMAGE:figures/full_fig_p029_31.png]
Figure 32
Figure 32. Figure 32: Layer 7: Minor changes with respect to Layer 6. [PITH_FULL_IMAGE:figures/full_fig_p030_32.png]
Figure 33
Figure 33. Figure 33: Layer 8: At this point there is clearly a different behavior between the tokens with most [PITH_FULL_IMAGE:figures/full_fig_p030_33.png]
Figure 34
Figure 34. Figure 34: Layer 9: SEP changes increasing the contribution, while the rest stays similar. [PITH_FULL_IMAGE:figures/full_fig_p030_34.png]
Figure 35
Figure 35. Figure 35: Layer 10: The contribution evolves with the same pattern as in previous layers. [PITH_FULL_IMAGE:figures/full_fig_p031_35.png]
Figure 36
Figure 36. Figure 36: Layer 11:The contribution evolves with the same pattern as in previous layers. [PITH_FULL_IMAGE:figures/full_fig_p031_36.png]
Figure 37
Figure 37. Figure 37: Layer 12: Finally, nouns, verbs, adjectives, adverbs, receive more contribution from their [PITH_FULL_IMAGE:figures/full_fig_p031_37.png]
Figure 38
Figure 38. Figure 38: Identifiability of contextual word embeddings at different layers on CoLA. [PITH_FULL_IMAGE:figures/full_fig_p032_38.png]
Figure 39
Figure 39. Figure 39: Identifiability of contextual word embeddings at different layers on a the first 500 sen [PITH_FULL_IMAGE:figures/full_fig_p033_39.png]
Figure 40
Figure 40. Figure 40: (a) Contribution of the input token to the embedding at the same position. (b) Percentage [PITH_FULL_IMAGE:figures/full_fig_p034_40.png]
Figure 41
Figure 41. Figure 41: (a) Relative contribution per layer of neighbours at different positions. (b) Total contri [PITH_FULL_IMAGE:figures/full_fig_p034_41.png]
Figure 42
Figure 42. Figure 42: (a) Contribution of the input token to the embedding at the same position. (b) Percentage [PITH_FULL_IMAGE:figures/full_fig_p035_42.png]
Figure 43
Figure 43. Figure 43: (a) Relative contribution per layer of neighbours at different positions. (b) Total contri [PITH_FULL_IMAGE:figures/full_fig_p035_43.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Provably Learning Multi-Head Attention with Queries

    cs.LG 2026-08 accept novelty 8.0 of 10

    Multi-head softmax attention can be provably recovered from black-box scalar value queries in O(H d^2) queries without orthogonal-subspace assumptions, using rational interpolation followed by head matching.

  2. PLEX: Perturbation-free Local Explanations for LLM-Based Text Classification

    cs.CL 2025-07 conditional novelty 6.0 of 10

    PLEX learns a mapping from BERT or RoBERTa token embeddings to word importance scores, reproducing LIME and SHAP style explanations without per-sentence perturbations.

  3. Towards Transparent AI: A Survey on Explainable Large Language Models

    cs.CL 2025-06 conditional novelty 3.0 of 10

    A review that groups LLM explainability methods by transformer architecture and discusses their evaluation and applications.

  4. Probing the Embedding Space of Transformers via Minimal Token Perturbations

    cs.LG 2025-06 conditional novelty 3.0 of 10

    Minimal single-token substitutions in BERT show that common tokens induce small embedding shifts, rare tokens induce larger shifts, and perturbation effects grow with layer depth, which the authors interpret as suppor...

Reference graph

Works this paper leans on

49 extracted references · 27 canonical work pages · cited by 4 Pith papers

  1. [1]

    A BERT B aseline for the N atural Q uestions

    Chris Alberti, Kenton Lee, and Michael Collins. A BERT B aseline for the N atural Q uestions. https://arxiv.org/abs/1901.08634, 2019

    arXiv 1901

  2. [2]

    Do Transformer Attention Heads Provide Transparency in Abstractive Summarization?

    Joris Baan, Maartje ter Hoeve, Marlies van der Wees, Anne Schuth, and Maarten de Rijke. Do transformer attention heads provide transparency in abstractive summarization? CoRR, abs/1907.00570, 2019

    work page Pith review arXiv 1907

  3. [3]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1409.0473

    arXiv 2015

  4. [4]

    Bellman and Karl Johan str \"o m

    R. Bellman and Karl Johan str \"o m. On structural identifiability. Mathematical Biosciences, 7: 0 329--339, 1970

    work page 1970

  5. [5]

    Brown, Stephen A

    Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19 0 (2): 0 263--311, 1993. URL https://www.aclweb.org/anthology/J93-2003

    work page 1993

  6. [6]

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of bert's attention. CoRR, abs/1906.04341, 2019

    arXiv 1906

  7. [7]

    Vi \' e gas, and Martin Wattenberg

    Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda B. Vi \' e gas, and Martin Wattenberg. Visualizing and measuring the geometry of BERT . CoRR, abs/1906.02715, 2019

    arXiv 1906

  8. [8]

    Universal transformers

    Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 , 2019

    work page 2019

Show all 49 references
  1. [9]

    BERT: pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan...

    2019

  2. [10]

    Dolan and Chris Brockett

    William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, IWP@IJCNLP 2005, Jeju Island, Korea, October 2005, 2005, 2005

    2005

  3. [11]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, AISTATS 2010, Chia Laguna Resort, Sardinia, Italy, May 13-15, 2010 ,...

    2010

  4. [12]

    Bridging nonlinearities and stochastic regularizers with gaussian error linear units

    Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415, 2016. URL http://arxiv.org/abs/1606.08415

    2016 arXiv

  5. [13]

    Sarthak Jain and Byron C. Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (L...

    2019

  6. [14]

    Ganesh Jawahar, Beno \^ t Sagot, and Djam \' e Seddah. What does BERT learn about the structure of language? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , pp.\...

    2019

  7. [15]

    Microsoft translator at wmt 2019: Towards large-scale document-level neural machine translation

    Marcin Junczys-Dowmunt. Microsoft translator at wmt 2019: Towards large-scale document-level neural machine translation. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pp.\ 225--233, 2019

    2019

  8. [16]

    Kingma and Jimmy Ba

    Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings , 2015. URL http://arxiv.org/abs/1412.6980

    2015 arXiv

  9. [17]

    Attention is (not) all you need for commonsense reasoning

    Tassilo Klein and Moin Nabi. Attention is (not) all you need for commonsense reasoning. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , pp.\ 4831--4836, 2019

    2019

  10. [18]

    Albert: A lite bert for self-supervised learning of language representations

    Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=H1eA7AEtvS

    2020

  11. [19]

    Open sesame: Getting inside bert's linguistic knowledge

    Yongjie Lin, Yi Chern Tan, and Robert Frank. Open sesame: Getting inside bert's linguistic knowledge. arXiv preprint arXiv:1906.01698, 2019

    1906 arXiv

  12. [20]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  13. [21]

    Extracting syntactic trees from transformer encoder self-attentions

    David Marecek and Rudolf Rosa. Extracting syntactic trees from transformer encoder self-attentions. In Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2018, Brussels, Belgium, November 1, 2018, pp.\ 347--349, 2018

    2018

  14. [22]

    Are sixteen heads really better than one? CoRR, abs/1905.10650, 2019

    Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? CoRR, abs/1905.10650, 2019. URL http://arxiv.org/abs/1905.10650

    1905 arXiv

  15. [23]

    Bruce Croft

    Harshith Padigela, Hamed Zamani, and W. Bruce Croft. Investigating the successes and failures of BERT for passage re-ranking. CoRR, abs/1905.01758, 2019

    1905 arXiv

  16. [24]

    Peters, Mark Neumann, Luke Zettlemoyer, and Wen - tau Yih

    Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen - tau Yih. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, ...

    2018

  17. [25]

    o rner, Hinrich Sch \

    Nina P \" o rner, Hinrich Sch \" u tze, and Benjamin Roth. Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Austr...

    2018

  18. [26]

    Learning to deceive with attention-based explanations

    Danish Pruthi, Mansi Gupta, Bhuwan Dhingra, Graham Neubig, and Zachary C Lipton. Learning to deceive with attention-based explanations. arXiv preprint arXiv:1909.07913, 2019

    1909 arXiv

  19. [27]

    Improving language understanding by generative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf, 2018

    2018

  20. [28]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1 0 (8), 2019

    2019

  21. [29]

    An analysis of encoder representations in transformer-based machine translation

    Alessandro Raganato and J \" o rg Tiedemann. An analysis of encoder representations in transformer-based machine translation. In Proceedings of the Workshop: Analyzing and Interpreting Neural Networks for NLP, BlackboxNLP@EMNLP 2018, Brussels, Belgium, November 1, 2018, pp.\ 2...

    2018

  22. [30]

    Sofia Serrano and Noah A. Smith. Is attention interpretable? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , pp.\ 2931--2951, 2019. URL https://www.aclweb.org/an...

    2019

  23. [31]

    Deep inside convolutional networks: Visualising image classification models and saliency maps

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Pro...

    2014

  24. [32]

    An analysis of attention mechanisms: The case of word sense disambiguation in neural machine translation

    Gongbo Tang, Rico Sennrich, and Joakim Nivre. An analysis of attention mechanisms: The case of word sense disambiguation in neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, October 31 - Nov...

    2018

  25. [33]

    BERT rediscovers the classical NLP pipeline

    Ian Tenney, Dipanjan Das, and Ellie Pavlick. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers , pp.\ 4593--4601, 2019

    2019

  26. [34]

    Manning, and Yoram Singer

    Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. Feature-rich part-of-speech tagging with a cyclic dependency network. In Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, HLT-NAACL 2003...

    2003

  27. [35]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-...

    2017

  28. [36]

    Visualizing attention in transformer-based language representation models

    Jesse Vig. Visualizing attention in transformer-based language representation models. CoRR, abs/1904.02679, 2019

    1904 arXiv

  29. [37]

    Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned

    Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Flore...

    2019

  30. [38]

    Attending to mathematical language with transformers

    Artit Wangperawong. Attending to mathematical language with transformers. CoRR, abs/1812.02825, 2018

    2018 arXiv

  31. [39]

    Neural network acceptability judgments

    Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018

    2018 arXiv

  32. [40]

    Attention is not not explanation

    Sarah Wiegreffe and Yuval Pinter. Attention is not not explanation. CoRR, abs/1908.04626, 2019. URL http://arxiv.org/abs/1908.04626

    1908 arXiv

  33. [41]

    A broad-coverage challenge corpus for sentence understanding through inference

    Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, ...

    2018

  34. [42]

    Wong, Fandong Meng, Lidia S

    Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. Modeling localness for self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, p...

    2018

  35. [43]

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. https://arxiv.org/abs/1906.08237, 2019 a

    1906 arXiv

  36. [44]

    Xlnet: Generalized autoregressive pretraining for language understanding

    Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in neural information processing systems, pp.\ 5754--5764, 2019 b

    2019

  37. [45]

    Adding interpretable attention to neural translation models improves word alignment

    Thomas Zenkel, Joern Wuebker, and John DeNero. Adding interpretable attention to neural translation models improves word alignment. CoRR, abs/1901.11359, 2019

    1901 arXiv

  38. [46]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  39. [47]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  40. [48]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  41. [49]

    妤 ' x P·uy n gʅ8Oj D q ^hژ

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.