Pith. sign in

REVIEW 5 major objections 5 minor 52 references

Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that prior LRP explanations for Transformers omit positional encoding, and that PE-aware LRP rules restore conservation and improve attribution in vision and NLP.

desk verdict PE-aware LRP rules are a real gap worth filling, but the conservation theory is mostly bookkeeping and the proof of Lemma 3.2 is circular; the empirical gains are real but overstated. read the letter →

arxiv 2506.02138 v1 pith:INUQJ2TO submitted 2025-06-02 cs.LG

Yarden Bakish , Itamar Zimerman , Hila Chefer , Lior Wolf This is my paper
classification cs.LG MSC 68T07
keywords Layer-wiseRelevancePropagationTransformerexplainabilitypositionalencodingPA-LRPRoPEconservationpropertyattentionattributionvisionsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that every existing LRP-based explanation method for Transformers has been ignoring a real contributor to predictions: the positional encoding. It argues that this omission breaks LRP's conservation property once the input is viewed as position-token pairs, and it throws away a distinct, structurally meaningful kind of relevance. The proposed method, PA-LRP, adds specialized LRP rules for learnable, sinusoidal, Rotary, and ALiBi positional encodings, with per-layer sinks that absorb positional relevance. If the paper is right, Transformer explanations become more faithful and more complete at no extra forward-pass cost, and the positional relevance maps alone are already competitive with the previous state of the art.

What carries the argument

The central object is the reformulated input space of position-token pairs, with each layer given its own positional sink so that positional relevance is not aggregated away. The rules that carry the argument are: for input-level PE, the LRP-$\epsilon$ addition rule $R(P_i') = P_i' R(z_i)/(P_i' + E_i + \epsilon)$; for RoPE, flattening the rotation matrix and setting $R(R_{i,k}) = \tfrac12 R(\tilde Q_i) + \tfrac12 R(\tilde K_i)$; and a multi-sink aggregation that sums only positive semantic and positional contributions across layers. Sinusoidal PE is reparameterized as a learnable-PE-style matrix so the same rule applies, and ALiBi gets an analogous distance-based rule.

What would settle it

On a synthetic task where the label is determined only by token position, compare AttnLRP's attribution map with the true position rule: if AttnLRP assigns zero or near-zero relevance to the decisive position while the model is accurate, the paper's claim that positional relevance is missing is directly confirmed, whereas if AttnLRP already highlights the position through the summed input $E+P$, the conservation-violation argument is exposed as bookkeeping rather than a computational gap.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that positional encodings are not a neutral preprocessing detail for Layer-wise Relevance Propagation; they carry a substantial, unique share of the relevance that standard LRP-based methods discard. By reformulating the input space as $\{(E_i, (P_{i,1}, \dots, P_{i,K}))\}$ rather than $\{E_i\}$ and by defining LRP rules that propagate relevance through the PE computation itself, the paper obtains explanations that satisfy conservation and, in its experiments, outperform the AttnLRP baseline on perturbation and segmentation tests for both language models and vision Transformers.

Load-bearing premise

The load-bearing premise is that the input space for explanation should be split into separate semantic embeddings and positional embeddings, so that relevance assigned to $E+P$ must be decomposed into two parts; if one treats $E+P$ as a single tensor, standard LRP already conserves relevance and the claimed violation disappears.

Editorial extensions

If this is right

  • PA-LRP can be layered on top of any LRP method that propagates through attention, not only the AttnLRP baseline, so the PE-aware rules are a modular addition.
  • Transformer explanations now include a separate positional channel that captures whole-object, structural, and ordering information; combining it with semantic relevance produces less fragmented maps.
  • Because the method uses only an equivalent forward and backward pass and no model modification, it applies to large zero-shot models such as LLaMA 3 at the cost of a single backward pass.
  • The PE-only ablation shows that positional relevance alone is a competitive explanation signal, meaning previous methods were discarding information that can by itself rival state-of-the-art attributions.
  • Conservation is restored for both input-level and attention-level PE, giving PA-LRP the same theoretical grounding as standard LRP.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The conservation violation in Lemma 3.1 is a bookkeeping result: standard LRP conserves relevance for the summed tensor $E+P$, and the paper's claim requires accepting that the input space should be split into separate semantic and positional components; if a reader rejects that split, the method's theoretical motivation shifts to its empirical gains.
  • The same position-token separation could be ported to other propagation-based attribution methods, where positional terms are currently folded into the input; a testable extension would be whether those methods also gain from explicit PE sinks.
  • PE-only maps might serve as a cheap unsupervised segmentation or object-localization signal in vision Transformers, since the paper's qualitative results show the positional channel highlights whole objects rather than sparse discriminative patches.
  • One could test the positional-concepts hypothesis directly by comparing PE-only relevance on images of objects in typical versus atypical contexts, since the paper predicts that typical-context objects receive more positional relevance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes PA-LRP, a Layer-wise Relevance Propagation (LRP) extension for Transformer explainability that attributes relevance to positional encodings (PE). The authors reformulate the explainability input space as a set of position-token pairs (Eq. 4), introduce LRP rules for input-level PEs (learnable and sinusoidal) and attention-level PEs (RoPE and ALiBi), and claim that existing LRP methods violate the conservation property because they ignore PE. They provide three lemmas intended to prove conservation violation, conservation satisfaction, and low faithfulness of existing methods, and they evaluate PA-LRP on DeiT vision models and on LLaMA-based NLP models using perturbation and segmentation tests, reporting improvements over the AttnLRP baseline. The paper also releases code and includes ablations isolating a 'PE Only' attribution map.

Significance. If the theoretical claims and empirical results held, PA-LRP would be a useful contribution to Transformer explainability: it identifies a genuinely overlooked component (positional encoding) and provides an open-source implementation with experiments across vision and NLP models. The 'PE Only' ablation is a valuable probe of how much positional signal LRP-based methods currently discard, and the multi-sink design is an interesting engineering choice. However, the paper's central theoretical grounding is not established: the conservation lemmas are largely definitional or circular, and the RoPE attribution rule contains a dimensional mismatch. The empirical gains are also small and inconsistent, with no error bars or significance tests, so the headline claim of 'significantly outperforming' the state of the art is not quantitatively supported. The empirical contribution may survive a revision, but the theoretical claims need substantial reframing or removal.

major comments (5)
  1. [Section 3.1 / Appendix E, Lemma 3.1] The claimed conservation violation is a consequence of the chosen bookkeeping rather than a property of the network's computation. The model consumes the summed input Z = P + E; for that tensor, standard LRP conservation holds. Attributing relevance only to E in the final explanation omits P by construction of the output space, not because the Transformer's computation violates conservation. Lemma 3.1 should be reframed as a design choice about which input components to explain, not as a violation of conservation.
  2. [Section 3.4 / Appendix E, Lemma 3.2] The proof of Lemma 3.2 is circular. Equation (22) asserts the recursion R(l) = R(l-1) + R_P(l), which is exactly the equality R(M) = R_E + sum_l R_P(l) that the lemma claims to prove. Under the paper's own conservation definition (output relevance equals input relevance per layer), diverting R_P(l) to a separate sink would imply R(l) = R(l-1) - R_P(l) if the sink is external, or R(l) = R(l-1) if the sink is included in the conserved quantity. The recursion must be derived from the propagation rules, not assumed.
  3. [Section 3.3, Eq. (10) and Eqs. (7)-(9)] There is a dimensional mismatch in the RoPE attribution rule. Equation (10) defines R(R_i,k) as the average of two D-dimensional relevance vectors R(\tilde Q_i) and R(\tilde K_i), but R_i,k is a D×D rotation matrix per Eq. (7), later flattened to a D^2-dimensional vector in Eqs. (8)-(9). The rule does not specify how a D^2-dimensional positional attribution is constructed from two D-dimensional vectors; as written, the RoPE attribution step is undefined.
  4. [Table 4 and Section 4.1, Tables 1 and 3] The claim that the method 'significantly outperforms' the baseline is not supported by the reported numbers. In Table 4, PA-LRP is worse than AttnLRP in three of six positive-perturbation settings (Base Target, Tiny Predicted, Tiny Target), and in Table 1 the Tiny-LLaMA generation AU-MSE of the full method (4.915) is worse than the PE-Only ablation (3.918). No error bars, confidence intervals, or significance tests are reported anywhere in the paper, so the aggregate 'significant' claim is not established.
  5. [Appendix E, Lemma 3.3] Lemma 3.3 is an illustrative existence proof for a deliberately degenerate model in which all semantic weights are zero and the model uses only positional information. It does not quantify faithfulness, and it does not show that existing LRP methods have low faithfulness in general. The lemma should be labeled as a motivating example rather than a general faithfulness theorem, or it should be replaced with a more rigorous statement.
minor comments (5)
  1. [Table 1] The row label 'Tiny-LLaMa-2 7B' is inconsistent with the model name TinyLLaMA used in the text and with the paper's own description in Section 4.1; this is likely a copy-paste error and should be corrected.
  2. [Section 3.3, Eq. (10)] The notation R(\tilde Q_i) and R(\tilde K_i) is used before \tilde Q_i and \tilde K_i are defined; the definitions in Eq. (21) appear only in the appendix proof. Please define them in Section 3.3.
  3. [Appendix B, Eq. (16)] The LRP rule for subtraction is written with the epsilon term in the denominator as i + (-j) + epsilon, which is confusing for an operation i - j; the sign convention should be stated explicitly.
  4. [Figure 8] The caption says the figure visualizes 'the violation of the conservation rule,' but the figure plots the ratio of positional to total relevance. The measured relevance shares do not by themselves demonstrate a conservation violation; the connection should be explained.
  5. [Abstract and Section 1] The abbreviation 'PA-LRP' is used in the abstract but introduced only in Section 1; please define it at first use.

Circularity Check

2 steps flagged · score 6.0 of 10

PA-LRP's central conservation claims are true by construction: Lemma 3.1 redefines the input space so that PE becomes a separate conserved feature, and Lemma 3.2's proof assumes the recursion that encodes the conclusion; the empirical benchmarks are independent and remain evidence.

  1. self definitional [Section 3.1, Eq. (4); Appendix E, Proof of Lemma 3.1]
    "In contrast, we reformulate the input space as token-position pairs, with positional features defined separately for each layer. ... Let Z be our input representation to the first transformer layer, such that Z = P + E, where P and E are the token and positional embeddings, respectively. ... When ignoring RP, we get the final relevancy attribution map Rinput, such that: sum R(l) = sum(RE + RP) ≠ RE = Rinput directly violating the conservation property rule."

    The alleged conservation violation is a consequence of the paper's redefined bookkeeping, not of the network's computation. Standard LRP propagates through the actual input tensor Z = P + E, and conservation holds for that tensor; no PE relevance is 'omitted' because PE is part of the embedding vector in the standard input space S = {Ei}. Only after Eq. (4) splits E and P into separate coordinates does failing to attribute P become a conservation error. Thus Lemma 3.1's conclusion is built into the redefinition of the input space rather than derived from the propagation rules.

  2. self definitional [Appendix E, Proof of Lemma 3.2, Eqs. (19)-(22)]
    "Recall that for our PA-LRP formulation, we achieve the final explanation map by summing together the semantic attribution RE ... and the positional relevancy R(l) P ... giving us the final relevancy map RE + P l R(l) P . We aim to prove: R(M) = RE + X l R(l) P ... The absorbing sink mechanism results in the following: R(0) = RE, ∀l ∈ [M] : R(l) = R(l−1) + R(l) P (22) Following this recursion we would get the exact same result as Eq. 20."

    Equation (22) states, as a recursion, the exact equality that Lemma 3.2 is supposed to prove. Under the paper's own conservation definition, R(output) = R(input) per layer, diverting R_P(l) to an absorbing sink would require R(l) = R(l-1) − R_P(l), with R_P(l) preserved in the sink, not added to the running total. Instead the proof adds the diverted relevance into the running sum and then asserts that the recursion reproduces the target equation. The conclusion is therefore assumed in the recursion rather than established by conservation.

full rationale

The empirical contribution is self-contained: the perturbation and segmentation comparisons against AttnLRP are external benchmarks, no fitted parameter is relabeled as a prediction, and the self-citations to Chefer et al. and Ali et al. are ordinary prior work, not load-bearing support for the central claim. However, the paper's headline theoretical claim that existing LRP methods violate conservation and that PA-LRP restores it is not independently established. Lemma 3.1 follows by definition from the new input-space bookkeeping in Eq. (4), and Lemma 3.2's proof assumes the recursion that encodes the result. These are internal reductions of the theory to its own definitions, which is why the circularity score is elevated despite the valid empirical experiments. The dimensional mismatch in Eq. (10) is a correctness concern rather than a circularity and does not affect the score.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or architectural entities. It relies on a redefinition of the input space, a uniform-relevance assumption for matrix products, and an additive combination of semantic and positional relevance. The 1/2 split in Eq. (10) and the equal weighting in Eq. (11) are free choices that affect the results but are not fitted to data.

free parameters (2)
  • Q/K relevance split weight = 0.5
    In Eq. (10), the relevance of the rotation matrix R(Ri,k) is set as the average of R(Q~) and R(K~). This equal weighting is chosen by hand, with no derivation from the model's computation.
  • semantic and positional balance weight = 1
    Eq. (11) sums semantic relevance and positional relevance with equal weight. No theoretical justification is given for this balance, which directly affects the final heatmap.
assumptions (5)
  • standard math Conservation of relevance is a defining requirement for LRP explanations.
    Adopted from LRP literature (Bach et al., Montavon et al.); used in Lemma 3.1 and 3.2.
  • ad hoc to paper The input space for Transformer explainability should be the set of position-token pairs (Eq. 4), so PE is part of the input to be explained.
    This redefinition is introduced in Section 3.1 and is the foundation of the claimed conservation violation.
  • domain assumption The standard LRP baseline (AttnLRP) is exactly conservative for the non-positional part of the network.
    Lemma 3.2 assumes R(M)=R(l)=RE for the baseline, i.e., the baseline conserves relevance when PE is excluded.
  • domain assumption Relevance from semantic and positional components can be combined additively to form the final explanation.
    Used in Eq. (11) and in Lemma 3.2's definition of the PA-LRP map.
  • standard math Uniform relevance propagation for matrix multiplication distributes relevance evenly between operands.
    Invoked in the proof of Lemma 3.2 and in the RoPE rule Eq. (10).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability." pith.science (2026). https://pith.science/paper/INUQJ2TO

@misc{pith2026250602138,
  author       = {Pith},
  title        = {Pith review of: Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/INUQJ2TO}},
  note         = {Machine review of arXiv:2506.02138}
}
read the original abstract

The development of effective explainability tools for Transformers is a crucial pursuit in deep learning research. One of the most promising approaches in this domain is Layer-wise Relevance Propagation (LRP), which propagates relevance scores backward through the network to the input space by redistributing activation values based on predefined rules. However, existing LRP-based methods for Transformer explainability entirely overlook a critical component of the Transformer architecture: its positional encoding (PE), resulting in violation of the conservation property, and the loss of an important and unique type of relevance, which is also associated with structural and positional features. To address this limitation, we reformulate the input space for Transformer explainability as a set of position-token pairs. This allows us to propose specialized theoretically-grounded LRP rules designed to propagate attributions across various positional encoding methods, including Rotary, Learnable, and Absolute PE. Extensive experiments with both fine-tuned classifiers and zero-shot foundation models, such as LLaMA 3, demonstrate that our method significantly outperforms the state-of-the-art in both vision and NLP explainability tasks. Our code is publicly available.

Figures

Figures reproduced from arXiv: 2506.02138 by the authors.

Figure 1
Figure 1. (a) Explainability heatmaps by the state￾of-the-art Attention-LRP method (AttnLRP) [2]. (b) The LRP heatmap obtained directly from our PE￾aware LRP rules. The relevancy captured by the PE is less fragmented and captures more of the object. Recently, several refinements have been pro￾posed to improve the stability and faithful￾ness of LRP rules for Transformers, leading to more robust and reliable interpretability te… view at source ↗
Figure 2
Figure 2. Visualization of our method for propagating PE-associated relevance. Purple arrows indicate the for￾ward path, while blue arrows represent the LRP propaga￾tion rules. Dashed arrows denote custom PE-aware rules defined in our method. In this section, we describe our PE-aware LRP rules. We first revise the input space used in the Transformer explainability problem in Sec￾tion 3.1. Then, building upon this formulation,… view at source ↗
Figure 3
Figure 3. Results of different explanation methods for DeiT. (a) Input image. (b) PA-LRP (ours), which includes PE attribution. (c) PE only LRP, (d) At￾tnLRP [2], which does not attribute relevancy to PE. For vision models, we present both quantitative and qualitative analysis. Qualitative Analysis. For qualitative analysis, we visualize the explainability maps obtained from our method, the AttnLRP [2] baseline, and the ablat… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Additional Qualitative Results In Vision. Results of different explanation methods for DeiT. (a) The input image. (b) PA-LRP (ours), which include PE relevancy attribution. (c) PE only LRP, (d) AttnLRP [2], which does not attribute relevancy to PE. 15 [PITH_FULL_IMAGE…
Figure 5
Figure 5. Figure 5: Additional Qualitative Results In Vision. Results of different explanation methods for DeiT. (a) The input image. (b) PA-LRP (ours), which include PE relevancy attribution. (c) PE only LRP, (d) AttnLRP [2], which does not attribute relevancy to PE. 16 [PITH_FULL_IMAGE…
Figure 6
Figure 6. Figure 6: Additional Qualitative Results In Vision. Results of different explanation methods for DeiT. (a) The input image. (b) PA-LRP (ours), which include PE relevancy attribution. (c) PE only LRP, (d) AttnLRP [2], which does not attribute relevancy to PE. 17 [PITH_FULL_IMAGE…
Figure 7
Figure 7. Figure 7: Qualitative Results in NLP. Both groups (a) and (b) present results from different explanation methods for the same example obtained from the IMDB benchmark. In each group, the first row represents the AttnLRP baseline, followed by the PE-only variant in the middle, an…
Figure 8
Figure 8. Figure 8: We assess both the positional relevance and the non-positional relevance for DeiT models at different capacities, visualizing the violation of conservation rule, with high non-negligible ratio between the entire relevance in the models ’ours’ and positional-associated …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 31 canonical work pages

  1. [1]

    Quantifying attention flow in transformers

    Samira Abnar and Willem Zuidema. Quantifying attention flow in transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics . Association for Computational Linguistics, 2020

    work page 2020

  2. [2]

    Attnlrp: attention-aware layer-wise relevance propagation for transformers

    Reduan Achtibat, Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Aakriti Jain, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. Attnlrp: attention-aware layer-wise relevance propagation for transformers. In Proceedings of the 41st International Conference on Machine Learning, pages 135–168, 2024

    work page 2024

  3. [3]

    Xai for transformers: Better explanations through conservative propagation

    Ameen Ali, Thomas Schnake, Oliver Eberle, Grégoire Montavon, Klaus-Robert Müller, and Lior Wolf. Xai for transformers: Better explanations through conservative propagation. In International conference on machine learning , pages 435–451. PMLR, 2022

    work page 2022

  4. [4]

    Video and text matching with conditioned embeddings

    Ameen Ali, Idan Schwartz, Tamir Hazan, and Lior Wolf. Video and text matching with conditioned embeddings. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 1565–1574, 2022

    work page 2022

  5. [5]

    The hidden attention of mamba models

    Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models. arXiv preprint arXiv:2403.01590, 2024

    arXiv 2024

  6. [6]

    Explaining recurrent neural network predictions in sentiment analysis

    Leila Arras, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek. Explaining recurrent neural network predictions in sentiment analysis. In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis , pages 159–168, 2017

    work page 2017

  7. [7]

    Explaining and interpreting lstms

    Leila Arras, José Arjona-Medina, Michael Widrich, Grégoire Montavon, Michael Gillhofer, Klaus-Robert Müller, Sepp Hochreiter, and Wojciech Samek. Explaining and interpreting lstms. Explainable ai: Interpreting, explaining and visualizing deep learning , pages 211–238, 2019

    work page 2019

  8. [8]

    Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai

    Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López, Daniel Molina, Richard Ben- jamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information fusion, 58:82–115, 2020

    work page 2020

Show all 52 references
  1. [9]

    On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation

    Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PloS one, 10(7):e0130140, 2015

    2015

  2. [10]

    How to explain individual classification decisions

    D Baehrens, T Schroeter, S Harmeling, M Kawanabe, K Hansen, and K-R Müller. How to explain individual classification decisions. Journal of Machine Learning Research, 2010

    2010

  3. [11]

    Qwen technical report

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023

    2023 arXiv

  4. [12]

    The shattered gradients problem: If resnets are the answer, then what is the question? In International conference on machine learning , pages 342–350

    David Balduzzi, Marcus Frean, Lennox Leary, JP Lewis, Kurt Wan-Duo Ma, and Brian McWilliams. The shattered gradients problem: If resnets are the answer, then what is the question? In International conference on machine learning , pages 342–350. PMLR, 2017

    2017

  5. [13]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In Intern...

    2023

  6. [14]

    Emerging properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650–9660, 2021

    2021

  7. [15]

    Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers

    Hila Chefer, Shir Gur, and Lior Wolf. Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 397–406, 2021. 10

    2021

  8. [16]

    Transformer interpretability beyond attention visualization

    Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visualization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 782–791, 2021

    2021

  9. [17]

    Unifying prediction and explanation in time-series transformers via shapley-based pretraining

    Qisen Cheng, Jinming Xing, Chang Xue, and Xiaoran Yang. Unifying prediction and explanation in time-series transformers via shapley-based pretraining. arXiv preprint arXiv:2501.15070, 2025

    2025 arXiv

  10. [18]

    What does bert look at? an analysis of bert’s attention

    Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention. InProceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , page 276. Association for Computational Ling...

    2019

  11. [19]

    Opportunities and challenges in explainable artificial intelligence (xai): A survey

    Arun Das and Paul Rad. Opportunities and challenges in explainable artificial intelligence (xai): A survey. arXiv preprint arXiv:2006.11371, 2020

    2006 arXiv

  12. [20]

    Position information in transformers: An overview

    Philipp Dufter, Martin Schmitt, and Hinrich Schütze. Position information in transformers: An overview. Computational Linguistics, 48(3):733–763, 2022

    2022

  13. [21]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  14. [22]

    Imagenet auto-annotation with segmentation propagation

    Matthieu Guillaumin, Daniel Küttel, and Vittorio Ferrari. Imagenet auto-annotation with segmentation propagation. International Journal of Computer Vision , 110:328 – 348, 2014. URL https://api.semanticscholar.org/CorpusID:1005559

    2014

  15. [23]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997

    1997

  16. [24]

    Explainable convolutional neural networks: a taxonomy, review, and future directions

    Rami Ibrahim and M Omair Shafiq. Explainable convolutional neural networks: a taxonomy, review, and future directions. ACM Computing Surveys, 55(10):1–37, 2023

    2023

  17. [25]

    Mambalrp: Explaining selective state space sequence models

    Farnoush Rezaei Jafari, Grégoire Montavon, Klaus-Robert Müller, and Oliver Eberle. Mambalrp: Explaining selective state space sequence models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  18. [26]

    Attention is not explanation

    Sarthak Jain and Byron C Wallace. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, V olume 1 (Long and Short Papers), pages 3543–3556, 2019

    2019

  19. [27]

    Serial order: A parallel distributed processing approach

    Michael I Jordan. Serial order: A parallel distributed processing approach. In Advances in psychology, volume 121, pages 471–495. Elsevier, 1997

    1997

  20. [28]

    Bert meets shapley: Extending shap explanations to transformer-based classifiers

    Enja Kokalj, Blaž Škrlj, Nada Lavraˇc, Senja Pollak, and Marko Robnik-Šikonja. Bert meets shapley: Extending shap explanations to transformer-based classifiers. In Proceedings of the EACL hackashop on news media content analysis and automated report generation , pages 16–21, 2021

    2021

  21. [29]

    Explaining in style: training a gan to explain a classifier in stylespace

    Oran Lang, Yossi Gandelsman, Michal Yarom, Yoav Wald, Gal Elidan, Avinatan Hassidim, William T Freeman, Phillip Isola, Amir Globerson, Michal Irani, et al. Explaining in style: training a gan to explain a classifier in stylespace. InProceedings of the IEEE/CVF International Co...

    2021

  22. [30]

    Explainable ai: A review of machine learning interpretability methods

    Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. Explainable ai: A review of machine learning interpretability methods. Entropy, 23(1):18, 2020

    2020

  23. [31]

    Explaining nonlinear classification decisions with deep taylor decomposition

    Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus- Robert Müller. Explaining nonlinear classification decisions with deep taylor decomposition. Pattern recognition, 65:211–222, 2017

    2017

  24. [32]

    Layer-wise relevance propagation: an overview

    Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, and Klaus- Robert Müller. Layer-wise relevance propagation: an overview. Explainable AI: interpreting, explaining and visualizing deep learning , pages 193–209, 2019. 11

    2019

  25. [33]

    Shap- based explanation methods: a review for nlp interpretability

    Edoardo Mosca, Ferenc Szigeti, Stella Tragianni, Daniel Gallagher, and Georg Groh. Shap- based explanation methods: a review for nlp interpretability. In Proceedings of the 29th international conference on computational linguistics , pages 4593–4603, 2022

    2022

  26. [34]

    Train short, test long: Attention with linear biases enables input length extrapolation

    Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. In International Conference on Learning Representations

  27. [35]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  28. [36]

    Sam 2: Segment anything in images and videos

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  29. [37]

    Explainable artificial intel- ligence: Understanding, visualizing and interpreting deep learning models

    Wojciech Samek, Thomas Wiegand, and Klaus-Robert Müller. Explainable artificial intel- ligence: Understanding, visualizing and interpreting deep learning models. arXiv preprint arXiv:1708.08296, 2017

    2017 arXiv

  30. [38]

    Grad-cam: Visual explanations from deep networks via gradient-based localization

    Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision , pages 618–626, 2017

    2017

  31. [39]

    Self-attention with relative position repre- sentations

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position repre- sentations. arXiv preprint arXiv:1803.02155, 2018

    2018 arXiv

  32. [40]

    Learning important features through propagating activation differences

    Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In International conference on machine learning , pages 3145–3153. PMLR, 2017

    2017

  33. [41]

    Roformer: Enhanced transformer with rotary position embedding

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024

    2024

  34. [42]

    Gemma: Open models based on gemini research and technology

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295, 2024

    2024 arXiv

  35. [43]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347–10357. PMLR, 2021

    2021

  36. [44]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  37. [45]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems , 2017

    2017

  38. [46]

    Analyzing the source and target contributions to predictions in neural machine translation

    Elena V oita, Rico Sennrich, and Ivan Titov. Analyzing the source and target contributions to predictions in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Na...

    2021

  39. [47]

    Towards explainable artificial intelligence (xai): A data mining perspective, 2024

    Haoyi Xiong, Xuhong Li, Xiaofei Zhang, Jiamin Chen, Xinhao Sun, Yuchen Li, Zeyi Sun, and Mengnan Du. Towards explainable artificial intelligence (xai): A data mining perspective, 2024. URL https://arxiv.org/abs/2401.04374

    2024 arXiv

  40. [48]

    Explaining information flow inside vision transformers using markov chain

    Tingyi Yuan, Xuhong Li, Haoyi Xiong, Hui Cao, and Dejing Dou. Explaining information flow inside vision transformers using markov chain. In eXplainable AI approaches for debugging and diagnosis., 2021

    2021

  41. [49]

    Tinyllama: An open-source small language model

    Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model. arXiv preprint arXiv:2401.02385, 2024. 12

    2024 arXiv

  42. [50]

    Visual interpretability for deep learning: a survey

    Quan-shi Zhang and Song-Chun Zhu. Visual interpretability for deep learning: a survey. Frontiers of Information Technology & Electronic Engineering, 19(1):27–39, 2018

    2018

  43. [51]

    A survey on neural network interpretability

    Yu Zhang, Peter Tiˇno, Aleš Leonardis, and Ke Tang. A survey on neural network interpretability. IEEE Transactions on Emerging Topics in Computational Intelligence , 5(5):726–742, 2021

    2021

  44. [52]

    They should have been giving a tribute to Branagh for bringing us one of the greatest films of all time

    Itamar Zimerman, Ameen Ali Ali, and Lior Wolf. Explaining modern gated-linear RNNs via a unified implicit attention formulation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=wnT8bfJCDx. 13 A Background for Ad...

    2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.