REVIEW 4 cited by
Transformer Interpretability Beyond Attention Visualization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Self-attention techniques, and specifically Transformers, are dominating the field of text processing and are becoming increasingly popular in computer vision classification tasks. In order to visualize the parts of the image that led to a certain classification, existing methods either rely on the obtained attention maps or employ heuristic propagation along the attention graph. In this work, we propose a novel way to compute relevancy for Transformer networks. The method assigns local relevance based on the Deep Taylor Decomposition principle and then propagates these relevancy scores through the layers. This propagation involves attention layers and skip connections, which challenge existing methods. Our solution is based on a specific formulation that is shown to maintain the total relevancy across layers. We benchmark our method on very recent visual Transformer networks, as well as on a text classification problem, and demonstrate a clear advantage over the existing explainability methods.
Forward citations
Cited by 4 Pith papers
-
Transformers as Unrolled Inference in Probabilistic Laplacian Eigenmaps: An Interpretation and Potential Improvements
Transformers can be viewed as unrolled inference in a probabilistic Laplacian Eigenmaps model, and replacing the attention matrix by attention minus identity improves validation performance.
-
Learning to Explain: Prototype-Based Surrogate Models for LLM Classification
A sentence-level prototype surrogate is trained to imitate LLM predictions and its prototype matches are used as explanations; the paper claims state-of-the-art faithfulness with moderate evidence.
-
Attention of a Kiss: Exploring Attention Maps in Video Diffusion for XAIxArts
A method and case study for visualizing cross-attention maps in Wan video diffusion transformers, showing token-region alignment over time and their use as artistic material.
-
Interpretable by AI Mother Tongue: Native Symbolic Reasoning in Neural Models
A VQ-based gated Transformer trained on AG News produces symbol traces that are supposed to be interpretable, but accuracy is low (50.94% then 47.32%) and central results are unreported.
Discussion (0). Sign in to comment.