REVIEW 4 major objections 5 minor 1 cited by
GMAR: Gradient-Driven Multi-Head Attention Rollout for Vision Transformer Interpretability
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Gradient-driven head weighting makes attention rollout more faithful for Vision Transformers.
desk verdict Gradient-weighted attention rollout is a reasonable incremental idea, but Algorithm 1 conflicts with Eq. (1), alpha is never specified, and the reported gains rest on a quarter-point margin — not ready as printed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the per-head gradient score $w_h = \mathrm{GR}_h / \sum_j \mathrm{GR}_j$, where $\mathrm{GR}_h = \sum |G_h|$ for L1 or $\sqrt{\sum G_h^2}$ for L2, and $G_h$ is the gradient of the predicted class logit with respect to the attention weights of head $h$. The weights are reshaped to $(1,H,1,1)$ and multiply each layer's attention map $A_\ell$ to form $A_{\text{weighted}}$. The rollout then follows $A_{\text{rollout}} = A_{\text{rollout}} \cdot A_{\text{weighted}} + \alpha I$, with $\alpha$ controlling the strength of the residual identity. This mechanism converts a class-specific, gradient-based importance signal at the head level into the spatial attention map, which is what lets GMAR keep rollout's global aggregation while adding head sensitivity.
What would settle it
Shuffle the GMAR head weights across heads and rerun the same rollout on the same images; if the shuffled maps match GMAR's Insertion and Deletion scores, then the specific assignment of gradient-based weights carries no information and only the act of weighting matters.
Extended reading notes
Core claim
The central claim, stated on the paper's own terms, is that gradient-driven head weighting makes attention rollout more faithful. For a given input, GMAR computes the gradient of the predicted logit with respect to each head's attention weights, aggregates it with an L1 or L2 norm, and normalizes across heads to get weights $w$. Each layer's attention map is scaled by $w$ before the recursive rollout $A_{\text{rollout}} = A_{\text{rollout}} \cdot A_{\text{weighted}} + \alpha I$, so heads that respond strongly to the predicted class dominate the final map. In the reported experiments, GMAR outperforms Attention Rollout across all evaluation metrics; with the L2 norm it reduces Average Drop from 25.78 to 22.13 and raises Insertion from 11.97 to 12.16, while also improving Deletion. The authors conclude that attention-weight-only methods are less faithful than gradient-informed ones, and that combining the two signals yields more reliable visual explanations.
Load-bearing premise
The load-bearing premise is that the gradient magnitude of the predicted class score with respect to a head's attention weights faithfully measures how much that head contributes to the prediction.
Editorial extensions
If this is right
- GMAR needs only one backward pass per input, so head-weighted explanations are nearly as cheap as standard attention rollout.
- Because the weights come from the predicted class logit, GMAR maps vary with the class being explained, unlike unweighted rollout's class-agnostic integration.
- On the reported Tiny-ImageNet/ViT-Large setup, GMAR improves all four fidelity metrics over attention rollout; the L2 variant's Average Drop (22.13) is even slightly below Grad-CAM's (22.61), while its Insertion (12.16) beats both baselines.
- The L1 and L2 variants behave differently: L1 tracks large abrupt changes, while L2 captures gradual cumulative variation, giving users two complementary explanation views.
Reading between the lines
- A natural but untested use of GMAR's weights is head pruning: if gradient magnitude really measures head importance, heads with near-zero weights should be removable with little accuracy loss, but the paper does not run that experiment.
- The residual weight $\alpha$ in Algorithm 1 is never given a value; comparing $\alpha=1$ against tuned values would separate the benefit of head weighting from the benefit of the residual recursion.
- Because the method only requires access to attention maps and gradients, it should transfer to non-vision transformers; verifying that on a text model would test whether the finding is about attention heads in general rather than ViT specifics.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Gradient-Driven Multi-Head Attention Rollout (GMAR), a method that computes per-head importance weights from the gradient of the predicted logit with respect to attention weights, normalizes them with L1 or L2 norms, and incorporates them into a weighted attention rollout for Vision Transformers. The method is evaluated on Tiny-ImageNet with a ViT-Large model using four faithfulness metrics (Average Drop, Average Increase, Insertion, Deletion), and the authors claim that GMAR consistently outperforms standard attention rollout. Qualitative overlay and difference-map visualizations are also provided.
Significance. If the reported results hold, the contribution is a moderate, incremental improvement to attention-rollout-based interpretability, with the useful property of producing head-level attribution. The paper uses four standard faithfulness metrics and compares against Grad-CAM and Attention Rollout, which is appropriate for a first validation. However, the experimental evidence is narrow (one model, one dataset, no error bars), and the method's definition contains an underspecified residual parameter and an inconsistency with the paper's own Eq. (1), so the central comparison is not yet reproducible. The idea of combining gradient-based head weighting with rollout is reasonable and worth pursuing, but the current manuscript does not yet establish the claimed advantage.
major comments (4)
- [Eq. (1), Algorithm 1, Eq. (3)] The definition of attention rollout in Eq. (1) is A_rollout = product over l of (A^(l) + I), which corresponds to the recursion A_rollout = A_rollout * (A^(l) + I). Algorithm 1 and Eq. (3) instead implement A_rollout = A_rollout * A_weighted + alpha * I. These two recursions coincide only for the first layer; with L=24 layers in ViT-Large they produce substantially different maps, and no value of alpha is given anywhere in the manuscript. Since the central claim is that GMAR outperforms attention rollout, the exact form of the recursion is load-bearing: the reported gains could be due to the ad-hoc residual term rather than to gradient-driven head weighting. Please reconcile Eq. (1) with Algorithm 1/Eq. (3), specify alpha (or remove it), and re-run the comparison.
- [Section 5.1, Table 1] Table 1 reports a single value per method and metric, with no error bars, no multiple runs, and no significance tests. Several differences that support the main claim are small (Insertion: 12.15 vs 11.97; Deletion: 10.62 vs 12.17), so without variance estimates or statistical testing the statement that GMAR 'consistently outperforms' attention rollout is not supported. Please report standard deviations or confidence intervals over multiple seeds or image subsets, and state the number of images used for evaluation.
- [Section 3.3.1] The central premise that the gradient norm of the predicted logit with respect to each head's attention weights measures head importance is asserted without justification or supporting citation. Gradients of attention weights are known to be susceptible to saturation and can be noisy; the paper should at least discuss this risk and provide a sensitivity analysis, for example comparing L1/L2 weighting against alternative head-importance measures such as attention variance or removal-based importance.
- [Algorithm 1] The description of gradient extraction is ambiguous: it says 'G <- Extract gradients of attention layers' and 'G_h <- split(G, num_head) by channel', but attention weight tensors in ViT are 4D (batch, heads, keys, queries) and gradients are typically taken with respect to the attention logits before softmax or the normalized attention weights after softmax. The paper does not specify which quantity G represents. This ambiguity, together with the unspecified alpha, prevents reproduction of the reported numbers.
minor comments (5)
- [Section 3.3.1, Eq. (2)] The typesetting of Eq. (2) and the normalization formula is garbled: 'pP G2 hi' should presumably be the square root of the sum of squared gradient elements, and 'w = GRP GR' is missing a division symbol. Please fix the notation and define the index ranges explicitly.
- [Related Work] Attention Flow is mentioned in Section 2.1 but never defined or compared against; either add a brief description or remove the reference, since the paper does not use it in the experiments.
- [Section 4.1] The citation for ViT-Large-Patch16-224 is given as [23, 24], but [23] is a paper titled 'Visual transformers' rather than the original ViT reference. Please verify the citations; the original ViT paper [1] appears to be the correct source for the architecture.
- [Section 5.2] The qualitative section says 'integrating Grad-CAM with attention rollout confirms the appropriate application of gradients', but no such integration is described in the method section; please clarify what integration was performed.
- [Section 4.3] The evaluation section does not state the number of images used for computing the metrics, the random seed, or the fine-tuned model's accuracy on Tiny-ImageNet. Providing these details would help readers assess the reliability of the results.
Circularity Check
No significant circularity: the proposed gradient-weighted rollout is an empirical heuristic compared against independent faithfulness metrics, not a fitted parameter renamed as a prediction.
full rationale
GMAR's contribution is a heuristic: compute per-head gradient norms, normalize them into weights, and re-run attention rollout with those weights. The core comparison in Table 1 (GMAR vs Attention Rollout vs Grad-CAM) is an external evaluation on four perturbation-based metrics; none of those metrics is defined as the weighted rollout itself, and the reported numbers could in principle have favored Attention Rollout if the weighting were harmful. The head weights w are derived from the model's gradients, not fitted to the evaluation metrics, so there is no fitted input renamed as a prediction. No load-bearing self-citation appears: the references are external works, and the cited attention-head-pruning literature is used only as motivation. Algorithm 1's residual recursion (A_rollout = A_rollout * A_weighted + alpha * I) differs from the stated Eq. (1) and leaves alpha unspecified, but this is a reproducibility/correctness defect, not circularity: it does not make the output equal to the input by construction. The use of the model's own gradients and logits is inherent to the faithfulness-evaluation setting for XAI and does not reduce the claimed comparison to a tautology.
Assumptions & free parameters
free parameters (1)
- alpha (residual ratio)
assumptions (3)
- domain assumption The L1 or L2 norm of the gradient of the predicted logit with respect to each head's attention weights measures that head's contribution to the prediction.
- domain assumption The identity matrix plus the attention matrix per layer (or the alpha-scaled identity in Algorithm 1) correctly represents residual connections in attention rollout.
- domain assumption Attention maps extracted from the Hugging Face model are normalized probability distributions over tokens, so weighting and multiplying them is a valid propagation.
Cite this review
Pith. "Pith review of GMAR: Gradient-Driven Multi-Head Attention Rollout for Vision Transformer Interpretability." pith.science (2026). https://pith.science/paper/VB7HI67L
@misc{pith2026250419414,
author = {Pith},
title = {Pith review of: GMAR: Gradient-Driven Multi-Head Attention Rollout for Vision Transformer Interpretability},
year = {2026},
howpublished = {\url{https://pith.science/paper/VB7HI67L}},
note = {Machine review of arXiv:2504.19414}
}
read the original abstract
The Vision Transformer (ViT) has made significant advancements in computer vision, utilizing self-attention mechanisms to achieve state-of-the-art performance across various tasks, including image classification, object detection, and segmentation. Its architectural flexibility and capabilities have made it a preferred choice among researchers and practitioners. However, the intricate multi-head attention mechanism of ViT presents significant challenges to interpretability, as the underlying prediction process remains opaque. A critical limitation arises from an observation commonly noted in transformer architectures: "Not all attention heads are equally meaningful." Overlooking the relative importance of specific heads highlights the limitations of existing interpretability methods. To address these challenges, we introduce Gradient-Driven Multi-Head Attention Rollout (GMAR), a novel method that quantifies the importance of each attention head using gradient-based scores. These scores are normalized to derive a weighted aggregate attention score, effectively capturing the relative contributions of individual heads. GMAR clarifies the role of each head in the prediction process, enabling more precise interpretability at the head level. Experimental results demonstrate that GMAR consistently outperforms traditional attention rollout techniques. This work provides a practical contribution to transformer-based architectures, establishing a robust framework for enhancing the interpretability of Vision Transformer models.
Forward citations
Cited by 1 Pith paper
-
Modeling Day-Long ECG Signals to Predict Heart Failure Risk with Explainable AI
DeepHHF, trained on day-long single-lead Holter ECGs from 40,174 patients, predicted incident heart failure within five years with AUROC 0.80 and external AUROC 0.81.
Reference graph
Works this paper leans on
-
[1]
INTRODUCTION The Vision Transformer (ViT) has emerged as a transforma- tive architecture in computer vision [1], achieving state-of- the-art performance in tasks like image classification and ob- ject detection. Unlike traditional convolutional neural net- works (CNNs) that rely on localized feature extraction, ViTs leverage a global perspective through t...
arXiv 2025
-
[2]
RELATED WORK 2.1. Explainable AI The increasing complexity of deep learning models high- lights the need for reliability and transparency, driving ad- vancements in XAI. Model-agnostic techniques, such as LIME(Local Interpretable Model-agnostic Explanations) [4], utilize local surrogate models to explain predictions, while SHAP(SHapley Additive exPlanatio...
-
[3]
PROPOSED METHOD 3.1. Overview of Attention Rollout Attention rollout is a technique designed to interpret and visu- alize the internal mechanisms of ViTs. It aggregates attention weights across multiple layers to quantify the influence of in- put tokens on the model’s final prediction. By recursively Fig. 2. Comparing attention head weights, with the high...
-
[4]
EXPERIMENT 4.1. Experimental Setup We utilize the Google ViT-Large-Patch16-224 model [23, 24] as the base architecture, fine-tuning it on the Tiny-ImageNet dataset [25]. This dataset was selected for its compact size, which enables rapid fine-tuning and experimentation. Al- though Tiny-ImageNet may yield slightly lower accuracy compared to larger datasets...
-
[5]
RESULT 5.1. Quantitative Results Table 1 summarizes the evaluation results of the interpretabil- ity methods for Vision Transformers. Grad-CAM outper- forms Attention Rollout on most metrics, highlighting its effectiveness in identifying class-specific regions. However, Attention Rollout excels in the Insertion metric, indicating its strength in capturing...
-
[6]
CONCLUSION Our paper introduced GMAR, a novel framework to enhance the interpretability of ViTs by assigning gradient-based im- portance scores to individual attention heads, enabling a more precise understanding of their contributions to model predic- tions. GMAR improves upon attention rollout by incorporat- ing these scores into the process, resulting ...
-
[7]
An image is worth 16x16 words: Transformers for image recognition at scale,
Alexey Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020
arXiv 2010
-
[8]
Xai for transformers: Better explanations through conserva- tive propagation,
Ameen Ali, Thomas Schnake, Oliver Eberle, Gr ´egoire Montavon, Klaus-Robert M ¨uller, and Lior Wolf, “Xai for transformers: Better explanations through conserva- tive propagation,” in ICML, 2022
work page 2022
Show all 32 references
-
[9]
Quantifying at- tention flow in transformers,
Samira Abnar and Willem Zuidema, “Quantifying at- tention flow in transformers,” in ACL, 2020
2020
-
[10]
” why should i trust you?
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin, “” why should i trust you?” explaining the predictions of any classifier,” in SIGKDD, 2016
2016
-
[11]
A unified approach to inter- preting model predictions,
M Scott, Lee Su-In, et al., “A unified approach to inter- preting model predictions,” NeurIPS, 2017
2017
-
[12]
Grad-cam: Visual explanations from deep net- works via gradient-based localization,
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra, “Grad-cam: Visual explanations from deep net- works via gradient-based localization,” in ICCV, 2017
2017
-
[13]
On pixel-wise explanations for non- linear classifier decisions by layer-wise relevance prop- agation,
Sebastian Bach, Alexander Binder, Gr ´egoire Montavon, Frederick Klauschen, Klaus-Robert M ¨uller, and Woj- ciech Samek, “On pixel-wise explanations for non- linear classifier decisions by layer-wise relevance prop- agation,” PloS one, vol. 10, no. 7, pp. e0130140, 2015
2015
-
[14]
Vit-cx: causal explanation of vision transform- ers,
Weiyan Xie, Xiao-Hui Li, Caleb Chen Cao, and Nevin L Zhang, “Vit-cx: causal explanation of vision transform- ers,” in IJCAI, 2023
2023
-
[15]
Human-inspired explanations for vision transformers and convolutional neural networks,
Mahadev Prasad Panda, Matteo Tiezzi, Martina Vilas, Gemma Roig, Bjoern M Eskofier, and Dario Zanca, “Human-inspired explanations for vision transformers and convolutional neural networks,” arXiv preprint arXiv:2408.02123, 2024
2024 arXiv
-
[16]
Attention is all you need,
A Vaswani, “Attention is all you need,” NeurIPS, 2017
2017
-
[17]
Bert: Pre-training of deep bidirec- tional transformers for language understanding,
Jacob Devlin, “Bert: Pre-training of deep bidirec- tional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018
2018 arXiv
-
[18]
Roberta: A robustly optimized bert pre- training approach,
Yinhan Liu, “Roberta: A robustly optimized bert pre- training approach,” arXiv preprint arXiv:1907.11692 , vol. 364, 2019
1907 arXiv
-
[19]
Language models are few-shot learners,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al., “Language models are few-shot learners,” NeurIPS, 2020
2020
-
[20]
Llama: Open and efficient foundation lan- guage models,
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al., “Llama: Open and efficient foundation lan- guage models,” arXiv preprint arXiv:2302.13971, 2023
2023 arXiv
-
[21]
Training data-efficient image transformers & distilla- tion through attention,
Hugo Touvron, Matthieu Cord, Matthijs Douze, Fran- cisco Massa, Alexandre Sablayrolles, and Herv ´e J´egou, “Training data-efficient image transformers & distilla- tion through attention,” in ICML, 2021
2021
-
[22]
Beit: Bert pre-training of image transformers,
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei, “Beit: Bert pre-training of image transformers,” arXiv preprint arXiv:2106.08254, 2021
2021 arXiv
-
[23]
Swin transformer: Hierarchical vision transformer us- ing shifted windows,
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo, “Swin transformer: Hierarchical vision transformer us- ing shifted windows,” in CVPR, 2021
2021
-
[24]
Roles and uti- lization of attention heads in transformer-based neural language models,
Jae-young Jo and Sung-Hyon Myaeng, “Roles and uti- lization of attention heads in transformer-based neural language models,” in ACL, 2020
2020
-
[25]
Transformer in- terpretability beyond attention visualization,
Hila Chefer, Shir Gur, and Lior Wolf, “Transformer in- terpretability beyond attention visualization,” in CVPR, 2021
2021
-
[26]
Analyzing multi-head self- attention: Specialized heads do the heavy lifting, the rest can be pruned,
Elena V oita, David Talbot, Fedor Moiseev, Rico Sen- nrich, and Ivan Titov, “Analyzing multi-head self- attention: Specialized heads do the heavy lifting, the rest can be pruned,” arXiv preprint arXiv:1905.09418, 2019
1905 arXiv
-
[27]
How does attention work in vision transform- ers? a visual analytics attempt,
Yiran Li, Junpeng Wang, Xin Dai, Liang Wang, Chin- Chia Michael Yeh, Yan Zheng, Wei Zhang, and Kwan- Liu Ma, “How does attention work in vision transform- ers? a visual analytics attempt,” IEEE transactions on visualization and computer graphics , 2023
2023
-
[28]
Moh: Multi-head attention as mixture-of-head atten- tion,
Peng Jin, Bo Zhu, Li Yuan, and Shuicheng Yan, “Moh: Multi-head attention as mixture-of-head atten- tion,” arXiv preprint arXiv:2410.11842, 2024
2024
-
[29]
Vi- sual transformers: Token-based image representation and processing for computer vision,
Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda, “Vi- sual transformers: Token-based image representation and processing for computer vision,” 2020
2020
-
[30]
Imagenet: A large-scale hierarchical image database,
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in CVPR, 2009
2009
-
[31]
Tiny imagenet visual recog- nition challenge,
Yann Le and Xuan Yang, “Tiny imagenet visual recog- nition challenge,” CS 231N, vol. 7, no. 7, pp. 3, 2015
2015
-
[32]
Evaluation metrics for xai: A review, taxonomy, and practical applications,
Md Abdul Kadir, Amir Mosavi, and Daniel Sonntag, “Evaluation metrics for xai: A review, taxonomy, and practical applications,” in IEEE INES, 2023
2023
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.