Pith. sign in

REVIEW 5 major objections 5 minor 2 cited by

DLBacktrace: A Model Agnostic Explainability for any Deep Learning Models

T0 review · 5 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read DLBacktrace claims a single, baseline-free backtrace gives faithful feature attribution for MLPs, CNNs, and transformer LLMs.

desk verdict DLBacktrace is LRP under a new name, and its attention extension doesn't type-check; the empirical claims conflict with the paper's own table. read the letter →

arxiv 2411.12643 v2 pith:UYPYA65U submitted 2024-11-19 cs.LG cs.AIcs.CL

Vinay Kumar Sankarapu , Chintan Chitroda , Yashwardhan Rathore , Neeraj Kumar Singh , Pratinav Seth This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords explainableAIfeatureattributionrelevancepropagationmodel-agnosticinterpretabilitydeeplearningtransformerattentiondeterministicexplanations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DLBacktrace is a proposed explainability method that traces a relevance score from a model's output back to its input, splitting each neuron's contribution into positive and negative parts according to the signed pre-activation products $W_{ij}x_{ij}$. The authors claim this gives deterministic, baseline-free local explanations that work across MLPs, CNNs, and transformer-based LLMs, with global importance obtained by averaging normalized local scores. They benchmark the method against LIME, SHAP, Grad-CAM, Vanilla Gradient, SmoothGrad, Integrated Gradients, and attention-rollout on tabular, image, and text tasks. If the central claim holds, a single method would provide faithful feature attribution for high-stakes applications without needing auxiliary white-box models or baseline samples.

What carries the argument

The carrying mechanism is the proportional relevance-distribution rule of Section 3.3: relevance arriving at a neuron is split into positive, negative, and bias shares in proportion to the positive pre-activation sum $X_{pj}$, the negative pre-activation sum $X_{nj}$, and the bias $b_j$, then distributed to the inputs in proportion to each signed product $W_{ij}x_{ij}$ within the same sign class. This rule is what makes the method deterministic and independent of sampling or auxiliary models, since it depends only on the model's weights, activations, and biases. For attention, the paper adds a separate recipe that routes relevance through the $QK^T$, $V$, $Q$, and $K$ outputs using elementwise products, treating any loss as bias.

What would settle it

For a two-layer ReLU network with known weights and bias, compute the sum of input relevances from DLBacktrace and compare it to the output relevance after subtracting bias shares; if the two diverge for a range of inputs, the rule is lossy. For the attention formulas, scale $Q$ and $K$ by a constant and check whether $r_{QK}$ remains invariant; a change would show the stated equations are dimensionally unstable.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that DLBacktrace is a model-agnostic explainability technique that reads only the trained network and the single test sample, requiring no auxiliary model, no background dataset, and no baseline input. The algorithm builds a graph from the network weights and propagates output relevance backward in a breadth-first manner. For each neuron, the total pre-activation is $T_j = X_{pj}+X_{nj}+b_j$, and the neuron's relevance is divided into positive, negative, and bias shares $R_{pj}=X_{pj}/T_j r_{yj}$, $R_{nj}=X_{nj}/T_j r_{yj}$, $R_{bj}=b_j/T_j r_{yj}$; these shares are then passed to each input unit in proportion to the signed contribution $W_{ij}x_{ij}$, with zeroing under saturation. A Contrastive Mode tracks separate positive and negative relevance streams, and Algorithm 3 extends the same idea to attention layers by distributing relevance through elementwise products of the attention outputs. The reported benchmarks show DLBacktrace with the best numbers on tabular stability (MPRT) and on image faithfulness, sensitivity, and pixel-flipping, while on the SST-2 text benchmark Integrated Gradients achieves the best scores and DLBacktrace is competitive rather than best.

Load-bearing premise

The method assumes that a neuron's relevance can be split into positive and negative parts in exact proportion to the signed pre-activation inputs $W_{ij}x_{ij}$ across every layer, including nonlinear, normalized, and attention layers, but the paper gives no proof that this split preserves the model's actual computation.

Editorial extensions

If this is right

  • One implementation of DLBacktrace could replace separate tabular, image, and NLP explainers with a single deterministic backtrace.
  • Layer-wise relevance, including bias-to-input ratios and saturation, could support network pruning and bias audits without retraining.
  • Global feature importance can be computed by averaging normalized local scores, giving a Shapley-like ranking without the cost of sampling.
  • In regulated settings, the deterministic and baseline-free nature would make explanations reproducible for audits.
  • The reported improvements on image metrics would mean DLBacktrace is more faithful to a ResNet's actual behavior than Grad-CAM, vanilla gradient, SmoothGrad, or Integrated Gradients.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test the paper does not run is to check relevance conservation on deeper nonlinear networks; the stated claim that any loss is due to bias is a testable identity that would settle whether the split rule is exact.
  • The attention formulas are presented without derivation, so a reader cannot yet tell whether they are correct for multi-head attention; a toy single-head attention model with known token contributions would resolve this.
  • The claim of universal model-agnosticism extends beyond the tested set of MLP, ResNet, U-Net, BERT, and one Llama demo; recurrent and multimodal architectures are plausible targets but remain unverified.
  • If the proportional rule holds, DLBacktrace could be a cheap way to compute counterfactual-style positive and negative attributions for bias detection in tabular models, though the paper only sketches that use.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes DLBacktrace, a deterministic, backpropagation-style relevance attribution method for deep networks. It presents propagation rules for dense layers in default and contrastive modes, a proposed extension to attention layers, and benchmarks on tabular (Lending Club MLP), image (CIFAR-10 ResNet-34), and text (SST-2 BERT) tasks against LIME, SHAP, Grad-CAM, Vanilla Gradient, SmoothGrad, and Integrated Gradients. The authors claim that DLBacktrace is model-agnostic and 'significantly outperforms traditional methods ... across all key metrics,' while also acknowledging in Section 5.3.3 that Integrated Gradients is the strongest method on the text benchmarks.

Significance. If the method worked as claimed, it would offer a useful deterministic, baseline-free alternative to gradient- and perturbation-based explanation, with attractive properties for live deployment. Strengths include the absence of auxiliary models or baselines, the deterministic nature, open-source availability, and the breadth of attempted modalities. However, the current manuscript does not establish the central claims: the attention propagation algorithm is mathematically underspecified, relevance conservation is asserted without proof, and the reported benchmarks either use nonstandard metrics without error bars or directly contradict the headline claim. The contribution is therefore not yet ready for publication.

major comments (5)
  1. [§3.4, Algorithm 3, Eqs. (10)–(14)] The attention-layer rule is not well defined. With standard shapes rO, xV ∈ R^{B×S×D} and xQK ∈ R^{B×H×S×S}, the dot products in Eqs. (10)–(13) are undefined; the softmax nonlinearity is never handled; and the claim after Algorithm 3 that 'any loss of relevance during propagation is due to network bias' is false for softmax and layer-norm. Because the BERT/LLM demonstrations rest entirely on this rule, the paper's central 'any deep learning models' claim is not established by a defined algorithm. The authors must either supply a complete, dimensionally consistent attention rule with a derivation or drop the transformer/LLM claims.
  2. [§3.3.1, Eq. (3)] The dense-layer redistribution rule also lacks a conservation argument. For a ReLU or saturating activation, the branch that sets terms to zero discards pre-activation relevance without transferring it to the bias or to saturated units; the statement that all lost relevance is due to network bias is therefore not generally true. Since the method's explanatory value depends on exactness of redistribution, the authors need a proof of conservation (or a precise statement of when it holds) before claiming the attributions are faithful.
  3. [§5.3.3, Table 3] The reported text results contradict the paper's headline. In Table 3, Integrated Gradients has the best MoRF AUC (lower), LeRF AUC (higher), and Delta AUC, while DLBacktrace is mid-ranked; Section 5.3.3 explicitly acknowledges IG's superiority. The abstract and Section 5.3.2's claim of 'significantly outperforms ... across all key metrics' is therefore not supported by the paper's own data. The claims need to be restricted to the metrics and modalities where they hold.
  4. [§5.2.1, Table 1 and §5.2.2, Table 2] The evaluation is not statistically grounded. MPRT and Complexity are introduced in this paper without validation as established metrics; no error bars or significance tests are given for any table; and Table 2 reports Max-Sensitivity values computed from a 'non-infinite mean' after discarding infinite values, a nonstandard procedure that can arbitrarily favor a method with finite outputs. The comparison should be repeated with standard XAI metrics (e.g., from Quantus) or with a principled justification and confidence intervals for the new metrics.
  5. [§5.1] Sample sizes and experimental detail are insufficient for the strength of the claims: only 1,024 tabular, 500 image, and the SST-2 evaluation set are used; no seeds, hyperparameters, or perturbation protocols are reported; and the Lending Club/ResNet/BERT checkpoints are not linked. This makes the claimed superiority non-reproducible from the manuscript alone.
minor comments (5)
  1. [§4] The reference 'Figure ??' is unresolved; it should be replaced with an actual figure reference or removed.
  2. [Eq. (3)] The cases in Eq. (3) are not mutually exclusive: the first and second branches both apply when W_ij x_ij > 0, and the third and fourth branches both apply when W_ij x_ij < 0. The intended precedence should be stated explicitly.
  3. [Algorithm 3] The input/output notation is confusing: the algorithm is declared with Input x and Output ry, but Step 3 computes rO from the linear projection layer without defining how rO relates to x and y. Clarify the variable mappings and the contraction operations in Eqs. (10)–(13).
  4. [§8 and §9] There are typographical errors in possessive forms: 'it’s usage' should be 'its usage' in both sections.
  5. [§5.3.3] The cross-reference 'as illustrated in Fig. 13' points to an appendix figure; use the main-text figure number or fix the cross-reference.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DLBacktrace's relevance propagation is a self-contained redistribution algorithm; the undefined attention formulas and self-defined metrics are correctness/validation concerns, not cases where a prediction is equivalent to its input by construction.

full rationale

The core derivation of DLBacktrace is self-contained: Eqs. (1)-(3) define a deterministic redistribution of output relevance to inputs proportional to signed pre-activations W_ij x_ij, with no parameter fitted to the benchmark data and no target explanation used as input. The paper does not invoke a prior result of its own authors to justify the method; all references are external, so no self-citation chain is load-bearing. The claimed benchmarks compare DLBacktrace against LIME, SHAP, Grad-CAM, IG, etc. using standard or paper-defined metrics; even if MPRT/Complexity are not externally standardized and the attention-layer formulas in Algorithm 3 are dimensionally unclear, those are issues of evaluation design and mathematical correctness, not circularity. Nothing in the manuscript exhibits a reduction of a claimed prediction to a fitted parameter or to the definition of the thing being explained. The 'any loss of relevance is due to network bias' note is a semantic claim, not a circular one. Hence no circular step is identifiable and the score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method introduces no new physical or computational entities, but it relies on several unproven assumptions about relevance decomposition and conservation. The free parameters are minimal but are not systematically tuned, and their influence on the reported results is not explored.

free parameters (2)
  • thresholding = 0.5
    Used for segmentation tasks to discard predictions below a confidence threshold; chosen by the authors per task.
  • scaler = 1
    Sets the initial relevance value at the last layer; the default of 1 is arbitrary and influences the scale of all attribution scores.
assumptions (3)
  • domain assumption Relevance can be decomposed proportionally into positive and negative pre-activation contributions, as in Eq. (3).
    This is a heuristic rule that lacks a formal derivation. It is the basis for all relevance propagation in the method and is assumed to hold for all layer types.
  • domain assumption Breadth-first propagation from the output node to input leaves yields a meaningful attribution.
    The paper assumes that a deterministic backward pass provides correct feature importance, but this is not proven and conflicts with known limitations of gradient- and relevance-based attributions.
  • ad hoc to paper Any loss of relevance during propagation is due to network bias.
    Stated in a note at the end of Section 3.4. This assumption is used to account for conservation violations but is not justified by the algorithm or theory.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DLBacktrace: A Model Agnostic Explainability for any Deep Learning Models." pith.science (2026). https://pith.science/paper/UYPYA65U

@misc{pith2026241112643,
  author       = {Pith},
  title        = {Pith review of: DLBacktrace: A Model Agnostic Explainability for any Deep Learning Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UYPYA65U}},
  note         = {Machine review of arXiv:2411.12643}
}
read the original abstract

The rapid growth of AI has led to more complex deep learning models, often operating as opaque "black boxes" with limited transparency in their decision-making. This lack of interpretability poses challenges, especially in high-stakes applications where understanding model output is crucial. This work highlights the importance of interpretability in fostering trust, accountability, and responsible deployment. To address these challenges, we introduce DLBacktrace, a novel, model-agnostic technique designed to provide clear insights into deep learning model decisions across a wide range of domains and architectures, including MLPs, CNNs, and Transformer-based LLM models. We present a comprehensive overview of DLBacktrace and benchmark its performance against established interpretability methods such as SHAP, LIME, and GradCAM. Our results demonstrate that DLBacktrace effectively enhances understanding of model behavior across diverse tasks. DLBacktrace is compatible with models developed in both PyTorch and TensorFlow, supporting architectures such as BERT, ResNet, U-Net, and custom DNNs for tabular data. The library is open-sourced and available at https://github.com/AryaXAI/DLBacktrace .

Figures

Figures reproduced from arXiv: 2411.12643 by the authors.

Figure 1
Figure 1. DLBacktrace Workflow: Generating Fine-Grained Explanations from Pre-Trained Models and Test Time [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustration Depicting DLBacktrace Calculation for a Sample Network [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Illustration of Explanations of a Correctly Classified Sample from the Lending Club Dataset where Loan was [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Visualizing ResNet’s decisions on a Truck image of CIFAR10 Dataset using various explanation methods. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Explanations by different methods for model decision making for Sentiment Analysis for a sample from [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Illustration of Explanations of a Correctly Classified Sample from the Lending Club Dataset where Loan was [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Illustration of Explanations of a Incorrectly Classified Sample from the Lending Club Dataset where Loan [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Visualizing ResNet’s decisions on a Horse image of CIFAR10 Dataset using various explanation methods. [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Analysis of a U-Net segmentation model’s decision-making on a CamVid Dataset Sample. The figure shows [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Analysis of a Tumour Segmentation Model’s decision-making on a ClinicdB Dataset Sample. The [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]
Figure 11
Figure 11. Figure 11: Explanations of the model’s decision-making process on a Bird image from the CUB-200 dataset, using [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Explanations of the model’s decision-making process on a duck image from the CUB-200 dataset, using [PITH_FULL_IMAGE:figures/full_fig_p015_12.png]
Figure 13
Figure 13. Figure 13: Explanations by different methods for model decision making for Sentiment Analysis for a sample from [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Explanations of the model’s decision-making process for multi-class topic detection for incorrect classi [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Explanations of the model’s decision-making process for multi-class topic detection for correct classification [PITH_FULL_IMAGE:figures/full_fig_p017_15.png]
Figure 16
Figure 16. Figure 16: Input Prompt: Classify the text into neutral, negative, or positive. Text: The product mentioned full coverage, but my claim wasn’t accepted, and the experience was not something I would recommend to others. Sentiment of the text is: Ouput : Negative. Reason: The text…
Figure 17
Figure 17. Figure 17: Input Prompt: Provide a function that calculates the n-th fibonacci number. Output: Fibonacci sequence is a series of numbers in which each number is the sum of the two preceding ones, usually starting with 0 and 1.Fibonacci Function. Here’s a Python function that cal…
Figure 18
Figure 18. Figure 18: Input Prompt: Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for $2 per fresh duck egg. How much in dollars does she make daily a…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Faithfulness to Refusal: A Causal Audit of Neuron Selectors

    cs.CL 2026-07 conditional novelty 6.0 of 10

    A causal audit via neuron-row zeroing shows attribution methods (LRP, IG) faithfully identify dispensable neurons and can install refusal behavior, while rank-stability proxies systematically miss selector failures.

  2. xai_evals : A Framework for Evaluating Post-Hoc Local Explanation Methods

    cs.LG 2025-02 reject novelty 2.0 of 10

    A technical report introducing xai_evals, a Python package that wraps existing explainability and metric libraries without adding new methods or validated results.

Reference graph

Works this paper leans on

32 extracted references · 13 canonical work pages · cited by 2 Pith papers

  1. [1]

    OpenAI ChatGPT

    OpenAI. OpenAI ChatGPT. https://openai.com/index/chatgpt/. [Accessed 12-11-2024]

    work page 2024

  2. [2]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971, 2023

    arXiv 2023

  3. [3]

    Rethinking interpretability in the era of large language models

    Chandan Singh, Jeevana Priya Inala, Michel Galley, Rich Caruana, and Jianfeng Gao. Rethinking interpretability in the era of large language models. ArXiv, abs/2402.01761, 2024

    arXiv 2024

  4. [4]

    Sensible ai: Re-imagining interpretability and explainability using sensemaking theory

    Harmanpreet Kaur, Eytan Adar, Eric Gilbert, and Cliff Lampe. Sensible ai: Re-imagining interpretability and explainability using sensemaking theory. Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, 2022

    work page 2022

  5. [5]

    Interpretability needs a new paradigm

    Andreas Madsen, Himabindu Lakkaraju, Siva Reddy, and Sarath Chandar. Interpretability needs a new paradigm. arXiv preprint arXiv:2405.05386, 2024

    arXiv 2024

  6. [6]

    Towards compositional interpretability for xai

    Sean Tull, Robin Lorenz, Stephen Clark, Ilyas Khan, and Bob Coecke. Towards compositional interpretability for xai. ArXiv, abs/2406.17583, 2024

    arXiv 2024

  7. [7]

    Concrete problems in ai safety

    Dario Amodei, Christopher Olah, Jacob Steinhardt, Paul Francis Christiano, John Schulman, and Dandelion Mané. Concrete problems in ai safety. ArXiv, abs/1606.06565, 2016

    arXiv 2016

  8. [8]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 114(13):3521...

    work page 2017

Show all 32 references
  1. [9]

    Scalable agent alignment via reward modeling: a research direction

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. Scalable agent alignment via reward modeling: a research direction. ArXiv, abs/1811.07871, 2018

    2018 arXiv

  2. [10]

    why should i trust you?

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. “why should i trust you?”: Explaining the predictions of any classifier. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016

    2016

  3. [11]

    Lundberg and Su-In Lee

    Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Neural Information Processing Systems, 2017

    2017

  4. [12]

    Axiomatic attribution for deep networks

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International Conference on Machine Learning, 2017

    2017

  5. [13]

    The disagreement problem in explainable machine learning: A practitioner’s perspective

    Satyapriya Krishna, Tessa Han, Alex Gu, Javin Pombra, Shahin Jabbari, Steven Wu, and Himabindu Lakkaraju. The disagreement problem in explainable machine learning: A practitioner’s perspective. Trans. Mach. Learn. Res., 2024, 2022

    2024

  6. [14]

    Bigham, J

    Jonathan Dinu, Jeffrey P. Bigham, J. Zico Kolter Unaffiliated, and Carnegie Mellon University. Challenging common interpretability assumptions in feature attribution explanations. ArXiv, abs/2012.02748, 2020

    2012 arXiv

  7. [15]

    When explanations lie: Why many modified bp attributions fail

    Leon Sixt, Maximilian Granz, and Tim Landgraf. When explanations lie: Why many modified bp attributions fail. In International Conference on Machine Learning, 2019

    2019

  8. [16]

    Applications of explainable artificial intelligence in finance—a systematic review of finance, information systems, and computer science literature

    Patrick Maximilian Weber, Kim Valerie Carl, and Oliver Hinz. Applications of explainable artificial intelligence in finance—a systematic review of finance, information systems, and computer science literature. Management Review Quarterly, 74:867–907, 2023

    2023

  9. [17]

    On behalf of the stakeholders: Trends in nlp model interpretability in the era of llms

    Nitay Calderon and Roi Reichart. On behalf of the stakeholders: Trends in nlp model interpretability in the era of llms. ArXiv, abs/2407.19200, 2024

    2024 arXiv

  10. [18]

    Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra

    Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision, 128:336 – 359, 2016

    2016

  11. [19]

    Deep inside convolutional networks: Visualising image classification models and saliency maps

    Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. CoRR, abs/1312.6034, 2013

    2013 arXiv

  12. [20]

    Viégas, and Martin Wattenberg

    Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg. Smoothgrad: removing noise by adding noise. ArXiv, abs/1706.03825, 2017

    2017 arXiv

  13. [21]

    Challenges and opportunities in text generation explainability

    Kenza Amara, Rita Sevastjanova, and Mennatallah El-Assady. Challenges and opportunities in text generation explainability. In xAI, 2024. 19 A PREPRINT - FEBRUARY 5, 2025

    2024

  14. [22]

    Latent concept-based explanation of nlp models

    Xuemin Yu, Fahim Dalvi, Nadir Durrani, and Hassan Sajjad. Latent concept-based explanation of nlp models. ArXiv, abs/2404.12545, 2024

    2024 arXiv

  15. [23]

    Incorporating attribution importance for improving faithfulness metrics

    Zhixue Zhao and Nikolaos Aletras. Incorporating attribution importance for improving faithfulness metrics. In Annual Meeting of the Association for Computational Linguistics, 2023

    2023

  16. [24]

    Christensen, Tuukka Ruotsalo, Lars Maaløe, and Maria Maistro

    Joakim Edin, Andreas Geert Motzfeldt, Casper L. Christensen, Tuukka Ruotsalo, Lars Maaløe, and Maria Maistro. Normalized aopc: Fixing misleading faithfulness metrics for feature attribution explainability. ArXiv, abs/2408.08137, 2024

    2024 arXiv

  17. [25]

    Zoom in: An introduction to circuits

    Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 2020. https://distill.pub/2020/circuits/zoom-in

    2020

  18. [26]

    Progress measures for grokking via mechanistic interpretability

    Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. ArXiv, abs/2301.05217, 2023

    2023 arXiv

  19. [27]

    Anna Hedström, Leander Weber, Dilyara Bareeva, Franz Motzkus, Wojciech Samek, Sebastian Lapuschkin, and Marina M.-C. Höhne. Quantus: An explainable ai toolkit for responsible evaluation of neural network explanations. ArXiv, abs/2202.06861, 2022

    2022 arXiv

  20. [28]

    Beexai: Benchmark to evaluate explainable ai

    Samuel Sithakoul, Sara Meftah, and Clément Feutry. Beexai: Benchmark to evaluate explainable ai. In xAI, 2024

    2024

  21. [29]

    Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N

    Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017

    2017

  22. [30]

    Umang Bhatt, Adrian Weller, and José M. F. Moura. Evaluating and aggregating feature-based model explanations. In International Joint Conference on Artificial Intelligence, 2020

    2020

  23. [31]

    Inouye, and Pradeep Ravikumar

    Chih-Kuan Yeh, Cheng-Yu Hsieh, Arun Sai Suggala, David I. Inouye, and Pradeep Ravikumar. On the (in)fidelity and sensitivity for explanations. arXiv: Learning, 2019

    2019

  24. [32]

    Attnlrp: Attention-aware layer-wise relevance propagation for transformers

    Reduan Achtibat, Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Aakriti Jain, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. Attnlrp: Attention-aware layer-wise relevance propagation for transformers. ArXiv, abs/2402.05602, 2024. 20

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.