Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Beyond the Black Box: Interpretability of LLMs in Finance

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that the internal machinery of language models can be read and modified for financial tasks, and it demonstrates the claim with logit lens, attribution patching, sparse-autoencoder features, and feature steering.

desk verdict A useful, honest survey-plus-prototype of mechanistic interpretability in finance; the proof-of-concept experiments support the direction but not yet the strong 'first application' claim. read the letter →

arxiv 2505.24650 v1 pith:ZKXTWG4T submitted 2025-05-14 cs.CE cs.LGq-fin.ST

Hariom Tatsat (Barclays) , Ariye Shater (Barclays) This is my paper · ORCID
classification cs.CEcs.LGq-fin.ST
keywords LargeLanguageModelsMechanisticInterpretabilitySparseAutoencodersHallucinationSentimentAnalysisBiasDetectionFinancialApplicationsRegulatoryCompliance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that mechanistic interpretability—reverse-engineering a language model's internal activations and circuits rather than relying on post-hoc explanations—can be applied to financial tasks, and claims to be the first work to do so in the finance domain. It argues that two families of tools, observation (logit lens, sparse-autoencoder feature reading) and intervention (attribution patching, feature steering), let practitioners not only see which components drive a financial prediction but also alter that prediction without retraining or extensive prompt engineering. The experiments cover sentiment classification, stock-movement prediction from news headlines, bias flagging in loan applications, and hallucination-triggered retrieval augmentation. If the claim holds, regulated financial institutions gain a way to audit and steer LLM behavior at the neuron level, which matters because regulators increasingly demand transparency and accountability in AI decisions.

What carries the argument

The load-bearing object is the sparse autoencoder (SAE), a two-layer network with a wide, sparsely activated hidden layer that decomposes a model's internal activations into a dictionary of roughly monosemantic features; each decoder column acts as a feature vector that can be read, clustered, and steered. Around this core sit logit lens (applying the model's output projection to intermediate layer activations to watch predictions form), attribution patching (replacing corrupted-run activations with clean-run ones to locate causally important components), and feature steering (adding a scaled feature vector to the hidden state during the forward pass). Together they convert an opaque model into a set of inspectable, adjustable components.

What would settle it

Run the credit-risk steering experiment on a held-out set of financial sentences not used in selecting the feature, comparing steered versus unsteered accuracy per sentiment class; if accuracy does not improve or another class degrades, the claim that steering a credit-risk feature improves classification fails. For the hallucination experiment, test the 20% activation threshold on a broader set of finance questions and compare RAG-trigger decisions against a random-feature baseline.

Watch

Extended reading notes

Core claim

The paper's central claim is that the internal machinery of a transformer language model encodes financially meaningful concepts, and that those concepts can be read out and deliberately modified. Concretely, the authors report that a sparse autoencoder trained on a small language model's activations yields features whose auto-generated labels include credit risk, financial terms, and market statistics; that a decision tree over these feature activations predicts short-term stock direction from news headlines; that high activation of bias-related features flags loan-application text containing discriminatory language; and that steering a credit-risk feature shifts sentiment ratings toward more balanced credit-aware judgments. They also use logit lens to trace how token probabilities for 'rise' versus 'fall' sharpen across layers, and attribution patching to identify the attention heads and layers that carry financial reasoning. The authors present these as demonstrations that observation and intervention are both practical in finance, not as production-ready systems.

Load-bearing premise

The load-bearing premise is that an auto-interpreted sparse-autoencoder feature labeled 'credit risk' is the right causal handle for financial sentiment; if steering that feature does not improve classification on unseen sentences beyond the reported examples, the practical case for feature steering in finance weakens.

Editorial extensions

If this is right

  • If the demonstrations hold, financial sentiment classification can be improved without fine-tuning by steering a single credit-risk feature, reducing the need for labeled retraining.
  • Feature-activation levels can serve as a hallucination alarm: when finance-related feature activation falls below a threshold, triggering retrieval-augmented generation grounds the answer in cited sources.
  • Bias detection becomes a quantitative flag: loan-application text whose activated features are dominated by bias-related features can be flagged for review before a decision.
  • Logit lens and attribution patching give model validators a layer-level map of where financial reasoning happens, supporting audit and risk-monitoring narratives.
  • Because steering modifies only activations, the same base model can be redirected for different regulatory or risk contexts without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A production-grade hallucination guard would need to calibrate the illustrative 20% activation threshold against a diverse set of financial questions and show that the two selected features beat random or simpler proxies.
  • The steering result rests on a single auto-interpreted credit-risk feature; whether the improvement generalizes to other features, other models, and other sentiment scales is untested, and the paper itself notes that excessive steering can produce entangled representations.
  • A natural extension the authors leave implicit is pairing SAE feature activations with traditional econometric risk factors, linking neuron-level signals to familiar financial indicators in a hybrid explainable model.
  • The bias-flagging system may inherit the subjectivity of auto-interpretation labels; a stronger design would validate flags against external fairness benchmarks and human review.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper argues that mechanistic interpretability, comprising observation-based methods (logit lens, sparse autoencoder feature discovery) and intervention-based methods (attribution patching, SAE feature steering), can be applied to financial LLM use cases including sentiment analysis, credit-risk assessment, bias detection, hallucination mitigation, and regulatory compliance. It provides background taxonomies, maps interpretability techniques to financial applications, and reports proof-of-concept experiments: logit-lens analysis on two financial prompts, attribution patching on eight prompts, SAE feature labeling and clustering on Gemma-2B, a decision-tree price-movement classifier built on SAE activations, a bias-flagging example, a feature-activation-triggered RAG example for a Form 8-K question, and credit-risk feature steering with confusion matrices. The abstract claims that this is the first finance-domain application of mechanistic interpretability.

Significance. If the central demonstrations were properly controlled and statistically supported, the paper could serve as a useful bridge between mechanistic interpretability tooling and financial practice. Its strengths include the use of publicly available components (Gemma Scope SAEs, GPT-2, Neuronpedia), a clear observation-versus-intervention framing, and an honest discussion of scalability and validation challenges in Section 7. However, the paper's practical-utility claim rests primarily on Section 6.3.2, where steering SAE feature 471 (labeled "credit risk") is reported to improve sentiment classification; that experiment lacks the controls needed to establish causal specificity. As presented, the manuscript is better described as a practitioner-oriented proof-of-concept than as a validated empirical study, and the novelty claim in the abstract is stronger than the evidence provided.

major comments (4)
  1. [Section 6.3.2, Figure 15] The steering experiment that carries the paper's central claim lacks essential controls. The confusion matrices report raw off-diagonal counts but no row/column totals, per-class precision or recall, aggregate accuracy, or statistical significance. There is no comparison condition in which an unrelated or random SAE feature is steered at the same magnitude, and no no-steer baseline with a matched activation-scale perturbation. Because increasing any SAE latent during the forward pass can shift output probabilities through a generic distributional change, the reported improvement (e.g., "very positive" correct classifications increasing from 654 to 953) could be a steering artifact rather than evidence that the auto-interpreted credit-risk feature is causally specific. The paper itself warns in Section 6.3.1 that excessive steering can entangle representations, which makes the missing control especially consequential. Please add a random/irrelevant-feature control, report variability across repeated runs or across feature-selection choices, and provide significance testing or confidence intervals.
  2. [Section 3.1.3 and Sections 6.3.1-6.3.2] The implementation details needed to reproduce the experiments are absent. The text states "see Methodological Details for implementation details," but no such section appears in the manuscript, and the experimental sections do not report the steering magnitude, the layer at which steering is applied, the specific SAE checkpoint for the Gemma-2B model, the number of annotated sentences, or the class distribution behind Figure 15. Without these details, the presented results cannot be checked or reproduced, which materially weakens the claim that feature steering is a practical alternative to prompt engineering and fine-tuning.
  3. [Section 6.2.3, Tables 6-7] The feature-activation-triggered RAG experiment is not validated as a hallucination-reduction method. The 20% trigger threshold is explicitly said to be "chosen for illustration," no validation set is used to show that this threshold separates hallucinated from non-hallucinated outputs, and the demonstration consists of a single Form 8-K question with one comparing table. The two selected features (15857 and 15354) are not shown to generalize across other financial questions, and no comparison is made to alternative detectors (e.g., simple perplexity or retrieval always on). The conclusion that this approach "enhances factual accuracy and reliability" requires a broader evaluation and a threshold analysis.
  4. [Section 6.2.2, Table 5] The bias-detection system is underspecified. The bias ratio is defined only verbally as the percentage of bias-related features among all activated features; the "predefined threshold" that separates flagged from approved applications is never given; and the flagged/approved labels in Table 5 appear to come from the same handful of constructed example sentences used to identify bias-related features, so the evaluation is partly circular. There are no false-positive/false-negative counts and no held-out loan text. Please specify the threshold, define the ratio formally, and evaluate the flagging rule on a held-out set with error rates.
minor comments (6)
  1. [Section 3.3.1] The logit-lens equation uses the final output projection W_out and bias b_out, but the manuscript does not define the intermediate activation h_l or clarify whether the same vocabulary head is used at every layer; please add notation and a one-sentence caution about layer-wise distribution shift.
  2. [Section 5.1, Figures 5-7] The logit-lens analysis is shown for only two hand-picked prompts; reporting the average logit difference over a larger set of financial sentences would make the claimed layer-wise trend more convincing.
  3. [Section 5.2, Figure 8] The color scale in the attribution-patching heatmap is called "normalized logit difference," but the normalization procedure is not defined; please state the exact quantity plotted.
  4. [Section 6.1.1, Table 4] The self-interpretability procedure steers "one feature at a time across all the features available in our SAE," but no details are given on how the finance-related subset in Table 4 was selected or how the LLM-generated descriptions in Table 4 were produced from the steering runs.
  5. [Table 6] The mean activation values for the Gemma-2B SAE features are reported without stating which layer or Gemma Scope checkpoint they come from; please specify the layer and model version so the numbers are reproducible.
  6. [References] Several references are inconsistent or incomplete: the text cites Hinton et al. (2023) for dictionary learning but the reference list entry lacks author names and venue, and there are two Olsson et al. entries with overlapping titles; please reconcile the bibliography.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the paper's claims are empirical demonstrations with externally sourced tools and measured outcomes, not derivations that reduce to their own inputs.

full rationale

The paper does not present a formal derivation chain in which an output is computed from an input; it applies established mechanistic-interpretability tools (logit lens, attribution patching, Gemma Scope SAEs, Neuronpedia) to financial examples and reports measured before/after behavior. The one passage that superficially resembles a self-validating loop is Section 6.3.2, where the authors state: "The experiment makes an assumption that steering a particular feature related to Credit Risk will improve the quality of classification using LLMs." But the assumption is explicitly labeled as an assumption and is then tested empirically via confusion matrices comparing unsteered and steered classifications; the reported improvement is a measured outcome, not a quantity entailed by the assumption. Similarly, the 20% RAG-trigger threshold in Section 6.2.3 is stated as "chosen for illustration, and adjustable based on application needs," so no fitted parameter is being relabeled as a prediction. Feature labels and identities come from external resources (Gemma Scope SAEs and Neuronpedia), not from the paper's own evaluation labels, and the authors cite no prior work of their own as load-bearing support. Any methodological concerns—such as the absence of a non-specific steering control or limited validation of the RAG threshold—are correctness or robustness issues, not circularity. Under the requirement to flag circularity only when a specific reduction can be exhibited, no circular step is present.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central demonstrations depend on pretrained sparse autoencoders from Gemma Scope, hand-chosen thresholds, and representative examples. No new theoretical entity is introduced; the paper's contribution is an application of existing machinery to finance.

free parameters (4)
  • RAG trigger threshold = 20%
    Set in Section 6.2.3 as the fraction of activation for features 15857 and 15354 below which RAG is triggered. The paper says it is 'chosen for illustration' and adjustable, so the hallucination-reduction result depends on this hand-chosen constant.
  • Bias flag threshold = not specified
    In Section 6.2.2, a 'predefined threshold' on the bias ratio decides whether loan text is flagged. The threshold is not reported, yet it determines the flagged/approved labels in Table 5.
  • Steering magnitude = not specified
    In Section 6.3.1, features are steered at 'high magnitude', but the exact delta applied to the SAE latent is not given. The observed sentiment shift from 3 to 4 and the confusion matrix improvements depend on this value.
  • Finance feature subset for hallucination RAG = features 15857, 15354 (two of sixteen top features)
    In Section 6.2.3, the two features used for the activation percentage were chosen from a ranked list by inspecting mean activations and labels; the demonstration would likely differ with a different subset.
assumptions (5)
  • domain assumption Gemma Scope SAE features are faithful, monosemantic decompositions of Gemma-2B/9B activations.
    Sections 6.1 to 6.3 treat the pretrained Gemma Scope dictionaries as ground truth for feature identification and steering, with no verification of feature quality on financial text.
  • domain assumption SAE activation magnitude measures how much the model focuses on a concept.
    Used in Section 6.2 to select features and set thresholds; the paper states this relationship but does not validate it.
  • ad hoc to paper Steering a feature labeled 'credit risk' improves financial sentiment classification.
    Section 6.3.2 explicitly says the experiment 'makes an assumption that steering a particular feature related to Credit Risk will improve the quality of classification'. The utility claim depends on this.
  • domain assumption One Form 8-K question-answer pair is representative of LLM hallucination in financial regulation.
    Section 6.2.3 generalizes from a single example to a claim about improving factual accuracy.
  • standard math The SAE reconstruction and L1 sparsity loss is a valid formulation for feature discovery.
    The loss in Section 3.1 is standard, but the paper does not prove or test the identifiability of the recovered features.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Beyond the Black Box: Interpretability of LLMs in Finance." pith.science (2026). https://pith.science/paper/ZKXTWG4T

@misc{pith2026250524650,
  author       = {Pith},
  title        = {Pith review of: Beyond the Black Box: Interpretability of LLMs in Finance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZKXTWG4T}},
  note         = {Machine review of arXiv:2505.24650}
}
read the original abstract

Large Language Models (LLMs) exhibit remarkable capabilities across a spectrum of tasks in financial services, including report generation, chatbots, sentiment analysis, regulatory compliance, investment advisory, financial knowledge retrieval, and summarization. However, their intrinsic complexity and lack of transparency pose significant challenges, especially in the highly regulated financial sector, where interpretability, fairness, and accountability are critical. As far as we are aware, this paper presents the first application in the finance domain of understanding and utilizing the inner workings of LLMs through mechanistic interpretability, addressing the pressing need for transparency and control in AI systems. Mechanistic interpretability is the most intuitive and transparent way to understand LLM behavior by reverse-engineering their internal workings. By dissecting the activations and circuits within these models, it provides insights into how specific features or components influence predictions - making it possible not only to observe but also to modify model behavior. In this paper, we explore the theoretical aspects of mechanistic interpretability and demonstrate its practical relevance through a range of financial use cases and experiments, including applications in trading strategies, sentiment analysis, bias, and hallucination detection. While not yet widely adopted, mechanistic interpretability is expected to become increasingly vital as adoption of LLMs increases. Advanced interpretability tools can ensure AI systems remain ethical, transparent, and aligned with evolving financial regulations. In this paper, we have put special emphasis on how these techniques can help unlock interpretability requirements for regulatory and compliance purposes - addressing both current needs and anticipating future expectations from financial regulators globally.

Figures

Figures reproduced from arXiv: 2505.24650 by the authors.

Figure 1
Figure 1. Interpretability techniques 2.3 Interpretability In Finance Financial AI models are integral to modern decision-making, driving risk management, fraud detection, credit assessments, and algorithmic trading and many other financial applications. However, their complex￾ity raises concerns about transparency, compliance, and accountability. This section explores how inter￾pretability techniques are applied in finance, … view at source ↗
Figure 2
Figure 2. Sparse Autoencoder Furthermore, the magnitude of feature activation in sparse autoencoders helps identify essential features, and enables performing numerical calculations, aiding in feature selection and statistical analysis for better decision-making. Section 6.1 shows our experiments related to the use of identifying the credit risk and bias using auto￾interpretation of sparse features. 3.1.2 Self-Interpretation … view at source ↗
Figure 3
Figure 3. Interpretability Methods 3.3 Observation-based Methods Observation-based methods in mechanistic interpretability analyze a large language model’s (LLM) internal activations without modifying their parameters. Sparse autoencoder-based feature detection helps reveal hidden patterns, showing how models organize and process information across layers. These techniques identify emerging features and the mechanisms driving… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Attribution patching analysis 3.4.2 Other Intervention Techniques In addition to attribution patching and SAE steering, several alternative intervention techniques exist: • Causal Abstraction: Maps neural computations onto human-understandable structures, making high-l…
Figure 5
Figure 5. Figure 5: Logit Lens heatmap showing predicted token probabilities at each layer for a financial prompt [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Logit difference chart between the words "rise" and "fall" across different layers [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Logit Lens heatmap showing predicted token probabilities at each layer for a financial prompt [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Heatmap visualizing attribution patching over attention heads in the model [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Feature clustering visualization highlighting finance-related topics [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Neuronpedia search results for credit risk related features 6.1.3 Example 2: Bias Related Features As shown in [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]
Figure 11
Figure 11. Figure 11: Neuronpedia search results for gender bias awareness features 6.2 Feature Activation-Based Experimental Results As mentioned earlier, the magnitude of feature activation provides a quantifiable measure of the model’s focus on specific input or output aspects, enabling…
Figure 12
Figure 12. Figure 12: Feature Importance For Price movement [PITH_FULL_IMAGE:figures/full_fig_p019_12.png]
Figure 13
Figure 13. Figure 13: Decision tree visualization for price movement [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]
Figure 14
Figure 14. Figure 14: Feature steering influence on sentiment analysis in financial statements [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Confusion matrices comparing sentiment classification performance before (left) and after (right) [PITH_FULL_IMAGE:figures/full_fig_p024_15.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Emergent Latent-State Computation under Stochastic Volatility

    cs.LG 2026-07 conditional novelty 6.0 of 10

    Volatility forecasters develop linearly decodable representations of the next hidden log-volatility state; in long cycles this appears immediately after the input projection and ℓ2 normalization.

  2. AlphaEval: A Comprehensive and Efficient Evaluation Framework for Formula Alpha Mining

    cs.AI 2025-08 conditional novelty 6.0 of 10

    AlphaEval scores alpha mining models on prediction, stability, robustness, logic, and diversity, replacing backtests with fast parallel metrics that the paper claims align with backtest outcomes.

  3. Why Bonds Fail Differently? Explainable Multimodal Learning for Multi-Class Default Prediction

    q-fin.RM 2025-09 reject novelty 5.0 of 10

    A multimodal deep learning model using financial time series and bond prospectus text predicts three-way bond outcomes (performing, extended, defaulted) and claims better recall and F1 than standard baselines.

Reference graph

Works this paper leans on

30 extracted references · 8 canonical work pages · cited by 3 Pith papers

  1. [6]

    Haozhe Chen, Carl Vondrick, and Chengzhi Mao

    URL https://arxiv.org/abs/1804.06620. Haozhe Chen, Carl Vondrick, and Chengzhi Mao. Selfie: Self-interpretation of large language model embed- dings,

    arXiv

  2. [7]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Jared Kaplan, Andy Chen, Tom Hennigan, Sam McCandlish, and Dario Amodei

    URL https://arxiv.org/abs/2403.10949. Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Jared Kaplan, Andy Chen, Tom Hennigan, Sam McCandlish, and Dario Amodei. A mathematical framework for transformer cir- cuits. Transformer Circuits Thread,

    arXiv

  3. [9]

    URL https://www.researchgate.net/publication/ 390873440_Balancing_Model_Performance_and_Explainability

    doi: 10.1007/s10462-023-10526-5. URL https://www.researchgate.net/publication/ 390873440_Balancing_Model_Performance_and_Explainability. Maarten Grootendorst. Bertopic: Neural topic modeling with a class-based tf-idf procedure,

    work page doi:10.1007/s10462-023-10526-5

  4. [11]

    URL https://arxiv.org/abs/1802.01933

    doi: 10.1007/s10462-018-09632-6. URL https://arxiv.org/abs/1802.01933. Yoshitsugu Hayashi et al. A review of recent studies on the development of smart cities.Journal of Urban Planning and Development, 142(4):04016019,

    arXiv

  5. [12]

    Geoffrey Hinton et al

    ArXiv:2301.04589. Geoffrey Hinton et al. Monosemantic features and dictionary learning.Transformer Circuits,

    arXiv

  6. [13]

    Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda

    URL https://transformer-circuits.pub/2023/monosemantic-features. Tom Lieberum, Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Nicolas Sonnerat, Vikrant Varma, János Kramár, Anca Dragan, Rohin Shah, and Neel Nanda. Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2,

    work page 2023

  7. [14]

    Xudong Lin, Manling Li, Richard Zemel, Heng Ji, and Shih-Fu Chang

    URLhttps://arxiv.org/abs/2408.05147. Xudong Lin, Manling Li, Richard Zemel, Heng Ji, and Shih-Fu Chang. Training-free deep concept injection enables language models for video question answering,

    arXiv

  8. [15]

    emnlp-main.1249/

    URLhttps://aclanthology.org/2024. emnlp-main.1249/. Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. Explainable ai: A review of machine learning interpretability methods. Artificial Intelligence Review, 53:4129–4190,

    work page 2024

Show all 30 references
  1. [16]

    URL https://www.mdpi.com/1099-4300/23/1/18

    doi: 10.1007/s10462-020-09829-5. URL https://www.mdpi.com/1099-4300/23/1/18. Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions.Advances in Neural Information Processing Systems,

    doi

  2. [19]

    Catherine Olsson, Deep Ganguli, et al

    URL https:// arxiv.org/abs/2301.05217. Catherine Olsson, Deep Ganguli, et al. Towards monosemanticity: Decomposing language mod- els with dictionary learning. Anthropic Research,

    arXiv

  3. [20]

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin

    URL https://arxiv.org/ abs/2207.13243. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. Why should i trust you? explaining the predictions of any classifier.ACM SIGKDD,

    arXiv

  4. [21]

    Muhammad Saeed and Christopher Omlin

    URL https://arxiv.org/abs/1602.04938. Muhammad Saeed and Christopher Omlin. A review of explainable ai: From statistical interpretation to ethical ai systems. Artificial Intelligence Review,

    arXiv

  5. [22]

    URL https: //doi.org/10.1007/s10462-022-10128-1

    doi: 10.1007/s10462-022-10128-1. URL https: //doi.org/10.1007/s10462-022-10128-1. J. Sakarvadia et al. Attention lens: Exploring attention maps in transformers. arXiv Preprint,

    doi

  6. [23]

    URL https://arxiv.org/abs/2310.16270. R. Saranya and V. Subhashini. A study on the role of explainability in machine learning models. Soft Computing,

    arXiv

  7. [24]

    URL https://www.sciencedirect.com/science/ article/pii/S277266222300070X

    doi: 10.1007/s00500-022-06883-4. URL https://www.sciencedirect.com/science/ article/pii/S277266222300070X. Ramprasaath R. Selvaraju et al. Grad-cam: Visual explanations from deep networks.IEEE International Conference on Computer Vision (ICCV),

    doi

  8. [25]

    Mukund Sundararajan, Ankur Taly, and Qiqi Yan

    URL https://arxiv.org/abs/1610.02391. Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks.International Conference on Machine Learning (ICML),

    arXiv

  9. [26]

    Gemma Team

    URL https://arxiv.org/abs/1703.01365. Gemma Team. Gemma 2: Improving open language models at a practical size, 2024a. URLhttps://arxiv. org/abs/2408.00118. Neuronpedia Team. Neuronpedia: A platform for mechanistic interpretability research. https://docs. neuronpedia.org/, 2024...

    2024 arXiv

  10. [27]

    Matthew D

    URLhttps://arxiv.org/abs/2401.02602. Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. arXiv Preprint,

    arXiv

  11. [29]

    URLhttps://arxiv.org/abs/ 2006.05525. 27 M. Zolanvari et al. Interpretable deep learning for cyber-physical systems: A model-agnostic perspective. IEEE Access,

    2006 arXiv

  12. [30]

    URL https://doi.org/10.1109/ACCESS

    doi: 10.1109/ACCESS.2021.3071274. URL https://doi.org/10.1109/ACCESS. 2021.3071274. Jurgita Černevičien˙ e and Audrius Kabašinskas. Explainable artificial intelligence (xai) in finance: A system- atic literature review.Artificial Intelligence Review,

    2021

  13. [31]

    doi: 10.1007/s10462-024-10854-8. 28

    doi

  14. [2014]

    John Zhang and Alice Smith

    URL https://arxiv.org/abs/1311.2901. John Zhang and Alice Smith. Understanding transformer circuits via resample ablation. Journal of AI Research, 45:123–145,

    arXiv

  15. [2016]

    Alejandro Barredo Arrieta et al

    URLhttps://arxiv.org/abs/1610.01644. Alejandro Barredo Arrieta et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportuni- ties, and challenges toward responsible ai.Information Fusion,

    arXiv

  16. [2017]

    26 Neel Nanda

    URL https://arxiv.org/abs/1705.07874. 26 Neel Nanda. In-context learning and induction heads in transformers.https://transformer-circuits. pub/2022/in-context-learning-and-induction-heads/,

    2022 arXiv

  17. [2018]

    URLhttps: //doi.org/10.1109/ACCESS.2018.2870052

    doi: 10.1109/ACCESS.2018.2870052. URLhttps: //doi.org/10.1109/ACCESS.2018.2870052. Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv Preprint,

    2018

  18. [2020]

    URL https://doi.org/10.1016/j.inffus.2019.12.012

    doi: 10.1016/j.inffus.2019.12.012. URL https://doi.org/10.1016/j.inffus.2019.12.012. Leonard Bereska and Efstratios Gavves. Mechanistic interpretability for ai safety: A review

    2019 doi

  19. [2021]

    Nelson Elhage et al

    URLhttps://transformer-circuits.pub/2021/framework/ index.html. Nelson Elhage et al. Toy models of superposition.ArXiv Preprint, 2022a. URL https://arxiv.org/abs/ 2209.10652. Nelson Elhage et al. A mechanistic interpretability analysis of superposition.ArXiv Preprint, 2022b. U...

    2021 arXiv

  20. [2022]

    Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi

    URL https://arxiv.org/abs/2203.05794. Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri, Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models.Artificial Intelligence Review, 51(1):93–114,

    arXiv

  21. [2023]

    Giuseppe Casalicchio, Christoph Molnar, and Bernd Bischl

    URL https://transformer-circuits.pub/2023/ monosemantic-features/index.html. Giuseppe Casalicchio, Christoph Molnar, and Bernd Bischl. Visualizing the feature importance for black box models. ArXiv Preprint,

    2023

  22. [2024]

    Retrieved from arXiv

    URL https://arxiv.org/abs/2404.14082. Retrieved from arXiv. 25 Trenton Bricken et al. Towards monosemanticity: Sparse autoencoders for feature disentangle- ment. Transformer Circuits Thread ,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.