REVIEW 30 cited by
Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
The interpretation of deep learning models is a challenge due to their size, complexity, and often opaque internal state. In addition, many systems, such as image classifiers, operate on low-level features rather than high-level concepts. To address these challenges, we introduce Concept Activation Vectors (CAVs), which provide an interpretation of a neural net's internal state in terms of human-friendly concepts. The key idea is to view the high-dimensional internal state of a neural net as an aid, not an obstacle. We show how to use CAVs as part of a technique, Testing with CAVs (TCAV), that uses directional derivatives to quantify the degree to which a user-defined concept is important to a classification result--for example, how sensitive a prediction of "zebra" is to the presence of stripes. Using the domain of image classification as a testing ground, we describe how CAVs may be used to explore hypotheses and generate insights for a standard image classification network as well as a medical application.
Forward citations
Cited by 30 Pith papers
-
When Are Two Networks the Same? Tensor Similarity for Mechanistic Interpretability
Tensor similarity is a symmetry-invariant metric that measures functional equivalence between tensor-based networks using a recursive algorithm for cross-layer mechanisms.
-
From Mechanistic to Compositional Interpretability
The paper introduces compositional interpretability as a category-theoretic framework that casts mechanistic explanations as commuting syntactic-semantic mappings optimized under faithfulness and complexity constraint...
-
Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Race
Alignment on Llama 3 reduces explicit bias but amplifies implicit bias, because aligned models no longer represent 'black' and 'white' as racial concepts in ambiguous contexts.
-
Unifying machine learning and quantum chemistry -- a deep neural network for molecular wavefunctions
Deep neural network predicts molecular wavefunctions in atomic orbital basis from which quantum properties are derived at force-field efficiency.
-
Finding and using interpretable latents in a neutrino foundation model with sparse autoencoders
Sparse autoencoders reveal clean-brightness and auxiliary-activity latents in PolarBERT's CLS representation that are causally inert for direction reconstruction but causally used by an uncertainty head, which improve...
-
CADENCE: A Cardiac Atom Dictionary for Interpretable Neural Concept Extraction from ECG Foundation Models
A sparse dictionary learned from an ECG foundation model's embeddings recovers interpretable cardiac concepts—PVCs, atrial fibrillation, bundle branch blocks, ST/T-wave segments—and transfers to an external dataset wi...
-
CLIF: Concept-Level Influence Functions for Transparent Bottleneck Models
CLIF applies influence functions to pinpoint influential training samples and key concepts in Concept Bottleneck Models, enabling data debugging and behavioral insights on CEBaB and Yelp datasets.
-
Correcting Influence: Unboxing LLM Outputs with Orthogonal Latent Spaces
A latent mediation framework with sparse autoencoders enables non-additive token-level influence attribution in LLMs by learning orthogonal features and back-propagating attributions.
-
UNBOX: Unveiling Black-box visual models with Natural-language
UNBOX recovers interpretable text concepts that maximally activate classes in black-box vision models by recasting activation maximization as semantic search with LLMs and diffusion models.
-
Learning Encoding-Decoding Direction Pairs to Unveil Concepts of Influence in Deep Vision Networks
An unsupervised method, EDDP, jointly learns encoding-decoding direction pairs for concepts in CNN latent spaces, recovering interpretable and influential concepts without labels, validated on synthetic and real data.
-
Concept Based Explanations and Class Contrasting
A concept-based explanation method filters hidden-layer activations by attribution, decomposes them with NMF, and validates by recombining concept crops, plus a class-contrast variant using a linear classifier.
-
Concept Boundary Vectors
Concept boundary vectors are derived from the boundary between latent concept clusters, and the paper reports they capture semantic relationships better than concept activation vectors.
-
Latent Fact-Checking: Detecting Misinformation through Activation Engineering
A projection onto a contrastive falsehood direction in frozen LLM activations, followed by a small MLP, outperforms zero-shot and few-shot prompting on LIAR and FACTors and supports the claim that truthfulness is line...
-
Runtime Uncertainty Monitoring for LLM-Based Multi-Agent Systems Using Bayesian Networks
A Bayesian-network monitor built on calibrated LLM log-probabilities gives workflow-level uncertainty scores for an actuarial multi-agent system, reproducing baseline RMSE but not clearly separating normal from pertur...
-
Steering LLMs? Actually, Sparse Autoencoders can outperform simple baselines
Sparse autoencoders with supervised feature selection achieve near-LoRA performance on the AxBench steering benchmark and identify causal features.
-
From Features to Actions: Explainability in Traditional and Agentic AI Systems
Attribution explanations that work for static classifiers do not diagnose failures in multi-step AI agents; trace-grounded rubric evaluation does, with state-tracking inconsistency 2.7x more common in failed agent runs.
-
A Concept-based approach to Voice Disorder Detection
Concept bottleneck and concept embedding models, trained on clinical concepts extracted from patient notes by a large language model, detect voice pathology from audio almost as accurately as an end-to-end transformer.
-
Wanting to Be Understood Explains the Meta-Problem of Consciousness
A social motivation to be understood, combined with the severe bandwidth limit of language, explains why conscious experience feels ineffable and why the hard problem of consciousness persists.
-
Towards a Science of Causal Interpretability in Deep Learning for Software Engineering
The dissertation presents docode, a causal interpretability method for neural code models, and uses a case study to show that some correlations between code properties and model performance are confounded rather than causal.
-
"All that Glitters": Approaches to Evaluations with Unreliable Model and Human Annotations
Encoder models trained on noisy classroom ratings look super-human under standard concordance metrics, but generalizability, disattenuation, and hierarchical rater analyses show the apparent advantage is partly spurio...
-
Generative Counterfactual Introspection for Explainable Deep Learning
A generative-model-driven introspection method produces counterfactual image edits to explain deep neural network predictions on MNIST and CelebA.
-
Towards explainable decision support using hybrid neural models for logistic terminal automation
The paper proposes a three-stage Interpretable Neural System Dynamics pipeline for interpretable-by-design decision support in intermodal logistics, but provides no validation.
-
Concept-Based Mechanistic Interpretability Using Structured Knowledge Graphs
BAGEL trains per-layer logistic-regression probes on CLIP-defined concepts and compares per-class concept probabilities with dataset-level concept frequencies, visualizing the alignment in a knowledge graph.
-
Perspectives for Direct Interpretability in Multi-Agent Deep Reinforcement Learning
A perspective paper that advocates direct, post hoc interpretability for multi-agent deep reinforcement learning and offers a taxonomy of where those methods might apply.
-
Explaining Model Overfitting in CNNs via GMM Clustering
A CNN filter whose feature-map GMM clustering has small outlier clusters is called an anomaly filter and is claimed to indicate model overfitting, but the supporting experiments are weakly consistent.
-
New Faithfulness-Centric Interpretability Paradigms for Natural Language Processing
The thesis shows that randomly masking input tokens during fine-tuning makes post-hoc explanations of NLP models consistently faithful under an erasure-based faithfulness metric.
-
FastCAV: Efficient Computation of Concept Activation Vectors for Explaining Deep Neural Networks
Concept activation vectors can be computed as the normalized difference between concept-mean and global-mean activations, giving a 46.4x average speedup over SVM-based CAVs with comparable quality.
-
Explainable Artificial Intelligence Techniques for Interpretation of Food Models: a Review
A survey proposing a taxonomy of XAI techniques for food quality research organized by data types and explanation methods.
-
Unexplainability and Incomprehensibility of Artificial Intelligence
Advanced AI systems are unexplainable in full and produce explanations that humans cannot comprehend.
-
Platonic Projection Structures: Operator-Induced Observability in Representation Learning
The paper introduces 'Platonic Projection Structures,' a reformulation of standard PSD operator theory applied to representation learning, with experiments that verify definitions rather than test predictions.
Discussion (0). Continue with ORCID to comment.