A Causal Lens for Peeking into Black Box Predictive Models: Predictive Model Interpretation via Causal Attribution

Aria Khademi; Vasant Honavar

RECORD

A Causal Lens for Peeking into Black Box Predictive Models: Predictive Model Interpretation via Causal Attribution

Aria Khademi, Vasant Honavar

Machine Learning · Submitted 1 Aug 2020

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim We present results of experiments that demonstrate the effectiveness of our approach to the interpretation of black box predictive models via causal attribution in the case of deep neural network models trained on one synthetic data set (where the input variables that impact the output variable are known by design) and two real-world data sets: Handwritten digit classification, and Parkinson's disease severity prediction.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2008.00357 (version 1)

Pith IDXAK6TX3O, Pith's permanent record of this paper and its review

CitationAria Khademi, Vasant Honavar. "A Causal Lens for Peeking into Black Box Predictive Models: Predictive Model Interpretation via Causal Attribution." arXiv:2008.00357 (2020). https://pith.science/paper/XAK6TX3O

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,912 input tokens.

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.51 · human participants 0.46 · controlled comparison 0.09 · randomized assignment 0.02 · states quantitative result 0.08 · compares with prior results 0.09 · mentions code or data availability 0.02.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

With the increasing adoption of predictive models trained using machine learning across a wide range of high-stakes applications, e.g., health care, security, criminal justice, finance, and education, there is a growing need for effective techniques for explaining such models and their predictions. We aim to address this problem in settings where the predictive model is a black box; That is, we can only observe the response of the model to various inputs, but have no knowledge about the internal structure of the predictive model, its parameters, the objective function, and the algorithm used to optimize the model. We reduce the problem of interpreting a black box predictive model to that of estimating the causal effects of each of the model inputs on the model output, from observations of the model inputs and the corresponding outputs. We estimate the causal effects of model inputs on model output using variants of the Rubin Neyman potential outcomes framework for estimating causal effects from observational data. We show how the resulting causal attribution of responsibility for model output to the different model inputs can be used to interpret the predictive model and to explain its predictions. We present results of experiments that demonstrate the effectiveness of our approach to the interpretation of black box predictive models via causal attribution in the case of deep neural network models trained on one synthetic data set (where the input variables that impact the output variable are known by design) and two real-world data sets: Handwritten digit classification, and Parkinson's disease severity prediction. Because our approach does not require knowledge about the predictive model algorithm and is free of assumptions regarding the black box predictive model except that its input-output responses be observable, it can be applied, in principle, to any black box predictive model.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	With the increasing adoption of predictive models trained using machine learning across a wide range of high-stakes applications, e.g., health care, security, criminal justice, finance, and education, there is a growing need for effective techniques for explaining such models and their predictions.	background 1.00	0.04	0.03	asserted 0.99
s1	We aim to address this problem in settings where the predictive model is a black box; That is, we can only observe the response of the model to various inputs, but have no knowledge about the internal structure of the predictive model, its parameters, the objective function, and the algorithm used to optimize the model.	background 0.56	0.23	0.03	asserted 0.92
s2	We reduce the problem of interpreting a black box predictive model to that of estimating the causal effects of each of the model inputs on the model output, from observations of the model inputs and the corresponding outputs.	method 0.72	0.90	0.23	asserted 1.00
s3	We estimate the causal effects of model inputs on model output using variants of the Rubin Neyman potential outcomes framework for estimating causal effects from observational data.	method 0.99	0.93	0.07	asserted 1.00
s4	We show how the resulting causal attribution of responsibility for model output to the different model inputs can be used to interpret the predictive model and to explain its predictions.	contribution 0.43	0.87	0.38	asserted 0.91
s5	We present results of experiments that demonstrate the effectiveness of our approach to the interpretation of black box predictive models via causal attribution in the case of deep neural network models trained on one synthetic data set (where the input variables that impact the output variable are known by design) and two real-world data sets: Handwritten digit classification, and Parkinson's disease severity prediction.	result 0.85	0.97	0.30	asserted 1.00
s6	Because our approach does not require knowledge about the predictive model algorithm and is free of assumptions regarding the black box predictive model except that its input-output responses be observable, it can be applied, in principle, to any black box predictive model.	interpretation 0.81	0.44	0.25	hedged possibility 0.73

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.89.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

With the increasing adoption of predictive models trained using machine learning across a wide range of high-stakes applications, e.g., health care, security, criminal justice, finance, and education, there is a growing need for effective techniques for explaining such models and their predictions. We aim to address this problem in settings where the predictive model is a black box; That is, we can only observe the response of the model to various inputs, but have no knowledge about the internal structure of the predictive model, its parameters, the objective function, and the algorithm used to optimize the model. We reduce the problem of interpreting a black box predictive model to that of estimating the causal effects of each of the model inputs on the model output, from observations of the model inputs and the corresponding outputs. We estimate the causal effects of model inputs on model output using variants of the Rubin Neyman potential outcomes framework for estimating causal effects from observational data. We show how the resulting causal attribution of responsibility for model output to the different model inputs can be used to interpret the predictive model and to explain its predictions. We present results of experiments that demonstrate the effectiveness of our approach to the interpretation of black box predictive models via causal attribution in the case of deep neural network models trained on one synthetic data set (where the input variables that impact the output variable are known by design) and two real-world data sets: Handwritten digit classification, and Parkinson's disease severity prediction. Because our approach does not require knowledge about the predictive model algorithm and is free of assumptions regarding the black box predictive model except that its input-output responses be observable, it can be applied, in principle, to any black box predictive model.

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

5 arXiv papers cite this in 7 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 3 sentences without a confident label

Cites it as background 3 papers

Although promising, most of these works still focus on feature-level explainability, such as feature attributions [ref] , feature-based counterfactual explanations [ref] , and causally induced model structure [ref] .

DiConStruct: Causal Concept-based Explanations through Black-Box Distillation · Introduction · 0.92
Khademi and Honavar [ref] compare different methods of causal effect estimation such as CBPS and NPCBPS to interpret predictive models and explain their prediction based on inputs average causal effect (ACE).

Explaining Image Classifiers Using Contrastive Counterfactuals in Generative Latent Spaces · Related works · 0.92
There has been much recent work on methods for explaining black box predictive models (reviewed in [ref] , typically, by attributing the responsibility for the model's predictions to different input features.

Explainable Multivariate Time Series Classification: A Deep Neural Network Which Learns To Attend To Important Variables As Well As Informative Time Intervals · Related Work · 0.99

What it cites · 9 sentences without a confident label

Uses the method or tool of 2 papers

To cope with this problem, we can use modern representation learning techniques [ref] to map the data to a low-dimensional latent space before estimating the counterfactual outcomes [ref] , or employ targeted maximum likelihood methods (TMLE) [ref] for counterfactual inference from observational data.

Perfect Match: A Simple Method for Learning Representations For Counterfactual Inference With Neural Networks · Estimating Causal Effects of Discrete Treatments · 0.83
To cope with this problem, we can use modern representation learning techniques [ref] to map the data to a low-dimensional latent space before estimating the counterfactual outcomes [ref] , or employ targeted maximum likelihood methods (TMLE) [ref] for counterfactual inference from observational data.

Recent Advances in Autoencoder-Based Representation Learning · Estimating Causal Effects of Discrete Treatments · 0.83

Cites as support 2 papers

As has been pointed out, they often fail to generate reliable attributions [ref] , let alone interpretations that support explanations [ref] .

Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI · Model Explanation and Interpretation · 0.99
Recent work has pointed out that mathematical problems arise when Shapley values are used for feature importance and that the solutions to mitigate these necessarily induce further complexity [ref] .

Problems with Shapley-value-based explanations as feature importance measures · Related Work · 0.96

Cites as background 7 papers

Hence, a large body of work has focused on methods for interpretation of black box predictive models (reviewed in [ref] ), also known as posthoc interpretations [ref] .

The Mythos of Model Interpretability · Model Explanation and Interpretation · 0.91
Hence, a large body of work has focused on methods for interpretation of black box predictive models (reviewed in [ref] ), also known as posthoc interpretations [ref] .

Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI · Model Explanation and Interpretation · 0.91
Such methods include techniques for visualizing the effect of the model inputs on its outputs [ref] , methods for extracting purportedly human interpretable rules from black box models [ref] , feature scoring methods that assess the importance of individual features on the prediction [ref] , gradient based methods that assess how changes in inputs impact the model predictions [ref] , and techniques for approximating local decision surfaces in the neighborhood of the input sample via localized regression [ref] .

Not Just a Black Box: Learning Important Features Through Propagating Activation Differences · Related Work · 0.90

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.