Inference on Variable Importance for Treatment Effect Heterogeneity: Shapley Values and Beyond

Alex Luedtke; Pawel Morzywolek; Peter B. Gilbert

arxiv: 2510.18843 · v2 · submitted 2025-10-21 · 📊 stat.ME · math.ST· stat.ML· stat.TH

Inference on Variable Importance for Treatment Effect Heterogeneity: Shapley Values and Beyond

Pawel Morzywolek , Peter B. Gilbert , Alex Luedtke This is my paper

Pith reviewed 2026-05-18 04:22 UTC · model grok-4.3

classification 📊 stat.ME math.STstat.MLstat.TH

keywords variable importancetreatment effect heterogeneityShapley valuessemiparametric inferencemachine learningcausal inference

0 comments

The pith

Researchers can now perform statistical inference on which variables matter for varying treatment effects across individuals.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper develops an inferential framework for measuring and testing the importance of variables in heterogeneous treatment effects. The measures can vary from person to person, but the tests determine if a variable plays a role for at least one person in the population. This is particularly relevant in medicine where black-box algorithms for treatment recommendations need explainability. The framework relies on semiparametric theory to remain valid when flexible machine learning methods are used to estimate the treatment effects. An application to infectious disease prevention illustrates the approach.

Core claim

The central discovery is a procedure for conducting valid hypothesis tests and constructing confidence intervals for local variable importance measures of the heterogeneous treatment effect function, where the importance is quantified using Shapley values or similar, and the procedure is asymptotically valid under standard conditions even if machine learning estimators are plugged in for the nuisance parameters.

What carries the argument

The key machinery is the application of semiparametric efficiency theory to function-valued parameters, specifically deriving influence functions for the variable importance functional applied to the conditional average treatment effect.

If this is right

Clinicians can test whether a particular covariate influences treatment effect heterogeneity for any patient.
The method provides uncertainty quantification for explainability in personalized medicine.
It extends beyond point estimates to allow rigorous statistical conclusions about variable roles.
Demonstrated utility in infectious disease contexts suggests broader applicability in public health.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This framework might be adapted to assess variable importance in other causal estimands, such as mediation effects.
Future work could explore finite-sample properties through targeted simulations.
It connects naturally to the growing literature on interpretable causal machine learning.

Load-bearing premise

The recent developments in semiparametric theory for function-valued parameters directly yield valid inference procedures when machine learning is used to estimate the heterogeneous treatment effects.

What would settle it

Finding that the proposed tests have incorrect type I error rates in a Monte Carlo simulation where the treatment effect heterogeneity is generated from a known model and machine learning estimators are used.

read the original abstract

We provide an inferential framework to assess variable importance for heterogeneous treatment effects. This assessment is especially useful in high-risk domains such as medicine, where decision makers hesitate to rely on black-box treatment recommendation algorithms. The variable importance measures we consider are local in that they may differ across individuals, while the inference is global in that it tests whether a given variable is important for any individual. Our approach builds on recent developments in semiparametric theory for function-valued parameters, and is valid even when statistical machine learning algorithms are employed to quantify treatment effect heterogeneity. We demonstrate the applicability of our method to infectious disease prevention strategies.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This paper gives a semiparametric way to test variable importance for heterogeneous treatment effects that stays valid with ML estimators, but the transfer of the required rate conditions is not obviously verified.

read the letter

The main thing to know is that the authors build an inferential procedure for local variable importance measures, such as Shapley values, in the setting of heterogeneous treatment effects. The test itself is global: it checks whether a covariate matters for any individual. They claim the whole thing remains valid when machine learning is used to estimate the conditional average treatment effect or the necessary nuisances, and they illustrate it on infectious disease data. That combination is the concrete advance over prior work on Shapley values or semiparametric functionals. It directly addresses a practical worry in medicine, where people want to know which variables actually drive personalized recommendations rather than just trusting a black box. The demonstration helps show the method is usable. The soft spot is the reliance on recent semiparametric results for function-valued parameters. Those results typically need the nuisance estimators to satisfy rate or entropy conditions, and many off-the-shelf ML methods for HTE do not meet them without extra structure like sample splitting. The abstract invokes the general theory but does not spell out how the conditions are checked or relaxed in this specific functional. If the full paper only cites the prior results without deriving the needed bounds for the variable-importance map, the validity claim for arbitrary black-box estimators is weaker than it appears. The citation pattern looks standard and the logic is internally consistent. This is for statisticians and causal-inference researchers who work on interpretable ML for treatment effects. A reader who already follows the semiparametric literature on function-valued parameters will get the most out of it. The paper is coherent enough on its own terms to deserve a serious referee who can check the derivations and any supporting simulations.

Referee Report

2 major / 2 minor

Summary. The paper develops an inferential framework for assessing variable importance (including Shapley values) in heterogeneous treatment effects. The measures are local (individual-specific) while inference is global (testing importance for any individual). The approach relies on recent semiparametric theory for function-valued parameters and claims validity when machine learning estimators are used for the conditional average treatment effect, with an application to infectious disease prevention strategies.

Significance. If the validity claims hold under the stated conditions, the work would offer a practically useful tool for interpreting black-box ML models in high-stakes settings such as medicine, by providing inferential guarantees on which variables drive treatment effect heterogeneity. The explicit connection to function-valued semiparametric efficiency theory and the empirical demonstration add relevance for applied researchers seeking trustworthy HTE analysis.

major comments (2)

[§4] §4 (theoretical results on influence functions for function-valued parameters): The central validity claim for arbitrary ML estimators of treatment effect heterogeneity rests on the transfer of semiparametric results without an explicit check that the chosen ML nuisance estimators satisfy the required rate (e.g., o_p(n^{-1/4})) or entropy conditions in the HTE setting. This is load-bearing because violation of these conditions would invalidate the asymptotic normality of the proposed test statistics.
[§5.1] §5.1 (application to infectious disease data): The empirical results use specific ML methods (e.g., random forests or neural nets) for CATE estimation, but no diagnostic or cross-validation evidence is provided to confirm that these estimators meet the Donsker or rate conditions invoked in the theory section; this weakens the demonstration that the framework remains valid in practice for black-box algorithms.

minor comments (2)

[§2] Notation for the variable importance functional (e.g., the definition of the Shapley value operator) could be introduced earlier and with a clearer link to the function-valued parameter space to improve readability for readers unfamiliar with Banach-space semiparametrics.
The abstract states validity 'even when statistical machine learning algorithms are employed' but does not mention the auxiliary conditions (sample splitting, rate requirements) that are standard in the cited semiparametric literature; a brief qualifier would help set expectations.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thoughtful comments, which help clarify the scope and practical applicability of our inferential framework. We address the major comments point by point below, indicating revisions where the manuscript will be updated.

read point-by-point responses

Referee: [§4] §4 (theoretical results on influence functions for function-valued parameters): The central validity claim for arbitrary ML estimators of treatment effect heterogeneity rests on the transfer of semiparametric results without an explicit check that the chosen ML nuisance estimators satisfy the required rate (e.g., o_p(n^{-1/4})) or entropy conditions in the HTE setting. This is load-bearing because violation of these conditions would invalidate the asymptotic normality of the proposed test statistics.

Authors: We appreciate this observation. Our results rely on the general semiparametric efficiency theory for function-valued parameters, which requires nuisance estimators (for the CATE and other components) to satisfy rate conditions such as o_p(n^{-1/4}) and appropriate entropy bounds to ensure asymptotic normality. The manuscript invokes these as standard assumptions from the cited semiparametric literature but does not provide an HTE-specific verification. In revision, we will add a dedicated paragraph in Section 4 explicitly stating the conditions, discussing how they can be met by common ML estimators for CATE (e.g., via cross-fitting and regularization as in double machine learning), and noting that validity is conditional on these rates being achieved. This clarifies the framework without changing its core claims. revision: yes
Referee: [§5.1] §5.1 (application to infectious disease data): The empirical results use specific ML methods (e.g., random forests or neural nets) for CATE estimation, but no diagnostic or cross-validation evidence is provided to confirm that these estimators meet the Donsker or rate conditions invoked in the theory section; this weakens the demonstration that the framework remains valid in practice for black-box algorithms.

Authors: We agree that additional evidence would strengthen the empirical section. The original application assumes the ML estimators satisfy the theory's conditions but provides no explicit diagnostics. In the revised manuscript, we will include cross-validation results, out-of-sample performance metrics, and approximate rate checks (via sample splitting) for the random forest and neural net CATE estimators in Section 5.1. These additions will support that the conditions hold in the infectious disease dataset, better demonstrating practical validity for black-box methods. revision: yes

Circularity Check

0 steps flagged

Framework extends external semiparametric theory; no derivation reduces to self-fit or self-citation by construction

full rationale

The paper's central claim is an inferential framework for variable importance in heterogeneous treatment effects that remains valid under machine learning nuisance estimators. This rests on cited developments in semiparametric theory for function-valued parameters rather than deriving those results internally or re-labeling fitted quantities as predictions. No equations or sections in the provided abstract and context show a self-definitional loop, a fitted input renamed as a prediction, or a load-bearing uniqueness theorem imported solely via self-citation. The approach is therefore self-contained against external benchmarks, yielding only a minor score for routine citation of prior semiparametric work.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Based on abstract only; limited visibility into parameters or assumptions.

axioms (1)

domain assumption Recent developments in semiparametric theory for function-valued parameters apply and yield valid inference here.
The approach explicitly builds on these developments as stated in the abstract.

pith-pipeline@v0.9.0 · 5642 in / 1040 out tokens · 39466 ms · 2026-05-18T04:22:52.309195+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Our approach builds on recent developments in semiparametric theory for function-valued parameters... valid even when statistical machine learning algorithms are employed
IndisputableMonolith/Foundation/AlexanderDuality.lean alexander_duality_circle_linking unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

embed the parameter in an RKHS... γK_ω(P)(x) := ∫ K(x,x′) γ_ω(P)(x′) P_X(dx′)

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

ConfoundingSHAP: Quantifying confounding strength in causal inference
cs.LG 2026-05 unverdicted novelty 7.0

ConfoundingSHAP defines a custom Shapley game to attribute confounding strength to individual covariates and uses TabPFN to estimate it scalably without exhaustive refitting.