REVIEW 5 major objections 6 minor 40 references
Removing Spurious Correlation from Neural Network Interpretations
T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that conversation topic confounds neuron-level attribution of toxicity in large language models, making toxic behavior look more localized than it is, and that entropy-balancing adjustment spreads the attribution across…
desk verdict Promising causal re-framing of neuron attribution, but the central de-localization claim needs error bars, positivity diagnostics, and validation before it can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Average Indirect Effect (AIE) estimator, a causal mediation quantity that measures how much of a query's effect on toxicity flows through a specific internal node while holding the topic fixed. It is computed with entropy balancing weights that reweight queries so that topic and node activations are balanced, and with a pairwise estimator that avoids generating out-of-distribution inputs. The cancellation of softmax normalization terms in the entropy balancing ratios lets the method compute propensity ratios without retraining, requiring only a single forward pass.
What would settle it
Use a synthetic dataset where a known topic confounder mediates a known fraction of toxicity through a known subset of units; if the estimator does not recover that ground-truth localization, the causal claim fails. Alternatively, if a finer-grained topic model (e.g., 20 clusters or a continuous topic model) reduces or reverses the reported de-localization in Bloomz or Phi-3-mini, the confounding adjustment is not capturing the true confounder.
Extended reading notes
Core claim
The central claim is that confounding by conversation topic creates spurious correlations in neuron-level attribution of toxicity, and that controlling for topic via entropy balancing changes the estimated contribution of each MLP to toxic generations. Using the DAG where topic $x$ affects both query $q$ and harmfulness $y$, and the query affects mediator $n$ (node activation), the paper estimates the Average Indirect Effect through each MLP with stabilized weights computed by entropy balancing, avoiding out-of-distribution counterfactuals by pairwise comparisons. In experiments with Bloomz 1B7 and Phi-3-mini 3B on RealToxicityPrompts, the normalized contributions to toxicity become more distributed across MLPs, i.e., toxicity is less localized, when topic adjustment is applied.
Load-bearing premise
The attribution is only unconfounded if the three k-means topic clusters capture everything about conversation topic that influences both toxicity and node activations, and if every question has positive probability under every node-activation and topic combination, conditions the paper itself notes may fail.
Editorial extensions
If this is right
- Attribution maps for toxicity in LLMs should be re-examined with confounder adjustment to avoid overstating localization.
- Neuron-editing or unlearning methods that rely on unadjusted attribution may suppress topic-general units rather than toxicity-specific ones.
- The same DAG and estimator can be applied to other confounded behaviors such as gender bias, hallucination, and factuality, as the paper notes.
- Because the estimator requires only one forward pass, confounder-adjusted attribution is feasible at scale for large models.
Reading between the lines
- If the de-localization holds under finer topic models, the localization hypothesis itself may need to be reformulated as conditional on unconfoundedness rather than a static property of the model.
- The pairwise AIE estimator could be adapted to other continuous mediators such as attention heads or feature circuits, not just MLP units.
- A natural stress test is to introduce a synthetic confounder with known ground-truth mediation, then check whether the estimator recovers the correct ranking of units.
- Because the paper acknowledges the positivity assumption may be violated by deterministic node activations, overlap diagnostics for the propensity weights would make the causal claim stronger.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a causal mediation approach to remove the effect of conversation topic when attributing toxic behavior to individual MLP units in large language models. The authors introduce a DAG in which topic x confounds the query q and the toxicity outcome y, with node activations n as a mediator, and use entropy balancing to estimate the Average Indirect Effect (AIE) for each MLP. Experiments with Bloomz 1B7 and Phi-3-mini 3B on RealToxicityPrompts show that after adjusting for topic, toxicity contributions become more distributed across MLPs. The paper does not provide code or data.
Significance. If the central claim is correct, the paper offers a principled way to correct neuron attribution for a common confounder, with direct implications for mechanistic interpretability and safety editing. The adaptation of entropy balancing to textual treatments and the focus on the localization hypothesis are timely. However, the empirical support is currently weak: the key estimator is not derived, the positivity assumption is acknowledged as possibly violated but never verified, the main results are presented without uncertainty quantification, and the topic confounder is estimated from the same queries used in the analysis. These issues prevent the paper from being accepted in its present form, though the core idea may be salvageable with additional diagnostics and analyses.
major comments (5)
- [Section 3, Eq. (2)] The AIE estimator in Eq. (2) is introduced without a formal derivation. The paper first defines the population quantity μ(q,q') in Eq. (1) and then states an empirical average over pairs, but Eq. (2) replaces the conditional expectation in Eq. (1) with the single observation y_i. As written, it is unclear why this plug-in estimator is unbiased or consistent for the stated causal contrast. The authors should provide a derivation, or at least a clear argument that the weighted sum over observed units estimates the population AIE.
- [Section 3, 'A Note on Positivity Assumption'] The paper acknowledges that f(q|n,x)>0 may fail because node activations are deterministic functions of the query, and it proposes PCA dimensionality reduction as a mitigation. However, no overlap diagnostics, weight distributions, or effective sample sizes are reported. Without evidence that the entropy balancing weights are stable and that the denominator terms in Eq. (2) are bounded away from zero, the observed flattening of the De-biased curve in Figure 2 could be an artifact of extreme reweighting, even after Winsorization.
- [Section 4, Figure 2] The main empirical claim that toxicity becomes less localized rests solely on visual comparison of two curves without error bars, confidence intervals, or significance tests. The authors should provide bootstrap confidence intervals that reflect both the stochastic approximation (K=200) and the entropy balancing estimation, and report a quantitative localization measure (e.g., a concentration index or slope) with uncertainty.
- [Section 4, topic construction] The confounder x is defined by k-means clustering on RoBERTa embeddings of the same queries, with k=3 fixed. Because x is derived from the treatment q, the DAG in Figure 1 (x -> q) is not directly satisfied, and the strong ignorability assumption depends on the cluster solution being a valid proxy for the true topic confounder. The paper should include sensitivity analyses over the number of clusters and the PCA dimension, and demonstrate that the main conclusion is robust to these choices.
- [Section 4, baseline definition] The 'Normal' baseline is described as obtained by Eq. (1) without confounding factors, but the exact formula is not given. To make the comparison reproducible, the authors should specify precisely how this baseline is computed, including the form of the propensity weights used when x is omitted.
minor comments (6)
- [Section 2.1] The heading 'Backround' should be 'Background'.
- [Section 2] The notation for random variables and vector values is introduced but not used consistently; for example, bold x appears both as a random vector and as a value, which can confuse the exposition.
- [Section 3, Eq. (1)-(2)] Eq. (1) uses δ = q'−q, while Eq. (2) uses the pair notation (q_i, q_j); the relation between q' and q_j should be stated explicitly.
- [Section 3, Algorithm 1] The stochastic approximation with K=200 is not justified; the authors should report sensitivity to this choice.
- [Appendix, Figure 3] Figure 3 is referenced as being in the appendix but appears after the references without a caption or explanation; the t-SNE plot and cluster labels should be described.
- [General] No code or data are provided, which limits the reproducibility of the empirical claims.
Circularity Check
No significant circularity: the de-localization finding is an empirical consequence of reweighting observed attributions, not an assumption encoded in the estimator.
full rationale
The paper's central claim is that adjusting for conversation topic makes toxicity attributions less localized. The AIE estimator in Eq. (2) is a weighted average over observed query pairs, with entropy-balancing weights fitted to balance treatment/confounder moments. The target quantity is not defined in terms of the localization conclusion, and the balance constraints (Gπ = 0) do not enforce any particular ordering or slope of the MLP-by-MLP AIE values. The 'Normal' baseline is the no-confounder special case of Eq. (1); the 'De-biased' estimate adds EB weights for [n,x]. The difference between the two curves is therefore not forced by construction. The topic variable is constructed from query embeddings via k-means, which is a measurement choice rather than a case of defining the confounder in terms of the outcome; weaknesses in that operationalization (k=3, PCA dimension, no overlap diagnostics) are validity concerns, not circularity. The paper explicitly acknowledges the positivity assumption may fail and defers node-masking validation to future work; these are honest limitation statements, not evidence that the result reduces to its inputs. The only self-citation, [Bahadori et al., 2022], supplies a general-purpose entropy-balancing method from prior published work and does not assume the present conclusion, so it is not load-bearing in a circular sense. The finding may be fragile or hard to validate, but no specific derivation step is equivalent to its own inputs.
Assumptions & free parameters
free parameters (5)
- Number of topic clusters =
3
- PCA reduced dimension =
25
- Entropy balancing regularization gamma =
not reported
- Stochastic approximation sample size K =
200
- Winsorization percentile =
5%
assumptions (6)
- domain assumption Strong ignorability: no hidden confounders for the effect of question q on toxicity y given topic x.
- domain assumption Positivity: f(q|n,x) > 0 for all q, n, x in the domain.
- ad hoc to paper The DAG in Figure 1 is the correct causal structure: topic x affects query q and harmfulness y, and query q affects node n and y.
- standard math The entropy balancing weights provide valid estimates of the generalized propensity score ratios.
- ad hoc to paper k-means clusters on RoBERTa embeddings capture the conversation topic confounder.
- domain assumption GPT-4 toxicity labels are accurate for the outcome.
Cite this review
Pith. "Pith review of Removing Spurious Correlation from Neural Network Interpretations." pith.science (2026). https://pith.science/paper/RKGRAZWW
@misc{pith2026241202893,
author = {Pith},
title = {Pith review of: Removing Spurious Correlation from Neural Network Interpretations},
year = {2026},
howpublished = {\url{https://pith.science/paper/RKGRAZWW}},
note = {Machine review of arXiv:2412.02893}
}
read the original abstract
The existing algorithms for identification of neurons responsible for undesired and harmful behaviors do not consider the effects of confounders such as topic of the conversation. In this work, we show that confounders can create spurious correlations and propose a new causal mediation approach that controls the impact of the topic. In experiments with two large language models, we study the localization hypothesis and show that adjusting for the effect of conversation topic, toxicity becomes less localized.
Figures
Reference graph
Works this paper leans on
-
[1]
Phi-3 technical report: A highly capable language model locally on your phone
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv:2404.14219, 2024
arXiv 2024
-
[2]
End-to-end balancing for causal continuous treatment-effect estimation
Taha Bahadori, Eric Tchetgen Tchetgen, and David Heckerman. End-to-end balancing for causal continuous treatment-effect estimation. In ICML, pages 1313--1326. PMLR, 2022
work page 2022
-
[3]
Doubly robust estimation in missing data and causal inference models
Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61 0 (4): 0 962--973, 2005
2005
-
[4]
Mechanistic Interpretability for AI Safety--A Review
Leonard Bereska and Efstratios Gavves. Mechanistic Interpretability for AI Safety--A Review . arXiv:2404.14082, 2024
arXiv 2024
-
[5]
Double/debiased machine learning for treatment and structural parameters
Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal, 21 0 (1): 0 C1--C68, 2018
2018
-
[6]
Applied Causal Inference Powered by ML and AI
Victor Chernozhukov, Christian Hansen, Nathan Kallus, Martin Spindler, and Vasilis Syrgkanis. Applied Causal Inference Powered by ML and AI. https://causalml-book.org/, 2024
work page 2024
-
[7]
Constructing inverse probability weights for marginal structural models
Stephen R Cole and Miguel A Hern \'a n. Constructing inverse probability weights for marginal structural models. American journal of epidemiology, 168 0 (6): 0 656--664, 2008
work page 2008
-
[8]
Dealing with limited overlap in estimation of average treatment effects
Richard K Crump, V Joseph Hotz, Guido W Imbens, and Oscar A Mitnik. Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96 0 (1): 0 187--199, 2009
2009
Show all 40 references
-
[9]
Causal inference without counterfactuals
A Philip Dawid. Causal inference without counterfactuals. JASA, 95 0 (450): 0 407--424, 2000
2000
-
[10]
Fitting science into legal contexts: assessing effects of causes or causes of effects? Sociological Methods & Research, 43 0 (3): 0 359--390, 2014
A Philip Dawid, David L Faigman, and Stephen E Fienberg. Fitting science into legal contexts: assessing effects of causes or causes of effects? Sociological Methods & Research, 43 0 (3): 0 359--390, 2014
2014
-
[11]
Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. R eal T oxicity P rompts: Evaluating neural toxic degeneration in language models. In EMNLP, 2020
2020
-
[12]
Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies
Jens Hainmueller. Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political analysis, pages 25--46, 2012
2012
-
[13]
Nonparametric estimation of mediation effects with a general treatment
Lukang Huang, Wei Huang, Oliver Linton, and Zheng Zhang. Nonparametric estimation of mediation effects with a general treatment. Econometric Reviews, 43 0 (2-4): 0 215--237, 2024
2024
-
[14]
Direct and indirect effects of continuous treatments based on generalized propensity score weighting
Martin Huber, Yu-Chin Hsu, Ying-Ying Lee, and Layal Lettry. Direct and indirect effects of continuous treatments based on generalized propensity score weighting. Journal of Applied Econometrics, 35 0 (7): 0 814--840, 2020
2020
-
[15]
A general approach to causal mediation analysis
Kosuke Imai, Luke Keele, and Dustin Tingley. A general approach to causal mediation analysis. Psychological methods, 15 0 (4): 0 309, 2010
2010
-
[16]
Causal inference in statistics, social, and biomedical sciences
Guido W Imbens and Donald B Rubin. Causal inference in statistics, social, and biomedical sciences. Cambridge university press, 2015
2015
-
[17]
Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data
Joseph DY Kang, Joseph L Schafer, et al. Demystifying double robustness: A comparison of alternative strategies for estimating a population mean from incomplete data. Statistical science, 22 0 (4): 0 523--539, 2007
2007
-
[18]
Probabilities of causation for continuous and vector variables
Yuta Kawakami, Manabu Kuroki, and Jin Tian. Probabilities of causation for continuous and vector variables. arXiv:2405.20487, 2024
2024 arXiv
-
[19]
Bloom: A 176b-parameter open-access multilingual language model
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili \'c , Daniel Hesslow, Roman Castagn \'e , Alexandra Sasha Luccioni, Fran c ois Yvon, Matthias Gall \'e , et al. Bloom: A 176b-parameter open-access multilingual language model. 2023
2023
-
[20]
RoBERTa: A Robustly Optimized BERT Pretraining Approach
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pretraining Approach . arXiv:1907.11692, 2019
1907 arXiv
-
[21]
Evaluating causes of effects by posterior effects of causes
Zitong Lu, Zhi Geng, Wei Li, Shengyu Zhu, and Jinzhu Jia. Evaluating causes of effects by posterior effects of causes. Biometrika, 110 0 (2): 0 449--465, 2023
2023
-
[22]
Sparse feature circuits: Discovering and editing interpretable causal graphs in language models
Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv:2403.19647, 2024
2024 arXiv
-
[23]
Locating and editing factual associations in gpt
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. NeurIPS, 35: 0 17359--17372, 2022
2022
-
[24]
Progress measures for grokking via mechanistic interpretability
Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. Progress measures for grokking via mechanistic interpretability. In ICLR, 2023
2023
-
[25]
Zoom in: An introduction to circuits
Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. Zoom in: An introduction to circuits. Distill, 5 0 (3), March 2020
2020
-
[26]
In-context learning and induction heads
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context learning and induction heads. arXiv preprint arXiv:2209.11895, 2022
2022 arXiv
-
[27]
Causality
Judea Pearl. Causality. Cambridge university press, 2009
2009
-
[28]
Mediation analysis methods used in observational research: a scoping review and recommendations
Judith JM Rijnhart, Sophia J Lamp, Matthew J Valente, David P MacKinnon, Jos WR Twisk, and Martijn W Heymans. Mediation analysis methods used in observational research: a scoping review and recommendations. BMC medical research methodology, 21: 0 1--17, 2021
2021
-
[29]
Marginal structural models and causal inference in epidemiology
JM Robins, MA Hern \'a n, and B Brumback. Marginal structural models and causal inference in epidemiology. Epidemiology, 11 0 (5): 0 550--560, 2000
2000
-
[30]
Does matching overcome lalonde's critique of nonexperimental estimators? Journal of econometrics, 125 0 (1-2): 0 305--353, 2005
Jeffrey A Smith and Petra E Todd. Does matching overcome lalonde's critique of nonexperimental estimators? Journal of econometrics, 125 0 (1-2): 0 305--353, 2005
2005
-
[31]
A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis
Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis. In EMNLP, 2023
2023
-
[32]
Investigating gender bias in language models using causal mediation analysis
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal mediation analysis. NeurIPS, 33: 0 12388--12401, 2020
2020
-
[33]
Minimal dispersion approximately balancing weights: asymptotic properties and practical considerations
Yixin Wang and Jose R Zubizarreta. Minimal dispersion approximately balancing weights: asymptotic properties and practical considerations. Biometrika, 107 0 (1): 0 93--105, 2020
2020
-
[34]
Unlearning bias in language models by partitioning gradients
Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. In ACL, pages 6032--6048, 2023
2023
-
[35]
Towards best practices of activation patching in language models: Metrics and methods
Fred Zhang and Neel Nanda. Towards best practices of activation patching in language models: Metrics and methods. In ICLR, 2024
2024
-
[36]
Conditional counterfactual causal effect for individual attribution
Ruiqi Zhao, Lei Zhang, Shengyu Zhu, Zitong Lu, Zhenhua Dong, Chaoliang Zhang, Jun Xu, Zhi Geng, and Yangbo He. Conditional counterfactual causal effect for individual attribution. In UAI, pages 2519--2528. PMLR, 2023
2023
-
[37]
A boosting algorithm for estimating generalized propensity scores with continuous treatments
Yeying Zhu, Donna L Coffman, and Debashis Ghosh. A boosting algorithm for estimating generalized propensity scores with continuous treatments. Journal of causal inference, 3 0 (1): 0 25--40, 2015
2015
-
[38]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[39]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[40]
@7N 37^3b#+]xI[bb ]CxAFgGxx=_ l ЛBPK 2Fi tY # W w(( qx 'ҋ9y |,BP
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
2024 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.