{"id":"d4f3bd8c-2319-4743-a3de-99288b39106e","arxiv_id":"2412.02893","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Adjusting for conversation topic with entropy balancing makes toxicity attributions in LLMs less localized, spreading across more MLP units.","lead":"This paper proposes a causal mediation method that adjusts for conversation topic when attributing toxicity to individual neurons in large language models. Applying it to two models shows that after removing the topic's influence, toxic behavior appears spread across more neurons rather than concentrated in a few.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Positivity is acknowledged as possibly violated but never verified; without overlap diagnostics or a placebo test, the reported de-localization could be an artifact of extreme reweighting rather than a causal finding.","rationale":"The paper proposes a useful extension of entropy balancing to text-based mediation and honestly notes the positivity risk. My read of the central claim is that it is plausible but conditional on the adjusted weights being stable and meaningful. The most load-bearing point is not the DAG itself but the acknowledged positivity violation: because n is a deterministic function of q, the ratio π_{n,x}(q_i,n_i,x_i)/π_{n,x}(q_j,n_i,x_i) in Eq. (2) can be extreme; PCA only approximates the conditional density and no evidence shows it restores overlap. This is an internal correctness risk, not merely a departure from consensus. A permutation test would distinguish a genuine topic-adjustment effect from a generic reweighting artifact, and overlap/percentile diagnostics would confirm or refute the positivity concern. The reader's weakest assumption (incomplete topic proxy plus positivity) partly overlaps with this; I focus on positivity because it is explicitly acknowledged and directly testable. Since the paper is already conditionally accepted subject to such validation, my concern does not shift the verdict; it strengthens the conditionality. No ad hominem intended; the method may well work after these checks.","tokens_in":7599,"tokens_out":6260,"duration_ms":61096,"concrete_test":"Run a permutation/placebo test on the published pipeline: randomly permute the topic cluster labels x_i (breaking any true topic–outcome confounding) and re-estimate the Normal-versus-De-biased AIE curves in Figure 2. If the flattening persists under permutation, the adjustment is not specifically removing topic confounding and the de-localization claim is an artifact of the estimator. As a supporting check, report the distribution of the within-pair weight ratios exp(−λ^T g(q_j,[n_i,x_i])) for each toxic i, including the fraction Winsorized; near-zero denominators would confirm positivity failure. Also re-run with k=5 and k=10 topic clusters and PCA dimensions 10 and 50 to test whether the conclusion is robust to the arbitrary cluster count.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that topic adjustment makes toxicity less localized—rests on Eq. (2)'s ratio weights being well-defined. Section 3's 'Note on Positivity Assumption' admits that f(q|n,x)>0 may fail because node activations are deterministic functions of the query, and offers PCA dimensionality reduction as a patch, but no overlap diagnostics are reported. If for a toxic query i the denominator exp(−λ^T g(q_j,[n_i,x_i])) is near zero for many sampled q_j, the AIE sum is dominated by a few huge terms; Winsorization and L1 regularization clip extremes but do not establish that the remaining estimate measures the mediated effect. Figures 2 report pointwise normalized contributions without error bars or significance tests, and the Discussion defers node-masking validation to future work. No code or data are provided. Consequently, the observed flattening of the De-biased curve is currently indistinguishable from a numerical artifact of unstable propensity weights; the causal interpretation is conditional on an unverified positivity/overlap condition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a causal mediation approach to remove the effect of conversation topic when attributing toxic behavior to individual MLP units in large language models. The authors introduce a DAG in which topic x confounds the query q and the toxicity outcome y, with node activations n as a mediator, and use entropy balancing to estimate the Average Indirect Effect (AIE) for each MLP. Experiments with Bloomz 1B7 and Phi-3-mini 3B on RealToxicityPrompts show that after adjusting for topic, toxicity contributions become more distributed across MLPs. The paper does not provide code or data.","tokens_in":7857,"tokens_out":6023,"duration_ms":92776,"significance":"If the central claim is correct, the paper offers a principled way to correct neuron attribution for a common confounder, with direct implications for mechanistic interpretability and safety editing. The adaptation of entropy balancing to textual treatments and the focus on the localization hypothesis are timely. However, the empirical support is currently weak: the key estimator is not derived, the positivity assumption is acknowledged as possibly violated but never verified, the main results are presented without uncertainty quantification, and the topic confounder is estimated from the same queries used in the analysis. These issues prevent the paper from being accepted in its present form, though the core idea may be salvageable with additional diagnostics and analyses.","major_comments":[{"comment":"The AIE estimator in Eq. (2) is introduced without a formal derivation. The paper first defines the population quantity μ(q,q') in Eq. (1) and then states an empirical average over pairs, but Eq. (2) replaces the conditional expectation in Eq. (1) with the single observation y_i. As written, it is unclear why this plug-in estimator is unbiased or consistent for the stated causal contrast. The authors should provide a derivation, or at least a clear argument that the weighted sum over observed units estimates the population AIE.","section":"Section 3, Eq. (2)"},{"comment":"The paper acknowledges that f(q|n,x)>0 may fail because node activations are deterministic functions of the query, and it proposes PCA dimensionality reduction as a mitigation. However, no overlap diagnostics, weight distributions, or effective sample sizes are reported. Without evidence that the entropy balancing weights are stable and that the denominator terms in Eq. (2) are bounded away from zero, the observed flattening of the De-biased curve in Figure 2 could be an artifact of extreme reweighting, even after Winsorization.","section":"Section 3, 'A Note on Positivity Assumption'"},{"comment":"The main empirical claim that toxicity becomes less localized rests solely on visual comparison of two curves without error bars, confidence intervals, or significance tests. The authors should provide bootstrap confidence intervals that reflect both the stochastic approximation (K=200) and the entropy balancing estimation, and report a quantitative localization measure (e.g., a concentration index or slope) with uncertainty.","section":"Section 4, Figure 2"},{"comment":"The confounder x is defined by k-means clustering on RoBERTa embeddings of the same queries, with k=3 fixed. Because x is derived from the treatment q, the DAG in Figure 1 (x -> q) is not directly satisfied, and the strong ignorability assumption depends on the cluster solution being a valid proxy for the true topic confounder. The paper should include sensitivity analyses over the number of clusters and the PCA dimension, and demonstrate that the main conclusion is robust to these choices.","section":"Section 4, topic construction"},{"comment":"The 'Normal' baseline is described as obtained by Eq. (1) without confounding factors, but the exact formula is not given. To make the comparison reproducible, the authors should specify precisely how this baseline is computed, including the form of the propensity weights used when x is omitted.","section":"Section 4, baseline definition"}],"minor_comments":[{"comment":"The heading 'Backround' should be 'Background'.","section":"Section 2.1"},{"comment":"The notation for random variables and vector values is introduced but not used consistently; for example, bold x appears both as a random vector and as a value, which can confuse the exposition.","section":"Section 2"},{"comment":"Eq. (1) uses δ = q'−q, while Eq. (2) uses the pair notation (q_i, q_j); the relation between q' and q_j should be stated explicitly.","section":"Section 3, Eq. (1)-(2)"},{"comment":"The stochastic approximation with K=200 is not justified; the authors should report sensitivity to this choice.","section":"Section 3, Algorithm 1"},{"comment":"Figure 3 is referenced as being in the appendix but appears after the references without a caption or explanation; the t-SNE plot and cluster labels should be described.","section":"Appendix, Figure 3"},{"comment":"No code or data are provided, which limits the reproducibility of the empirical claims.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem, but the current evidence is not strong enough. The most concerning issue is the unverified positivity assumption combined with the absence of overlap diagnostics; since node activations are deterministic functions of the query, the estimator in Eq. (2) could be numerically unstable. Additionally, deriving the topic confounder from the treatment itself raises conceptual questions about the DAG that the authors should address head-on. I would encourage the editor to seek a major revision where the authors provide a rigorous derivation of the estimator, add uncertainty quantification and sensitivity analyses, and ideally release code and data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this paper proposes a causal adjustment for neuron attribution in LLMs—controlling for conversation topic when measuring how much each MLP contributes to toxicity—and reports that after adjustment, toxicity looks less localized. If that holds, it's a real challenge to the localization assumption that underlies a lot of interpretability-for-safety work. But as it stands, the evidence is too thin to accept the finding.\n\nWhat's genuinely new: the idea of treating topic as a confounder in attribution is timely, and the adaptation of entropy balancing to textual queries (using the embedding and the ratio form of the weights) is a sensible move. The pairwise average indirect effect estimator in Eq. (2) is a useful contribution in itself—it avoids generating out-of-distribution counterfactual queries. The DAG is clean and the writing is honest: they flag the positivity assumption explicitly and try to patch it with PCA.\n\nThe soft spots are real and, for the central claim, load-bearing. The main result is two curves with no error bars, confidence intervals, or tests. I can't tell whether the flattening of the De-biased curve is signal or a numerical artifact of the reweighting. The positivity concern they themselves raise is never checked: no overlap diagnostics, no distribution of weights, no placebo test. PCA to dimension 25 and Winsorization may help, but they don't establish that the denominator in Eq. (2) is well-behaved for the toxic queries that drive the estimate. The topic confounder is three k-means clusters on RoBERTa embeddings, with no justification for k=3 or the PCA dimension. And there is no validation—the Discussion defers node-masking to future work, so we have no intervention to confirm that the de-biased attributions point at units that actually mediate toxicity.\n\nNone of this means the idea is wrong. The method is plausible and the direction is worth pursuing. But the paper currently reads as a working note: the estimator is stated without derivation or consistency proof, and the empirical claim is not yet backed by the required evidence.\n\nFor you: if you work on mechanistic interpretability or safety evaluations, this is worth reading as a caution about confounders. I wouldn't cite the de-localization claim yet, and I'd want code and data before trusting the estimator.\n\nRecommendation: send it to peer review. The topic matters, the framing is new, and the authors are clearly competent—but a serious referee should demand error bars, positivity diagnostics, a sensitivity analysis on k and the PCA dimension, and at least one intervention or placebo check before the claim is accepted.\n\nBest","headline":"Promising causal re-framing of neuron attribution, but the central de-localization claim needs error bars, positivity diagnostics, and validation before it can be trusted.","tokens_in":8331,"tokens_out":2677,"would_cite":false,"duration_ms":23554,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that conversation topic confounds neuron-level attribution of toxicity in large language models, making toxic behavior look more localized than it is, and that entropy-balancing adjustment spreads the attribution across…","keywords":["causal mediation","neuron attribution","toxicity localization","spurious correlation","entropy balancing","interpretability","large language models","confounder adjustment"],"falsifier":"Use a synthetic dataset where a known topic confounder mediates a known fraction of toxicity through a known subset of units; if the estimator does not recover that ground-truth localization, the causal claim fails. Alternatively, if a finer-grained topic model (e.g., 20 clusters or a continuous topic model) reduces or reverses the reported de-localization in Bloomz or Phi-3-mini, the confounding adjustment is not capturing the true confounder.","tokens_in":7423,"feed_emoji":"🧠","tokens_out":3767,"duration_ms":36548,"temperature":0.7,"pith_summary":"The paper argues that standard methods for attributing toxic output to specific neurons in large language models are confounded by conversation topic, and it proposes a causal mediation estimator that reweights queries so that topic no longer distorts the attribution. Applying the estimator to two instruction-tuned LLMs, the paper reports that toxicity appears less localized after adjustment: more multilayer perceptron units share responsibility. If correct, this means that mechanistic interpretability findings on toxicity localization may overstate the concentration of harmful behavior, and safety editing based on those attributions may target the wrong units.","feed_headline":"Topic confounders make LLM toxicity look too localized","feed_subtitle":"Adjusting for conversation topic spreads toxicity attribution across more MLP units in two LLMs.","key_machinery":"The central object is the Average Indirect Effect (AIE) estimator, a causal mediation quantity that measures how much of a query's effect on toxicity flows through a specific internal node while holding the topic fixed. It is computed with entropy balancing weights that reweight queries so that topic and node activations are balanced, and with a pairwise estimator that avoids generating out-of-distribution inputs. The cancellation of softmax normalization terms in the entropy balancing ratios lets the method compute propensity ratios without retraining, requiring only a single forward pass.","core_discovery":"The central claim is that confounding by conversation topic creates spurious correlations in neuron-level attribution of toxicity, and that controlling for topic via entropy balancing changes the estimated contribution of each MLP to toxic generations. Using the DAG where topic $x$ affects both query $q$ and harmfulness $y$, and the query affects mediator $n$ (node activation), the paper estimates the Average Indirect Effect through each MLP with stabilized weights computed by entropy balancing, avoiding out-of-distribution counterfactuals by pairwise comparisons. In experiments with Bloomz 1B7 and Phi-3-mini 3B on RealToxicityPrompts, the normalized contributions to toxicity become more distributed across MLPs, i.e., toxicity is less localized, when topic adjustment is applied.","pith_inferences":["If the de-localization holds under finer topic models, the localization hypothesis itself may need to be reformulated as conditional on unconfoundedness rather than a static property of the model.","The pairwise AIE estimator could be adapted to other continuous mediators such as attention heads or feature circuits, not just MLP units.","A natural stress test is to introduce a synthetic confounder with known ground-truth mediation, then check whether the estimator recovers the correct ranking of units.","Because the paper acknowledges the positivity assumption may be violated by deterministic node activations, overlap diagnostics for the propensity weights would make the causal claim stronger."],"forward_implications":["Attribution maps for toxicity in LLMs should be re-examined with confounder adjustment to avoid overstating localization.","Neuron-editing or unlearning methods that rely on unadjusted attribution may suppress topic-general units rather than toxicity-specific ones.","The same DAG and estimator can be applied to other confounded behaviors such as gender bias, hallucination, and factuality, as the paper notes.","Because the estimator requires only one forward pass, confounder-adjusted attribution is feasible at scale for large models."],"supporting_citations":[{"why":"Supplies the entropy-balancing-based mediation estimator for a general treatment that the paper adapts to textual queries.","marker":"[Huang et al., 2024]"},{"why":"Provides the weighted natural indirect effect equation for continuous treatments that is the basis of the paper's estimator.","marker":"[Huber et al., 2020]"},{"why":"Introduces entropy balancing for reweighting observational data, the core weighting method used here.","marker":"[Hainmueller, 2012]"},{"why":"Supplies the regularized approximate-balancing formulation that stabilizes the entropy balancing weights.","marker":"[Wang and Zubizarreta, 2020]"},{"why":"Provides the RealToxicityPrompts dataset used in the experiments.","marker":"[Gehman et al., 2020]"},{"why":"Defines the direct and indirect effect decomposition via nested counterfactuals that the mediation analysis relies on.","marker":"[Pearl, 2009]"},{"why":"Prior causal mediation analysis for bias in language models that the paper contrasts as ignoring confounders.","marker":"[Vig et al., 2020]"},{"why":"Represents the activation patching approach whose attributions the paper argues are confounded by topic.","marker":"[Zhang and Nanda, 2024]"}],"fun_headline_variants":["Topic adjustment spreads LLM toxicity attribution","Confounders inflate apparent localization of toxicity","Causal fix reveals toxicity is less localized in LLMs","Controlling for topic redistributes toxicity in LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The attribution is only unconfounded if the three k-means topic clusters capture everything about conversation topic that influences both toxicity and node activations, and if every question has positive probability under every node-activation and topic combination, conditions the paper itself notes may fail.","fun_headline_variants_meta":{"raw":{"variants":["Topic adjustment spreads LLM toxicity attribution","Confounders inflate apparent localization of toxicity","Causal fix reveals toxicity is less localized in LLMs","Controlling for topic redistributes toxicity in LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00016,"raw_usage":{"total_tokens":1133,"prompt_tokens":748,"completion_tokens":385,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":364,"completion_tokens_details":{"reasoning_tokens":337}},"tokens_in":364,"tokens_out":385,"duration_ms":4165,"temperature":1.0,"reasoning_tokens":337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:58:52.697221+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a synthetic dataset where a known topic confounder mediates a known fraction of toxicity through a known subset of units; if the estimator does not recover that ground-truth localization, the causal claim fails. Alternatively, if a finer-grained topic model (e.g., 20 clusters or a continuous topic model) reduces or reverses the reported de-localization in Bloomz or Phi-3-mini, the confounding adjustment is not capturing the true confounder.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the RealToxicityPrompts dataset used in the experiments."},{"cited_title":"Nonparametric estimation of mediation effects with a general treatment","cited_arxiv_id":null,"evidence_quote":"Supplies the entropy-balancing-based mediation estimator for a general treatment that the paper adapts to textual queries."},{"cited_title":"Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies","cited_arxiv_id":null,"evidence_quote":"Introduces entropy balancing for reweighting observational data, the core weighting method used here."},{"cited_title":"Investigating gender bias in language models using causal mediation analysis","cited_arxiv_id":null,"evidence_quote":"Prior causal mediation analysis for bias in language models that the paper contrasts as ignoring confounders."}],"review_version":1}