{"id":"9fe2d322-fdae-44fb-be20-d4756af3006c","arxiv_id":"2607.25546","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A post-hoc, model-agnostic score based on cross-environment sensitivity variation identifies causal versus spurious features used by a trained model under structured shift.","lead":"This paper introduces a post-hoc statistic, NSR, that tells which features a trained model relies on causally versus spuriously by measuring how model sensitivity varies across environments. If it works, deployed clinical and genomics models can be audited without retraining, catching silent reliance on site-specific correlations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Causal identification hinges on fixed P(X_C): the theorem assumes causal marginals do not shift, but the paper's own bike-sharing experiment violates this for the causal feature temp, leaving the assumption's boundaries untested.","rationale":"The reader's weakest_assumption correctly identifies marginal causal stability as the key fragility of Theorem 4.1. I agree that this assumption is load-bearing: if P(X_C) shifts, the proof of part (i) collapses and causal features are misclassified. However, I do not fully agree with the reader's second fragile premise — the uniform scalar shift β_e. The proof of Proposition 3(ii) only requires that the mean of each spurious feature shifts by some amount (not necessarily equal across features); the scalar assumption is a simplification, not a necessary condition for the theorem. The more important gap is that the paper provides no empirical or theoretical quantification of how sensitive NSR is to violations of marginal causal stability, and its headline real-world experiment seems to violate the assumption for tmp, a ground-truth causal feature. This is an addressable concern: a simple simulation would map the boundary of the regime. The theorem itself is logically correct under its assumptions, so the verdict should remain CONDITIONAL rather than be rejected; the paper's own claims would be strengthened by a direct test of the assumption's role.","tokens_in":21195,"tokens_out":14707,"duration_ms":140597,"concrete_test":"Simulate the linear SCM of Definition 1 with K=5 environments, p_C=p_S=4, and causal means µ_C^(e)=µ_C+η_e for η_e ~ N(0, σ_η^2 I), while keeping P(Y|X_C) fixed. Sweep σ_η from 0 to 2 (relative to within-environment noise), compute NSR for all features, and measure AUROC for causal/spurious separation. If AUROC drops significantly away from 1 as σ_η increases, marginal causal stability is confirmed as load-bearing; if AUROC remains near 1, the assumption is less critical than stated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Theorem 4.1's claim that NSR_j=0 for causal features rests entirely on equation (1), which fixes X_C ~ N(µ_C, Σ_C) across environments. If causal-feature marginals shift, Proposition 3(i) fails: δ_j for causal features becomes environment-dependent and NSR_j>0, causing false positives. This 'marginal causal stability' is strictly stronger than the P(Y|X_C) invariance used by IRM/ICP, as the paper admits in Section 3. The paper does not quantify how much shift in P(X_C) is tolerable, and the real-world validation in Section 5.5 appears to violate its own assumption: temp is listed as ground-truth causal yet its environment means shift by 6.2–9.4°C across month-quintile environments, while the paper's stability check only verifies hr and workingday. Thus the central claim's boundary is not rigorously tested, and the practical utility of the method outside a very narrow regime is uncertain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Normalised Sensitivity Ratio (NSR), a post-hoc diagnostic that uses multi-environment data to identify which features of an already-trained model are causally vs. spuriously relied upon. Under a linear SCM in which causal feature marginals are fixed across environments while spurious feature means shift by a scalar β_e (Definition 1, eqs. (1)–(3)), NSR is defined as the squared coefficient of variation of per-environment model sensitivity (eq. (5)). Theorem 4.1 claims exact identification: NSR_j = 0 iff j is causal and NSR_j > 0 iff j is spurious, independent of the learned weight ŵ_j. The paper also provides finite-sample rates (Theorem 4.2), a failure-mode characterisation for weak shifts, degenerate geometry, and proxy features (Theorem 4.3), and a formal comparison with IRM/ICP showing incomparable detection conditions (Proposition 1). Synthetic experiments confirm the theoretical operating regime, and real-world experiments on bike-sharing and wine quality demonstrate the method's practical use. The paper is explicit that the regime requires stable causal marginals and that violations can cause false positives.","tokens_in":21461,"tokens_out":5885,"duration_ms":64371,"significance":"If the identification claim holds, NSR is a valuable addition to the post-hoc interpretability toolbox: it requires no retraining, no gradient access, and no causal graph, and it provably separates causal from spurious features under a structured-shift regime. The proof of Theorem 4.1 is clean and self-contained: the cancellation of ŵ_j in the CV² ratio is explicit (Appendix A.2, eq. (7)), and the folded-normal monotonicity argument for spurious features is correct. The synthetic validation is thorough, with AUROC = 1.000 under the stated regime and sharp confirmation of the predicted failure modes. The explicit comparison with IRM and ICP is a useful conceptual contribution. The main weaknesses are that the exact guarantee is narrower than the abstract suggests (only linear predictors, fixed causal marginals, scalar shifts), and the real-data validation does not cleanly satisfy the stated assumptions. Nevertheless, the core theorem and its proof are sound, and the paper is honest about the principal limitation.","major_comments":[{"comment":"The bike-sharing experiment is presented as satisfying marginal causal stability, but the text states that the causal feature `temp` shifts by 6.2–9.4°C across environments while `hr` and `workingday` are stable. Under eq. (1), X_C is fixed across environments; a causal feature with a shifting mean directly violates the assumption needed for Theorem 4.1(i). The 'stability check' only verifies `hr` and `workingday`, not `temp`, which is listed as a ground-truth causal feature. Since the experiment is the paper's main real-world evidence, the authors should either (a) correct the assertion that the checks are satisfied, (b) report the NSR values for the shifting causal feature and discuss how the partial recovery nonetheless occurs, or (c) reframe the result as a robustness test under assumption violation rather than a clean validation.","section":"Section 5.5 / Table 2"},{"comment":"The abstract claims a 'post-hoc, model-agnostic diagnostic' with exact identification, but Theorem 4.1 is stated and proved only for linear predictors. The extension to nonlinear models via SHAP/permutation importance is empirical only and lacks a theoretical justification that the CV² of the attribution equals the CV² of the marginal-replacement sensitivity signal. Section 5.3 reports consistent rankings across model families, but this does not substitute for a formal statement of the conditions under which the identification guarantee carries over. The paper should either soften the model-agnostic claim in the abstract or provide a theorem for a class of nonlinear models.","section":"Abstract / Theorem 4.1 / Section 5.3"},{"comment":"The finite-sample result Theorem 4.2(ii) states that under H0, empirical NSR = O_p(n^{-1}). This rate is used to justify a permutation test with a growing signal-to-noise ratio. However, the appendix does not provide a calibration study demonstrating that the permutation null indeed controls Type I error at finite n, or a power analysis under H1. The real-data FDR claim in Section 5.5 ('correctly classify 10 of 12 features (4 false negatives, 0 false positives)') is arithmetically inconsistent: 10 correct with 0 false positives implies 2 false negatives, not 4. This undermines the interpretability of the reported real-world performance and should be checked.","section":"Theorem 4.2 and Appendix A.5"}],"minor_comments":[{"comment":"The research-question labels are inconsistent: 'RQ1' is used for three different questions (operating conditions, IRM comparison, real data), and the appendix refers to 'RQ1' for model-agnostic and proxy experiments that are distinct from the main RQ1. Please renumber to RQ1–RQ5 consistently.","section":"Section 5, intro paragraph"},{"comment":"The sentence '(Theorems 4.3–??)' contains a missing cross-reference; the proxy-attenuation result is Theorem 4.3(iii), not a separate numbered theorem.","section":"Introduction, contribution (ii)"},{"comment":"The check (2) about leading PCs of environment centroids is not formally connected to the scalar-shift assumption. Clarify whether this is a heuristic or a condition whose violation provably breaks Theorem 4.1.","section":"Section 3, 'Scope and assumptions'"},{"comment":"Table 2 is labeled 'RQ1' but is the RQ5 real-world validation. Correct the caption.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The central theorem is sound and the paper is honest about its main limitation (fixed P(X_C)). The main issue is that the real-world validation is presented as satisfying the regime when it does not, and the abstract's 'model-agnostic' claim goes beyond the proved linear result. These are fixable with a more careful framing and additional analysis, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core idea is new and the theory is mostly solid: NSR, the squared coefficient of variation of per-environment sensitivity, is weight-invariant, and the proof that it is exactly zero for causal features and positive for spurious features under the linear SCM is clean. The failure-mode analysis—weak-shift collapse at O(ε^4), degenerate geometry, proxy attenuation at O((1−α)^4)—is a real contribution, and the incomparability result with IRM/ICP is genuinely useful. Synthetic validation matches the theory, including the phase transition in K and ε, and the model-agnostic ranking across five families is a nice check.\n\nSecond, the paper overclaims. Exact identification requires equation (1): causal feature marginals P(X_C) are fixed across environments. That is stronger than the standard invariance of P(Y|X_C), and the paper admits it. But it never quantifies how much shift in P(X_C) breaks the result. That would be acceptable if the real-world test stayed in the regime. On bike-sharing, the environments are month-quintiles; the paper labels temp causal, yet reports temp's environment means shifting by 6.2–9.4 °C, and the stability check only covers hr and workingday. So the headline real-world result may violate the paper's own central assumption. This is the main soft spot.\n\nOther issues are minor but real. The nonlinear extension is empirical only; the theorem is for linear predictors. Real-world AUROC is modest (around 0.66) and depends on tuning a regularization target toward within-environment R²≈0.75, a free parameter whose choice is not justified. No code or data are provided, despite a checklist statement that code will come with the camera-ready—for a methods paper aimed at practitioners, that hurts reproducibility. There are also internal labeling errors (repeated RQ1, a broken cross-reference to Theorem 4.3–??), a sign of a rushed final pass.\n\nWho should read it: researchers in invariant learning, model auditing, and multi-site/batch-effect settings. The paper gives a clear map of when this post-hoc sensitivity-variance test works and when it fails. It deserves peer review because the question is important and the theory is coherent; but a good referee will demand a revision that aligns the abstract with the fixed-marginal assumption, tests that assumption's boundary directly (including on the bike-sharing data), and ships code. I would cite the theorem if I wrote on this topic, though I would not rely on the real-world numbers.\n\nRecommendation: send to peer review with an expectation of major revision.","headline":"NSR is a genuinely new post-hoc causal/spurious feature diagnostic with a clean linear-SCM identification theorem, but the exactness claim hides a strong fixed-causal-marginal assumption that the paper's own bike-sharing experiment appears to violate.","tokens_in":21920,"tokens_out":3829,"would_cite":true,"duration_ms":37356,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single statistic—the squared coefficient of variation of per-environment sensitivity—exactly separates causal from spurious features in an already-trained linear model, provided environments shift only spurious featu","keywords":["Causal feature identification","Post-hoc model auditing","Spurious correlation","Normalised Sensitivity Ratio","Structured distribution shift","Structural causal model","Sensitivity analysis","Feature attribution"],"falsifier":"A direct test of the central claim is to simulate the paper's linear SCM with at least three environments whose absolute shifts from the chosen reference are unequal, train any linear model with nonzero weights, and compute NSR for every feature; the theorem predicts exactly zero for causal features and strictly positive for spurious ones. Observing a nonzero NSR for a causal feature or a zero NSR for a spurious feature under these conditions would refute the identification claim. A complementary test is to let causal-feature means vary across environments while keeping P(Y|X_C) fixed; the pap","tokens_in":21111,"feed_emoji":"🔍","tokens_out":7958,"duration_ms":82802,"temperature":0.7,"pith_summary":"This paper tries to solve a practical problem: once a model is deployed, how do you tell which features it relies on for genuinely causal reasons and which only because of environment-specific correlations? Its answer is the Normalised Sensitivity Ratio (NSR), computed by measuring, for each feature, how much the model's output changes when the feature is replaced by a draw from another environment's distribution, and then taking the squared coefficient of variation of that change across environments. The paper proves that, for any linear model with nonzero weights, NSR is exactly zero for causal features and strictly positive for spurious features, as long as there are at least three environments that shift only spurious feature means and at unequal distances from a reference. That would matter because it means deployed models can be audited post-hoc—without retraining, gradient access, or a causal graph—in the multi-site and multi-batch settings where spurious correlations silently break. The paper also gives exact rates and conditions for the method's failure, so practitioners know when to trust it.","feed_headline":"Sensitivity ratio separates causal and spurious features post-hoc","feed_subtitle":"A score of zero marks causal features; positive scores mark spurious ones, letting deployed models be audited without retraining.","key_machinery":"The central object is the Normalised Sensitivity Ratio: for each feature j, sample the model's mean absolute output change when values of j are replaced by draws from a reference environment's marginal, do this for each environment, and take the squared coefficient of variation (variance over environments divided by squared mean). For a linear predictor the per-environment sensitivity factorises as |w_j|·δ_j, where δ_j depends only on the feature's marginal across environments; the |w_j| factor and its square cancel in the CV² ratio, so the statistic isolates δ_j. Because δ_j is constant across environments for causal features and strictly increasing in the absolute environment shift for spu","core_discovery":"The central claim is Theorem 4.1: under the paper's linear structural causal model (Definition 1), if the causal mechanism and causal-feature marginals are identical across environments while spurious features shift by a scalar per environment, then for any linear predictor with nonzero weights the Normalised Sensitivity Ratio of a feature j equals zero if and only if j is causal, and is strictly positive if and only if j is spurious. Crucially, the ratio does not depend on the learned weight w_j, because the squared weight cancels when the variance across environments is normalised by the square of the mean sensitivity. The identification signal therefore lives in the geometry of the enviro","pith_inferences":["An extension the paper leaves implicit: NSR could be used in reverse as a check on the structured-shift regime itself—if no feature scores near zero, the practitioner has evidence either that the regime is violated or that the model ignores causal features.","The weight-cancellation property suggests the method may keep working when only attribution estimates (e.g., permutation-based or additive attribution values) are available instead of model weights; the paper experiments this way, but the exact identification proof is linear, so the nonlinear guarantee is an extrapolation.","The scalar-uniform-shift assumption is the least realistic part of the model; replacing per-feature replacement with subspace-level interventions along environment-centroid principal components would be a natural way to handle feature-specific batch effects, but the paper does not prove guarantees for that relaxation.","A concrete testable extension: run NSR on the same model under two different reference environments; if rankings flip, the choice of reference is doing work, and the paper's recommended maximum-over-references variant would be the safer default."],"forward_implications":["A practitioner with only environment labels, held-out data, and forward passes of a trained linear model can identify the causal/spurious partition exactly, without retraining or knowing the model class.","Features with small coefficients are not hidden: because the ratio normalises out w_j², a spurious feature with negligible weight is still detected.","Before applying NSR, a practitioner can check the three quantitative conditions—at least three environments, non-constant absolute shifts from the reference, and shift-to-noise ratio not near zero—and know in advance whether the diagnostic will work or collapse as O(ε⁴).","NSR and training-time invariant-learning approaches are complements, not competitors: each detects spurious features under shift schedules the other misses, so the right tool depends on the environment geometry.","The O_p(n^{-1}) null rate versus O_p(n^{-1/2}) alternative rate gives a distribution-free permutation test with growing separation as the sample size increases."],"fun_headline_variants":["Post-hoc metric pinpoints causal vs spurious features","Zero sensitivity ratio marks causal features in any model","Audit deployed models without retraining: NSR finds causes","Sensitivity ratio: zero means causal, positive means spurious"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that causal features' own distributions are identical across environments—not just the causal law—so if demographic or case-mix variation shifts causal covariates, NSR will label those causal features as spurious.","fun_headline_variants_meta":{"raw":{"variants":["Post-hoc metric pinpoints causal vs spurious features","Zero sensitivity ratio marks causal features in any model","Audit deployed models without retraining: NSR finds causes","Sensitivity ratio: zero means causal, positive means spurious"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0007,"raw_usage":{"total_tokens":3036,"prompt_tokens":820,"completion_tokens":2216,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":2162}},"tokens_in":564,"tokens_out":2216,"duration_ms":15864,"temperature":1.0,"reasoning_tokens":2162,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T02:06:48.900512+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test of the central claim is to simulate the paper's linear SCM with at least three environments whose absolute shifts from the chosen reference are unequal, train any linear model with nonzero weights, and compute NSR for every feature; the theorem predicts exactly zero for causal features and strictly positive for spurious ones. Observing a nonzero NSR for a causal feature or a zero NSR for a spurious feature under these conditions would refute the identification claim. A complementary test is to let causal-feature means vary across environments while keeping P(Y|X_C) fixed; the pap","supporting_citations":[],"review_version":1}