{"id":"77d24275-fb95-42dd-97c2-85faa97f1aa4","arxiv_id":"2505.21422","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Under hidden confounding shifts, predictive information reduces to conditional informativeness minus a residual, a result the authors use to explain ERM's surprising OOD competitiveness and the value of all-covariate models.","lead":"This paper argues that hidden confounders, unmeasured causes of both inputs and outcomes, explain why simple models often match specialized out-of-distribution methods. It derives an information-theoretic decomposition and presents evidence that using extra variables tied to the hidden confounder can improve robustness.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 4.2's proof relies on two mutually incompatible inequalities, so the central theorem is not established and the abstract's proof claim fails.","rationale":"The reader identified a real gap: maximizing predictive information I(Y;Yhat) is not the same as maximizing accuracy, and Appendix C concedes that mutual information can be maximal for a systematically wrong classifier. That concern is about the bridge from the theorem to OOD generalization. The stress-test found a more fundamental problem: the theorem itself, Proposition 4.2, is not established. Its proof relies on two inequalities that are mutually contradictory under a standard information-theoretic identity. This is not a matter of interpretation or overclaim; the internal mathematics of the central result is flawed. The empirical sections may still be valuable as exploratory evidence, and the Proposition 4.1 decomposition appears algebraically sound, but the abstract's explicit claim 'we prove that effective generalization requires learning environment-specific relationships' is unsupported by the formal argument. Because the central theoretical contribution fails at its proof stage, the appropriate verdict moves from a conditional accept to reject unless the authors can supply a corrected theorem under additional assumptions. The concrete test would settle the issue definitively: a simple simulation satisfying the paper's causal graph would show whether the asserted equality holds. If it fails, the paper's main claim cannot stand as stated.","tokens_in":41683,"tokens_out":9057,"duration_ms":91351,"concrete_test":"Run a discrete or linear-Gaussian experiment satisfying U→X, U→Y, X→Y and E→U, for example U = E, X = U + N(0,0.1), Y = X + U + N(0,0.1), with E binary and N small. Set φ(X) = X and Yhat = X. Compute I(Y;Yhat), I(X;Y|E), I(X;Y|Yhat), and the four terms in inequalities (8) and (9). If I(Y;Yhat) differs from I(X;Y|E) − I(X;Y|Yhat), Proposition 4.2 is false as stated. Separately, verify numerically that I(X;E|Y) − I(X;E) = I(Y;E|X) − I(Y;E); this identity shows inequalities (8) and (9) cannot both hold unless all differences vanish.","verdict_should_be":"REJECT","load_bearing_attack":"The abstract's claim that hidden confounding shift provably requires environment-specific relationships rests entirely on Proposition 4.2, whose proof in Appendix A is internally inconsistent. For any variables φ, E, Y, the identity I(φ;E|Y) − I(φ;E) = I(Y;E|φ) − I(Y;E) holds, since both sides equal I(φ;Y|E) − I(φ;Y). In Case X→Y the paper asserts I(φ;E|Y) ≥ I(φ;E) (inequality (8)) and I(Y;E) ≥ I(Y;E|φ) (inequality (9)). These are opposite directions of the same identity: (8) is equivalent to I(Y;E|φ) ≥ I(Y;E), not to (9). Both can hold only if all differences are zero, which is not a stated assumption. The same contradiction appears in Case Y→X with inequalities (10) and (11). Consequently the sandwich argument cannot establish equality I(Y;Yhat) = I(φ(X);Y|E) − I(φ(X);Y|Yhat). A concrete linear-Gaussian example with U→X, U→Y, X→Y can make I(X;E|Y) < I(X;E), falsifying inequality (8). Thus the formal basis for the paper's central claim is not merely overinterpreted; the theorem as stated is not proven and is likely false in general.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies out-of-distribution generalization under hidden confounding shifts. It proposes an information-theoretic decomposition of predictive information I(Y;Ŷ) into conditional informativeness, variation, label shift, feature shift, concept shift, and residual terms (Proposition 4.1), and claims that under hidden confounding with either X→Y or Y→X this reduces to I(φ(X);Y|E) - I(φ(X);Y|Ŷ) (Proposition 4.2). From this, the authors conclude that effective OOD generalization requires environment-specific relationships rather than invariant ones, and that adding informative non-causal covariates as proxies for hidden confounders improves generalization (Proposition 4.3). The paper supports these claims with real-world tabular benchmarks (TableShift) and synthetic experiments showing correlations between the information-theoretic components and ID/OOD accuracy.","tokens_in":41986,"tokens_out":10189,"duration_ms":104175,"significance":"If the theoretical results were sound, the paper would make a valuable conceptual contribution to the OOD generalization debate by explaining the empirical success of ERM and all-covariate models in the presence of hidden confounding. It would also provide practical guidance for covariate selection and for methods that learn environment-specific experts. The authors should be credited for a clean derivation of Proposition 4.1, for a transparent empirical protocol that reports mutual information estimates alongside accuracies, and for including an explicit limitation paragraph in Appendix C. However, the central proof of Proposition 4.2 is internally inconsistent; the main theoretical claim is therefore not established, and the interpretation of the information-theoretic identities as statements about OOD risk is not justified. The current contribution, while empirically suggestive, does not meet the bar for a theoretical claim of proof.","major_comments":[{"comment":"The proof of Proposition 4.2 is invalid: inequalities (8) and (9) cannot both hold in general. For arbitrary variables φ, E, Y, the identity I(φ;E|Y)-I(φ;E)=I(Y;E|φ)-I(Y;E) holds, since both sides equal I(φ;Y|E)-I(φ;Y). Hence (8), I(φ;E|Y)≥I(φ;E), is equivalent to I(Y;E|φ)≥I(Y;E), which is the exact reverse of (9), I(Y;E)≥I(Y;E|φ). Both can hold only when the differences are zero, which is not an assumed condition. The same contradiction appears in Case 2 between inequalities (10) and (11). A concrete linear-Gaussian example with E∼N(0,1), U=E+N(0,1), X=U+N(0,1), and Y=X-2U+N(0,1) satisfies the graph U→X, U→Y, X→Y but gives I(X;E|Y)≈0.16<I(X;E)≈0.20 when φ is the identity map, directly falsifying inequality (8). Consequently, the sandwich argument in the proof collapses and Proposition 4.2 is not established; the abstract's claim that generalization is proven to require environment-specific relationships is unsupported.","section":"Appendix A, Proposition 4.2, Case 1 (inequalities (8) and (9))"},{"comment":"The central interpretive step from Equation (4) to OOD generalization is not justified. The theorem, even if correct, concerns the mutual information I(Y;Ŷ), not the OOD risk Re(f∘ϕ). The manuscript itself concedes in Appendix C that a binary classifier can achieve maximal I(Y;Ŷ) by perfectly anti-correlating with Y while having 0% accuracy. No bound or monotonicity relation between I(Y;Ŷ) and expected accuracy or loss is provided. Therefore maximizing I(φ(X);Y|E)-I(φ(X);Y|Ŷ) does not entail better OOD generalization, and the conclusion that effective generalization requires environment-specific relationships does not follow from the information-theoretic identities.","section":"Section 4 and Appendix C (limitations of predictive information decomposition)"},{"comment":"Proposition 4.3(iii) is not rigorously established. The proof relies on the statement that 'conditioning usually reduces mutual information unless the additional conditioning variable opens any collider paths,' but mutual information can increase under conditioning in non-collider settings depending on the joint distribution; no formal condition (e.g., faithfulness or a specific conditional independence) is given. Since the covariate-selection guidance in Section 4.3 depends on part (iii), this inequality needs either a proper proof under explicit assumptions or should be relabeled as an empirical observation.","section":"Appendix A, Proposition 4.3 (iii)"}],"minor_comments":[{"comment":"The text says 'the first six datasets... listed in Table 2,' but Table 2 lists eight datasets; please clarify which datasets are used and why.","section":"Section 5, Figure 4"},{"comment":"The sentence 'The relative ID/OOD performance of the methods shown in Figure 1' should refer to Figure 4, not Figure 1.","section":"Section 5"},{"comment":"The sentence 'the predictive information can be maximized by maximizing the difference: conditional informativeness - residual' is tautological given the equality; the substantive claim about environment-specific relationships requires a formal argument that invariant representations cannot achieve this difference.","section":"Section 4.2, Equation (4)"},{"comment":"The sentence 'any covariate that is informative to U, is also informative to Y and vice-versa' is not generally implied by U→Y and needs assumptions or should be softened.","section":"Definition 4.3"}],"recommendation":"reject","confidential_remarks":"The central proof error in Proposition 4.2 is decisive; I would not consider a minor revision. The empirical analysis could be resubmitted as a study of information-theoretic correlates of OOD accuracy without the proof claim. The authors should also be asked to address the relation between predictive information and risk before any resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper is worth reading for Proposition 4.1, which gives a valid decomposition of predictive information I(Y;Yhat) into conditional informativeness, variation, label/feature/concept shift, and residual. That identity is clean and could be a useful way to think about what different OOD objectives are doing. The empirical study is also solid: eight TableShift datasets, careful covariate partitions, a sign-consistency metric, and synthetic experiments that back the story that hidden confounder shifts make environment-specific structure matter. The authors also deserve credit for being upfront in Appendix C that predictive information doesn't guarantee accuracy, since a perfectly anti-correlated classifier gets maximal MI and zero accuracy.\n\nThe central claim, however, is not established. Proposition 4.2 is the load-bearing theorem, and its proof in Appendix A is internally inconsistent. For any phi, E, Y, the identity I(phi;E|Y) - I(phi;E) = I(Y;E|phi) - I(Y;E) holds. In the X->Y case, inequality (8) asserts I(phi;E|Y) >= I(phi;E), which is equivalent to I(Y;E|phi) >= I(Y;E). Inequality (9) asserts exactly the opposite: I(Y;E) >= I(Y;E|phi). Both can only hold if the differences are zero. So the sandwich argument cannot work, and the equality I(Y;Yhat) = I(phi(X);Y|E) - I(phi(X);Y|Yhat) is not proven. The Y->X case has the same problem with inequalities (10) and (11). A simple linear-Gaussian example can violate (8). So the abstract's \"we prove\" is not supported.\n\nEven if the inequality issue were fixed, the step from maximizing predictive information to OOD generalization is a leap. The paper proves facts about mutual information, not about risk on test environments, and the appendix's own limitation paragraph concedes that gap. To get a theorem about generalization, the authors would need a control on loss or accuracy in terms of the information terms, or an explicit risk bound.\n\nNet: there is a salvageable core here — the decomposition and the empirical findings — but the headline result doesn't hold as stated. I'd send it to review only if the authors are willing to downgrade the claim to a conjecture or motivation, and to fix the proof or restrict it to cases where the countervailing shift terms cancel for an independent reason. As is, I wouldn't trust the central conclusion. The right move for a referee is major revision or reject, depending on how much weight they put on the theorem.","headline":"The paper has a clean information decomposition and a lot of empirical work, but the central theorem rests on a contradictory pair of inequalities, so the proof claim in the abstract fails.","tokens_in":42471,"tokens_out":3446,"would_cite":false,"duration_ms":38985,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Under hidden confounding shift, out-of-distribution generalization provably requires environment-specific relationships rather than invariant ones alone.","keywords":["out-of-distribution generalization","hidden confounding","predictive information","causal invariance","distribution shift","covariate selection","empirical risk minimization","proxy variables"],"falsifier":"Train a binary classifier under a hidden confounding shift with $U \\to X$ and $U \\to Y$ and $X \\to Y$, where the model deliberately outputs the label opposite to the true one; it achieves maximum $I(Y;\\hat{Y}) = 1$ bit yet 0 percent accuracy. If the paper's own conditional-informativeness-minus-residual objective ranks this predictor as optimal, that would falsify the claim that maximizing predictive information is the right OOD generalization objective.","tokens_in":41503,"feed_emoji":"🔄","tokens_out":6608,"duration_ms":67050,"temperature":0.7,"pith_summary":"Conventional out-of-distribution (OOD) generalization wisdom says to learn causal invariant features, yet standard empirical risk minimization (ERM) often matches or beats specialized invariance methods, and using all covariates instead of only causal ones helps. This paper argues that hidden confounding is the reason. When an unobserved variable drives both the input and the outcome, and its distribution shifts across environments, the usual invariance assumptions break; the paper proves that the predictive information between predictions and labels then equals conditional informativeness minus residual information, so effective generalization requires modeling environment-specific relationships. It also shows that informative non-causal covariates act as proxies for the hidden confounder and improve OOD accuracy. If correct, this reframes covariate selection and justifies methods that use environment information rather than only hunting invariant features.","feed_headline":"Hidden confounding shift makes invariant-only models insufficient","feed_subtitle":"A decomposition proves OOD generalization needs environment-specific information under confounding.","key_machinery":"The carrying object is the predictive-information decomposition: an identity that rewrites $I(Y;\\hat{Y})$, how much the prediction tells about the label, as a signed sum of conditional informativeness, variation, label shift, feature shift, concept shift, and residual. Under hidden confounding with $X \\to Y$ or $Y \\to X$, the four shift terms cancel, leaving the reduced identity $I(Y;\\hat{Y}) = I(\\phi(X);Y|E) - I(\\phi(X);Y|\\hat{Y})$, where $I(\\phi(X);Y|E)$ is the information the features provide about the label within each environment and $I(\\phi(X);Y|\\hat{Y})$ is the information the features carry about the label beyond what the prediction already captures. This identity does the work: it converts the question of what to learn into a two-term objective and shows why environment-specific information and proxy covariates matter.","core_discovery":"The central claim is that hidden confounding shift flips the goal of OOD generalization. For causal graphs where an unobserved confounder $U$ causes both $X$ and $Y$ and either $X$ causes $Y$ or $Y$ causes $X$, the paper derives the identity $I(Y;\\hat{Y}) = I(\\phi(X);Y|E) - I(\\phi(X);Y|\\hat{Y})$: predictive information is exactly conditional informativeness minus residual. The variation, label-shift, feature-shift, and concept-shift terms from the general decomposition cancel. Consequently maximizing predictive information requires maximizing the information the representation gives about the label within each environment while minimizing what remains unexplained by the prediction, which means invariant representations alone are insufficient and environment-specific input-output mappings are needed. The paper further proves that adding informative covariates that are not causally related to $Y$ increases conditional informativeness and reduces concept shift, explaining why all-covariate models can beat causal-subset models.","pith_inferences":["Beyond the paper, the decomposition suggests a practical diagnostic: compute conditional informativeness minus residual on validation environments to predict which model will generalize and to decide whether a candidate covariate is worth adding before running a full training experiment.","Beyond the paper, when an adversary can choose test environments with extreme confounder values, maximizing conditional informativeness might encourage fitting environment artifacts, so the strategy may need to be paired with regularization on the residual or on feature shift.","Beyond the paper, the identity suggests that the distinction between causation and prediction blurs under hidden confounding: variables that are merely informative about the hidden confounder are as useful as causal parents, which could reshape feature-engineering practice in tabular machine learning."],"forward_implications":["If correct, the superior OOD performance of ERM observed in benchmarks is not an accident: under hidden confounding shift, pooling all data and fitting environment-specific structure is the mathematically justified strategy, which may explain why ERM often beats invariance-constrained methods.","Methods that enforce invariant representations will underperform when hidden confounding shift is present, because they suppress the very environment-specific information the identity says is needed.","Adding non-causal covariates informative about the hidden confounder should improve OOD performance, giving a principled covariate-selection rule: prefer covariates that raise conditional informativeness and reduce concept shift, even if they are not causes of the outcome.","Mixture-of-experts and other environment-conditional models receive theoretical backing, and the paper extends this rationale to settings where confounding support does not overlap and the confounder is continuous-valued.","Practitioners should treat environment labels or environment statistics as a resource for OOD generalization rather than as noise to be removed."],"supporting_citations":[{"why":"This is the benchmark finding that all-covariate models Pareto-dominate causal-subset models on both ID and OOD data, the phenomenon the paper explains.","marker":"[57]"},{"why":"This supplies the invariant-risk-minimization objective that the paper contrasts with environment-specific learning.","marker":"[6]"},{"why":"This introduces the mixture-of-experts approach for hidden confounding shift whose environment-per-expert structure Proposition 4.2 justifies.","marker":"[62]"},{"why":"This supplies the benchmark datasets and environment splits used in the paper's real-world experiments.","marker":"[24]"},{"why":"This establishes that ERM is competitive with OOD methods under careful model selection, the empirical pattern the paper attributes to hidden confounding.","marker":"[30]"},{"why":"This provides the language of distribution shifts and the empirical claim that additional covariates stabilize the conditional outcome distribution, which Proposition 4.3 formalizes.","marker":"[50]"},{"why":"This is the proxy-based domain adaptation line the paper builds on for treating informative covariates as proxies for hidden confounders.","marker":"[4]"},{"why":"This supplies the proxy-method formulation for domain adaptation that motivates the paper's argument that observable proxies can substitute for the hidden confounder.","marker":"[85]"}],"fun_headline_variants":["Hidden confounding shift demands environment-specific models","All-covariate models beat causal subsets under hidden confounding","OOD generalization needs environment info, not just invariants","Confounding shift flips the case for invariant-only models","Hidden confounders shift: why invariant learning fails OOD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument treats how much the prediction tells you about the true label as the same thing as generalizing well; the paper itself notes that a predictor which always chooses the wrong label can get maximum mutual information but zero accuracy, so if that identification fails, the conclusion does not follow.","fun_headline_variants_meta":{"raw":{"variants":["Hidden confounding shift demands environment-specific models","All-covariate models beat causal subsets under hidden confounding","OOD generalization needs environment info, not just invariants","Confounding shift flips the case for invariant-only models","Hidden confounders shift: why invariant learning fails OOD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000211,"raw_usage":{"total_tokens":1395,"prompt_tokens":905,"completion_tokens":490,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":521,"tokens_out":490,"duration_ms":7258,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:28:07.587202+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a binary classifier under a hidden confounding shift with $U \\to X$ and $U \\to Y$ and $X \\to Y$, where the model deliberately outputs the label opposite to the true one; it achieves maximum $I(Y;\\hat{Y}) = 1$ bit yet 0 percent accuracy. If the paper's own conditional-informativeness-minus-residual objective ranks this predictor as optimal, that would falsify the claim that maximizing predictive information is the right OOD generalization objective.","supporting_citations":[{"cited_title":"Do causal predictors generalize better to new domains? In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024","cited_arxiv_id":null,"evidence_quote":"This is the benchmark finding that all-covariate models Pareto-dominate causal-subset models on both ID and OOD data, the phenomenon the paper explains."},{"cited_title":"Scalable out-of-distribution robustness in the presence of unobserved confounders","cited_arxiv_id":null,"evidence_quote":"This introduces the mixture-of-experts approach for hidden confounding shift whose environment-per-expert structure Proposition 4.2 justifies."},{"cited_title":"In search of lost domain generalization","cited_arxiv_id":null,"evidence_quote":"This establishes that ERM is competitive with OOD methods under careful model selection, the empirical pattern the paper attributes to hidden confounding."},{"cited_title":"On the need for a language describing distribution shifts: Illustrations on tabular datasets","cited_arxiv_id":null,"evidence_quote":"This provides the language of distribution shifts and the empirical claim that additional covariates stabilize the conditional outcome distribution, which Proposition 4.3 formalizes."},{"cited_title":"Proxy methods for domain adaptation","cited_arxiv_id":null,"evidence_quote":"This supplies the proxy-method formulation for domain adaptation that motivates the paper's argument that observable proxies can substitute for the hidden confounder."}],"review_version":1}