{"id":"0d71864c-f82d-4bbd-89dc-a11f2131914d","arxiv_id":"2501.02012","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"Information Subtraction trains a generator against two mutual information estimators to represent conditional entropy H(Y|X), but the objective does not reliably remove the conditioned variable's information.","lead":"This paper proposes a neural architecture that learns a representation of one variable with a conditioned variable's information removed, to capture conditional entropy. It reports fair-learning and domain-generalization applications, but the experiments show the information removal is unreliable and partly contradicts the central claim.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 reports I(Z;X)=2.98 bits for a three-level categorical X (H(X)≤log2(3)≈1.585), an impossible value for mutual information; the central 'eliminating undesired' result is therefore an estimator artifact, not evidence.","rationale":"The reader's verdict is REJECT, and I agree; my concern does not change that verdict, so I recommend UNCHANGED. My load-bearing concern is a sharper, more specific instance of the reader's weakest_assumption about the MINE/SMILE estimator. The reader located the problem as the Lagrangian surrogate failing to remove true X-information; I argue that the failure is even more basic: the reported information values are mathematically impossible (I(Z;X) > H(X), and also I(Z;X) > I(X;Y) under the X→Y→Z Markov structure enforced by the generator). This directly invalidates the experimental evidence for the central claim and shows that the estimator used for both training and evaluation is unreliable. I considered the Appendix B admission that the framework cannot disentangle sectors, but that is an acknowledged limitation and less decisive than the impossibility of the headline numbers. A single concrete re-estimation check can settle whether the numbers are artifacts. The central claim, as stated in the abstract and Section 5.1, is therefore unsupported by the paper's own data.","tokens_in":13763,"tokens_out":7038,"duration_ms":67962,"concrete_test":"Re-estimate I(X;Z) for the trained generator from Section 5.3 using a non-neural plug-in estimator over a discretization of Z (or a kNN MI estimator), and compare to the trivial upper bound H(X) ≤ log2(3) ≈ 1.585 bits and to I(X;Y) = 0.25 bits. If the re-estimate violates either bound, Table 2's values are artifacts of the SMILE estimator; if it does not, the reported 2.98 bits cannot be reproduced and the central claim lacks empirical support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that Information Subtraction generates representations which preserve desired information while eliminating undesired information—rests on the experimental numbers in Tables 1–3, and these numbers are produced by the same neural MI estimators (MINE/SMILE) used in the training loss (Eq. 10). In Section 5.3, X is a three-level categorical variable (Eq. 13), so H(X) ≤ log2(3) ≈ 1.585 bits. Table 2 reports I(Z;X) = 2.98 bits, which exceeds H(X) and is therefore impossible for true mutual information. Moreover, since the generator NA takes only Y as input, X→Y→Z is a Markov chain, and the data processing inequality requires I(X;Z) ≤ I(X;Y) = 0.25 bits; the reported 2.98 bits violates this bound by more than an order of magnitude. These violations show that the SMILE/MINE estimates are not valid measures of mutual information in this setting, so the claimed 'elimination' of sensitive information is an artifact of the estimator. Because the same estimator drives the training objective in Eq. 10, the optimized Lagrangian may bear no relationship to the true I(X;Z), undermining the theoretical justification of the method. The paper's own Appendix B concedes that the framework 'cannot effectively disentangle' information sectors, which is consistent with this failure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Information Subtraction, a generative framework intended to learn representations Z of conditional entropy H(Y|X) (and, by iteration, other information-theoretic sectors of a Venn diagram) by simultaneously maximizing a neural estimate of I(Y;X,Z) and minimizing a neural estimate of I(X;Z). The method is demonstrated on a Lotka-Volterra synthetic system, a synthetic fair-learning setup, the Adult income dataset, and the Cover Type dataset, with claims that the learned representations carry semantic features of conditional entropy, improve fairness, and aid domain generalization. The central claim is that the framework can preserve desired information while eliminating undesired information, and can be iterated to represent arbitrary information components between continuous variables.","tokens_in":14088,"tokens_out":8258,"duration_ms":82371,"significance":"If valid, the paper would address a real gap: extending conditional-entropy representations beyond discrete conditioning variables and providing a compositional way to isolate information sectors. The authors release code, and the downstream fairness evaluation in Table 4 uses metrics that are independent of the neural mutual-information estimators, which is a strength. However, the core empirical validation is not sound: Table 2 reports impossible mutual-information values, Appendix B explicitly concedes that the framework cannot disentangle information sectors, and the evaluation of the conditional-entropy claim uses the same estimator family that appears in the training objective. These issues are load-bearing, so the paper's main contributions are not currently established.","major_comments":[{"comment":"The synthetic fair-learning experiment has a three-level categorical variable X, so H(X) <= log2(3) approximately 1.585 bits. Table 2 reports I(Z;X) = 2.98 bits, which exceeds this information-theoretic upper bound. Moreover, Algorithm 1 generates Z as a function of Y only, so X -> Y -> Z is a Markov chain; the data-processing inequality then requires I(Z;X) <= I(Y;X), which Table 2 reports as 0.25 bits. The reported 2.98 bits therefore cannot be a true mutual information. This means the SMILE estimates used in this experiment are not valid measures of mutual information, and the claim that Z contains H(Y|X) while eliminating X is an estimator artifact rather than evidence. Because the same estimator family enters the training objective in Eq. (10), the optimized representation need not satisfy the claimed information properties.","section":"Section 5.3, Table 2, Eq. (13)"},{"comment":"Appendix B explicitly concedes that \"the framework cannot effectively disentangle and represent the information within each individual sector.\" The numbers in Table B.2 support this: for the first sector, I(Z1;G|S,W) = 0.08 while the leakage I(Z1;S,W) = 1.14, and I(Z7;G|Z1,Z4,Z5) = 0.00. This contradicts the Section 5.2 claim that iteratively applying Information Subtraction can represent arbitrary sectors such as conditional mutual information and trivariate mutual information. This is not a minor implementation detail; the multivariate decomposition is one of the paper's stated contributions and is central to the general framework.","section":"Appendix B, Table B.2"},{"comment":"The success metrics used to verify the representation are estimated with the same family of neural mutual-information estimators (MINE/SMILE) as the training objective. For example, Table 2 reports I(Z;Y|X) = 2.76 as evidence that Z represents H(Y|X), but this is computed by the same type of estimator that the generator is trained against in Eq. (10). Without an independent ground-truth estimate for the synthetic Gaussian-mixture model in Eq. (13), or a non-neural estimator, the high reported conditional mutual information could reflect fitting the estimator rather than actual information content. The downstream fairness metrics in Table 4 are more independent, but they do not by themselves establish that Z is a representation of conditional entropy.","section":"Section 4.3, Eq. (10); Tables 1-3"},{"comment":"No theoretical guarantee is provided that a finite value of the trade-off parameter lambda in the Lagrangian relaxation Eq. (6) yields a representation with I(X;Z) = 0 or with I(Z;Y|X) = H(Y|X). Appendix G shows that I(X;Z) varies by orders of magnitude as lambda changes, but the paper gives no criterion for selecting lambda to satisfy the constraint, and the algorithm in Section 4.3 does not actually implement lambda (Algorithm 1 line 8 uses lambda = 1 implicitly). The claimed equivalence between the optimized objective and the target conditional-entropy sector is therefore not established.","section":"Section 2, Eq. (6); Appendix G"},{"comment":"Algorithm D.1, which underpins the real-data fair-learning and domain-generalization results, is internally inconsistent. The inner loop samples (x,c), but line 12 sets z <- NA(y) with y undefined, and line 13 uses NC(x,y,z) even though the task is to debias X with respect to C; the outer loop correctly uses z <- NA(x). As written, the algorithm cannot be executed, so the Adult and Cover Type results are not reproducible from the pseudocode.","section":"Appendix D, Algorithm D.1"}],"minor_comments":[{"comment":"Table 3 reproduces the column headers of Table 1 (\"I(S;G)\", \"H(G)\", \"H(G|S)\", etc.) even though the Adult experiment concerns X, C, and Z; the quantity reported in each column is therefore undefined.","section":"Table 3"},{"comment":"Algorithm 1 line 8 minimizes I(X;Z) - I(Y;X,Z), which corresponds to lambda = 1, while Eq. (10) and Appendix G describe a tunable lambda; the sensitivity analysis is not connected to the algorithm as stated.","section":"Algorithm 1, Eq. (10)"},{"comment":"The sentence beginning \"The generated representation Z provides an additional 1.44 bits...\" ends with \"contain minimal information about G,\" but the preceding discussion and Table 1 concern information about S; this appears to be a typo for S.","section":"Section 5.1, text near Eq. (12)"},{"comment":"The three panels of Figure 4 all share the caption label \"(c)\"; distinct letters are needed for the dynamics of S and G, Z and G, and Z and S.","section":"Figure 4"},{"comment":"Equation (A.1) writes the discrete increment as (1/Delta t) times the derivative, which is not the standard Euler update; with Delta t = 800 this would produce a very small increment, so this is presumably a typo for Delta t times the derivative.","section":"Appendix A, Eq. (A.1)"},{"comment":"The real-data results are reported without error bars or repeated-seed variation, and the fairness metrics are not accompanied by confidence intervals or significance tests.","section":"Tables 4 and 5"}],"recommendation":"reject","confidential_remarks":"The paper's own Table 2 contains impossible mutual-information values, and Appendix B concedes that the framework cannot disentangle information sectors. These are not local presentation issues: they invalidate the central claim. The code release and the general idea of iterative information subtraction may be worth developing, but the current manuscript does not provide a sound validation of its main contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline is that the paper's central empirical claim is dead on arrival: Table 2 reports I(Z;X)=2.98 bits when X is a three-level categorical variable, so H(X) ≤ log2(3) ≈ 1.585 bits. Mutual information cannot exceed the entropy of one of the variables, so that number is impossible. Moreover, since Z is generated from Y alone, X→Y→Z forms a Markov chain, and the data processing inequality forces I(X;Z) ≤ I(X;Y) = 0.25 bits (their own estimate). The reported value is off by more than an order of magnitude. This is not a small estimation error; it means the SMILE/MINE estimators are producing garbage in this regime.\n\nWhat's genuinely new here is the framing: describing conditional entropy representation as a simultaneous maximization/minimization of two information terms, and the iterative sector-by-sector decomposition of the Venn diagram. That conceptual packaging is clean, and the two-discriminator generative architecture is a reasonable way to attempt it. The authors also provide code, which is commendable.\n\nThe soft spots are serious, though. The core evaluation relies on the same family of neural MI estimators used as the training objective. Since the estimators are demonstrably unreliable here, the reported 'representations of conditional entropy' may be nothing more than artifacts of a bad estimator. Appendix B concedes that the framework 'cannot effectively disentangle' the sectors, which undercuts the main claim of Section 5.2. There are no baselines anywhere, so even the fair learning and domain generalization numbers are hard to interpret. The domain generalization result is particularly weak: Z alone is worse than X alone, and the 'less is more' explanation for why X+Z helps is post hoc.\n\nThe paper is written for an audience interested in information-theoretic representation learning, and the idea might be salvageable. But in its current form the evidence does not support the abstract's promise of 'eliminating the undesired' information. I would not send this to a serious referee as is; a desk reject with encouragement to redo the experiments with proper MI estimation (or ground-truth checks on synthetic data) and add baselines would be appropriate.\n\nRecommendation: reject with a clear, concrete path to revision.","headline":"The paper's main empirical evidence is impossible—Table 2 reports I(Z;X) exceeding H(X) for a 3-level categorical variable—so the central claim collapses, despite an interesting conceptual framing.","tokens_in":14559,"tokens_out":3782,"would_cite":false,"duration_ms":36095,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","94A17"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a generator trained to maximize $I(Y;X,Z)$ while minimizing $I(X;Z)$ learns a representation of the conditional entropy $H(Y|X)$ in continuous variables, and that iterating the procedure decomposes multivariate…","keywords":["conditional entropy representation","information subtraction","mutual information neural estimation","fair representation learning","domain generalization","generative model","continuous conditioning variables","disentanglement"],"falsifier":"Compute the exact mutual informations $I(X;Z)$ and $I(Y;X,Z)$ on a small synthetic system where $H(Y|X)$ is known analytically, after training with the proposed objective; the claim stands only if $I(X;Z)$ is near zero and $I(Y;X,Z)$ approaches $H(Y)$. A compact check already lies in the paper's own synthetic fair-learning table, where the trained $Z$ has $I(Z;X)=2.98$ against $I(Z;Y)=0.19$, and in Appendix B where sector representations retain substantial conditional-variable information; reproducing those experiments with tighter estimators would settle whether the subtraction is real.","tokens_in":13562,"feed_emoji":"🧩","tokens_out":10474,"duration_ms":92865,"temperature":0.7,"pith_summary":"This paper introduces Information Subtraction, a way to train a representation $Z$ of a target variable $Y$ so that $Z$ keeps the information $Y$ carries while dropping the information supplied by a conditioning variable $X$. Concretely, the training objective maximizes the mutual information $I(Y;X,Z)$ and simultaneously minimizes $I(X;Z)$; a generator maps $Y$ to $Z$, and two neural discriminators estimate the two information terms. The paper argues that the optimizer of this objective represents the conditional entropy $H(Y|X)$—the part of $Y$ that $X$ does not explain—and that applying the subtraction iteratively yields representations of conditional mutual informations and even the trivariate mutual information, for continuous variables and without distributional assumptions. Experiments on a predator-prey system, the Adult income data, and the Cover Type data show the representation carrying semantic features of the conditional entropy, improving group fairness when sensitive attributes are subtracted, and improving out-of-distribution accuracy when universal and domain-specific features are combined. A sympathetic reader would care because it promises a flexible, distribution-free tool for saying what a variable uniquely explains, with direct uses in fair learning and domain generalization.","feed_headline":"Subtraction training isolates what X can't explain about Y","feed_subtitle":"The objective drops one mutual information while keeping another, giving fairer and more general models.","key_machinery":"The central machinery is a three-network architecture with one generator and two discriminators. The generator $N_A$ maps the target $Y$ to the representation $Z$. Discriminator $N_C$ estimates $I(Y;X,Z)$ through the Donsker–Varadhan representation, and discriminator $N_D$ estimates $I(X;Z)$; both are implemented as MINE/SMILE neural estimators, which make the information terms differentiable with respect to $Z$ and therefore trainable by backpropagation. Training first reconstructs $Y$ from $Z$ to initialize useful features and then updates the generator along $\\nabla(\\hat{I}(Y;X,Z)-\\lambda \\hat{I}(X;Z))$. Because the method only needs neural estimates of two mutual informations, it makes no distributional assumption on $X$, $Y$, or $Z$ and naturally handles continuous conditioning variables. The iterative extension reuses the same objective with different target/condition pairs, so the same machinery produces $Z_1,\\ldots,Z_7$ for all sectors of a three-variable Venn diagram.","core_discovery":"On the paper's own terms, the central discovery is that conditional information can be generated rather than sampled: instead of selecting training samples that share the conditioning value, as conditional contrastive methods do, one can write down the objective $\\max_{P_{Z|Y}} I(Y;X,Z) - \\lambda I(X;Z)$ and train a generator against two mutual-information discriminators to realize it. The paper claims the resulting $Z$ contains the information of $H(Y|X)$, and it reports evidence in a Lotka–Volterra ecosystem: with $I(S;G)=0.89$, $H(G)=3.28$, and $H(G|S)=2.39$, the learned representation achieves $I(Z;G)=2.06$, $I(Z;S)=0.28$, and $I(Z;G|S)=1.44$ bits. Iterating the procedure on three variables is claimed to decompose the joint information into seven mutually exclusive sectors—conditional entropies, conditional mutual informations, and the trivariate interaction—so that arbitrary information terms can be assembled by adding sectors. In the fair-learning experiment, subtracting the protected attribute $C$ from features $X$ yields $Z$ with near-zero mutual information to $C$, and prediction from $Z$ raises balanced accuracy from 0.628 to 0.810 while cutting the group gap roughly in half. In domain generalization, adding the subtracted representation $Z$ to original features $X$ raises test accuracy from 0.566 to 0.598 on an unseen region.","pith_inferences":["Because the training objective is a weighted sum of two separately estimated information terms, the method's guarantee is only as good as the mutual-information estimators; an obvious next test is swapping SMILE for tighter estimators or for exact MI in small synthetic systems and checking whether the reported leakage shrinks.","The iterative sector decomposition assumes that subtracting a conditioned code from an already-subtracted code yields an independent sector; enforcing orthogonality or statistical independence between the generated $Z_i$ seems a natural extension the paper leaves open.","A testable application beyond the paper: use Information Subtraction on temporal data to isolate the part of one time series not explained by another, which would connect the method to Granger-causality and unobserved-confounding analyses.","The fair-learning result suggests treating protected/discrete variables and continuous nuisance variables in one unified subtraction scheme; extending the current discrete protected-attribute experiments to continuous sensitive variables (e.g., age) would test the method's central promise."],"forward_implications":["For any pair of continuous variables, this objective gives a trainable code that purports to isolate the information in $Y$ not carried by $X$, eliminating the need for conditional sampling or clustering.","Iterating the subtraction decomposes a multivariate system's joint information into mutually exclusive sectors, so representations of conditional entropy, conditional mutual information, and higher-order interactions become obtainable from one architecture.","Subtracting protected attributes from features before classification can produce substantially fairer predictors: the Adult experiment reports balanced accuracy rising from 0.628 to 0.810 and group-gap metrics dropping by more than half.","In domain generalization, adding the subtracted 'universal' representation to the original domain-specific features can improve accuracy on an unseen domain, here from 0.566 to 0.598 on the held-out Cover Type region.","If the representation truly equals $H(Y|X)$, the same code can serve as a semantic decomposition: in the predator-prey system it encodes peaks of grass population that sheep population alone cannot predict."],"supporting_citations":[{"why":"Supplies the MINE lower-bound estimator that the discriminators use to make both information terms differentiable and trainable.","marker":"Belghazi et al. 2018"},{"why":"Provides SMILE, the stabilized variant of MINE used in the architecture for convergent information estimates.","marker":"Song and Ermon 2020"},{"why":"Gives the earlier objective and architecture (maximizing $I(Y;X,Z)$) that this work extends with the subtraction term.","marker":"Leong et al. 2023"},{"why":"The conditional-contrastive baseline for discrete sensitive variables that this paper's continuous-variable framework is designed to generalize.","marker":"Ma et al. 2021"},{"why":"Proposes kernel/clustering conditioning for continuous variables, the approach Information Subtraction aims to replace with direct information optimization.","marker":"Tsai et al. 2022"},{"why":"Provides the Adult census dataset used in the fair-learning experiment.","marker":"Becker and Kohavi 2024"},{"why":"Provides the Cover Type dataset used in the domain-generalization experiment.","marker":"Blackard 1998"},{"why":"Supplies the information-plane view used in the sensitivity analysis of the trade-off hyperparameter $\\lambda$.","marker":"Zhao et al. 2022"}],"fun_headline_variants":["Generative subtraction learns conditional entropy without sampling","Subtraction objective isolates conditional information in continuous data","New generative framework extracts conditional entropy for fairer models","Information subtraction distills what Y hides from X","Continuous conditional info via generative mutual-information trade-off"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that training on the surrogate objective $I(Y;X,Z)-\\lambda I(X;Z)$, using neural-network estimates of the two mutual informations and gradient descent, actually drives the true information $I(X;Z)$ to zero while preserving the true $I(Y;X,Z)$; if the estimator gradients diverge from the true information quantities, the learned code can be heavily contaminated by $X$ even when the training loss looks correct.","fun_headline_variants_meta":{"raw":{"variants":["Generative subtraction learns conditional entropy without sampling","Subtraction objective isolates conditional information in continuous data","New generative framework extracts conditional entropy for fairer models","Information subtraction distills what Y hides from X","Continuous conditional info via generative mutual-information trade-off"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000515,"raw_usage":{"total_tokens":2532,"prompt_tokens":1008,"completion_tokens":1524,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":624,"completion_tokens_details":{"reasoning_tokens":1452}},"tokens_in":624,"tokens_out":1524,"duration_ms":11434,"temperature":1.0,"reasoning_tokens":1452,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:31:52.962885+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the exact mutual informations $I(X;Z)$ and $I(Y;X,Z)$ on a small synthetic system where $H(Y|X)$ is known analytically, after training with the proposed objective; the claim stands only if $I(X;Z)$ is near zero and $I(Y;X,Z)$ approaches $H(Y)$. A compact check already lies in the paper's own synthetic fair-learning table, where the trained $Z$ has $I(Z;X)=2.98$ against $I(Z;Y)=0.19$, and in Appendix B where sector representations retain substantial conditional-variable information; reproducing those experiments with tighter estimators would settle whether the subtraction is real.","supporting_citations":[],"review_version":1}