{"id":"f0c26eaa-0ee0-4af1-b1af-f2c8af737543","arxiv_id":"1908.09961","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"New information-theoretic metrics (RMIG, JEMMIG, WSEPIN) rank disentanglement models consistently, but they are sensitive to quantization choices and high latent dimensionality.","lead":"This paper defines disentangled representations with information theory and proposes new evaluation metrics, including RMIG, JEMMIG, and WSEPIN, to compare models like FactorVAE and beta-VAE. The metrics rank these models consistently on dSprites and CelebA, but some scores depend on user-chosen settings and one metric becomes unstable with many latent variables.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own ablations show WSEPIN becomes negative for >=15 latents and JEMMIG rises with bin count, yet the conclusion still names these 'key' metrics; no quantitative ranking-stability evidence supports the robustness claim.","rationale":"The reader's verdict of CONDITIONAL is appropriate. I searched for internal inconsistencies in the formal definition (Def. 1) and found that Eq. 6, while at first glance odd, is actually equivalent to Eq. 4 under standard entropy inequalities, so the theory is not invalid on that ground. The load-bearing issue is the robustness of the proposed estimators: the paper's own ablation (A.5) reports WSEPIN instability for latents >=15 and monotone bin-count dependence for JEMMIG. The text asserts that relative order is preserved but provides no quantitative evidence. This directly bears on the strongest claim (Section 5) about robustness. The concrete ranking-stability check would settle whether the concern lands. If rankings are stable across #bins and latent counts, the conditional can be lifted; if not, the metrics' advertised robustness fails. Because the paper discloses the ablation and the core idea (classifier-free information-theoretic metrics) is still valuable, CONDITIONAL remains the right verdict rather than REJECT.","tokens_in":28396,"tokens_out":11125,"duration_ms":112543,"concrete_test":"Re-run the model evaluations on dSprites using the published code, for all FactorVAE/beta-VAE checkpoints, computing (i) JEMMIG and RMIG with #bins = 50, 100, 200, 400 and value range [-4,4], and (ii) WSEPIN after appending 0, 5, 10, 15 i.i.d. N(0,1) dummy latents to each model's encoder output. Compute Kendall's tau between the model rankings at each setting. If any pairwise tau is below 0.9, or if WSEPIN changes sign/magnitude materially with dummy latents, the claim that relative order is preserved fails and the metrics cannot be called robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of Section 5 is that the proposed metrics are \"more robust\" and produce evaluations \"compatible with visual results.\" That claim rests on the metrics ordering models consistently under user-chosen estimation settings. The paper's own ablation in Appdx A.5 contradicts this: Fig. 19e,f shows I(x,zi|z-i) (and hence WSEPIN) becomes negative and unstable when the number of latents is 15 or 20, and Fig. 19f shows WSEPIN scores that are clearly not invariant to latent count. In the same section, Fig. 17 shows normalized JEMMIG increases monotonically with #bins; the paper states that \"the relative order among different models remains the same\" but gives no numerical measure (e.g., rank correlation) to support this. Moreover, the main experiments use #bins=100 (Appdx A.4) while the RMIG ablation shows stability only for #bins>200 (Appdx A.5). Thus a user following the paper's defaults is in the non-robust regime for RMIG, and a user comparing models with different latent dimensionality cannot rely on WSEPIN at all. Without a quantitative demonstration that rankings are invariant over reasonable settings, the headline robustness claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an information-theoretic characterization of disentangled representations along three dimensions—informativeness, separability, and interpretability—and introduces several quantitative metrics: MISJED, WSEPIN/SEPIN@k, WINDIN/INDIN@k, RMIG, and JEMMIG. The definitions are used to analyze FactorVAE, β-VAE, and AAE on dSprites and CelebA, and the proposed metrics are compared with existing ones such as MIG, DCI scores, modularity/explicitness, and Z-diff. The central claim is that the new metrics are robust, require no classifiers, support real and continuous-factor settings, and produce rankings consistent with visual inspection.","tokens_in":28667,"tokens_out":6434,"duration_ms":65501,"significance":"If the robustness claim holds, the paper would provide a useful, classifier-free evaluation toolkit for disentanglement learning and a formal framework connecting three commonly discussed properties. The empirical study is broad, includes comparisons with MIG and DCI-style metrics, and provides external anchors via dSprites ground-truth factors and visualizations. The paper also ships code, which aids reproducibility. However, the headline robustness claim is weakened by the paper's own ablations: WSEPIN becomes negative and unstable for 15+ latents, and JEMMIG scores change monotonically with the number of bins. These are load-bearing limitations because WSEPIN and JEMMIG are named as the two key metrics in the conclusion. The framework is still valuable, but the robustness claim needs to be either repaired with additional evidence or substantially qualified.","major_comments":[{"comment":"The paper's own ablation shows that I(x,zi|z_not_i), the quantity underlying WSEPIN, becomes negative when the number of latents is 15 or 20, and the text states that \"This causes the instability of WSEPIN.\" Since Section 5 names WSEPIN as one of the two key metrics, and since the paper itself uses 65 latents on CelebA and 10 on dSprites, WSEPIN cannot be used to compare models with different latent dimensionalities—exactly the situation in which a robust metric is needed. The robustness claim is therefore unsupported as stated. The paper should either repair the estimator, restrict WSEPIN's scope explicitly, or provide quantitative evidence that WSEPIN rankings are stable across latent counts in the regime where it is nonnegative.","section":"Appendix A.5, Fig. 19(e)-(f)"},{"comment":"The ablation shows normalized JEMMIG increasing monotonically with the number of bins, while RMIG stabilizes only for #bins > 200, yet the main experimental tables and figures (e.g., Tables 6-9, Figs. 13 and 25) report scores at #bins = 100. The text asserts that \"the relative order among different models remains the same\" without presenting any rank-correlation or order-flip analysis. Since JEMMIG is introduced as a robust metric and its value depends on a user-chosen quantization granularity, the paper needs to demonstrate ranking invariance quantitatively (e.g., Spearman rank correlation over a grid of #bins values) and either report metrics in the stable regime or justify the choice of #bins = 100.","section":"Appendix A.5, Fig. 17"}],"minor_comments":[{"comment":"Definition 1 writes the full interpretability condition as I(zi,yk) = H(zi,yk), whereas Section 2 defines full interpretability as I(zi,yk) = H(zi) = H(yk). The two conditions are equivalent only under equal entropy; please state the intended condition explicitly to avoid confusion.","section":"Section 2, Eq. (6)"},{"comment":"The definitions of WSEPIN and WINDIN do not show the normalization by H(zi) that is mentioned in the preceding paragraph; please make the formulas self-contained so that the reported numerical values can be reproduced unambiguously.","section":"Section 3.2"},{"comment":"Several tables report normalized scores but do not state in the captions whether normalization is by log(#bins) or by H(yk); adding this information would improve reproducibility.","section":"Appendix A.4, Tables 5-9"},{"comment":"The claim that FactorVAE learns \"statistically equivalent factors\" across runs is supported only by visual inspection; a quantitative measure, such as the correlation of factor orderings across seeds, would strengthen this claim.","section":"Appendix A.15, Figs. 26-28"}],"recommendation":"major_revision","confidential_remarks":"The paper's own ablation study materially undercuts the headline robustness claim, and the conclusion still names the unstable WSEPIN and the bin-count-sensitive JEMMIG as the key metrics. If the authors can add quantitative rank-stability evidence and appropriately qualify the claims, the paper could be defensible; otherwise the central claim is not established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I just read arXiv:1908.09961. The short version: the paper has a solid core—new information-theoretic metrics for disentangled representations with code and broad experiments—but the 'robust' claim in the title and conclusion is not supported by their own ablation study. I'd still send it to a serious reviewer, but the authors need to either soften the claim or add rank-stability numbers.\n\nThe genuinely new pieces: they frame disentanglement as three quantities—informativeness, separability, interpretability—and instantiate them as MI-based scores. RMIG is a sensible fix to MIG's probabilistic assumption, JEMMIG is a self-contained interpretability metric, WSEPIN/SEPIN@k address the label-free case, and MISJED handles pair independence. The point that many prior metrics use E_q(zi|x)[zi] rather than the full q(zi|x) is correct and matters in practice. The dSprites comparisons against MIG and DCI are informative, and the code is public.\n\nThe soft spots are right where the stress-test note puts them. In Appendix A.5, WSEPIN goes negative and unstable at 15–20 latents; the paper acknowledges this but still calls WSEPIN a 'key metric' in the conclusion. JEMMIG varies monotonically with the number of bins, and the claim that model rankings are preserved is asserted without any rank-correlation measurement. The main experiments use #bins=100, while the RMIG stability is only demonstrated for #bins>200. So a user following the paper's defaults is in the non-robust regime. That is not a fatal flaw—many metrics need tuning—but the robustness framing in Table 1 and Section 5 overstates what is shown.\n\nI don't think the metrics are circular: they anchor on dSprites ground truth and agree with MIG/DCI where expected, so the framework is doing real work. But the definition and metric are close together, and the paper would be stronger if it validated rankings under settings variation rather than just mean scores.\n\nWho should read it: anyone evaluating disentanglement methods, especially VAE variants. I would cite RMIG as a baseline if I were working in that area, but I'd be careful with WSEPIN in high-dimensional latent spaces. For peer review: yes, it deserves a referee. The core is sound, the empirical work is broad, and the overclaim is fixable with a few extra analyses.","headline":"Solid metrics paper, but the robustness claim outruns the ablation evidence.","tokens_in":29183,"tokens_out":3803,"would_cite":true,"duration_ms":37156,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper defines disentangled representations as informative, separable, and interpretable, then builds metrics that measure all three with mutual information.","keywords":["disentangled representations","mutual information","evaluation metrics","representation learning","variational autoencoders","informativeness","separability","interpretability"],"falsifier":"Rank two trained models on dSprites with JEMMIG and WSEPIN at bin counts 50, 100, 200, and 500 while keeping all other settings fixed; if the relative order of the two models flips at any step, the claimed consistency of the metrics across settings fails. Equivalently, evaluate a beta-VAE with 20 latents: WSEPIN becoming negative, as in the paper's Appendix Figure 19f, would show the separability metric is not robust at latent counts the paper itself targets.","tokens_in":28200,"feed_emoji":"🧩","tokens_out":7842,"duration_ms":70750,"temperature":0.7,"pith_summary":"This paper tries to give disentanglement learning a measurement standard. It defines a disentangled representation as one that is informative about the data, separable from the other learned latents, and interpretable as a match to a human-labeled factor of variation, and it expresses all three properties as mutual-information conditions. On top of that definition it builds evaluation metrics—MISJED for pairwise independence, WSEPIN and SEPIN@k for separability-plus-informativeness without labels, and RMIG and JEMMIG for interpretability with labels—that require no trained classifier and can run on real, continuous-factor data. The authors' central claim is that these metrics rank existing methods such as FactorVAE, beta-VAE, and AAE the way a human inspecting generated traversals would, giving the field a fair ground for comparison.","feed_headline":"Mutual-information metrics put a number on disentanglement","feed_subtitle":"WSEPIN, RMIG and JEMMIG match visual inspection when ranking FactorVAE, beta-VAE and AAE.","key_machinery":"The load-bearing object is the learned encoder distribution q(zi|x) treated as a stochastic map, with all quantities computed from it rather than from decoder reconstructions or classifier weights. The identities that carry the argument are the multivariate mutual-information decomposition I(x, zi, zj) = I(x, zi) + I(x, zj) − I(x, (zi, zj)) = I(zi, zj) − I(zi, zj|x), which defines separability with respect to x, and the conditional mutual information I(x, zi | z≠i) = I(x, zi) − I(zi, z≠i) + I(zi, z≠i|x), whose informativeness-weighted average becomes WSEPIN. For interpretability the key construct is the gap between the top two latents' mutual information with factor yk: RMIG = I(zi*, yk) − I(zj°, yk), and JEMMIG = H(zi*, yk) − I(zi*, yk) + I(zj°, yk), with JEMMIG being the single metric that certifies the paper's full definition. Entropy and mutual-information terms are estimated by quantization on a fixed value range [−4, 4] with a chosen number of bins, or by Monte Carlo sampling.","core_discovery":"On the paper's own terms, the central discovery is that 'disentangled' need not be a vague visual impression or a property of a particular architecture. It is the conjunction of three conditions on the learned encoder q(z|x): informativeness I(x, zi) large; full separability I(zi, z≠i) = 0; and, relative to a labeled factor yk, interpretability I(zi, yk) = H(yk), or the stronger invertibility condition I(zi, yk) = H(zi) = H(yk). The paper then claims the proposed metrics measure these conditions robustly. RMIG subtracts the second-best latent's mutual information with yk from the best one under a probabilistic assumption p(zi, yk, x) = q(zi|x) p(yk|x) p(x), avoiding MIG's need to model p(x|yk). JEMMIG adds the joint-entropy term H(zi*, yk) so that only a latent matching yk alone scores well. WSEPIN weights each latent's conditional information I(x, zi | z≠i) by its informativeness so noisy latents cannot drag the score down. Empirically the paper finds FactorVAE's representations score more interpretable than beta-VAE's or AAE's, matching visual traversals, and that FactorVAE learns a consistent set of factors across runs, analogous to PCA.","pith_inferences":["If the metrics are adopted, the practical bottleneck shifts from choosing an architecture to standardizing the evaluation protocol; a shared bin-count and value-range convention would make published JEMMIG numbers comparable, and the paper's own bin-sensitivity suggests that convention must be chosen before numbers are reported.","The stability failure of WSEPIN at 15 to 20 latents hints that a corrected estimator, for example replacing the joint entropy H(z≠i) with pairwise or blockwise conditional entropies, could extend the method to high-dimensional latent spaces; this is a testable extension the paper does not pursue.","The analogy between FactorVAE and PCA could be pushed further: if independence constraints force a unique solution to the information-bottleneck objective, then the same consistent representations should appear in other total-correlation-penalized models, making identifiability a property of the loss rather than of the data.","A natural downstream check of the metrics' value is whether models ranked higher by JEMMIG also transfer better to unseen tasks; the paper does not run that check, but its definition would predict a positive relationship."],"forward_implications":["Model comparison no longer needs to train a classifier for each factor, so results from different papers can be compared as long as they fix the same binning and value range.","JEMMIG can be used as a self-contained test for whether a learned latent is disentangled with respect to a known factor, because it combines separability and interpretability in one number.","WSEPIN gives a ground-truth-free monitor of disentanglement on real datasets like CelebA, where the true generative factors are only partially known.","The empirical ranking, with FactorVAE above beta-VAE and AAE in interpretability and beta = 10 peaking WSEPIN, gives a benchmark expectation for future models on dSprites and CelebA.","Because RMIG supports continuous factors and can take p(yk|x) from an external expert model, the metrics extend to supervised and semi-supervised settings where MIG cannot be applied."],"supporting_citations":[{"why":"Supplies the beta-VAE objective and the Z-diff baseline that the interpretability metrics are compared against.","marker":"Higgins et al., 2017a"},{"why":"Supplies the FactorVAE model and total-correlation objective that the experiments evaluate.","marker":"Kim & Mnih, 2018"},{"why":"Supplies the MIG metric whose probabilistic assumption RMIG replaces and with which RMIG is compared on dSprites.","marker":"Chen et al., 2018"},{"why":"Supplies the disentanglement, completeness, and informativeness metrics shown to disagree with visual results.","marker":"Eastwood & Williams, 2018"},{"why":"Supplies modularity and explicitness metrics whose use of conditional means the paper argues distorts mutual-information estimates.","marker":"Ridgeway & Mozer, 2018"},{"why":"Supplies the AAE model used as a third evaluation target.","marker":"Makhzani et al., 2015"},{"why":"Supplies the dSprites toy dataset with known ground-truth factors used to validate RMIG against MIG.","marker":"Matthey et al., 2017"},{"why":"Supplies the CelebA real-image dataset with 40 binary attributes used to test the metrics on real data.","marker":"Liu et al., 2015"},{"why":"Supplies the common-assumptions challenge and the MIG implementation whose bin-range choices the paper identifies as unstable.","marker":"Locatello et al., 2019"},{"why":"Supplies the multivariate mutual-information decomposition used to define separability with respect to the data.","marker":"McGill, 1954"}],"fun_headline_variants":["New metrics make disentanglement measurable","Disentanglement defined and scored via information theory","FactorVAE tops new disentanglement metrics","RMIG, JEMMIG, WSEPIN quantify disentanglement","Disentanglement: a precise definition plus robust metrics"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The metrics are only fair if every model is evaluated with the same binning and sampling settings, because the entropy estimates depend on those settings—the paper's own ablation shows WSEPIN turns negative and unstable when the latent count reaches 15 or 20, and JEMMIG rises as the number of bins grows.","fun_headline_variants_meta":{"raw":{"variants":["New metrics make disentanglement measurable","Disentanglement defined and scored via information theory","FactorVAE tops new disentanglement metrics","RMIG, JEMMIG, WSEPIN quantify disentanglement","Disentanglement: a precise definition plus robust metrics"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000228,"raw_usage":{"total_tokens":1501,"prompt_tokens":999,"completion_tokens":502,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":430}},"tokens_in":615,"tokens_out":502,"duration_ms":5255,"temperature":1.0,"reasoning_tokens":430,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:56:33.429037+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rank two trained models on dSprites with JEMMIG and WSEPIN at bin counts 50, 100, 200, and 500 while keeping all other settings fixed; if the relative order of the two models flips at any step, the claimed consistency of the metrics across settings fails. Equivalently, evaluate a beta-VAE with 20 latents: WSEPIN becoming negative, as in the paper's Appendix Figure 19f, would show the separability metric is not robust at latent counts the paper itself targets.","supporting_citations":[{"cited_title":"Disentangling by factorising","cited_arxiv_id":null,"evidence_quote":"Supplies the FactorVAE model and total-correlation objective that the experiments evaluate."},{"cited_title":"A framework for the quantitative evaluation of disentangled representations","cited_arxiv_id":null,"evidence_quote":"Supplies the disentanglement, completeness, and informativeness metrics shown to disagree with visual results."},{"cited_title":"Learning deep disentangled embeddings with the f-statistic loss","cited_arxiv_id":null,"evidence_quote":"Supplies modularity and explicitness metrics whose use of conditional means the paper argues distorts mutual-information estimates."},{"cited_title":"Challenging common assumptions in the unsupervised learning of disentangled representations","cited_arxiv_id":null,"evidence_quote":"Supplies the common-assumptions challenge and the MIG implementation whose bin-range choices the paper identifies as unstable."},{"cited_title":"Multivariate information transmission","cited_arxiv_id":null,"evidence_quote":"Supplies the multivariate mutual-information decomposition used to define separability with respect to the data."}],"review_version":1}