{"id":"0bb227dd-5557-4249-a6c4-acea3146d3ef","arxiv_id":"2501.15705","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper proposes a PCA-based method to find latent directions for each generative factor, creating disentanglement metrics that work for latent variable models with non-axis-aligned encodings, and reports improved scores on DSprites and 3D Shapes.","lead":"This paper proposes a way to evaluate disentanglement in deep latent variable models by finding latent directions that match ground-truth generative factors, rather than assuming factors align with coordinate axes. A reader might care because many modern models, such as those matching aggregate posterior distributions, encode factors in rotated directions and score poorly under existing metrics.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The min-variance eigenvector in Algorithm 1 may select dead or uninformative latent dimensions rather than the true factor direction; without a random-encoder control, the PCA-MIG improvements in Table I could be artifacts of the metric rather than evidence of disentanglement.","rationale":"The reader's weakest assumption matches my primary concern: Algorithm 1's choice of the minimum-variance PCA eigenvector is load-bearing because the entire evaluation rests on it. I agree with the conditional verdict. The paper does not provide code, a random-encoder baseline, or any evidence that the estimated directions correspond to the true generative factors beyond visual latent traversals for two models. The min-variance criterion in particular is vulnerable to dead latent dimensions, which the paper itself identifies as a common failure mode ('posterior collapse'). If a dimension has zero variance, it will be selected for every factor, and the resulting 'directions' are meaningless. A random-encoder control would settle whether the reported score improvements are an artifact of the metric's flexibility. The method could be salvageable with such a control and with a comparison against known factor directions, but as written the central claim is only conditionally supported. I therefore keep the reader's verdict unchanged and agree that additional controls are required.","tokens_in":7974,"tokens_out":8047,"duration_ms":75045,"concrete_test":"Run Algorithm 1 and the PCA FactorVAE/PCA MIG metrics exactly as described in Section II, but substitute a randomly initialized (untrained) encoder with the same architecture and dataset. If the untrained encoder gives PCA MIG scores substantially above zero (or PCA FactorVAE accuracy above 1/6), the minimum-variance selection is capturing noise/dead directions, and the improvements in Table I cannot be attributed to disentanglement. A complementary check: take a trained VAE, set three of the six latent units to a constant value, and observe whether Algorithm 1 selects those dead dimensions; if it does, the method fails precisely in the regime the paper's own introduction states is common.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1 (lines 10-11) selects the PCA eigenvector with minimum variance from encodings of samples in which generative factor Fi is held fixed. The implicit assumption is that the latent direction encoding Fi is invariant when Fi is fixed and therefore has the smallest variance, while other factors induce variance along other directions. This assumption is not secure in two common regimes. First, if any latent dimension is unused (posterior collapse or dead units), that dimension has near-zero variance in every subset, so the minimum-variance eigenvector is a dead dimension no matter which factor is fixed; repeated runs produce arbitrary directions in that near-zero subspace, and the mean-outer-product step (lines 15-20) may output a direction that does not correspond to Fi at all. Second, if the representation is entangled, no direction is invariant under Fi, and the minimum-variance direction can be a spurious combination of axes with accidentally low variance. The paper provides no validation that the discovered directions actually encode the intended factors: no random/untrained encoder baseline, no test with injected dead dimensions, and no comparison of the estimated directions against known ground-truth factor axes. Consequently, the large PCA-MIG gains in Table I (e.g., AVAE 0.67 vs 0.08 on 3D Shapes) could arise from the metric's ability to fit arbitrary low-variance directions to the labeled evaluation data, not from genuine disentanglement. The central empirical claim is therefore not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a method for evaluating disentanglement in deep latent variable models (DLVMs) that does not assume the generative factors are aligned with cardinal latent axes. The method first estimates a latent direction for each ground-truth factor by fixing that factor, encoding the resulting samples, and taking the minimum-variance PCA eigenvector of the encoded set (Algorithm 1). These directions are then used to define two metrics, PCA FactorVAE and PCA MIG, which are evaluated on DSprites and 3D Shapes across several VAE variants, including aggregate-posterior matching models such as AVAE. Table I reports substantial improvements over the standard FactorVAE and MIG metrics, particularly for AVAE on 3D Shapes (PCA MIG 0.67 vs MIG 0.08).","tokens_in":8316,"tokens_out":3147,"duration_ms":30366,"significance":"The paper addresses a real gap: existing disentanglement metrics assume axis-aligned latent factors, and models that match the aggregate posterior to an isotropic prior may encode factors along rotated directions. If the proposed direction-finding procedure were validated, the PCA-based metrics could be a useful extension of FactorVAE and MIG to a broader class of DLVMs. The paper also provides a useful pairwise-angle diagnostic (Figure 2) and compares multiple models and seeds. However, as presented, the empirical claims are not independently supported: the direction-finding step uses the same labeled evaluation data that is later used for scoring, and no control is provided to show that the discovered directions are not spurious or overfit. The central claim therefore requires substantial additional validation before it can be accepted.","major_comments":[{"comment":"The direction-discovery procedure in Algorithm 1 uses the ground-truth factor labels of the evaluation dataset to select latent directions, and the same labeled data is then used to compute the PCA FactorVAE and PCA MIG scores reported in Table I. This creates a form of label leakage: the metric is fit to the labels and then evaluated on the same labels, so the reported improvements (e.g., AVAE PCA MIG 0.67 vs MIG 0.08 on 3D Shapes) are not independent measurements of disentanglement. The authors should either split the data into a direction-discovery set and a separate evaluation set, or report cross-validated scores, to ensure that the directions are not overfitting the evaluation data.","section":"Section II-B and II-C; Table I"},{"comment":"The key assumption is that when a generative factor is fixed, the minimum-variance PCA eigenvector of the encoded samples points along the latent direction encoding that factor. This assumption is not validated. If a latent dimension is unused or has near-zero variance (posterior collapse or dead units), that dimension will be selected as the minimum-variance direction regardless of which factor is fixed, and repeated runs may produce arbitrary directions in the near-zero subspace. The authors should provide a random-encoder baseline, an untrained-encoder control, or an analysis of the eigenvalue spectra to demonstrate that the discovered directions are not artifacts of low-variance or unused dimensions.","section":"Algorithm 1, lines 10-11"},{"comment":"The aggregation step estimates the final direction as the principal eigenvector of the mean outer product of per-trial minimum-variance eigenvectors. This step is not robust when the per-trial eigenvectors are poorly determined: when variances are near zero, the eigenvector direction can be arbitrary, and the sign ambiguity of eigenvectors is not addressed. The authors do not provide any statistical justification for this estimator, nor do they discuss conditions under which the principal eigenvector is uniquely defined. A simulation study with known factor directions and controlled noise would help establish whether the estimator recovers the intended directions.","section":"Algorithm 1, lines 15-20"},{"comment":"The text states, 'Other than the regular VAE, we consider different variations of the VAE ... such as the FactorVAE, and beta-TCVAE, for comparison. We do not consider the FactorVAE as it uses a discriminator...' This is contradictory: FactorVAE is first listed as a method for comparison and then explicitly not considered. The reader cannot determine which methods were actually evaluated. Please clarify the list of benchmark methods and remove the contradiction.","section":"Section III-A"}],"minor_comments":[{"comment":"The phrase 'variatonal autoencoder' in the abstract text contains a typo; it should read 'variational autoencoder'.","section":"Abstract"},{"comment":"The sentence 'DLVMs learn a joint distribution distribution, pθ(x, z), ...' contains a duplicated word 'distribution'.","section":"Section I"},{"comment":"The color coding for improvements and drops (blue and red) may not be visible in grayscale print; consider adding symbols such as arrows or plus/minus signs.","section":"Table I"},{"comment":"The caption describes 'pairwise angle' but does not specify whether the angles are in degrees or radians, nor how the angles are computed from the estimated directions; please add this information.","section":"Figure 2"},{"comment":"The conclusion states 'we show limitations in the existing metrics that rely on cardinal latent axes', but the paper does not directly compare against other non-axis-aligned disentanglement metrics such as DCI or the directional extension in DCI-ES; a brief discussion of related work would help position the contribution.","section":"Section IV"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and the proposed metric is intuitively appealing, but the current validation is insufficient: the direction-finding procedure is fit to the same labeled data used for evaluation, and the central assumption about minimum-variance eigenvectors is untested. With a proper held-out evaluation, a random-encoder baseline, and a robustness analysis of the eigenvector step, the paper could become a useful contribution. I recommend major revision rather than rejection because these issues are fixable within the scope of the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this paper proposes a simple, sensible extension of FactorVAE and MIG metrics to latent spaces where generative factors are not axis-aligned. Instead of assuming cardinal axes, they run PCA on encodings of data with a given factor held fixed, take the min-variance eigenvector as that factor's direction, aggregate over repeats, and then compute standard metrics on the projected representations. That is a legitimate idea, and their results show large score increases for aggregate-posterior methods like AVAE and WAE, with qualitative latent traversal figures that partially corroborate.\n\nWhat's genuinely new: no prior work in their citations uses PCA-derived directions for disentanglement evaluation. The generalization is modest—a fairly direct adaptation of standard PCA—but it addresses a real gap, since AAE/WAE/AVAE-like models don't align factors with the cardinal axes.\n\nWhere it gets soft. The main assumption in Algorithm 1 is that when factor Fi is fixed, the min-variance PCA direction corresponds to Fi's encoding. That is not secure under posterior collapse or dead units: an unused latent dimension has near-zero variance in every subset, so the min-variance eigenvector can be a dead dimension, and the mean-outer-product aggregation can lock onto that arbitrary direction. The paper does not test this with a random/untrained encoder baseline or with injected dead dimensions, and it does not compare estimated directions to known ground-truth axes. Without those controls, the big PCA-MIG gains in Table I (AVAE 0.67 vs 0.08 on 3D Shapes) may partly reflect the metric fitting arbitrary low-variance directions to labeled data rather than true disentanglement. Also, the direction-finding step uses the same labeled evaluation data, so the scores are not fully independent measurements. \"New SOTA\" is overreach given the scope.\n\nMinor: the text says they consider FactorVAE as a baseline, then says they skip it because it is hard to train. That needs cleanup.\n\nNet: the core idea is worth taking seriously, but the empirical evidence is incomplete. A serious referee should see it, with requests for code, a random-encoder control, actual direction-accuracy checks, and softer claims. I would send it to review rather than desk reject.","headline":"A useful PCA-based adaptation of disentanglement metrics for non-axis-aligned DLVMs, but the validation leaves the key min-variance assumption unexamined.","tokens_in":8763,"tokens_out":2591,"would_cite":false,"duration_ms":24070,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that standard disentanglement metrics misjudge deep latent variable models whose generative factors are not aligned with the latent axes, and that the lost directions can be recovered with a PCA-based search.","keywords":["disentanglement","deep latent variable models","aggregate posterior matching","variational autoencoder","mutual information gap","PCA MIG","FactorVAE metric","latent directions"],"falsifier":"Run Algorithm 1 and the PCA MIG evaluation on a randomly initialized, untrained encoder over DSprites or 3D Shapes; if the estimated directions are stable or the reported PCA MIG remains high, the min-variance heuristic is picking up dataset structure rather than learned disentanglement, and the metric's claim to measure model representations fails. A complementary check is to replace the estimated directions with random unit vectors and see whether PCA MIG changes materially.","tokens_in":7787,"feed_emoji":"🧩","tokens_out":5812,"duration_ms":47111,"temperature":0.7,"pith_summary":"Deep latent variable models that match the distribution of encoded data (the aggregate posterior) to the prior, such as the AAE, WAE, and AVAE, can encode each generative factor along a direction that is not one of the coordinate axes. The paper argues that standard disentanglement metrics such as FactorVAE and mutual information gap (MIG) silently assume axis alignment, and therefore under-report how disentangled these models really are. It proposes a PCA-based procedure that, for each known generative factor, inspects latent encodings of data in which that factor is held fixed and the others vary, and extracts the direction of minimal variance as the factor's latent direction. Re-scoring the models along these recovered directions improves disentanglement scores across all studied methods, with the largest gains for the AVAE. The conclusion is that axis-aligned evaluation is a major source of apparent poor disentanglement in aggregate-posterior matching models.","feed_headline":"PCA directions reveal the disentanglement axis-aligned metrics miss","feed_subtitle":"For aggregate-posterior models like AVAE, re-scoring along learned directions lifts MIG from 0.08 to 0.67 on 3D Shapes.","key_machinery":"The load-bearing mechanism is Algorithm 1's min-variance eigenvector heuristic: with the i-th generative factor pinned to a fixed value and all other factors sampled freely, the latent encodings should spread in every direction except the one associated with Fi, so the smallest-variance principal component of those encodings is taken as Fi's latent direction. Repeating the procedure for N random fixed values and taking the leading eigenvector of the mean outer-product matrix stabilizes the estimate. These recovered directions then replace the coordinate axes in otherwise standard FactorVAE and MIG computations, which is what allows a model like the AVAE to score highly.","core_discovery":"On the paper's own terms, the central discovery is that any trained DLVM's generative factors can be associated with unit vectors in latent space, and these vectors need not coincide with cardinal axes. Algorithm 1 estimates, for each ground-truth factor Fi, the eigenvector with minimum variance from PCA on a batch of encodings in which Fi is fixed and other factors vary; repeating over multiple fixed values and averaging the outer products yields a robust direction. Replacing the coordinate axes with these directions in the FactorVAE and MIG evaluation procedures gives the PCA FactorVAE and PCA MIG metrics. On DSprites and 3D Shapes, the re-scoring lifts scores for nearly every model, with the AVAE improving from 0.08 to 0.67 in MIG on 3D Shapes; the paper reads this as evidence that axis-aligned metrics misjudge aggregate-posterior matching DLVMs.","pith_inferences":["An implication the paper leaves implicit is that the same min-variance-direction search could serve as a diagnostic on real-world datasets where labels are unavailable, by comparing directions found from different attribute groupings for stability.","The procedure assumes the encoder maps each factor to a linear subspace; if true latent structure is curved or multi-modal, a single PCA direction per factor may be insufficient, and a piecewise or manifold-aware variant would be a natural test.","A direct stress test is to run Algorithm 1 on an untrained encoder: if it still yields high PCA MIG scores or stable directions, the metric is measuring dataset geometry rather than the model's learned representation."],"forward_implications":["Standard FactorVAE and MIG scores systematically underestimate disentanglement for DLVMs that match the aggregate posterior to the prior; the PCA re-scoring gives consistent improvements across eight model variants on two datasets.","The AVAE, which closely matches the aggregate posterior, is the clearest beneficiary: its PCA MIG on 3D Shapes rises from 0.08 to 0.67, and its estimated latent directions are nearly mutually orthogonal.","Estimated latent directions that are orthogonal indicate well-separated generative factors; near-collinear directions indicate entanglement and low scores, as seen for the WAE.","Because the method only requires encodings and known factor labels, it applies to any trained DLVM, not just VAEs, without retraining or modifying the model."],"supporting_citations":[{"why":"Supplies the FactorVAE metric and the data-generation strategy of fixing one factor while varying the others, which the proposed method generalizes.","marker":"[17]"},{"why":"Supplies the MIG metric and the axis-aligned assumption that PCA MIG modifies by projecting onto learned directions.","marker":"[18]"},{"why":"Introduces the AVAE, the aggregate-posterior matching model whose reported scores improve most dramatically under the proposed metrics.","marker":"[5]"},{"why":"Introduces the WAE with IMQ kernel, one of the aggregate-posterior matching models evaluated in the experiments.","marker":"[27]"},{"why":"Explains why VAEs align latent factors with coordinate axes and why rotation makes representations non-interpretable, motivating the search for non-axis directions.","marker":"[3]"},{"why":"Provides the benchmark architecture, hyperparameter settings, and evaluation protocol used for fair comparison across methods.","marker":"[15]"},{"why":"Supplies the 3D Shapes dataset with known generative factors used for computing disentanglement scores.","marker":"[29]"},{"why":"Supplies the DSprites dataset with known generative factors used for computing disentanglement scores.","marker":"[31]"}],"fun_headline_variants":["PCA directions reveal the disentanglement axis-aligned metrics miss","New metric scores disentanglement along learned latent directions","Re-scoring with PCA lifts MIG from 0.08 to 0.67 on 3D Shapes","Axis-aligned disentanglement metrics underrate AAE and WAE models","For any deep latent model, evaluate disentanglement via PCA directions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Algorithm 1 assumes that when a generative factor is held fixed while the others vary, the smallest-variance PCA direction of the encoded samples points along that factor's latent direction; if the encoder is entangled or an irrelevant latent dimension has near-zero variance, that direction can be arbitrary and the resulting scores would not reflect true factor encoding.","fun_headline_variants_meta":{"raw":{"variants":["PCA directions reveal the disentanglement axis-aligned metrics miss","New metric scores disentanglement along learned latent directions","Re-scoring with PCA lifts MIG from 0.08 to 0.67 on 3D Shapes","Axis-aligned disentanglement metrics underrate AAE and WAE models","For any deep latent model, evaluate disentanglement via PCA directions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1566,"prompt_tokens":926,"completion_tokens":640,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":545}},"tokens_in":542,"tokens_out":640,"duration_ms":6048,"temperature":1.0,"reasoning_tokens":545,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:02:03.875062+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Algorithm 1 and the PCA MIG evaluation on a randomly initialized, untrained encoder over DSprites or 3D Shapes; if the estimated directions are stable or the reported PCA MIG remains high, the min-variance heuristic is picking up dataset structure rather than learned disentanglement, and the metric's claim to measure model representations fails. A complementary check is to replace the estimated directions with random unit vectors and see whether PCA MIG changes materially.","supporting_citations":[{"cited_title":"Disentangling by factorising,","cited_arxiv_id":null,"evidence_quote":"Supplies the FactorVAE metric and the data-generation strategy of fixing one factor while varying the others, which the proposed method generalizes."},{"cited_title":"Isolating sources of disentanglement in vaes,","cited_arxiv_id":null,"evidence_quote":"Supplies the MIG metric and the axis-aligned assumption that PCA MIG modifies by projecting onto learned directions."},{"cited_title":"Wasserstein auto-encoders,","cited_arxiv_id":null,"evidence_quote":"Introduces the WAE with IMQ kernel, one of the aggregate-posterior matching models evaluated in the experiments."},{"cited_title":"Variational autoencoders pursue pca directions (by accident),","cited_arxiv_id":null,"evidence_quote":"Explains why VAEs align latent factors with coordinate axes and why rotation makes representations non-interpretable, motivating the search for non-axis directions."},{"cited_title":"Challenging common assumptions in the unsupervised learning of disentangled representations,","cited_arxiv_id":null,"evidence_quote":"Provides the benchmark architecture, hyperparameter settings, and evaluation protocol used for fair comparison across methods."},{"cited_title":"3d shapes dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the 3D Shapes dataset with known generative factors used for computing disentanglement scores."},{"cited_title":"dsprites: Disentanglement testing sprites dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the DSprites dataset with known generative factors used for computing disentanglement scores."}],"review_version":1}