Pith. sign in

REVIEW 4 major objections 6 minor 33 references

Splitting a VAE's latent into per-factor subspaces, each trained with an InfoNCE contrastive loss, yields explicit and controllable disentanglement of selected factors without adversarial training or classifiers.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:52 UTC pith:B2KNF2KK

load-bearing objection A genuinely new architecture for weakly-supervised disentanglement, but the missing I(T;S) term in the implemented loss and evaluation metrics that track the training objective make the SOTA claim unsupported as written. the 4 major comments →

arxiv 2601.21688 v2 pith:B2KNF2KK submitted 2026-01-29 cs.LG cs.AI

XFACTORS: Disentangled Information Bottleneck via Contrastive Supervision

Alexandre Myara , Nicolas Bourriez , Thomas Boyer , Thomas Lemercier , Ihab Bendidi , Auguste Genovesio This is my paper
classification cs.LG cs.AI
keywords disentangled representation learningweakly-supervised VAEInfoNCE contrastive lossinformation bottleneckfactor subspace decompositionattribute swappingcounterfactual generationCelebA
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

XFACTORS proposes a weakly-supervised VAE that disentangles a user-selected set of factors of variation by assigning each factor its own low-dimensional latent subspace T_i and a residual subspace S for everything else. The key claim is that an InfoNCE contrastive loss applied per subspace, together with KL regularization on the marginals, is enough to make each T_i carry its own factor while S carries the remaining information — no adversarial objectives or auxiliary classifiers are needed. If this holds, practitioners can choose which factors to disentangle, keep the rest implicit, and edit images by swapping the corresponding latent code. The paper reports state-of-the-art disentanglement scores across four synthetic datasets and one real dataset (CelebA) with a fixed hyperparameter setting, and shows attribute-swapping generations.

Core claim

The central claim is that explicit factor-wise disentanglement can be achieved architecturally rather than by adversarial games: the latent space is decomposed as a direct sum Z = S ⊕ T_1 ⊕ ... ⊕ T_K, and each T_i is trained with an InfoNCE loss that pulls together latent codes sharing the same value of the i-th labeled factor and pushes apart mismatched pairs. This acts as a tractable lower bound on the mutual information I(T_i; y_fi), directly instantiating a Disentangled Information Bottleneck style Lagrangian without the unstable min-max terms of prior work. Gaussian KL regularization on S and on the aggregate T organizes the geometry of both subspaces, and a reconstruction loss preserve

What carries the argument

The direct-sum latent decomposition Z = S ⊕ (⊕_i T_i) with a per-factor InfoNCE loss on each T_i. Each T_i is a low-dimensional subspace (dim 2 in the main experiments) whose InfoNCE objective, L_NCE(i), encourages clustering by the value of its assigned labeled factor y_fi, serving as a lower bound on I(T_i; y_fi). KL terms on S and on the aggregated T impose Gaussian structure, and reconstruction from [z_s ∥ z_t] forces the joint code to retain information about x. The load-bearing effect is that the explicit subspace assignment plus contrastive geometry pushes each factor into its own T_i, leaving S free to carry residual variation; no explicit term enforces I(T;S)=0, which is the paper's

Load-bearing premise

The method assumes that encoding z_s and z_t with two separate encoders and reconstructing from their concatenation makes z_s and z_t independent, even though no term in the implemented loss (Equation 10) enforces I(T;S)=0; the KL constraints act on marginals only.

What would settle it

Train XFACTORS on a dataset like 3DShapes, then train a linear probe or small classifier to predict each labeled factor from the residual code z_s alone. If any factor can be decoded from z_s with accuracy significantly above chance, the claimed decomposition into a factor-free residual and factor-specific subspaces is violated, and attribute swapping could be carrying leaked factor information through S. Alternatively, measure I(z_s; z_t) with a neural MI estimator on held-out data.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Disentanglement becomes a selectable feature: users can choose exactly which labeled factors get their own subspace and leave all others in the residual space, making the approach feasible on real-world data like CelebA.
  • Controlled counterfactual editing follows directly: replacing a single T_i code from a target image while keeping everything else fixed yields an edited image that changes only that factor, as shown for attributes like eyeglasses, smiling, and lipstick.
  • The method avoids adversarial training and auxiliary classifiers, which are often unstable or susceptible to adversarial perturbations, replacing them with a single contrastive loss that is stable and scales to many factors.
  • Constant hyperparameters across datasets suggest the method is not overfit to particular benchmark statistics, a practical advantage for deployment on new datasets.
  • Scaling the residual subspace S up to 126 dimensions does not degrade disentanglement as long as the target subspace regularization is strong, indicating the architecture does not leak targeted factors into the residual.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct test of the paper's central assumption would estimate the mutual information between z_s and z_t (or train a classifier to predict each factor from z_s). If a factor's value can be decoded from z_s above chance, the reported disentanglement may reflect clustering in T_i while S still carries a leaked copy of the same factor, undermining the justification for factor swapping.
  • The success of the approach suggests a general recipe for other information-bottleneck-style disentanglement problems: replace adversarial min-max terms with explicit subspace assignments plus a contrastive lower bound, which could apply to scientific settings with partially labeled cofactors such as batch-effect removal in high-throughput screens.
  • Because each T_i is only 2-dimensional in the main experiments, the representation is an extreme bottleneck; one might expect higher-dimensional T_i to preserve finer-grained within-factor variation, and the paper's own dim=3 variation suggests stability, but a systematic capacity-per-factor study would clarify the trade-off.
  • The reconstruction quality is acknowledged as a clear weakness (visible on CelebA); if combined with a stronger generative head, the same subspace architecture could potentially be used for higher-fidelity counterfactual generation, a direction the paper flags as future work.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces XFACTORS, a weakly-supervised VAE that decomposes the latent representation into a residual subspace S and factor-specific subspaces T_1,...,T_K. Each target factor is encouraged to be encoded in its assigned T_i through an InfoNCE loss, while KL regularization imposes Gaussian structure. The authors claim state-of-the-art disentanglement scores across several datasets with fixed hyperparameters, explicit control over which factors are disentangled, and the ability to perform factor swapping via latent replacement. The theoretical objective (Eq. 9) is presented as an extension of the Disentangled Information Bottleneck, but the implemented loss (Eq. 10) omits the I(T;S) penalty, relying instead on architectural separation and marginal KL regularizers. Experiments include latent visualizations, FactorVAE/DCI metrics, factor swapping generations, scaling studies, and ablations.

Significance. If the claims hold, XFACTORS would be a practically useful contribution: it avoids adversarial training, works with a constant hyperparameter set, and offers targeted disentanglement for a subset of factors on real-world data. The paper provides code, performs ablations, and reports extensive comparisons. However, the central theoretical-to-implementation gap—the missing I(T;S) term—undermines the claim that S and T are independent, which is load-bearing for explicit control and factor swapping. The reported FactorVAE scores are also partially circular with the InfoNCE objective, since FactorVAE selects dimensions with low variance given a factor value, exactly what InfoNCE promotes. The scaling experiment in §4.4 cannot fail by construction because the factor-specific 2D subspaces are selected regardless of dimS. These issues do not necessarily falsify the empirical method, but they require additional evidence to support the paper's stronger claims.

major comments (4)
  1. [§3.2, Eqs. (9)-(10)] The stated X-DisenIB objective (Eq. 9) explicitly includes an I(T;S) penalty to encourage independence between the residual subspace and the factor subspaces. The implemented loss (Eq. 10) contains no term corresponding to this penalty: L_KL acts on the marginals of S and T separately, and each InfoNCE term acts only on T_i with its own label y_fi. Nothing in the objective prevents z_s from encoding y_fi, nor prevents T_i from encoding y_fj for j≠i. This gap is load-bearing for Algorithm 2 (factor swapping) and §4.3: if S encodes the target factor, replacing T_i from the target will not remove the source factor's value from S; if T_j encodes a different target factor, swapping T_i will not cleanly transfer the intended factor. The paper provides no direct evidence that I(S;T)≈0 or that the T_i are mutually exclusive. The authors should either add a term that explicitly penalizes these mu
  2. [§4.2, Tables 1-2] The FactorVAE metric (Kim & Mnih, 2019) identifies the latent dimension with the lowest variance conditioned on a fixed factor value. Since the InfoNCE loss in Eq. (6) directly pulls together T_i codes that share the same factor value and pushes apart those with different values, one expects that T_i will have low within-factor variance for factor i. Thus, high FactorVAE scores may be largely a reflection of the InfoNCE objective being optimized, rather than an independent confirmation of disentanglement. This is especially true because XFACTORS evaluates FactorVAE over all latent dimensions, but the T_i subspaces are 2D and the metric will naturally select them for the corresponding factor. To strengthen the claim, the authors should report disentanglement metrics on held-out combinations (e.g., correlated factors) or evaluate whether the residual S remains free of target-factor informa
  3. [§4.4, Figure 6] The scaling experiment claims that XFACTORS is robust to increasing dimS. However, the FactorVAE metric is computed by selecting the dimension with the lowest variance conditioned on a factor value. For each factor, that dimension will always be in the corresponding 2D subspace T_i (since dimT_i=2 is fixed). Therefore, increasing dimS from 0 to 126 does not affect the FactorVAE selection process, and the experiment cannot fail by construction. This does not test whether S absorbs or leaks factor information when its capacity grows. To make the scaling claim meaningful, the authors should report reconstruction quality, downstream task performance, or measure whether S starts to encode target factors as dimS increases. The current plot only shows that FactorVAE is insensitive to dimS, which is not evidence of robust scaling.
  4. [§6.3, CelebA details] For CelebA, the authors aggregate all 35 non-target attributes into a single factor 's' and then compute FactorVAE/DCI with that aggregated factor. This conflates all residual attributes and masks any entanglement among them. The paper does not claim that S is internally disentangled, but treating the 35 attributes as a single block means that a high score for the 5 target factors can be achieved even if S entangles all residual attributes in an arbitrary way. Since the paper emphasizes robustness to 'other / not of interest' variations, it should at least report whether the residual attributes remain separable in S or provide a more granular evaluation (e.g., compute DCI on S alone with the 35 attributes). The current treatment overstates the completeness of the residual representation.
minor comments (6)
  1. [Abstract] The abstract contains a duplicated phrase: 'a residual subspace S and factor-specific subspaces T1,...,T_K and a residual subspace S.' Please correct.
  2. [§3.1, Eq. (7)] In Eq. (7), the notation N=|T_i| is confusing; N should be defined as the number of negative samples (or batch size) used in InfoNCE, not the dimensionality of T_i. Clarify.
  3. [§4.3, Algorithm 2] Algorithm 2 uses the means (µ_src, µ_tgt) without reparameterization sampling. Since the model is trained with stochastic sampling, the inference procedure should be explicitly justified or matched to training (e.g., using means as a point estimate).
  4. [§6.2] Inconsistent capitalization: 'XFactors' appears in 'We train XFactors' while the method is named 'XFACTORS' throughout. Minor style issue.
  5. [§4.1] Figure 3: the claim that 'the free factor is well represented in the S latent space' is based on a 2D PCA with 67% explained variance. The PCA projection may hide structure; consider showing all S dimensions or using a different visualization.
  6. [§5, Discussion] The discussion mentions 'weakly combinatorially full' for dSprites, but this is not reflected in the experimental design or metric computation. Consider adding a footnote or experiment that addresses this dataset limitation.

Circularity Check

3 steps flagged

Eq. 9's I(T;S) penalty vanishes from Eq. 10; FactorVAE and scaling results are wired to the InfoNCE objective, so headline metrics largely restate the training loss.

specific steps
  1. other [Section 3.2, Equations (9)-(10)]
    "LX-IB =− K∑ i=1 I(Ti;yfi )−I(X,(S,Y)) +I(T;S)(9) ... In order to optimize Equation (9) we propose the final objective L: L=L reco +βs·LS KL +βt·LT KL + K∑ i=1 λi·L (i) InfoNCE (10)"

    Equation (9) is the stated X-DisenIB objective and contains the explicit term +I(T;S), which is what would keep residual S free of the target factors and keep the T_i subspaces mutually exclusive. Equation (10), called its optimization, has no term corresponding to I(T;S); the KL terms act on marginals and the InfoNCE terms act only on each T_i relative to its own label. The independence that the central claims require is therefore assumed, not derived. Algorithm 2 then swaps a single T_i while keeping S, and the paper reports clean factor transfer; but if y_fi leaks into S or into T_j for j≠i, the implemented loss does nothing to prevent it. The 'explicit control' result is thus attributed to an objective that is not the one being optimized.

  2. fitted input called prediction [Section 4.2 and Supplementary 6.3, Algorithm 3]
    "We report scores for two widely reported disentanglement metrics ... We use the FactorV AE (Kim & Mnih, 2019) and DCI (Eastwood & Williams, 2018) scores ... Compute the per-dimension variances σ2 d = Var(zi,d) for alld; Identify d∗ = arg mind σ2 d"

    The InfoNCE term in Eq. (10) is trained to make each T_i tightly clustered according to y_fi, i.e., to minimize within-factor-value variance in each factor subspace. FactorVAE's score selects the single dimension with minimum variance and asks whether it predicts the factor. So a high FactorVAE score is essentially a readout of whether the InfoNCE objective was optimized; it is not an independent confirmation that S excludes target factors or that factor-specific subspaces exclude each other's factors. Reporting near-1.0 FactorVAE as 'state-of-the-art disentanglement' therefore renames successful InfoNCE optimization as external validation rather than testing the disentanglement claim.

  3. fitted input called prediction [Section 4.4, Figure 6 and surrounding text]
    "Contrary to this trend, our results demonstrate that XFACTORS scales robustly. As shown in Figure 6, increasing dimS (up to 126) does not reduce the FactorV AE score ... This indicates that our architecture effectively isolates the factors of interest in T, regardless of the capacity allocated to the residual information in S."

    The scaling experiment cannot fail for the chosen metric. FactorVAE always selects d* = argmin_d Var(z_d) for each factor value. Because InfoNCE makes the factor-specific T_i dimensions have near-zero within-factor-value variance, d* is guaranteed to be in T_i; dimensions added to S never enter the argmin. Hence FactorVAE is invariant to dimS by construction, and the conclusion that XFACTORS 'effectively isolates' factors from residual capacity is not tested by this experiment.

full rationale

XFACTORS is not a self-citation-circular paper: the core architecture, Eq. (10), and the qualitative/ablative experiments are presented openly and compare against external baselines. The main circularity concern is that the paper's headline quantitative claims are, to a significant degree, restatements of the training signal. The theoretical objective Eq. (9) contains +I(T;S), but the implemented loss Eq. (10) drops it; the paper nevertheless uses language from Eq. (9) ('minimize MI between S and T', 'explicit control', clean factor swapping) when interpreting results. Additionally, FactorVAE's minimum-variance-dimension rule is aligned with InfoNCE's clustering of each T_i by its factor value, so high FactorVAE/DCI scores mostly verify that InfoNCE was optimized, not that the subspaces are free of cross-information. The scaling claim 'increasing dimS does not reduce FactorVAE' is forced by the metric's argmin construction. The CelebA aggregation of 35 residual attributes into a single factor 's' further prevents detection of entanglement inside S. There is independent content—latent visualizations, swapping on synthetic data, ablations, and a clear architectural contribution—so a score of 6 (partial circularity) is appropriate rather than 8 or 10.

Axiom & Free-Parameter Ledger

7 free parameters · 5 axioms · 0 invented entities

The method's central design—per-factor InfoNCE subspaces plus a residual—introduces hyperparameters whose values are fixed but with no selection protocol, and the independence between S and T is assumed rather than derived.

free parameters (7)
  • λNCE = 0.5
    Weight of the InfoNCE loss in Eq (10); chosen once and kept constant but no selection protocol is reported.
  • βs = 100
    KL weight for residual subspace S.
  • βt = 100
    KL weight for aggregated factor subspaces; scaling experiment shows βt=100 is needed to avoid degradation at large dimS.
  • dimS = 126
    Residual subspace dimension; chosen as a hyperparameter.
  • dimTi = 2 per factor
    Set 'for ease of visualization' (Sec 4.1); directly affects FactorVAE because the factor-specific 2D subspace is easy to identify.
  • InfoNCE temperature τ = not reported
    Critical hyperparameter in Eq (6); not specified, making the method underdetermined.
  • selection of which factor is left in S = one factor per synthetic dataset; 35 attributes on CelebA
    They left one factor out per dataset to be encoded in S; choice is a modeling decision.
axioms (5)
  • standard math VAE reparameterization and Gaussian prior p(z)=N(0,I) are used to regularize latents
    Standard VAE assumption.
  • standard math InfoNCE provides a lower bound on mutual information I(T_i;y_fi)
    From van den Oord et al. (2019); used to justify contrastive losses. This is a bound, not an equality, and the tightness depends on batch size.
  • domain assumption The two-encoder architectural split (ψs and ψt) yields factorized posterior q(zs)q(zt)
    The paper assumes separate encoders produce independent subspaces; no explicit independence penalty is used (Eq 10 omits I(T;S)).
  • ad hoc to paper KL on aggregated factor space plus per-subspace InfoNCE organizes the whole latent
    The KL terms constrain marginals only, so they do not directly enforce factorization; the paper presents this as an organizing mechanism (Section 3.2).
  • domain assumption The reconstruction loss L2 is a sufficient proxy for I(X;Z)
    Standard in VAEs, but the method favors disentanglement over reconstruction, as the authors admit (Section 4.3 footnote).

pith-pipeline@v1.3.0-alltime-deepseek · 3972 in / 4016 out tokens · 150716 ms · 2026-08-03T06:52:18.192578+00:00 · methodology

0 comments
read the original abstract

Disentangled representation learning aims to map independent factors of variation to independent representation components. On one hand, purely unsupervised approaches have proven successful on fully disentangled synthetic data, but fail to recover semantic factors from real data without strong inductive biases. On the other hand, supervised approaches are unstable and hard to scale to large attribute sets because they rely on adversarial objectives or auxiliary classifiers. We introduce \textsc{XFactors}, a weakly-supervised VAE framework that disentangles and provides explicit control over a chosen set of factors. Building on the Disentangled Information Bottleneck perspective, we decompose the representation into a residual subspace $\mathcal{S}$ and factor-specific subspaces $\mathcal{T}_1,\ldots,\mathcal{T}_K$ and a residual subspace $\mathcal{S}$. Each target factor is encoded in its assigned $\mathcal{T}_i$ through contrastive supervision: an InfoNCE loss pulls together latents sharing the same factor value and pushes apart mismatched pairs. In parallel, KL regularization imposes a Gaussian structure on both $\mathcal{S}$ and the aggregated factor subspaces, organizing the geometry without additional supervision for non-targeted factors and avoiding adversarial training and classifiers. Across multiple datasets, with constant hyperparameters, \textsc{XFactors} achieves state-of-the-art disentanglement scores and yields consistent qualitative factor alignment in the corresponding subspaces, enabling controlled factor swapping via latent replacement. We further demonstrate that our method scales correctly with increasing latent capacity and evaluate it on the real-world dataset CelebA. Our code is available at \href{https://github.com/ICML26-anon/XFactors}{github.com/ICML26-anon/XFactors}.

Figures

Figures reproduced from arXiv: 2601.21688 by Alexandre Myara, Auguste Genovesio, Ihab Bendidi, Nicolas Bourriez, Thomas Boyer, Thomas Lemercier.

Figure 1
Figure 1. Figure 1: Factor swapping generations on CelebA. Represen￾tation can be edited by replacing the code of a factor Ti from the source image (row 1) with the corresponding code Ti from the target image (row 2). Our decoder can then be used to decode the representation for inspection. Each row 3-8 shows swapping performed on a single factor; all other latent components are kept fixed. Sources and targets displayed here … view at source ↗
Figure 2
Figure 2. Figure 2: Our architecture. XFACTORS processes the input x using two parallel encoders: ψs(·), which captures the residual information in the latent code zs, and ψt(·), which encodes the factors of interest in zt. The factor latent zt is explicitly disentan￾gled by aligning specific subspaces zt,fi with their corresponding ground-truth labels yfi via InfoNCE objectives (LNCE). The latent spaces are regularized using… view at source ↗
Figure 3
Figure 3. Figure 3: Latent space visualizations on MPI3D. Each Ti is colored by the values of the factors that it should encode. For S the first two principal components are shown (67% explained variance). The other Ti subspaces are 2D. sion. This behavior is consistent with the organizing effect of the KL regularization, in agreement with Burgess et al. (2018). This space seems less contrasted since it is a projec￾tion with … view at source ↗
Figure 4
Figure 4. Figure 4: Factor swapping generations on Shapes3D and dSprites For each row (source image), we replace one latent code Ti with the corresponding code from the target image (shown in the header) and decode. Each column corresponds to swapping a single factor; all other latent components are kept fixed. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Factor swapping generations on MPI3D For each row (source image), we replace one latent code Ti with the correspond￾ing code from the target image (shown in the header) and decode. Each column corresponds to swapping a single factor; all other latent components are kept fixed. dimension of S (dimS) and the target KL weight βt. Scalability with dimS. A common failure mode in disen￾tanglement methods is the … view at source ↗
Figure 6
Figure 6. Figure 6: Impact of residual dimension dimS and target reg￾ularization βT . XFACTORS scales robustly: increasing dimS does not degrade FactorVAE scores. However, higher target reg￾ularization (βT = 100, right) proves essential to maintain this performance in large latent spaces, preventing the decay observed with weaker regularization (βT = 1, left). dimensions. Contrary to this trend, our results demonstrate that X… view at source ↗
Figure 7
Figure 7. Figure 7: FactorVAE scores on Shapes3D for different batch sizes and numbers of iterations. DCI computational details Following Eastwood & Williams (2018) we compute DCI metric as a triplet. We choose as a classifier a Random Forest Classifier [PITH_FULL_IMAGE:figures/full_fig_p012_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Learned latents on Shapes3D. Rows: encoding space (top to bottom: (S, T0, T1, T2, T3, T4)). Columns: color being used, corresponding to the different possible values of a single factor (left to right: (floor hue, wall hue, object hue, scale, shape, orientation), only the coloring changes between columns). For the S space the first 2 components of its PCA are shown. 13 [PITH_FULL_IMAGE:figures/full_fig_p01… view at source ↗
Figure 9
Figure 9. Figure 9: Learned latents on Shapes3D when the InfoNCE term of the loss is ablated. Rows: encoding space (top to bottom: (S, T0, T1, T2, T3, T4)). Columns: color being used, corresponding to the different possible values of a single factor (left to right: (floor hue, wall hue, object hue, scale, shape, orientation), only the coloring changes between columns). For the S space the first 2 components of its PCA are sho… view at source ↗
Figure 10
Figure 10. Figure 10: Learned latents of Shapes3D when the S space is ablated. Rows: encoding space (top to bottom: (T0, T1, T2, T3, T4)). Columns: color being used, corresponding to the different possible values of a single factor (left to right: (floor hue, wall hue, object hue, scale, shape, orientation), only the coloring changes between columns). 15 [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Learned latents on dSprites. Rows: encoding space (top to bottom: (S, T0, T1, T2, T3)). Columns: color being used, corresponding to the different possible values of a single factor (left to right: (shape, scale, orientation, x position, y position), only the coloring changes between columns). For the S space the first 2 components of its PCA are shown. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Learned latents on MPI3D. Rows: encoding space (top to bottom: (S, T0, T1, T2, T3, T4, T5)). Columns: color being used, corresponding to the different possible values of a single factor (left to right: (object color, object shape, object size, camera height, background color, horizontal axis, vertical axis), only the coloring changes between columns). For the S space the first 2 components of its PCA are … view at source ↗
Figure 13
Figure 13. Figure 13: Learned latents on cars3D. Rows: encoding space (top to bottom: (S, T0, T1)). Columns: color being used, corresponding to the different possible values of a single factor (left to right: (elevation angle, azimuth angle, identity), only the coloring changes between columns). For the S space the first 2 components of its PCA are shown. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Learned latents on CelebA. Rows: encoding space (top to bottom: (S, T0, T1, T2, T3, T4, T5)). Columns: color being used, corresponding to the different possible values of a single factor (left to right: (eyeglasses, male, pale skin, smiling, wearing hat, wearing lipstick), only the coloring changes between columns). For the S space the first 2 components of its PCA are shown. The colorings for the S facto… view at source ↗
Figure 15
Figure 15. Figure 15: Factor swapping generations on Shapes3D For each row (source image), we replace one latent code Ti with the corresponding code from the target image (shown in the header) and decode. Each column corresponds to swapping a single factor; all other latent components are kept fixed. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Factor swapping generations on CelebA For each row (source image), we replace one latent code Ti with the corresponding code from the target image (shown in the header) and decode. Each column corresponds to swapping a single factor; all other latent components are kept fixed. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Factor swapping generations on CelebA For each row (source image), we replace one latent code Ti with the corresponding code from the target image (shown in the header) and decode. Each column corresponds to swapping a single factor; all other latent components are kept fixed. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Factor swapping generations on CelebA For each row (source image), we replace one latent code Ti with the corresponding code from the target image (shown in the header) and decode. Each column corresponds to swapping a single factor; all other latent components are kept fixed. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Factor swapping generations on MPI3D For each row (source image), we replace one latent code Ti with the corresponding code from the target image (shown in the header) and decode. Each column corresponds to swapping a single factor; all other latent components are kept fixed. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Factor swapping generations on Cars3D For each row (source image), we replace one latent code Ti with the corresponding code from the target image (shown in the header) and decode. Each column corresponds to swapping a single factor; all other latent components are kept fixed. 25 [PITH_FULL_IMAGE:figures/full_fig_p025_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Factor swapping generations on dSprites For each row (source image), we replace one latent code Ti with the corresponding code from the target image (shown in the header) and decode. Each column corresponds to swapping a single factor; all other latent components are kept fixed. 26 [PITH_FULL_IMAGE:figures/full_fig_p026_21.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

33 extracted references · 22 linked inside Pith

  1. [5]

    nips.cc/paper_files/paper/2022/hash/ 0850e04a62e0f3407780852581c5fcf4- Abstract-Conference.html

    URL https://papers. nips.cc/paper_files/paper/2022/hash/ 0850e04a62e0f3407780852581c5fcf4- Abstract-Conference.html. Chen, R. T. Q., Li, X., Grosse, R., and Duvenaud, D. Isolating sources of disentanglement in variational au- toencoders,

    2022

  2. [6]

    Eastwood, C

    URL https://arxiv.org/abs/ 1802.04942. Eastwood, C. and Williams, C. A framework for the quanti- tative evaluation of disentangled representations. InSixth International Conference on Learning Representations (ICLR 2018), May

    Pith/arXiv arXiv 2018

  3. [7]

    6th International Conference on Learning Representations, ICLR 2018 ; Conference date: 30-04-2018 Through 03-05-2018

    URL https://iclr.cc/ Conferences/2018. 6th International Conference on Learning Representations, ICLR 2018 ; Conference date: 30-04-2018 Through 03-05-2018. Funke, C. M., Vicol, P., Wang, K., Kümmerer, M., Zemel, R. S., and Bethge, M. Disentanglement and generaliza- tion under correlation shifts. In Chandar, S., Pascanu, R., and Precup, D. (eds.),Conferen...

    2018

  4. [10]

    URLhttps://arxiv.org/abs/2004.02546. Kim, H. and Mnih, A. Disentangling by factorising,

    Pith/arXiv arXiv 2004

  5. [11]

    Lample, G., Zeghidour, N., Usunier, N., Bordes, A., De- noyer, L., and Ranzato, M

    URLhttps://arxiv.org/abs/1802.05983. Lample, G., Zeghidour, N., Usunier, N., Bordes, A., De- noyer, L., and Ranzato, M. Fader networks: Manipu- lating images by sliding attributes,

    Pith/arXiv arXiv

  6. [12]

    URL https: //arxiv.org/abs/1706.00409. Leek, J. T., Scharpf, R. B., Bravo, H. C., Simcha, D., Lang- mead, B., Johnson, W. E., Geman, D., Baggerly, K., and Irizarry, R. A. Tackling the widespread and critical impact of batch effects in high-throughput data.Nature Reviews Genetics, 11(10):733–739,

    Pith/arXiv arXiv

  7. [14]

    Liu, Z., Luo, P., Wang, X., and Tang, X

    URL https://arxiv.org/abs/1906.06034. Liu, Z., Luo, P., Wang, X., and Tang, X. Deep learning face attributes in the wild. InProceedings of International Conference on Computer Vision (ICCV), December

    Pith/arXiv arXiv 1906

  8. [15]

    Locatello, F., Poole, B., Raetsch, G., Schölkopf, B., Bachem, O., and Tschannen, M

    URL https://arxiv.org/ abs/1811.12359. Locatello, F., Poole, B., Raetsch, G., Schölkopf, B., Bachem, O., and Tschannen, M. Weakly-supervised disentangle- ment without compromises. In III, H. D. and Singh, A. (eds.),Proceedings of the 37th International Confer- ence on Machine Learning, volume 119 ofProceedings of Machine Learning Research, pp. 6348–6359. ...

    Pith/arXiv arXiv

  9. [18]

    Matthey, L., Higgins, I., Hassabis, D., and Lerchner, A

    URL https://arxiv.org/abs/1611.03383. Matthey, L., Higgins, I., Hassabis, D., and Lerchner, A. dsprites: Disentanglement testing sprites dataset. https://github.com/deepmind/dsprites-dataset/,

    Pith/arXiv arXiv

  10. [19]

    org/abs/2012.07372

    URL https://arxiv. org/abs/2012.07372. Poole, B., Ozair, S., Van Den Oord, A., Alemi, A., and Tucker, G. On variational bounds of mutual information. In Chaudhuri, K. and Salakhutdinov, R. (eds.),Proceed- ings of the 36th International Conference on Machine Learning, volume 97 ofProceedings of Machine Learn- ing Research, pp. 5171–5180. PMLR, 09–15 Jun

    Pith/arXiv arXiv 2012

  11. [22]

    Steenbrugge, X., Leroux, S., Verbelen, T., and Dhoedt, B

    URL https://arxiv.org/abs/ 2101.06046. Steenbrugge, X., Leroux, S., Verbelen, T., and Dhoedt, B. Improving generalization for abstract reasoning tasks using disentangled feature representations,

    Pith/arXiv arXiv

  12. [23]

    Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R

    URL https://arxiv.org/abs/1811.04784. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., and Fergus, R. Intriguing properties of neural networks,

    Pith/arXiv arXiv

  13. [24]

    Tishby, N., Pereira, F

    URL https://arxiv.org/ abs/1312.6199. Tishby, N., Pereira, F. C., and Bialek, W. The informa- tion bottleneck method,

    Pith/arXiv arXiv

  14. [26]

    van den Oord, A., Li, Y ., and Vinyals, O

    URLhttps://arxiv.org/abs/2006.07886. van den Oord, A., Li, Y ., and Vinyals, O. Representation learning with contrastive predictive coding,

    Pith/arXiv arXiv 2006

  15. [27]

    V oynov, A

    URL https://arxiv.org/abs/1807.03748. V oynov, A. and Babenko, A. Unsupervised discovery of interpretable directions in the gan latent space,

    Pith/arXiv arXiv

  16. [28]

    Yang, T., Wang, Y ., Lv, Y ., and Zheng, N

    URL https://arxiv.org/abs/2002.03754. Yang, T., Wang, Y ., Lv, Y ., and Zheng, N. Disdiff: Unsu- pervised disentanglement of diffusion probabilistic mod- els,

    Pith/arXiv arXiv 2002

  17. [31]

    We report standard deviations w.r.t

    disentangles only 2 attributes. We report standard deviations w.r.t. the computation of metrics for XFACTORS(we run 5 evaluations). Computational details for FactorV AE-scoreFollowing Kim & Mnih (2019), we compute FactorV AE-score with the following algorithm: Algorithm 3FactorV AE Computation 1: Inputs:Factors (f1,...,f K), standardized latent codes {zi}...

    2019

  18. [32]

    Other implementation for FactorV AE-scoreEncDiff (Yang et al., 2024), Disco (Ren et al.,

    2024

  19. [33]

    We choose to stay close to the Kim & Mnih (2019) algorithm

    and DisDiff (Yang et al., 2023), choose to fit a PCA at the first order on their subspaces before computing the FactorV AE-score. We choose to stay close to the Kim & Mnih (2019) algorithm. 11 XFACTORS: Disentangled Information Bottleneck via Contrastive Supervision Figure 7.FactorV AE scores on Shapes3D for different batch sizes and numbers of iterations...

    2023

  20. [100]

    We train XFactors on a NVIDIA A100 in 15 hours for 3DShapes. 6.3. Disentanglement metrics⇈ The methods reported in Table 1 are: FactorV AE (Kim & Mnih, 2019),β-TCV AE (Chen et al., 2018), InfoGAN- CR (Lin et al., 2020), GAN-LD (V oynov & Babenko, 2020), GANSpace (Härkönen et al., 2020), DisCo (Ren et al., 2021), DisDiff (Yang et al., 2023), EncDiff (Yang ...

    2019

  21. [2000]

    org/abs/physics/0004057

    URL https://arxiv. org/abs/physics/0004057. Träuble, F., Creager, E., Kilbertus, N., Locatello, F., Dittadi, A., Goyal, A., Schölkopf, B., and Bauer, S. On disentan- gled representations learned from correlated data,

    Pith/arXiv arXiv

  22. [2010]

    URLhttps://doi.org/10.1038/nrg2825

    doi: 10.1038/nrg2825. URLhttps://doi.org/10.1038/nrg2825. Lin, Z., Thekumparampil, K. K., Fanti, G., and Oh, S. Infogan-cr and modelcentrality: Self-supervised model 9 XFACTORS: Disentangled Information Bottleneck via Contrastive Supervision training and selection for disentangling gans,

    doi:10.1038/nrg2825

  23. [2014]

    Burgess, C

    URL https://arxiv.org/abs/1206.5538. Burgess, C. and Kim, H. 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/,

    Pith/arXiv arXiv

  24. [2015]

    cc/paper_files/paper/2015/file/ e07413354875be01a996dc560274708e- Paper.pdf

    URL https://proceedings.neurips. cc/paper_files/paper/2015/file/ e07413354875be01a996dc560274708e- Paper.pdf. Ren, X., Yang, T., Wang, Y ., and Zeng, W. Learning dis- entangled representation by exploiting pretrained gener- ative models: A contrastive learning view,

    2015

  25. [2016]

    Augustin, M., Boreiko, V ., Croce, F., and Hein, M

    URL https://arxiv.org/abs/1612.00410. Augustin, M., Boreiko, V ., Croce, F., and Hein, M. Diffusion visual counterfactual explanations,

    Pith/arXiv arXiv

  26. [2018]

    Cao, J., Nai, R., Yang, Q., Huang, J., and Gao, Y

    URL https://arxiv.org/ abs/1804.03599. Cao, J., Nai, R., Yang, Q., Huang, J., and Gao, Y . An empirical study on disentanglement of negative-free contrastive learning. In Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., and Oh, A. (eds.),Advances in Neural Information Process- ing Systems, volume 35, pp. 1210–1222. Curran Associates, Inc.,

    Pith/arXiv arXiv

  27. [2019]

    neurips.cc/paper/2019/file/ d97d404b6119214e4a7018391195240a- Paper.pdf

    URL https://proceedings. neurips.cc/paper/2019/file/ d97d404b6119214e4a7018391195240a- Paper.pdf. Higgins, I., Matthey, L., Pal, A., Burgess, C. P., Glo- rot, X., Botvinick, M. M., Mohamed, S., and Lerch- ner, A. beta-vae: Learning basic visual concepts with a constrained variational framework. In5th Interna- tional Conference on Learning Representations,...

    2019

  28. [2020]

    doi: 10.1038/s42256-020-00257-z

    ISSN 2522-5839. doi: 10.1038/s42256-020-00257-z. URL http://dx. doi.org/10.1038/s42256-020-00257-z. Gondal, M. W., Wuthrich, M., Miladinovic, D., Locatello, F., Breidt, M., V olchkov, V ., Akpo, J., Bachem, O., Schölkopf, B., and Bauer, S. On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. In Wallach, H., L...

    doi:10.1038/s42256-020-00257-z

  29. [2021]

    Sauer, A

    URL https://arxiv.org/abs/2102.10543. Sauer, A. and Geiger, A. Counterfactual generative net- works,

    Pith/arXiv arXiv

  30. [2022]

    Belghazi, M

    URL https: //arxiv.org/abs/2210.11841. Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Ben- gio, Y ., Courville, A., and Hjelm, R. D. Mutual in- formation neural estimation. In Dy, J. and Krause, A. (eds.),Proceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Ma- chine Learning Research, pp. 531–540. PMLR,

    Pith/arXiv arXiv

  31. [2023]

    URL https://doi.org/ 10.15252/msb.202211517

    doi: 10.15252/msb.202211517. URL https://doi.org/ 10.15252/msb.202211517. Makino, T., Park, J. W., Tagasovska, N., Kudo, T., Coelho, P., Huetter, J.-C., Yao, H., Hoeckendorf, B., Leote, A. C., Ra, S., Richmond, D., Cho, K., Regev, A., and Lopez, R. Supervised contrastive block disentanglement,

    doi:10.15252/msb.202211517

  32. [2024]

    10 XFACTORS: Disentangled Information Bottleneck via Contrastive Supervision

    URL https://arxiv.org/abs/ 2402.09712. 10 XFACTORS: Disentangled Information Bottleneck via Contrastive Supervision

    Pith/arXiv arXiv

  33. [2025]

    Mathieu, M., Zhao, J., Sprechmann, P., Ramesh, A., and LeCun, Y

    URLhttps://arxiv.org/abs/2502.07281. Mathieu, M., Zhao, J., Sprechmann, P., Ramesh, A., and LeCun, Y . Disentangling factors of variation in deep representations using adversarial training,

    Pith/arXiv arXiv