REVIEW 4 major objections 5 minor 36 references
Disentanglement in Difference: Directly Learning Semantically Disentangled Representations by Maximizing Inter-Factor Differences
T0 review · 4 major / 5 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read A contrastive loss on inter-sample differences, rather than statistical independence, is the direct driver of semantic disentanglement in the proposed DiD method.
desk verdict The novelty is real but the core objective doesn't implement the paper's stated same-factor/different-factor principle; a genuinely interesting idea undermined by a loss that can be maximized without semantic alignment. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Difference Encoder $H$, realized as an MLP, which turns an image pair into one difference vector. The contrastive loss $L_H=-\lVert H(g_1)-H(g_2)\rVert_2$ amplifies the separation between variations caused by different latent axes; the WGAN-GP generator and discriminator provide a high-quality sample distribution over a bounded uniform latent space $c\sim U[-1,1]^n$; and the Sample Encoder $E$ with MSE reconstruction loss $\lVert c-E(G(c))\rVert_2^2$ ties generated samples back to latent codes. The whole system is optimized so that the generator's axes become directions of single-factor variation.
What would settle it
Train DiD on a benchmark where one ground-truth factor is deliberately split across two latent axes or two factors always change together, then measure per-axis factor purity (e.g., SAP per dimension). If the model maintains high MIG while no single axis predicts a single factor better than chance, then maximizing inter-axis difference does not by itself yield semantic disentanglement.
Extended reading notes
Core claim
The central discovery claimed here is that semantic disentanglement can be learned directly from inter-sample differences, without any penalty on total correlation or mutual information. Define three latent codes $c$, $c'$, $c''$ where $c'$ moves a small step from $c$ along axis $a_1$ and $c''$ moves the same step along axis $a_2$; generate image pairs $g_1=[G(c),G(c')]$ and $g_2=[G(c),G(c'')]$; then train a Difference Encoder $H$ to maximize $-\lVert H(g_1)-H(g_2)\rVert_2$ while a Sample Encoder $E$ is trained with $\lVert c-E(G(c))\rVert_2^2$. The paper argues that this contrastive difference objective directly encodes the prior that different semantic factors should be far apart in representation space, and reports that on dSprites and 3DShapes the method consistently achieves significantly higher SAP, MIG, and DCI-D scores compared to all baseline methods.
Load-bearing premise
The method assumes that nudging one raw coordinate of the sampled latent code produces an image pair that differs in exactly one semantic factor; the training loss never checks this, so a latent axis that mixes two factors can still satisfy the contrastive objective without any factor being cleanly separated.
Editorial extensions
If this is right
- If the contrastive difference objective is what produces disentanglement, then disentanglement can be optimized as a first-class objective rather than as a side effect of a KL or TC penalty.
- The method extends to any generative model with a continuous latent space and a differentiable generator, since the difference encoder operates on image pairs rather than on dataset labels.
- DiD also gives an operational definition of semantic difference: two factors are disentangled when their induced difference vectors are far apart, which could be used as a training signal in settings where ground-truth factors are unknown.
- The ablation result implies that without the difference comparison the model reduces to an ordinary WGAN-GP and disentanglement scores drop, so the contrastive term, not the GAN backbone, is the source of the reported gains.
Reading between the lines
- A natural next test is whether a learned or normalized metric on difference vectors behaves better than raw Euclidean distance; when factors have very different variability, raw Euclidean distance can be dominated by the highest-variance axis, a case the paper does not examine.
- The same difference-contrastive prior could be applied to video or multi-view sequences, where temporal or viewpoint changes provide natural same-factor pairs, extending the method beyond synthetic generative benchmarks.
- The paper's TC-versus-MIG scatter suggests a mathematical gap between statistical independence and semantic disentanglement, but it does not formalize it; a formal condition stating when inter-axis difference maximization implies factor-wise alignment would settle the theoretical status of the method.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Disentanglement in Difference (DiD), a GAN-based method for learning semantically disentangled representations. DiD trains a generator with WGAN-GP, a Difference Encoder that maps pairs of generated images to difference vectors, and a Sample Encoder that reconstructs the latent code. The central idea is to contrast sample pairs generated by perturbing different latent axes, using the loss LH = -||v1-v2||2 (Eq. 3) to maximize the distance between difference vectors of different axes. Experiments on dSprites and 3DShapes report higher MIG, DCI-D, and SAP scores than several VAE- and GAN-based baselines. The paper also presents an analysis showing that total correlation and MIG are not monotonically related in baseline models, and an ablation of the difference module.
Significance. If the proposed objective genuinely enforced the stated prior that same-factor variations are compact and different-factor variations are separated, the idea of directly optimizing semantic differences rather than statistical independence would be a meaningful contribution to unsupervised disentanglement. The paper has some strengths: it evaluates on standard benchmarks, includes an ablation of the difference module, and makes the interesting empirical observation that lower TC does not consistently improve MIG. However, the central objective as formulated does not implement its stated prior, and the quantitative claims in Table 1 are overstated, so the significance of the reported results is not established. The paper does not ship code or provide sufficient implementation details to reproduce the experiments, which further limits its impact.
major comments (4)
- [Section 3.4, Eq. (3)] The loss LH = -||v1 - v2||2 is an unbounded negative Euclidean distance with no margin, no normalization, no Lipschitz constraint on H, and no within-factor compactness term. Minimizing this loss (as implied by Eq. 5) can be trivially driven to -infinity by scaling the output of H, independent of the generator G. Moreover, the objective only separates one pair of difference vectors (one from axis a1 and one from axis a2) at a time; it contains no term that compresses H(G(c), G(c+δe_a)) across different base codes c, and no term relating the difference vectors to the actual semantic content of the generated images. Consequently, maximizing Eq. (3) does not implement the paper's stated principle that same-factor variations are small and different-factor variations are large; it only pushes two arbitrary pair encodings apart. The reported disentanglement metrics, even if reproducible, cannot be attributed to the contrastive objective as written.
- [Section 3.2 and Section 3.4] The paper states in Section 3.2 that 'it is assumed that within each sample pair, the two samples differ only in one latent space dimension, while remaining consistent across other factors.' This assumption holds by construction for the input code c, but it is not enforced for the generated images: the loss does not align the generator's axes with the semantic factors of the dataset. If a single input axis mixes multiple semantic factors, the Difference Encoder can maximize inter-axis distances without achieving factor disentanglement. The paper provides no mechanism, such as a reconstruction or classification term on semantic factors, that would force the generator to map each latent axis to a single factor. The central claim therefore relies on an unenforced and untested assumption.
- [Section 4.4, Table 1] The claim that DiD 'consistently achieves significantly higher SAP, MIG, and DCI-D scores compared to all baseline methods' is contradicted by Table 1. On dSprites, DiD's SAP is 0.17 ± 0.02, which is lower than FactorVAE's 0.19 ± 0.03 and only equal to InfoGAN-CR within error; on DCI-D, DiD's 0.58 ± 0.06 overlaps with FactorVAE's 0.55 ± 0.05. No statistical significance tests are reported, so 'significantly higher' is unsupported. The claim should be corrected to reflect the actual numbers, and the discussion should acknowledge where DiD does not dominate.
- [Section 3.5 and Section 4.3] The Sample Encoder E is trained exclusively on generated samples G(c) with the loss Lenc = ||c - E(G(c))||2_2. The paper does not specify how E is applied to real dataset images when computing the disentanglement metrics (MIG, DCI-D, SAP), which are typically computed on the encoder's representations of the training data. If real images are encoded, there is a domain shift because E has only seen generated images; if generated images are used, the metrics may be biased because the codes c are known. The experimental protocol for metric computation is not described, making the quantitative results non-reproducible.
minor comments (5)
- [Throughout] The terminology is inconsistent: the module is called 'Difference Encoder' in Section 3.4 and the abstract, but 'Difference Estimator' in Figure 1 and Section 4.6. Please unify.
- [Section 3.4, Eq. (3)] The sign of LH is confusing: the text says 'Our objective is to maximize the distance, d(v1, v2)', but Eq. (3) defines LH as the negative distance. Since Eq. (5) sums LH into a loss to be minimized, the intended effect is to maximize ||v1-v2||; this should be stated explicitly to avoid ambiguity.
- [Section 4.3] Several free parameters are not specified: the perturbation step size δ along latent axes, the number of dimensions used for comparison in the Difference Encoder, and the relative weights of LH and Lenc in Eq. (5). Without these, the experiments cannot be reproduced.
- [Section 4.1] The paper refers to 'Differentiable Information Disentanglement (DiD)' in the first sentence of Section 4.1, but the acronym was defined as 'Disentanglement in Difference' in the abstract and introduction. This appears to be a typo.
- [Section 4.6, Figure 3] The ablation study varies the number of comparison dimensions, but the x-axis label is not readable in the figure, and the text does not state the default number used in Table 1. Please clarify the experimental setup.
Circularity Check
The central 'inter-factor difference' loss is defined over the model's own input-code axes, so the claim of direct semantic disentanglement reduces to separating those axes.
-
self definitional
[Section 3.2, Section 3.4, Eq. (3)]
"We then select two dimensions, denoted a1 and a2, from S centered at c. These dimensions correspond to two mutually independent latent factors that we aim to disentangle. ... LH = −||v1 − v2||2 = −||H(g1) − H(g2)||2 (3)"
The 'factors' that the loss separates are, by construction, the input-code axes a1 and a2: g1 and g2 are created by perturbing one coordinate of c, and Eq. (3) maximizes only the distance between the two resulting difference encodings. No term verifies that these axes correspond to dataset semantic factors. Therefore 'maximizing inter-factor differences' is defined to be 'maximizing inter-axis differences,' and the paper's conclusion that DiD directly learns semantic differences restates this definition rather than deriving alignment with the data's generative factors.
-
fitted input called prediction
[Section 3.5, Eq. (4)]
"Our goal is to make the encoder’s output c′enc of this sample in the latent space as close as possible to the original latent code c used to generate the sample. ... Lenc = ||c − c′enc||22 = ||c − E(G(c))||22 (4)"
This objective trains E to return exactly the code c that was sampled and then used to define the two 'factor' axes in Eq. (3). Consequently, for any generated image, E's output is fitted to the generator's input by construction, and representations evaluated in Table 1 measure how faithfully the encoder inverts G's own latent axes. The paper presents this as learning a disentangled representation of the data, but the target of the fitting is the same self-generated code whose axes were already labeled 'factors' in the loss, so the claimed prediction of a semantic latent code is the training target itself.
full rationale
The paper contains no load-bearing self-citations: all cited baselines and architectural components are external, and none of the central claims is justified by a prior work of these authors. The main circularity is self-definitional. In Section 3.2, the 'factors' to be disentangled are introduced as axes a1 and a2 of the sampled latent code, with the assumption that perturbing one coordinate produces a change in exactly one factor. Equation (3) then maximizes the distance between the difference encodings of two such axis-perturbation pairs. If 'factor' is read as 'axis of the input code' — the only meaning the loss can access — then maximizing inter-factor differences is identical to separating the model's own axes; the semantic correspondence to dataset factors is assumed, not learned. Equation (4) completes the loop by training the sample encoder to reproduce the sampled code c, so the representation is anchored to the generator's own inputs. The reported MIG, DCI-D, and SAP numbers are external measurements and could in principle refute the claim, so the circularity is partial rather than total. However, the paper's core narrative that DiD 'directly learn[s] semantic differences' rests on equating code axes with semantic factors, which is the very correspondence the method is supposed to establish. The lack of any compactness constraint on same-axis difference vectors and the unboundedness of H are correctness risks, not circularity, and are not scored here.
Assumptions & free parameters
free parameters (4)
- latent dimension n =
not reported
- perturbation step size along latent axes =
not reported
- number of comparison dimensions =
not reported
- loss weights for LH and Lenc =
1 (implicit)
assumptions (5)
- ad hoc to paper WGAN-GP training yields a generator whose input axes are aligned with the true semantic factors of the dataset
- domain assumption Euclidean distance between difference-encoding vectors reflects semantic distinctness
- domain assumption The latent uniform prior U[-1,1]^n is sufficient to cover all semantic variations
- domain assumption Dataset factors are the same as the axes the model is expected to discover
- standard math Disentanglement metrics (MIG, DCI, SAP) reliably measure semantic disentanglement
invented entities (2)
-
Difference Encoder (H)
-
semantic difference vector
Cite this review
Pith. "Pith review of Disentanglement in Difference: Directly Learning Semantically Disentangled Representations by Maximizing Inter-Factor Differences." pith.science (2026). https://pith.science/paper/OVLWSGR6
@misc{pith2026250203123,
author = {Pith},
title = {Pith review of: Disentanglement in Difference: Directly Learning Semantically Disentangled Representations by Maximizing Inter-Factor Differences},
year = {2026},
howpublished = {\url{https://pith.science/paper/OVLWSGR6}},
note = {Machine review of arXiv:2502.03123}
}
read the original abstract
In this study, Disentanglement in Difference(DiD) is proposed to address the inherent inconsistency between the statistical independence of latent variables and the goal of semantic disentanglement in disentanglement representation learning. Conventional disentanglement methods achieve disentanglement representation by improving statistical independence among latent variables. However, the statistical independence of latent variables does not necessarily imply that they are semantically unrelated, thus, improving statistical independence does not always enhance disentanglement performance. To address the above issue, DiD is proposed to directly learn semantic differences rather than the statistical independence of latent variables. In the DiD, a Difference Encoder is designed to measure the semantic differences; a contrastive loss function is established to facilitate inter-dimensional comparison. Both of them allow the model to directly differentiate and disentangle distinct semantic factors, thereby resolving the inconsistency between statistical independence and semantic disentanglement. Experimental results on the dSprites and 3DShapes datasets demonstrate that the proposed DiD outperforms existing mainstream methods across various disentanglement metrics.
Figures
Reference graph
Works this paper leans on
-
[1]
Wasserstein generative adversarial networks
Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein generative adversarial networks. In International conference on machine learning, pp.\ 214--223. PMLR, 2017
2017
-
[2]
Representation learning: A review and new perspectives
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35 0 (8): 0 1798--1828, 2013
2013
-
[3]
Scaling learning algorithms toward ai
Bottou, L., Chapelle, O., DeCoste, D., and Weston, J. Scaling learning algorithms toward ai. 2007
work page 2007
-
[4]
and Kim, H
Burgess, C. and Kim, H. 3d shapes dataset. https://github.com/deepmind/3dshapes-dataset/, 2018
2018
-
[5]
P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A
Burgess, C. P., Higgins, I., Pal, A., Matthey, L., Watters, N., Desjardins, G., and Lerchner, A. Understanding disentangling in -vae. arXiv preprint arXiv:1804.03599, 2018
arXiv 2018
-
[6]
Measuring disentanglement: A review of metrics
Carbonneau, M.-A., Zaidi, J., Boilard, J., and Gagnon, G. Measuring disentanglement: A review of metrics. IEEE transactions on neural networks and learning systems, 2022
work page 2022
-
[7]
T., Li, X., Grosse, R
Chen, R. T., Li, X., Grosse, R. B., and Duvenaud, D. K. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018
2018
-
[8]
Infogan: Interpretable representation learning by information maximizing generative adversarial nets
Chen, X., Duan, Y., Houthooft, R., Schulman, J., Sutskever, I., and Abbeel, P. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. Advances in neural information processing systems, 29, 2016
2016
Show all 36 references
-
[9]
and Williams, C
Eastwood, C. and Williams, C. K. A framework for the quantitative evaluation of disentangled representations. In 6th International Conference on Learning Representations, 2018
2018
-
[10]
and Hoshen, Y
Gabbay, A. and Hoshen, Y. Demystifying inter-class disentanglement. arXiv preprint arXiv:1906.11796, 2019
1906 arXiv
-
[11]
and Hoshen, Y
Gabbay, A. and Hoshen, Y. Scaling-up disentanglement for image translation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 6783--6792, 2021
2021
-
[12]
Gulrajani, I., Ahmed, F., Arjovsky, M., Dumoulin, V., and Courville, A. C. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017
2017
-
[13]
P., Glorot, X., Botvinick, M
Higgins, I., Matthey, L., Pal, A., Burgess, C. P., Glorot, X., Botvinick, M. M., Mohamed, S., and Lerchner, A. beta-vae: Learning basic visual concepts with a constrained variational framework. ICLR (Poster), 3, 2017
2017
-
[14]
Towards a definition of disentangled representations
Higgins, I., Amos, D., Pfau, D., Racaniere, S., Matthey, L., Rezende, D., and Lerchner, A. Towards a definition of disentangled representations. arXiv preprint arXiv:1812.02230, 2018
2018 arXiv
-
[15]
and Mnih, A
Kim, H. and Mnih, A. Disentangling by factorising. In International conference on machine learning, pp.\ 2649--2658. PMLR, 2018
2018
-
[16]
Kingma, D. P. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013
2013 arXiv
-
[18]
Variational inference of disentangled latent concepts from unlabeled observations
Kumar, A., Sattigeri, P., and Balakrishnan, A. Variational inference of disentangled latent concepts from unlabeled observations. arXiv preprint arXiv:1711.00848, 2017 b
2017 arXiv
-
[19]
M., Ullman, T
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gershman, S. J. Building machines that learn and think like people. Behavioral and brain sciences, 40: 0 e253, 2017
2017
-
[20]
Gradient-based learning applied to document recognition
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86 0 (11): 0 2278--2324, 1998
1998
-
[21]
Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans
Lin, Z., Thekumparampil, K., Fanti, G., and Oh, S. Infogan-cr and modelcentrality: Self-supervised model training and selection for disentangling gans. In international conference on machine learning, pp.\ 6127--6139. PMLR, 2020
2020
-
[22]
Challenging common assumptions in the unsupervised learning of disentangled representations
Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Sch \"o lkopf, B., and Bachem, O. Challenging common assumptions in the unsupervised learning of disentangled representations. In international conference on machine learning, pp.\ 4114--4124. PMLR, 2019 a
2019
-
[23]
a tsch, G., Sch \
Locatello, F., Tschannen, M., Bauer, S., R \"a tsch, G., Sch \"o lkopf, B., and Bachem, O. Disentangling factors of variation using few labels. arXiv preprint arXiv:1905.01258, 2019 b
1905 arXiv
-
[24]
dsprites: Disentanglement testing sprites dataset
Matthey, L., Higgins, I., Hassabis, D., and Lerchner, A. dsprites: Disentanglement testing sprites dataset. https://github.com/deepmind/dsprites-dataset/, 2017
2017
-
[25]
Elements of causal inference: foundations and learning algorithms
Peters, J., Janzing, D., and Sch \"o lkopf, B. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017
2017
-
[26]
Unsupervised representation learning with deep convolutional generative adversarial networks
Radford, A. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015
2015 arXiv
-
[27]
and Mozer, M
Ridgeway, K. and Mozer, M. C. Learning deep disentangled embeddings with the f-statistic loss. Advances in neural information processing systems, 31, 2018
2018
-
[28]
Semilinear predictability minimization produces well-known feature detectors
Schmidhuber, J., Eldracher, M., and Foltin, B. Semilinear predictability minimization produces well-known feature detectors. Neural Computation, 8 0 (4): 0 773--786, 1996
1996
-
[29]
Dynamicvae: Decoupling reconstruction error and disentangled representation learning
Shao, H., Lin, H., Yang, Q., Yao, S., Zhao, H., and Abdelzaher, T. Dynamicvae: Decoupling reconstruction error and disentangled representation learning. arXiv preprint arXiv:2009.06795, 2020
2009 arXiv
-
[30]
Recent advances in autoencoder-based representation learning
Tschannen, M., Bachem, O., and Lucic, M. Recent advances in autoencoder-based representation learning. arXiv preprint arXiv:1812.05069, 2018
2018 arXiv
-
[31]
Are disentangled representations helpful for abstract visual reasoning? Advances in neural information processing systems, 32, 2019
Van Steenkiste, S., Locatello, F., Schmidhuber, J., and Bachem, O. Are disentangled representations helpful for abstract visual reasoning? Advances in neural information processing systems, 32, 2019
2019
-
[32]
Information theoretical analysis of multivariate correlation
Watanabe, S. Information theoretical analysis of multivariate correlation. IBM Journal of research and development, 4 0 (1): 0 66--82, 1960
1960
-
[33]
Zeiler, M. D. and Fergus, R. Visualizing and understanding convolutional networks. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13, pp.\ 818--833. Springer, 2014
2014
-
[34]
Y., Carlsson, G., and Ermon, S
Zhou, S., Zelikman, E., Lu, F., Ng, A. Y., Carlsson, G., and Ermon, S. Evaluating the disentanglement of deep generative models through manifold topology. arXiv preprint arXiv:2006.03680, 2020
2006 arXiv
-
[35]
Visual object networks: Image generation with disentangled 3d representations
Zhu, J.-Y., Zhang, Z., Zhang, C., Wu, J., Torralba, A., Tenenbaum, J., and Freeman, B. Visual object networks: Image generation with disentangled 3d representations. Advances in neural information processing systems, 31, 2018
2018
-
[36]
Where and what? examining interpretable disentangled representations
Zhu, X., Xu, C., and Tao, D. Where and what? examining interpretable disentangled representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 5861--5870, 2021
2021
-
[37]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.