REVIEW 3 major objections 6 minor 36 references
Learning Disentangled Representations via Independent Subspaces
T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper proposes a resnet autoencoder whose latent space is split into independent subspaces, each tied by a mask loss to a facial region, so that swapping one subspace between two faces transfers exactly that region and leaves the…
desk verdict Face-part swapping works, but the 'independent subspaces' claim is unsupported: the entropy loss in Sec. 2.4 has a trivial optimum and does not measure independence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the independent-subspace decomposition layer: a non-singular matrix $A$ that maps the encoder output $z_{enc}$ to subspace coordinates $s = A^{-1}z_{enc}$ and back to $z_{dec} = A s$, with the requirement $z_{dec}=z_{enc}$ so the layer can be bypassed during reconstruction backpropagation. Two losses train it: the mask loss $\mathcal{L}_m$ uses diagonal gating matrices $D_m$ to mix two images' subspace coordinates and penalizes deviations from the input outside the selected mask and from the target inside it, and the entropy loss $\mathcal{L}_e$ maps each subspace through functions $F_i$ and trains a $C$-way classifier to distinguish which subspace a sample came from, using binary cross-entropy. The matrix $A$ is what lets the same latent space be factored into groups that may be correlated within but independent across; the losses are what attach those groups to image regions.
What would settle it
Measure pairwise mutual information between the five learned subspace coordinate sets on held-out face images; if the mutual information between any pair is substantially above zero, or if swapping one subspace in a large batch systematically changes pixels outside the corresponding segmentation region beyond reconstruction noise, then the independence claim would be refuted. A simpler control: two dependent but separable distributions, such as two perfectly correlated Gaussian blobs, would pass a classifier-based independence check even though they are not independent.
Extended reading notes
Core claim
The central claim is that disentanglement of face images can be achieved by decomposing the latent space of a resnet autoencoder into subspaces that are mutually statistically independent, with each subspace aligned to one facial region. The encoder output $z_{enc}$ is mapped through a non-singular matrix $A$ to source coordinates $s = A^{-1}z_{enc}$; the decoder input is $z_{dec} = A s$, and these decomposition layers are excluded from the reconstruction backpropagation so they are trained only by the mask loss and entropy loss. The mask loss interpolates between two encoded images while gating which subspace coordinates are exchanged, then penalizes differences in image regions according to segmentation masks; the entropy loss maps each subspace through a learned function and trains a classifier to identify the subspace, extending the single-dimension independence idea to multi-dimensional subspaces. If the claim holds, the network performs localized semantic editing with one end-to-end trainable model, and the segmentation is required only during training.
Load-bearing premise
The method's claimed independence of subspaces rests on the assumption that training a classifier to determine which subspace a latent sample came from makes the subspace distributions statistically independent, yet separable distributions do not have to be independent, and the paper offers no proof that the entropy loss implies independence.
Editorial extensions
If this is right
- An unseen face can be edited by swapping one or more subspace blocks with another face, and the swapped parts appear as realistic, coherent facial features.
- Removing the entropy loss makes cross-subspace contamination visible: changing the background-and-hair subspace can create facial parts such as hair where none should appear.
- Region-local attributes such as an open mouth are separated mostly in the mouth subspace, while global attributes such as gender are distributed across all subspaces, showing the decomposition follows image regions rather than arbitrary factors.
- Once trained, the network can perform test-time edits without any semantic segmentation or manual mask input.
- Because only chosen subspace coordinates are exchanged, the method can change one region while preserving identity and the rest of the face, unlike attribute-editing baselines that change the whole image.
Reading between the lines
- If the independence result holds, the same subspace-decomposition training should extend to other aligned multi-part objects such as full bodies, vehicles, or medical scans, with masks used only for training; this is a direct testable extension the paper only gestures at.
- The entropy loss most plausibly enforces class separability, not statistical independence; replacing it with an explicit mutual-information penalty or an independence test would make the central claim directly measurable.
- The PCA-based subspace analysis implies a diagnostic rule: an attribute whose mean-distance concentrates in one subspace is a region-local edit direction, while an attribute spread across all subspaces (such as gender) is global, which could predict which attribute edits will stay localized without retraining.
- Because masking is needed only during training, coupling the method with an unsupervised segmentation model would remove the remaining training-time supervision and make the approach applicable where masks are unavailable.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a ResNet autoencoder whose latent space is decomposed into five fixed subspaces, each associated with a semantic face region: background+hair, face, eyebrows, eyes, and mouth. Training combines a reconstruction loss, a gradient loss, a mask loss (Eq. 5) that mixes latent coordinates from two images so that the decoded image matches the target's masked regions, and an entropy loss (Sec. 2.4) intended to make the subspaces statistically independent. The authors demonstrate qualitative face-part swapping, an ablation of the entropy loss, a PCA-based subspace analysis, and a comparison with AttGAN for attribute editing.
Significance. If the independence claim were established, the work would be a valuable step toward interpretable face editing with a single, end-to-end trainable autoencoder that does not require semantic segmentation at test time. The mask-supervised exchange of facial parts is visually compelling, the ablation in Fig. 6 suggests that the entropy loss reduces cross-subspace leakage, and the comparison with AttGAN clarifies the advantage of localized editing over global attribute changes. The paper provides a clear architecture, a reproducible training setup, and a useful discussion of limitations in Sec. 3.4.
major comments (3)
- [Sec. 2.4, Eq. (6)] The entropy loss does not enforce statistical independence between subspaces. The C-way classifier trained on F_i-mapped samples can achieve perfect accuracy whenever the five post-mapping distributions are distinguishable, which is a property of the marginals, not of their joint factorization. Furthermore, the loss has a trivial global optimum that leaves the latent representation unchanged: each F_i can be a constant function (e.g., a zero-weight fully-connected layer with a per-subspace bias), making the classifier perfect and L_e = 0 for any encoder output and any A. In that state, no gradient flows through F_i to the encoder or to A, so the loss cannot shape the representation. The paper must either provide a formal argument that cross-entropy on this classifier penalizes joint dependence, or replace the loss with a proper independence criterion (e.g., an estimator of total correlation, mutual information, or Hilbert-Schmidt independence criterion).
- [Sec. 3.6, Tab. 1 and Fig. 8] The subspace analysis is not valid evidence of independence. Because the mask loss (Eq. 5) explicitly supervises each subspace to reconstruct the corresponding facial region, it is expected that an attribute like 'mouth open' has its largest PCA-distance in the mouth subspace even without any statistical-independence mechanism. The experiment therefore mainly reflects the mask supervision, not an emergent property of the entropy loss. Additionally, L2 distances between class-conditional PCA means are a weak proxy for dependence; no error bars, confidence intervals, or formal independence tests are provided.
- [Sec. 3.5, Fig. 6 and Eq. (9)] The quantitative claim that the entropy loss improves disentanglement is based on a single comparison without error bars, and the metric uses the same masks used for supervision. While the with/without-ISA comparison is informative, the evaluation should be repeated over several random seeds and supplemented with a direct measure of cross-subspace leakage that does not presuppose the mask correspondence. As written, the experiment does not substantiate the central claim of statistical independence.
minor comments (6)
- [Sec. 1.1] There is a typo: 'Insteadt' should be 'Instead'.
- [Sec. 2.4] For C > 2 classes with a softmax output, the loss should be categorical cross-entropy, not binary cross-entropy. Please clarify the implementation.
- [Sec. 2.3, Eq. (4)] The definitions of D_m and D_-m are confusing; clarify that D_m selects the coordinates of subspace m and D_-m selects the complement.
- [Sec. 3.5, Eq. (9)] The normalization 'divided by the sum of pixels x,y of the corresponding mask' is ambiguous; specify whether the sum is over all pixels in the mask area and whether soft masks are thresholded.
- [Sec. 3.1] The hyperparameters lambda_1..lambda_4 are fixed without any ablation; a sensitivity analysis would strengthen the paper.
- [Fig. 6] Error bars are missing; include variance over multiple runs or batches.
Circularity Check
Quantitative evaluation re-uses the mask training loss, and the entropy loss has a trivial optimum that leaves the latent distribution unchanged.
-
fitted input called prediction
[Sec. 3.5, Eqs. (5), (8), (9)]
"Lm = (Imix−Iin)· (1− max(Mi,in,M i,t)) + (Imix−It)· min(Mi,in,M i,t). (5) ... ej = ∑x∑y|Imix,j(x,y)−Ij,masked(x,y)| / ∑x∑y Mj,j(x,y) (9)"
The quantitative disentanglement metric in Sec. 3.5 is a normalized version of the exact mask loss used to train the network. Eq. 9 sums absolute differences between the mixed image and the original inside each mask region, which is the same per-mask term minimized in Eq. 5. Thus 'the smaller the error, the less influence other subspaces have' measures the training objective itself, not an independent test of disentanglement. The comparison with and without the entropy loss only shows that adding the loss helps optimize the mask-loss objective; it cannot validate the claimed statistical independence of subspaces.
-
other
[Sec. 2.4, Entropy Loss]
"All di variables Xi of a batch corresponding to the ith out of C subspaces are selected, and mapped by a function Fi : Rdi → Rdmax ... The matrix X = [X1 ··· XC] consisting of stacked matrices Xi can now be used to learn a decision problem with C classes. ... For the loss Le, we use binary cross entropy."
The entropy loss is presented as enforcing 'mutual statistical independence between the groups,' but cross-entropy classification into subspace indices measures separability, not factorization of the joint distribution. Because Fi is subspace-specific, a trivial optimum exists: each Fi can output a distinct constant (e.g., a zero-weight fully-connected layer with a per-subspace bias), making the classifier perfect and Le = 0 for any latent distribution. In that regime the loss has no gradient with respect to the encoder or A, so it cannot shape the representation or establish independence. The central 'independent subspaces' claim is therefore not derived from the stated loss.
1 more flagged steps
-
fitted input called prediction
[Sec. 3.6, Subspace Analysis]
"This result confirms our claim that the subspaces are independent: An attribute that should only affect the mouth area of the image, mouth open, only affects the mouth subspace..."
This confirmation uses the same trained s-vectors whose subspace-to-mask correspondence was directly enforced by the mask loss in Sec. 2.3. Finding that mouth-open information concentrates in the mouth subspace is a consequence of training a subspace to control the mouth region, not an independent test of statistical independence. Moreover, the gender example, which the paper says is 'spread across all subspaces,' is itself evidence of dependence across subspaces, undermining the independence claim rather than supporting it.
full rationale
The paper's self-citations (e.g., [28], [33]) appear only as related or future work and are not load-bearing for the central derivation; there is no imported uniqueness theorem and no ansatz smuggled in by citation. The main circularity is in the evaluation: Sec. 3.5's error metric is the normalized mask loss from Eq. 5, so it measures whether the training objective was optimized rather than providing independent evidence of disentanglement. Sec. 3.6 similarly treats a consequence of mask supervision as confirmation of independence. Separately, the entropy loss of Sec. 2.4 does not enforce statistical independence as claimed, because a classifier over subspace indices can be trivially satisfied by subspace-specific constant outputs, leaving the latent distribution unchanged. The qualitative internet-image transfers in Sec. 3.4 do provide some independent evidence that localized face-part swapping generalizes outside the training set, which prevents a higher score. Overall, the face-swapping capability is largely supervised by the mask loss and is not presented as a first-principles prediction, but the paper's quantitative support and its central independence claim are substantially circular or vacuous.
Assumptions & free parameters
free parameters (6)
- lambda_1 =
2
- lambda_2 =
1
- lambda_3 =
1
- lambda_4 =
1
- subspace_dimensions =
512, 256, 128, 128, 128
- number_of_masks =
5
assumptions (5)
- domain assumption The C-way classifier in the entropy loss promotes statistical independence between subspaces
- domain assumption The semantic segmentation masks from the VGG network are accurate enough to provide reliable supervision
- ad hoc to paper The latent space can be meaningfully split into fixed subspaces pre-assigned to masks
- standard math Standard autoencoder losses (MSE, gradient loss) are adequate for the task
- standard math PCA on the learned subspace vectors preserves enough structure to measure information content
Cite this review
Pith. "Pith review of Learning Disentangled Representations via Independent Subspaces." pith.science (2026). https://pith.science/paper/UR4RBN7G
@misc{pith2026190808989,
author = {Pith},
title = {Pith review of: Learning Disentangled Representations via Independent Subspaces},
year = {2026},
howpublished = {\url{https://pith.science/paper/UR4RBN7G}},
note = {Machine review of arXiv:1908.08989}
}
read the original abstract
Image generating neural networks are mostly viewed as black boxes, where any change in the input can have a number of globally effective changes on the output. In this work, we propose a method for learning disentangled representations to allow for localized image manipulations. We use face images as our example of choice. Depending on the image region, identity and other facial attributes can be modified. The proposed network can transfer parts of a face such as shape and color of eyes, hair, mouth, etc.~directly between persons while all other parts of the face remain unchanged. The network allows to generate modified images which appear like realistic images. Our model learns disentangled representations by weak supervision. We propose a localized resnet autoencoder optimized using several loss functions including a loss based on the semantic segmentation, which we interpret as masks, and a loss which enforces disentanglement by decomposition of the latent space into statistically independent subspaces. We evaluate the proposed solution w.r.t. disentanglement and generated image quality. Convincing results are demonstrated using the CelebA dataset.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
M. Awiszus and B. Rosenhahn. Markov chain neural net- works. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops , pages 2180– 2187, 2018. 8
work page 2018
- [2]
- [3]
-
[4]
A. Brunton, T. Bolkart, and S. Wuhrer. Multilinear wavelets: A statistical shape space for human faces. In European Con- ference on Computer Vision, pages 297–312. Springer, 2014. 2
work page 2014
-
[5]
P. P. Busto, C. Eisenacher, S. Lefebvre, M. Stamminger, et al. Instant texture synthesis by numbers. In VMV, pages 81–85,
-
[6]
A. J. Champandard. Semantic style transfer and turn- ing two-bit doodles into fine artworks. arXiv preprint arXiv:1603.01768, 2016. 3
work page Pith review arXiv 2016
-
[7]
Q. Chen and V . Koltun. Photographic image synthesis with cascaded refinement networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 1511– 1520, 2017. 3
work page 2017
-
[8]
X. Chen, Y . Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in neural information processing systems , pages 2172–2180, 2016. 2, 3, 4
work page 2016
Show all 36 references
-
[9]
Goodfellow, J
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y . Bengio. Gen- erative adversarial nets. In Advances in neural information processing systems, pages 2672–2680, 2014. 1, 2
2014
-
[10]
Graßhof, H
S. Graßhof, H. Ackermann, S. Brandt, and J. Ostermann. Ap- athy is the root of all expressions. 12th IEEE Conference on Automatic Face and Gesture Recognition, 2017. 2
2017
-
[11]
S. Gu, J. Bao, H. Yang, D. Chen, F. Wen, and L. Yuan. Mask-guided portrait editing with conditional gans. In The IEEE Conference on Computer Vision and Pattern Recogni- tion, June 2019. 2, 3
2019
-
[12]
Hays and A
J. Hays and A. A. Efros. Scene completion using millions of photographs. ACM Transactions on Graphics, 26(3):4, 2007. 3
2007
-
[13]
Z. He, W. Zuo, M. Kan, S. Shan, and X. Chen. Attgan: Fa- cial attribute editing by only changing what you want. arXiv preprint arXiv:1711.10678, 2017. 2, 4, 7, 8
2017 arXiv
-
[14]
G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504–507, 2006. 1
2006
-
[15]
Hyv ¨arinen, J
A. Hyv ¨arinen, J. Karhunen, and E. Oja. Independent compo- nent analysis, volume 46. John Wiley & Sons, 2004. 2
2004
-
[16]
Isola, J.-Y
P. Isola, J.-Y . Zhu, T. Zhou, and A. A. Efros. Image-to- image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017. 3
2017
-
[17]
Jo and J
Y . Jo and J. Park. Sc-fegan: Face editing generative adver- sarial network with user’s sketch and color. arXiv preprint arXiv:1902.06838, 2019. 3
1902 arXiv
-
[18]
Jolliffe
I. Jolliffe. Principal component analysis. Springer, 2011. 2
2011
-
[19]
D. Joo, D. Kim, and J. Kim. Generating a fusion image: One’s identity and another’s shape. InThe IEEE Conference on Computer Vision and Pattern Recognition, June 2018. 2
2018
-
[20]
D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013. 1, 2
2013 arXiv
-
[21]
Kluger, H
F. Kluger, H. Ackermann, M. Y . Yang, and B. Rosenhahn. Deep learning for vanishing point detection using an inverse gnomonic projection. In German Conference on Pattern Recognition, pages 17–28. Springer, 2017. 8
2017
-
[22]
Lalonde, D
J.-F. Lalonde, D. Hoiem, A. A. Efros, C. Rother, J. Winn, and A. Criminisi. Photo clip art. In ACM transactions on graphics, volume 26, page 3. ACM, 2007. 3
2007
-
[23]
V . Le, J. Brandt, Z. Lin, L. Bourdev, and T. S. Huang. Inter- active facial feature localization. In European conference on computer vision, pages 679–692. Springer, 2012. 5
2012
-
[24]
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Con- ference on Computer Vision, Dec. 2015. 1, 5
2015
-
[25]
S. Mika, G. Ratsch, J. Weston, B. Scholkopf, and K.-R. Mullers. Fisher discriminant analysis with kernels. In Neural networks for signal processing IX: Proceedings of the 1999 IEEE signal processing society workshop (cat. no. 98th8468), pages 41–48. Ieee, 1999. 2
1999
-
[26]
P. J. Phillips, H. Moon, P. Rauss, and S. A. Rizvi. The feret evaluation methodology for face-recognition algorithms. In Proceedings of IEEE Computer Society Conference on Com- puter Vision and Pattern Recognition, pages 137–143. IEEE,
-
[27]
P. J. Phillips, H. Wechsler, J. Huang, and P. J. Rauss. The feret database and evaluation procedure for face-recognition algorithms. Image and vision computing , 16(5):295–306,
-
[28]
Rudolph, B
M. Rudolph, B. Wandt, and B. Rosenhahn. Structuring au- toencoders. In The IEEE International Conference on Com- puter Vision Workshops, 2019. 8
2019
-
[29]
Simonyan and A
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 5
2014 arXiv
-
[30]
B. M. Smith, L. Zhang, J. Brandt, Z. Lin, and J. Yang. Exemplar-based face parsing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3484–3491, 2013. 5
2013
-
[31]
Thies, M
J. Thies, M. Zollh ¨ofer, M. Nießner, L. Valgaerts, M. Stam- minger, and C. Theobalt. Real-time expression transfer for facial reenactment. ACM Trans. Graph., 34(6):183–1, 2015. 2
2015
-
[32]
Vlasic, M
D. Vlasic, M. Brand, H. Pfister, and J. Popovi´c. Face transfer with multilinear models. In ACM transactions on graphics, volume 24, pages 426–433. ACM, 2005. 2
2005
-
[33]
Wandt and B
B. Wandt and B. Rosenhahn. Repnet: Weakly supervised training of an adversarial reprojection network for 3d human pose estimation. In Computer Vision and Pattern Recogni- tion, June 2019. 8
2019
-
[34]
Wang, M.-Y
T.-C. Wang, M.-Y . Liu, J.-Y . Zhu, A. Tao, J. Kautz, and B. Catanzaro. High-resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- tion, pages 8798–8807, 2018. 3
2018
-
[35]
M. Y . Yang, W. Liao, H. Ackermann, and B. Rosenhahn. On support relations and semantic scene graphs. ISPRS journal of photogrammetry and remote sensing, 131:15–25, 2017. 8
2017
-
[36]
J.-Y . Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image- to-image translation using cycle-consistent adversarial net- works. In The IEEE International Conference on Computer Vision, Oct 2017. 2, 3, 4
2017
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.