REVIEW 2 major objections 5 minor 33 references
Structuring Autoencoders
T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A Structuring Autoencoder can learn a latent space in which user-chosen class distances are preserved, so a linear classifier on the latent variables outperforms comparable models trained directly for classification when labels are scarce.
desk verdict A useful idea from an integration of MDS and autoencoder training, with a real mathematical error in the alignment step that needs fixing, but the core approach deserves peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the per-epoch calculation of target latent positions using multidimensional scaling (MDS), combined with an orthogonal alignment step and a structural loss $L_S = \|f_{enc}(x) - \tilde{z}\|_2^2$. MDS derives target class centers from a fixed distance matrix $D$, and the alignment rotates these targets to match the current latent positions $Z$ before the encoder is pulled toward $\tilde{Z}=RZ^*$; the reconstruction loss $L_{AE}$ anchors the representation to the data. This treats class geometry as coordinates rather than a classifier boundary, so a linear classifier on the latent variables reads off the intended structure.
What would settle it
Take a three-class problem with latent dimension two, train the SAE on a standard dataset, and at an epoch compute the SVD of the MDS target $Z^*$: if $Z^*$ has a zero singular value, or the orthogonal alignment $R=US^*V^T$ fails the orthogonality check $R^TR=I$, then the targets are not reachable by rotation and any observed class separation cannot be attributed to the stated structural-loss mechanism. Reimplementing the algorithm and looking for such checkpoints settles whether the mechanism is doing the work.
Extended reading notes
Core claim
The paper's central claim is that an autoencoder can be trained to honor a user-specified class distance structure in its latent space, even when those classes are invisible in the raw data. At each epoch the encoder projects the whole training set to latent variables $Z$; multidimensional scaling, seeded with the current $Z$, produces target positions $Z^*$ from the fixed class-distance matrix $D$, and an orthogonal alignment rotates $Z^*$ to fit $Z$ as closely as possible. The structural loss $\mathcal{L}_S = \|f_{enc}(x) - \tilde{z}\|_2^2$ then drives the encoder toward those positions while the reconstruction loss keeps the representation decodable. With this mechanism, class separation appears in latent space for MNIST digits grouped into arbitrary classes, for season-based groupings of Fashion-MNIST, for skirts versus shorts in DeepFashion2, and for male versus female 3D body shapes; a linear SVM on the latent variables outperforms standard classifiers, and the margin information doubles as a calibrated confidence and a guide for which samples to label next.
Load-bearing premise
The load-bearing premise is that the MDS target matrix $Z^*$ has full column rank whenever there are more data points than latent dimensions, so that the orthogonal alignment can rotate the target positions to fit the current latent space; with only a few classes the target configuration lies in a lower-dimensional subspace, the pseudoinverse condition $Z^*Z^+=I$ can fail, and the structural loss may then pull the encoder toward positions that do not encode the intended class distances.
Editorial extensions
If this is right
- A linear SVM on the SAE latent space gives lower test error than a same-architecture classifier and an adversarial autoencoder baseline, and the gap grows as the number of labeled samples shrinks (Fig. 6).
- The reconstruction loss stays close to that of a traditional autoencoder for a wide range of the balancing parameter $\gamma$; the structural constraint can be added without visibly degrading image or body-shape reconstruction (Figs. 11-13).
- The SVM margin in the SAE latent space is monotonically related to actual precision, so the produced confidence values are informative where a softmax classifier's scores are noisy (Fig. 8).
- In guided labeling, adding the 100 most uncertain points found by the latent margin to a 600-sample MNIST training set lowers test error from about 4% to 3%, outperforming random sample selection (Fig. 10).
- Moving a latent code along the vector between male and female class centers produces a smooth morph between body shapes while the pose stays fixed (Fig. 14).
Reading between the lines
- Editorial inference: the same construction should carry over to ordinal or hierarchical label distances, where the MDS target is genuinely high-dimensional and the rank condition is easier to satisfy; the paper only tests equal-distance, few-class settings.
- Editorial inference: the monotone score-to-precision relation suggests the latent margin could serve as a reject option for automated decisions in safety-critical settings, an application the paper does not develop.
- Editorial inference: comparing SAE-guided labeling with standard uncertainty sampling, such as softmax entropy, would isolate whether the benefit comes from the structured latent geometry or from the boundary-distance heuristic itself.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Structuring AutoEncoders (SAE), which augment a standard autoencoder with a structural loss that encourages a user-defined class structure in the latent space. The desired structure is specified by a distance matrix between classes, and at each training iteration the encoder's current latent positions are used to initialize Multidimensional Scaling (MDS), producing target positions Z*. An "orthogonal alignment" step computes a rotation R to align Z* with the current latent matrix Z, and the structural loss penalizes the squared distance between encoded points and the aligned targets R Z*. The authors report experiments on MNIST, Fashion-MNIST, DeepFashion2, and a 3D body-shape dataset, showing improved classification with few labels, more reliable decision confidence, effective guided labeling, and class morphing.
Significance. If the proposed method works as claimed, it offers a simple and general mechanism for injecting weak supervision into autoencoders, with practical applications in semi-supervised classification, active labeling, and latent-space interpretability. The paper covers a broad range of datasets and architectures, and it explicitly analyzes the trade-off between reconstruction error and structure via the balancing parameter gamma. The guided-labeling idea is appealing and the morphing demonstration is visually compelling. However, the central Procrustes alignment derivation contains a mathematical error that, as written, invalidates the claimed guarantee that the structural loss enforces the desired class distances. The experimental evaluation also lacks error bars and does not fully control for the use of unlabeled data in the semi-supervised setting. These issues prevent the paper from being accepted in its current form, but they are addressable in a revision.
major comments (2)
- [Sec. 2.3] The derivation of the rotation R is mathematically incorrect. The paper assumes that the Moore-Penrose inverse Z+ of Z* satisfies Z*Z+ = I whenever there are more data points than latent dimensions. For an n x d row-stacked matrix with n>d, the meaningful identity would be Z+Z* = I, and it requires Z* to have full column rank d. In all experiments, Z* is obtained by MDS from class-level distances over k=2 or k=3 classes, so rank(Z*) <= k-1 (1 or 2), while the latent dimension d is 10, 30, 64, or 192. Hence Z*Z+ != I and P* = ZZ+ is singular. Setting nonzero singular values to 1 then produces a matrix R that is a partial isometry rather than an orthogonal rotation, and it is not the standard Procrustes solution. Consequently, tilde{Z} = R Z* need not preserve the distances in D, and the structural loss in Eq. (2) does not provably enforce the intended class structure. This is a load-bearing issue because the structural loss is the core mechanism of the SAE. Please correct the derivation, state the exact pseudo-inverse/Procrustes computation used in the implementation, and verify that the resulting tilde{Z} preserves the target distances in all reported settings.
- [Sec. 3.3] The classification curves in Fig. 6 are reported as single runs without error bars or repeated-run statistics. Given that the claimed advantage over baselines is especially pronounced at low labeled-sample counts, the results may be sensitive to random initialization and data subsampling. Please report mean and standard deviation over at least five independent runs with different seeds and subsamples. In addition, clarify whether the "comparable neural network" baseline is trained only on the labeled subset while the SAE additionally uses all unlabeled data through its reconstruction loss; if so, the comparison is semi-supervised versus supervised and should be described as such, and the AAE baseline should be clearly distinguished.
minor comments (5)
- [Algorithm 1] Algorithm 1 says "set all singular values >= 0 to 1", which would include zero singular values; the text in Sec. 2.3 correctly says "setting all nonzero singular values to 1". Please align the algorithm listing with the text.
- [Sec. 3.1] The paper describes 6000 labeled samples as "a very sparse set of data", but 6000 out of 60000 is 10%. Later experiments use 600 samples. Please use consistent and precise terminology for the amount of supervision.
- [Fig. 4] The 2D projections in Fig. 4 would benefit from axis labels and a legend; the current figure is hard to read in print.
- [Sec. 3.6] The optimal gamma for the 3D HumanPose dataset is reported as 0.0041, but Fig. 11 does not clearly show the scale of the curves. A log-scale inset or a table would make the behavior more transparent.
- [General] No code or implementation details for the MDS and Procrustes steps are provided; making the code available would substantially improve reproducibility, especially given the mathematical issue raised above.
Circularity Check
No circularity: the structural target is user-defined and classification is evaluated on held-out test data.
full rationale
The derivation chain is self-contained. In Sec. 2.2 the target structure is defined by a user-supplied distance matrix D, not derived from the encoder's latent variables. MDS in Sec. 2.3 computes Z* from D, and the orthogonal alignment (Eqs. 5-8) only rotates Z* to align with the current latent coordinates; it does not fit the distance structure to the data. The structural loss (Eq. 2) then pulls encoder outputs toward tilde-Z = R Z*, whose pairwise distances remain those of D. No quantity fitted to the training data is later reported as a prediction: the classification experiments in Sec. 3.3 train a linear SVM on the latent variables and evaluate on held-out test data, so the reported accuracy is not forced by construction. The hyperparameter gamma is selected by examining error curves (Sec. 3.6), which is normal model selection rather than a self-definitional reduction. The paper's self-citations (references 1, 2, 30, 32) appear only in related-work and future-work remarks and are not load-bearing for the central claims. The rank-deficiency concern about the Moore-Penrose inverse in Sec. 2.3 is a mathematical correctness issue, not circularity: even if the alignment is not a true rotation, the target still originates from D and the empirical claims are evaluated on held-out data. Therefore no circular step is present.
Assumptions & free parameters
free parameters (3)
- gamma =
0.5 (MNIST), 0.75 (Fashion-MNIST), 0.0041 (3D HumanPose)
- latent_dim =
10 (MNIST), 64 (Fashion-MNIST), 30 (3D HumanPose), 192 (DeepFashion2)
- class_distance_scale =
1.0
assumptions (4)
- domain assumption Data can be partitioned into classes that are not obvious in the raw data.
- domain assumption There exists a latent space in which class distances can be preserved while still allowing reconstruction of the input.
- standard math MDS (Shepard-Kruskal) produces a configuration that preserves the pairwise class-distance matrix D.
- standard math Procrustes analysis via SVD gives the optimal rotation aligning Z* to Z.
Cite this review
Pith. "Pith review of Structuring Autoencoders." pith.science (2026). https://pith.science/paper/UQGDCO4I
@misc{pith2026190802626,
author = {Pith},
title = {Pith review of: Structuring Autoencoders},
year = {2026},
howpublished = {\url{https://pith.science/paper/UQGDCO4I}},
note = {Machine review of arXiv:1908.02626}
}
read the original abstract
In this paper we propose Structuring AutoEncoders (SAE). SAEs are neural networks which learn a low dimensional representation of data which are additionally enriched with a desired structure in this low dimensional space. While traditional Autoencoders have proven to structure data naturally they fail to discover semantic structure that is hard to recognize in the raw data. The SAE solves the problem by enhancing a traditional Autoencoder using weak supervision to form a structured latent space. In the experiments we demonstrate, that the structured latent space allows for a much more efficient data representation for further tasks such as classification for sparsely labeled data, an efficient choice of data to label, and morphing between classes. To demonstrate the general applicability of our method, we show experiments on the benchmark image datasets MNIST, Fashion-MNIST, DeepFashion2 and on a dataset of 3D human shapes.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
M. Awiszus, H. Ackermann, and B. Rosenhahn. Learning disentangled representations via independent subspaces. In Third International Workshop on ”Robust Subspace Learn- ing and Applications in Computer Vision”, 2019
work page 2019
-
[2]
M. Awiszus and B. Rosenhahn. Markov chain neural net- works. In Computer Vision and Pattern Recognition Work- shops (CVPRW), June 2018
work page 2018
-
[3]
H. Bourlard and Y . Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59(4):291–294, Sep 1988
work page 1988
-
[4]
M. A. Carreira-Perpi ˜n´an and R. Raziperchikolaei. Hash- ing with binary autoencoders. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 557–566, June 2015
work page 2015
-
[5]
M. Chen, Z. Xu, K. Weinberger, and F. Sha. Marginalized denoising autoencoders for domain adaptation. In J. Lang- ford and J. Pineau, editors, Proceedings of the 29th Interna- tional Conference on Machine Learning (ICML-12) , ICML ’12, pages 767–774. ACM, New York, NY , USA, July 2012
work page 2012
-
[6]
Y . Chen, L. Zhang, and Z. Yi. Subspace clustering using a low-rank constrained autoencoder. Information Sciences, 424:27–38, 2018
work page 2018
-
[7]
C. Donahue, A. Balsubramani, J. McAuley, and Z. C. Lip- ton. Semantically decomposing the latent spaces of gener- ative adversarial networks. In International Conference on Learning Representations, 2018
work page 2018
-
[8]
Y . Ge, R. Zhang, L. Wu, X. Wang, X. Tang, and P. Luo. Deepfashion2: A versatile benchmark for detection, pose es- timation, segmentation and re-identification of clothing im- ages. CoRR, abs/1901.07973, 2019
work page Pith review arXiv 1901
Show all 33 references
-
[9]
Graßhof, H
S. Graßhof, H. Ackermann, S. S. Brandt, and J. Ostermann. Apathy is the root of all expressions. In 2017 12th IEEE In- ternational Conference on Automatic Face & Gesture Recog- nition (FG 2017), pages 658–665. IEEE, 2017
2017
-
[10]
S. Gu, J. Bao, H. Yang, D. Chen, F. Wen, and L. Yuan. Mask- guided portrait editing with conditional gans. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3436–3445, 2019
2019
-
[11]
G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504–507, 2006
2006
-
[12]
D. P. Kingma and M. Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013
2013 arXiv
-
[13]
J. B. Kruskal. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 29(1):1–27, Mar 1964
1964
-
[14]
T. D. Kulkarni, W. F. Whitney, P. Kohli, and J. Tenenbaum. Deep convolutional inverse graphics network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2539–2547. Curran Associates, Inc., 2015
2015
-
[15]
Y . LeCun. Generalization and network design strategies. In R. Pfeifer, Z. Schreter, F. Fogelman, and L. Steels, editors, Connectionism in Perspective , Zurich, Switzerland, 1989. Elsevier. an extended version was published as a technical report of the University of Toronto
1989
-
[16]
LeCun and C
Y . LeCun and C. Cortes. MNIST handwritten digit database. 2010
2010
-
[17]
F. Li, H. Qiao, and B. Zhang. Discriminatively boosted im- age clustering with fully convolutional auto-encoders. Pat- tern Recognition, 83:161 – 173, 2018
2018
-
[18]
H. Liu, M. Shao, S. Li, and Y . Fu. Infinite ensemble for image clustering. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 1745–1754, New York, NY , USA,
-
[19]
Loper, N
M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) , 34(6):248:1– 248:16, Oct. 2015
2015
-
[20]
Makhzani, J
A. Makhzani, J. Shlens, N. Jaitly, and I. Goodfellow. Adver- sarial autoencoders. In International Conference on Learn- ing Representations, 2016
2016
-
[21]
McInnes and J
L. McInnes and J. Healy. UMAP: Uniform Manifold Ap- proximation and Projection for Dimension Reduction. ArXiv e-prints, Feb. 2018
2018
-
[22]
Roberts, J
A. Roberts, J. Engel, C. Raffel, C. Hawthorne, and D. Eck. A hierarchical latent vector model for learning long-term struc- ture in music. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machin...
2018
-
[23]
P. H. Sch ¨onemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1–10, Mar 1966
1966
-
[24]
Simonyan and A
K. Simonyan and A. Zisserman. Very deep convolu- tional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014
2014 arXiv
-
[25]
C. Song, F. Liu, Y . Huang, L. Wang, and T. Tan. Auto- encoder based data clustering. In J. Ruiz-Shulcloper and G. Sanniti di Baja, editors, Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications , pages 117–124, Berlin, Heidelberg, 2013. Springer B...
2013
-
[26]
van der Maaten
L. van der Maaten. Accelerating t-sne using tree-based al- gorithms. Journal of Machine Learning Research, 15:3221– 3245, 2014
2014
-
[27]
van der Maaten and G
L. van der Maaten and G. Hinton. Visualizing high- dimensional data using t-sne. Journal of Machine Learning Research, 9:2579–2605, 2008
2008
-
[28]
V . N. Vapnik and A. Y . Chervonenkis. Theory of Pattern Recognition. Nauka, USSR, 1974
1974
-
[29]
Vincent, H
P. Vincent, H. Larochelle, I. Lajoie, Y . Bengio, and P.-A. Manzagol. Stacked denoising autoencoders: Learning use- ful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res., 11:3371–3408, Dec. 2010
2010
-
[30]
Wandt and B
B. Wandt and B. Rosenhahn. Repnet: Weakly supervised training of an adversarial reprojection network for 3d human pose estimation. In The IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), June 2019
2019
-
[31]
H. Xiao, K. Rasul, and R. V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning al- gorithms. arXiv preprint arXiv:1708.07747, 2017
2017 arXiv
-
[32]
M. Y . Yang, W. Liao, Y . Cao, and B. Rosenhahn. Video event recognition and anomaly detection by combining gaussian process and hierarchical dirichlet process models. In Pho- togrammetric Engineering & Remote Sensing, 2018
2018
-
[33]
F. W. Young. Multidimensional Scaling: History, Theory, and Applications. Lawrence, Erlbaum Associates, Publish- ers (Hillsdale, New Jersey; London), 1987
1987
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.