Pith. sign in

REVIEW 2 major objections 5 minor 33 references

Structuring Autoencoders

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A Structuring Autoencoder can learn a latent space in which user-chosen class distances are preserved, so a linear classifier on the latent variables outperforms comparable models trained directly for classification when labels are scarce.

desk verdict A useful idea from an integration of MDS and autoencoder training, with a real mathematical error in the alignment step that needs fixing, but the core approach deserves peer review. read the letter →

arxiv 1908.02626 v1 pith:UQGDCO4I submitted 2019-08-07 cs.LG cs.CVstat.ML

classification cs.LGcs.CVstat.ML
keywords structuringautoencoderlatentspacestructureweaksupervisionmultidimensionalscalingfew-labelclassificationguidedlabelingdecisionconfidenceclassmorphing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Structuring Autoencoders (SAEs), autoencoders whose training includes a weak-supervision structural loss that pushes the latent representation to match positions computed by multidimensional scaling from a user-defined distance matrix between classes. The claim is that this makes hidden semantic groupings, ones a traditional autoencoder ignores because they are not dominant in reconstruction error, visible and separable in latent space. A consequence the paper demonstrates is that a linear SVM on the structured latent variables classifies sparsely labeled data more accurately than neural networks trained solely for classification, while reconstruction quality stays close to that of an ordinary autoencoder. The same structure also yields meaningful decision confidences and a guided-labeling procedure that identifies which unlabeled examples are most useful to annotate.

What carries the argument

The central mechanism is the per-epoch calculation of target latent positions using multidimensional scaling (MDS), combined with an orthogonal alignment step and a structural loss $L_S = \|f_{enc}(x) - \tilde{z}\|_2^2$. MDS derives target class centers from a fixed distance matrix $D$, and the alignment rotates these targets to match the current latent positions $Z$ before the encoder is pulled toward $\tilde{Z}=RZ^*$; the reconstruction loss $L_{AE}$ anchors the representation to the data. This treats class geometry as coordinates rather than a classifier boundary, so a linear classifier on the latent variables reads off the intended structure.

What would settle it

Take a three-class problem with latent dimension two, train the SAE on a standard dataset, and at an epoch compute the SVD of the MDS target $Z^*$: if $Z^*$ has a zero singular value, or the orthogonal alignment $R=US^*V^T$ fails the orthogonality check $R^TR=I$, then the targets are not reachable by rotation and any observed class separation cannot be attributed to the stated structural-loss mechanism. Reimplementing the algorithm and looking for such checkpoints settles whether the mechanism is doing the work.

Watch

Extended reading notes

Core claim

The paper's central claim is that an autoencoder can be trained to honor a user-specified class distance structure in its latent space, even when those classes are invisible in the raw data. At each epoch the encoder projects the whole training set to latent variables $Z$; multidimensional scaling, seeded with the current $Z$, produces target positions $Z^*$ from the fixed class-distance matrix $D$, and an orthogonal alignment rotates $Z^*$ to fit $Z$ as closely as possible. The structural loss $\mathcal{L}_S = \|f_{enc}(x) - \tilde{z}\|_2^2$ then drives the encoder toward those positions while the reconstruction loss keeps the representation decodable. With this mechanism, class separation appears in latent space for MNIST digits grouped into arbitrary classes, for season-based groupings of Fashion-MNIST, for skirts versus shorts in DeepFashion2, and for male versus female 3D body shapes; a linear SVM on the latent variables outperforms standard classifiers, and the margin information doubles as a calibrated confidence and a guide for which samples to label next.

Load-bearing premise

The load-bearing premise is that the MDS target matrix $Z^*$ has full column rank whenever there are more data points than latent dimensions, so that the orthogonal alignment can rotate the target positions to fit the current latent space; with only a few classes the target configuration lies in a lower-dimensional subspace, the pseudoinverse condition $Z^*Z^+=I$ can fail, and the structural loss may then pull the encoder toward positions that do not encode the intended class distances.

Editorial extensions

If this is right

  • A linear SVM on the SAE latent space gives lower test error than a same-architecture classifier and an adversarial autoencoder baseline, and the gap grows as the number of labeled samples shrinks (Fig. 6).
  • The reconstruction loss stays close to that of a traditional autoencoder for a wide range of the balancing parameter $\gamma$; the structural constraint can be added without visibly degrading image or body-shape reconstruction (Figs. 11-13).
  • The SVM margin in the SAE latent space is monotonically related to actual precision, so the produced confidence values are informative where a softmax classifier's scores are noisy (Fig. 8).
  • In guided labeling, adding the 100 most uncertain points found by the latent margin to a 600-sample MNIST training set lowers test error from about 4% to 3%, outperforming random sample selection (Fig. 10).
  • Moving a latent code along the vector between male and female class centers produces a smooth morph between body shapes while the pose stays fixed (Fig. 14).

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same construction should carry over to ordinal or hierarchical label distances, where the MDS target is genuinely high-dimensional and the rank condition is easier to satisfy; the paper only tests equal-distance, few-class settings.
  • Editorial inference: the monotone score-to-precision relation suggests the latent margin could serve as a reject option for automated decisions in safety-critical settings, an application the paper does not develop.
  • Editorial inference: comparing SAE-guided labeling with standard uncertainty sampling, such as softmax entropy, would isolate whether the benefit comes from the structured latent geometry or from the boundary-distance heuristic itself.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces Structuring AutoEncoders (SAE), which augment a standard autoencoder with a structural loss that encourages a user-defined class structure in the latent space. The desired structure is specified by a distance matrix between classes, and at each training iteration the encoder's current latent positions are used to initialize Multidimensional Scaling (MDS), producing target positions Z*. An "orthogonal alignment" step computes a rotation R to align Z* with the current latent matrix Z, and the structural loss penalizes the squared distance between encoded points and the aligned targets R Z*. The authors report experiments on MNIST, Fashion-MNIST, DeepFashion2, and a 3D body-shape dataset, showing improved classification with few labels, more reliable decision confidence, effective guided labeling, and class morphing.

Significance. If the proposed method works as claimed, it offers a simple and general mechanism for injecting weak supervision into autoencoders, with practical applications in semi-supervised classification, active labeling, and latent-space interpretability. The paper covers a broad range of datasets and architectures, and it explicitly analyzes the trade-off between reconstruction error and structure via the balancing parameter gamma. The guided-labeling idea is appealing and the morphing demonstration is visually compelling. However, the central Procrustes alignment derivation contains a mathematical error that, as written, invalidates the claimed guarantee that the structural loss enforces the desired class distances. The experimental evaluation also lacks error bars and does not fully control for the use of unlabeled data in the semi-supervised setting. These issues prevent the paper from being accepted in its current form, but they are addressable in a revision.

major comments (2)
  1. [Sec. 2.3] The derivation of the rotation R is mathematically incorrect. The paper assumes that the Moore-Penrose inverse Z+ of Z* satisfies Z*Z+ = I whenever there are more data points than latent dimensions. For an n x d row-stacked matrix with n>d, the meaningful identity would be Z+Z* = I, and it requires Z* to have full column rank d. In all experiments, Z* is obtained by MDS from class-level distances over k=2 or k=3 classes, so rank(Z*) <= k-1 (1 or 2), while the latent dimension d is 10, 30, 64, or 192. Hence Z*Z+ != I and P* = ZZ+ is singular. Setting nonzero singular values to 1 then produces a matrix R that is a partial isometry rather than an orthogonal rotation, and it is not the standard Procrustes solution. Consequently, tilde{Z} = R Z* need not preserve the distances in D, and the structural loss in Eq. (2) does not provably enforce the intended class structure. This is a load-bearing issue because the structural loss is the core mechanism of the SAE. Please correct the derivation, state the exact pseudo-inverse/Procrustes computation used in the implementation, and verify that the resulting tilde{Z} preserves the target distances in all reported settings.
  2. [Sec. 3.3] The classification curves in Fig. 6 are reported as single runs without error bars or repeated-run statistics. Given that the claimed advantage over baselines is especially pronounced at low labeled-sample counts, the results may be sensitive to random initialization and data subsampling. Please report mean and standard deviation over at least five independent runs with different seeds and subsamples. In addition, clarify whether the "comparable neural network" baseline is trained only on the labeled subset while the SAE additionally uses all unlabeled data through its reconstruction loss; if so, the comparison is semi-supervised versus supervised and should be described as such, and the AAE baseline should be clearly distinguished.
minor comments (5)
  1. [Algorithm 1] Algorithm 1 says "set all singular values >= 0 to 1", which would include zero singular values; the text in Sec. 2.3 correctly says "setting all nonzero singular values to 1". Please align the algorithm listing with the text.
  2. [Sec. 3.1] The paper describes 6000 labeled samples as "a very sparse set of data", but 6000 out of 60000 is 10%. Later experiments use 600 samples. Please use consistent and precise terminology for the amount of supervision.
  3. [Fig. 4] The 2D projections in Fig. 4 would benefit from axis labels and a legend; the current figure is hard to read in print.
  4. [Sec. 3.6] The optimal gamma for the 3D HumanPose dataset is reported as 0.0041, but Fig. 11 does not clearly show the scale of the curves. A log-scale inset or a table would make the behavior more transparent.
  5. [General] No code or implementation details for the MDS and Procrustes steps are provided; making the code available would substantially improve reproducibility, especially given the mathematical issue raised above.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the structural target is user-defined and classification is evaluated on held-out test data.

full rationale

The derivation chain is self-contained. In Sec. 2.2 the target structure is defined by a user-supplied distance matrix D, not derived from the encoder's latent variables. MDS in Sec. 2.3 computes Z* from D, and the orthogonal alignment (Eqs. 5-8) only rotates Z* to align with the current latent coordinates; it does not fit the distance structure to the data. The structural loss (Eq. 2) then pulls encoder outputs toward tilde-Z = R Z*, whose pairwise distances remain those of D. No quantity fitted to the training data is later reported as a prediction: the classification experiments in Sec. 3.3 train a linear SVM on the latent variables and evaluate on held-out test data, so the reported accuracy is not forced by construction. The hyperparameter gamma is selected by examining error curves (Sec. 3.6), which is normal model selection rather than a self-definitional reduction. The paper's self-citations (references 1, 2, 30, 32) appear only in related-work and future-work remarks and are not load-bearing for the central claims. The rank-deficiency concern about the Moore-Penrose inverse in Sec. 2.3 is a mathematical correctness issue, not circularity: even if the alignment is not a true rotation, the target still originates from D and the empirical claims are evaluated on held-out data. Therefore no circular step is present.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central mechanism depends on the user-defined distance matrix D and the balancing weight gamma, neither of which is derived from first principles. The method introduces no new physical or mathematical entities; the 'structure' is an external input.

free parameters (3)
  • gamma = 0.5 (MNIST), 0.75 (Fashion-MNIST), 0.0041 (3D HumanPose)
    Balancing parameter between structural and reconstruction loss, chosen per dataset based on validation classification error (Sec. 3.6, Fig. 11).
  • latent_dim = 10 (MNIST), 64 (Fashion-MNIST), 30 (3D HumanPose), 192 (DeepFashion2)
    Latent space dimensionality chosen by hand for each dataset; affects the capacity of the structure to be expressed.
  • class_distance_scale = 1.0
    The inter-class distance in matrix D is set to 1; the absolute scale affects the weighting of the structural loss and is not derived from data.
assumptions (4)
  • domain assumption Data can be partitioned into classes that are not obvious in the raw data.
    The motivation for SAE depends on this; if classes were already separated, a standard autoencoder would discover them.
  • domain assumption There exists a latent space in which class distances can be preserved while still allowing reconstruction of the input.
    The method assumes the encoder can simultaneously satisfy the structural loss and the reconstruction loss; if these conflict, the training target is unattainable.
  • standard math MDS (Shepard-Kruskal) produces a configuration that preserves the pairwise class-distance matrix D.
    The algorithm relies on MDS to compute target positions from D, a standard result from multivariate statistics.
  • standard math Procrustes analysis via SVD gives the optimal rotation aligning Z* to Z.
    Orthogonal Procrustes is a standard technique, but the paper's claim about the Moore-Penrose inverse (Z*Z+ = I) is only valid under full column rank, which is not guaranteed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Structuring Autoencoders." pith.science (2026). https://pith.science/paper/UQGDCO4I

@misc{pith2026190802626,
  author       = {Pith},
  title        = {Pith review of: Structuring Autoencoders},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UQGDCO4I}},
  note         = {Machine review of arXiv:1908.02626}
}
read the original abstract

In this paper we propose Structuring AutoEncoders (SAE). SAEs are neural networks which learn a low dimensional representation of data which are additionally enriched with a desired structure in this low dimensional space. While traditional Autoencoders have proven to structure data naturally they fail to discover semantic structure that is hard to recognize in the raw data. The SAE solves the problem by enhancing a traditional Autoencoder using weak supervision to form a structured latent space. In the experiments we demonstrate, that the structured latent space allows for a much more efficient data representation for further tasks such as classification for sparsely labeled data, an efficient choice of data to label, and morphing between classes. To demonstrate the general applicability of our method, we show experiments on the benchmark image datasets MNIST, Fashion-MNIST, DeepFashion2 and on a dataset of 3D human shapes.

Figures

Figures reproduced from arXiv: 1908.02626 by the authors.

Figure 1
Figure 1. Latent spaces of the autoencoders for the 3D HumanPose [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our Structuring AutoEncoder (SAE) projects data into a [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Visualization of the iteration steps. With each iteration the two classes are separated better in the latent space. The images show [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Comparison of two projections of the latent space using [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 4
Figure 4. Figure 4: The scatterplots show 2D projections of the latent space [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 6
Figure 6. Figure 6: Test error for different sizes of the training set without using data augmentation. The SAE outperforms a comparable traditional [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]
Figure 9
Figure 9. Figure 9: Histogram of prediction scores when using a standard [PITH_FULL_IMAGE:figures/full_fig_p006_9.png]
Figure 10
Figure 10. Figure 10: Comparison of the guided and unguided sampling ap [PITH_FULL_IMAGE:figures/full_fig_p006_10.png]
Figure 11
Figure 11. Figure 11: Influence of the balancing parameter γ on the autoencoder error and the classification error. For MNIST and Fashion-MNIST only 6000 labeled training samples (10% of the data) were used. The training set of 3D body shape dataset consists of 1000 body shapes. tropy loss…
Figure 14
Figure 14. Figure 14: Visualization of the body shape morphing in the latent [PITH_FULL_IMAGE:figures/full_fig_p008_14.png]
Figure 13
Figure 13. Figure 13: Reconstructions (green) obtained by the SAE of the 3D [PITH_FULL_IMAGE:figures/full_fig_p008_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 27 canonical work pages

  1. [1]

    Awiszus, H

    M. Awiszus, H. Ackermann, and B. Rosenhahn. Learning disentangled representations via independent subspaces. In Third International Workshop on ”Robust Subspace Learn- ing and Applications in Computer Vision”, 2019

  2. [2]

    Awiszus and B

    M. Awiszus and B. Rosenhahn. Markov chain neural net- works. In Computer Vision and Pattern Recognition Work- shops (CVPRW), June 2018

  3. [3]

    Bourlard and Y

    H. Bourlard and Y . Kamp. Auto-association by multilayer perceptrons and singular value decomposition. Biological Cybernetics, 59(4):291–294, Sep 1988

  4. [4]

    M. A. Carreira-Perpi ˜n´an and R. Raziperchikolaei. Hash- ing with binary autoencoders. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 557–566, June 2015

  5. [5]

    M. Chen, Z. Xu, K. Weinberger, and F. Sha. Marginalized denoising autoencoders for domain adaptation. In J. Lang- ford and J. Pineau, editors, Proceedings of the 29th Interna- tional Conference on Machine Learning (ICML-12) , ICML ’12, pages 767–774. ACM, New York, NY , USA, July 2012

  6. [6]

    Y . Chen, L. Zhang, and Z. Yi. Subspace clustering using a low-rank constrained autoencoder. Information Sciences, 424:27–38, 2018

  7. [7]

    Donahue, A

    C. Donahue, A. Balsubramani, J. McAuley, and Z. C. Lip- ton. Semantically decomposing the latent spaces of gener- ative adversarial networks. In International Conference on Learning Representations, 2018

  8. [8]

    Y . Ge, R. Zhang, L. Wu, X. Wang, X. Tang, and P. Luo. Deepfashion2: A versatile benchmark for detection, pose es- timation, segmentation and re-identification of clothing im- ages. CoRR, abs/1901.07973, 2019

Show all 33 references
  1. [9]

    Graßhof, H

    S. Graßhof, H. Ackermann, S. S. Brandt, and J. Ostermann. Apathy is the root of all expressions. In 2017 12th IEEE In- ternational Conference on Automatic Face & Gesture Recog- nition (FG 2017), pages 658–665. IEEE, 2017

  2. [10]

    S. Gu, J. Bao, H. Yang, D. Chen, F. Wen, and L. Yuan. Mask- guided portrait editing with conditional gans. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3436–3445, 2019

  3. [11]

    G. E. Hinton and R. R. Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504–507, 2006

  4. [12]

    D. P. Kingma and M. Welling. Auto-encoding variational bayes. CoRR, abs/1312.6114, 2013

  5. [13]

    J. B. Kruskal. Multidimensional scaling by optimizing goodness of fit to a nonmetric hypothesis. Psychometrika, 29(1):1–27, Mar 1964

  6. [14]

    T. D. Kulkarni, W. F. Whitney, P. Kohli, and J. Tenenbaum. Deep convolutional inverse graphics network. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2539–2547. Curran Associates, Inc., 2015

  7. [15]

    Y . LeCun. Generalization and network design strategies. In R. Pfeifer, Z. Schreter, F. Fogelman, and L. Steels, editors, Connectionism in Perspective , Zurich, Switzerland, 1989. Elsevier. an extended version was published as a technical report of the University of Toronto

  8. [16]

    LeCun and C

    Y . LeCun and C. Cortes. MNIST handwritten digit database. 2010

  9. [17]

    F. Li, H. Qiao, and B. Zhang. Discriminatively boosted im- age clustering with fully convolutional auto-encoders. Pat- tern Recognition, 83:161 – 173, 2018

  10. [18]

    H. Liu, M. Shao, S. Li, and Y . Fu. Infinite ensemble for image clustering. In Proceedings of the 22Nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, pages 1745–1754, New York, NY , USA,

  11. [19]

    Loper, N

    M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) , 34(6):248:1– 248:16, Oct. 2015

  12. [20]

    Makhzani, J

    A. Makhzani, J. Shlens, N. Jaitly, and I. Goodfellow. Adver- sarial autoencoders. In International Conference on Learn- ing Representations, 2016

  13. [21]

    McInnes and J

    L. McInnes and J. Healy. UMAP: Uniform Manifold Ap- proximation and Projection for Dimension Reduction. ArXiv e-prints, Feb. 2018

  14. [22]

    Roberts, J

    A. Roberts, J. Engel, C. Raffel, C. Hawthorne, and D. Eck. A hierarchical latent vector model for learning long-term struc- ture in music. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machin...

  15. [23]

    P. H. Sch ¨onemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1):1–10, Mar 1966

  16. [24]

    Simonyan and A

    K. Simonyan and A. Zisserman. Very deep convolu- tional networks for large-scale image recognition. CoRR, abs/1409.1556, 2014

  17. [25]

    C. Song, F. Liu, Y . Huang, L. Wang, and T. Tan. Auto- encoder based data clustering. In J. Ruiz-Shulcloper and G. Sanniti di Baja, editors, Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications , pages 117–124, Berlin, Heidelberg, 2013. Springer B...

  18. [26]

    van der Maaten

    L. van der Maaten. Accelerating t-sne using tree-based al- gorithms. Journal of Machine Learning Research, 15:3221– 3245, 2014

  19. [27]

    van der Maaten and G

    L. van der Maaten and G. Hinton. Visualizing high- dimensional data using t-sne. Journal of Machine Learning Research, 9:2579–2605, 2008

  20. [28]

    V . N. Vapnik and A. Y . Chervonenkis. Theory of Pattern Recognition. Nauka, USSR, 1974

  21. [29]

    Vincent, H

    P. Vincent, H. Larochelle, I. Lajoie, Y . Bengio, and P.-A. Manzagol. Stacked denoising autoencoders: Learning use- ful representations in a deep network with a local denoising criterion. J. Mach. Learn. Res., 11:3371–3408, Dec. 2010

  22. [30]

    Wandt and B

    B. Wandt and B. Rosenhahn. Repnet: Weakly supervised training of an adversarial reprojection network for 3d human pose estimation. In The IEEE Conference on Computer Vi- sion and Pattern Recognition (CVPR), June 2019

  23. [31]

    H. Xiao, K. Rasul, and R. V ollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning al- gorithms. arXiv preprint arXiv:1708.07747, 2017

  24. [32]

    M. Y . Yang, W. Liao, Y . Cao, and B. Rosenhahn. Video event recognition and anomaly detection by combining gaussian process and hierarchical dirichlet process models. In Pho- togrammetric Engineering & Remote Sensing, 2018

  25. [33]

    F. W. Young. Multidimensional Scaling: History, Theory, and Applications. Lawrence, Erlbaum Associates, Publish- ers (Hillsdale, New Jersey; London), 1987

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.