Pith. sign in

REVIEW 2 major objections 2 minor 29 references

The role of class encoding in neural collapse

T0 review · 2 major / 2 minor · reviewed 2026-06-28 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For one-hot labels and balanced data, increasing bias regularization on the final classifier shifts uncentered class mean features from a simplex equiangular tight frame to an orthogonal frame.

desk verdict The paper derives a bias-regularization shift from ETF to orthogonal class means in the UFM under MSE and one-hot labels, but only with exact balance. read the letter →

arxiv 2606.00344 v1 pith:M4SVQRLH submitted 2026-05-29 cs.LG

Bastien Massion , Roy Makhlouf , Estelle Massart This is my paper · ORCID
classification cs.LG
keywords neuralcollapselabelencodingone-hotunrestrictedfeaturemodelmeansquarederrorbiasregularizationsimplexequiangulartightframeorthogonal
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper examines how the choice of label encoding shapes the geometry of last-layer features under neural collapse. It relies on the unrestricted feature model together with mean-squared-error loss to isolate the effect of encoding. For one-hot labels and perfectly balanced classes, the mean feature vectors per class move from a simplex equiangular tight frame toward an orthogonal frame as the regularization strength on the classifier bias is raised. The bias term itself is shown to center the encoded labels, offsetting any offset between the label mean and the origin. The analysis extends to arbitrary encodings and notes that encoding choices influence additional collapse properties.

What carries the argument

Unrestricted feature model (last-layer activations treated as free variables) under mean-squared-error loss, with bias regularization as the control parameter that drives the frame transition.

What would settle it

Train a network on balanced one-hot data while sweeping the bias-regularization coefficient and check whether the empirical class-mean vectors switch from simplex equiangular tight frame geometry to orthogonal geometry.

Watch

Extended reading notes

Core claim

Under the unrestricted feature model with mean-squared-error loss, one-hot encoded labels and balanced data produce uncentered class-mean features that transition from a simplex equiangular tight frame to an orthogonal frame when the bias-regularization coefficient of the final classifier is increased; for arbitrary encodings the bias acts to center the labels relative to the origin.

Load-bearing premise

The last-layer features behave exactly as free variables under mean-squared-error loss and the data classes are perfectly balanced.

Editorial extensions

If this is right

  • The bias of the final classifier compensates for any discrepancy between the global mean of the encoded labels and the coordinate origin.
  • The particular geometry that emerges in neural collapse depends on the chosen label encoding.
  • Other neural-collapse properties besides the mean-feature frame are also modulated by the encoding scheme.
  • The orthogonal-frame limit recovered at high bias regularization matches the geometry of the one-hot labels themselves.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Practitioners could deliberately tune bias regularization to enforce a desired target geometry in the last-layer means.
  • The centering role of the bias suggests that label encodings with non-zero mean may require explicit mean-subtraction or adjusted regularization to reach the same collapse regime.
  • If real networks deviate from the unrestricted-feature idealization, the predicted transition may appear only in simplified linear or two-layer models.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The manuscript claims that within the unrestricted feature model (UFM) trained under mean-squared-error loss, one-hot encoded labels on perfectly balanced data produce a transition in the uncentered class-mean features from a simplex equiangular tight frame to an orthogonal frame as the bias-regularization coefficient λ increases; for arbitrary label encodings the optimal bias centers the labels to compensate for the offset between the label global mean and the origin; the work further discusses implications for other neural-collapse properties.

Significance. If the derivations are rigorous, the result supplies an explicit, closed-form account of how label encoding and bias regularization shape last-layer geometry inside the UFM, thereby clarifying why orthogonal-frame structures appear under strong regularization and extending prior NC analyses that typically fix λ = 0. The parameter-free limiting characterizations (λ → 0 and λ → ∞) constitute a genuine analytical contribution.

major comments (2)
  1. [derivation of the transition (around the bias-regularization analysis)] The central transition claim is obtained from the stationarity condition that equates the weighted sum of features to the label means; this identity holds only under exact class balance (N/K samples per class). The manuscript states the balanced-data assumption but does not display the explicit stationarity equation or show how the orthogonal fixed point ceases to be critical once the global mean shifts under imbalance.
  2. [main results and abstract] No numerical verification of the claimed ETF-to-orthogonal transition is provided inside the UFM, nor is the full step-by-step derivation of the stationarity conditions reproduced. Without either, it is impossible to confirm the absence of hidden assumptions or post-hoc choices in the optimization.
minor comments (2)
  1. Notation for the bias-regularization coefficient λ and the uncentered means ar h_k should be introduced with a single consolidated table or equation block rather than scattered across paragraphs.
  2. The discussion of arbitrary encodings would benefit from an explicit statement of the centering condition (bias = –global label mean) as a separate proposition or corollary.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful review and constructive suggestions. We address each major comment below and commit to revisions that strengthen the manuscript's clarity and rigor.

read point-by-point responses

  1. Referee: The central transition claim is obtained from the stationarity condition that equates the weighted sum of features to the label means; this identity holds only under exact class balance (N/K samples per class). The manuscript states the balanced-data assumption but does not display the explicit stationarity equation or show how the orthogonal fixed point ceases to be critical once the global mean shifts under imbalance.

    Authors: We agree that the stationarity condition is derived under the exact class-balance assumption (N/K samples per class), which is stated but not expanded upon. In the revision we will insert the explicit stationarity equation obtained from the UFM Lagrangian, and we will add a short analysis showing that the orthogonal fixed point is no longer critical when the global mean is shifted by imbalance; the transition result will be explicitly qualified to the balanced case. revision: yes

  2. Referee: No numerical verification of the claimed ETF-to-orthogonal transition is provided inside the UFM, nor is the full step-by-step derivation of the stationarity conditions reproduced. Without either, it is impossible to confirm the absence of hidden assumptions or post-hoc choices in the optimization.

    Authors: We acknowledge the absence of both numerical verification inside the UFM and the complete derivation. The revised manuscript will include (i) a dedicated numerical experiment that sweeps λ inside the UFM and plots the transition of the uncentered class means from simplex ETF to orthogonal frame, and (ii) the full step-by-step derivation of the stationarity conditions placed either in the main text or as a self-contained appendix so that every algebraic step and assumption is transparent. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; central transition derived directly from stationarity conditions of UFM+MSE objective under explicit balance assumption.

full rationale

The paper performs a closed-form analysis of the joint minimization of the unrestricted feature model objective (features as free variables, linear classifier) under MSE loss plus bias regularization term. The ETF-to-orthogonal transition follows from the stationarity equations when class counts are exactly equal; the orthogonal fixed point is shown to satisfy the KKT conditions only under that balance. This is a mathematical consequence of the stated model, not a redefinition, a fitted parameter renamed as prediction, or a self-citation chain. No load-bearing step reduces to its own input by construction. The result is falsifiable inside the model and does not rely on external data or prior self-authored uniqueness theorems.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

The work rests on the unrestricted feature model and MSE loss; these are modeling choices rather than new axioms, but they are not independently verified in the provided abstract.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The role of class encoding in neural collapse." pith.science (2026). https://pith.science/paper/M4SVQRLH

@misc{pith2026260600344,
  author       = {Pith},
  title        = {Pith review of: The role of class encoding in neural collapse},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M4SVQRLH}},
  note         = {Machine review of arXiv:2606.00344}
}
read the original abstract

Neural collapse is a structural property of the last-hidden-layer activations in neural network classification models, when trained beyond a zero classification error. In this work, we explore the role of label encoding in neural collapse by relying on the unrestricted feature model with mean squared error training loss. We demonstrate that, for one-hot encoded labels and balanced data, the uncentered mean features associated with each class transition from a simplex equiangular tight frame to an orthogonal frame when increasing the bias regularization coefficient associated with the final classifier. These structures are reminiscent of the orthogonal frame structure of one-hot encoded labels. For any arbitrary encoding, we also show that the final classifier's bias aims at centering the labels, compensating for the discrepancy between the global mean of the labels and the origin. We further discuss the role of the encoding in other neural collapse properties.

Figures

Figures reproduced from arXiv: 2606.00344 by the authors.

Figure 1
Figure 1. Centering an OF (blue) always results in a simplex ETF (green). For [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 2 canonical work pages

  1. [1]

    Andriopoulos, Z

    G. Andriopoulos, Z. Dong, L. Guo, Z. Zhao, and K. Ross. The prevalence of neural collapse in neural multivariate regression. In Proceedings of the 38th Conference on Neural Information Processing Systems, 2024

    2024

  2. [2]

    Demirkaya, J

    A. Demirkaya, J. Chen, and S. Oymak. Exploring the role of loss functions in multiclass classification. InProceedings of the 54th Annual Conference on Information Sciences and Systems, 2020

    2020

  3. [3]

    W. E and S. Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Proceedings of 2nd Annual Conference on Mathematical and Scientific Machine Learning, 2021

    2021

  4. [4]

    C. Fang, H. He, Q. Long, and W. J. Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training.Proceedings of the National Academy of Sciences, 118(43), 2021

    2021

  5. [5]

    Galanti, A

    T. Galanti, A. Gy ¨orgy, and M. Hutter. On the role of neural collapse in transfer learning. InProceedings of the 10th International Conference on Learning Representations, 2022

    2022

  6. [6]

    F. Graf, C. D. Hofer, M. Niethammer, and R. Kwitt. Dissecting su- pervised constrastive learning. InProceedings of the 38th International Conference on Machine Learning, 2021

    2021

  7. [7]

    L. Guo, G. Andriopoulos, Z. Zhao, S. Ling, Z. Dong, and K. Ross. Cross entropy versus label smoothing: A neural collapse perspective. arXiv:2402.03979, 2025

    work page arXiv 2025

  8. [8]

    J. Haas, W. Yolland, and B. T. Rabus. Linking neural collapse and l2 normalization with improved out-of-distribution detection in deep neural networks.Transactions on Machine Learning Research, 2022

    2022

Show all 29 references
  1. [9]

    X. Y . Han, V . Papyan, and D. L. Donoho. Neural collapse under MSE loss: proximity to and dynamics on the central path. InProceedings of the 10th International Conference on Learning Representations, 2022

    2022

  2. [10]

    Hui and M

    L. Hui and M. Belkin. Evaluation of neural architectures trained with square loss vs cross-entropy in classification tasks.Proceedings of the 9th International Conference on Learning Representations, 2021

    2021

  3. [11]

    Jiang, J

    J. Jiang, J. Zhou, P. Wang, Q. Qu, D. G. Mixon, C. You, and Z. Zhu. Generalized neural collapse for a large number of classes. InProceed- ings of the 41st International Conference on Machine Learning, 2024

    2024

  4. [12]

    Kothapalli

    V . Kothapalli. Neural collapse: A review on modelling principles and generalization.Transactions on Machine Learning Research, 2023

    2023

  5. [13]

    H. Liu. The exploration of neural collapse under imbalanced data. arXiv:2411.17278, 2024

    2024

  6. [14]

    X. Liu, J. Zhang, T. Hu, H. Cao, L. Pan, and Y . Yao. Inducing neural collapse in deep long-tailed learning. InProceedings of the 26th International Conference on Artificial Intelligence and Statistics, 2023

    2023

  7. [15]

    Lu and S

    J. Lu and S. Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59:224–241, 2022

    2022

  8. [16]

    D. G. Mixon, H. Parshall, and J. Pi. Neural collapse with unconstrained features.Sampling Theory, Signal Processing, and Data Analysis, 20(11), 2020

    2020

  9. [17]

    Papyan, X

    V . Papyan, X. Y . Han, and D. L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training.Proceedings of the National Academy of Sciences, 117(40):24652–24663, 2020

    2020

  10. [18]

    Rangamani, M

    A. Rangamani, M. Lindegaard, T. Galanti, and T. Poggio. Feature learning in deep classifiers through intermediate neural collapse. In Proceedings of the 40th International Conference on Machine Learning, 2023

    2023

  11. [19]

    Sukenik, C

    P. Sukenik, C. H. Lampert, and M. Mondelli. Neural collapse versus low-rank bias: Is deep neural collapse really optimal? InProceedings of the 38th Conference on Neural Information Processing Systems, 2024

    2024

  12. [20]

    Sukenik, M

    P. Sukenik, M. Mondelli, and C. H. Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. In Proceedings of the 37th Conference on Neural Information Processing Systems, 2023

    2023

  13. [21]

    Szegedy, V

    C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the Inception architecture for computer vision. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016

    2016

  14. [22]

    Thrampoulidis, G

    C. Thrampoulidis, G. R. Kini, V . Vakilian, and T. Behnia. Imbalance trouble: Revisiting neural-collapse geometry. InProceedings of the 36th Conference on Neural Information Processing Systems, 2022

    2022

  15. [23]

    Tirer and J

    T. Tirer and J. Bruna. Extended unconstrained features model for exploring deep neural collapse. InProceedings of the 39th International Conference on Machine Learning, 2022

    2022

  16. [24]

    Tirer, H

    T. Tirer, H. Huang, and J. Niles-Weed. Perturbation analysis of neural collapse. InProceedings of the 40th International Conference on Machine Learning, 2023

    2023

  17. [25]

    Wu and V

    R. Wu and V . Papyan. Linguistic collapse: Neural collapse in (large) language models. InProceedings of the 38th Conference on Neural Information Processing Systems, 2024

    2024

  18. [26]

    Yaras, P

    C. Yaras, P. Wang, Z. Zhu, L. Balzano, and Q. Qu. Neural collapse with normalized features: A geometric analysis over the Riemannian manifold. InProceedings of the 36th Conference on Neural Information Processing Systems, 2022

    2022

  19. [27]

    J. Zhou, X. Li, T. Ding, C. You, Q. Qu, and Z. Zhu. On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features. InProceedings of the 39th International Conference on Machine Learning, 2022

    2022

  20. [28]

    J. Zhou, C. You, X. Li, K. Liu, S. Liu, Q. Qu, and Z. Zhu. Are all losses created equal: A neural collapse perspective. InProceedings of the 36th Conference on Neural Information Processing Systems, 2022

    2022

  21. [29]

    Z. Zhu, T. Ding, J. Zhou, X. Li, C. You, J. Sulam, and Q. Qu. A geometric analysis of neural collapse with unconstrained features. In Proceedings of the 35th Conference on Neural Information Processing Systems, 2021

    2021

Pith tools

Reviewed June 28, 2026 · model on record in the stance chip above.