Pith. sign in

REVIEW 3 major objections 4 minor 76 references

Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Deep ResNets and transformers provably collapse at global optima.

desk verdict Strong idea, shaky proof: the depth-splitting reduction to a GUFM is novel and worth engaging, but the main theorem relies on an asserted construction whose ReLU step does not check out, so the paper needs major repair before it can be trusted. read the letter →

arxiv 2505.15239 v1 pith:CYEJQUMA submitted 2025-05-21 cs.LG cs.AIstat.ML

Peter Súkeník , Christoph H. Lampert , Marco Mondelli This is my paper · ORCID
classification cs.LGcs.AIstat.ML MSC 68T07
keywords neuralcollapseunconstrainedfeaturesmodelResNettransformerLayerNormglobaloptimalityweightregularizationcross-entropyloss
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Neural collapse is the empirical tendency of a trained classifier's last-layer features to shrink within each class and for the class means to form a maximally separated simplex. This paper proves that, for deep LayerNorm residual networks and transformers with weight regularization, every global optimum of the end-to-end training problem is approximately collapsed, and the approximation becomes exact as the number of blocks grows. The proof covers both cross-entropy and mean-squared-error losses, and architectures with one or two linear layers per block, provided that in the two-layer case the regularization strength decays with depth. A direct consequence is that the end-to-end objective of a deep-enough ResNet or transformer is equivalent to a generalized unconstrained features model in which the penultimate features are free variables, which justifies using that simpler model to analyze modern architectures.

What carries the argument

The load-bearing mechanism is a split-the-map-across-depth construction. In a one-dimensional model, the map $x \mapsto \exp(a)x$ can be realized by $L$ residual layers each scaling by $1+a/L$, so the sum of squared Frobenius norms of all weight matrices is $a^2/L$ and vanishes as $L$ grows. The paper generalizes this by assigning each training sample a smooth curve on the zero-sum sphere of radius $\sqrt{d}$ from its initial feature to its GUFM-optimal target, and building rank-one ReLU layers whose pre-activations are non-negative for the moving sample and negative for all others, so that LayerNorm takes exactly the intended step along the curve while all other samples stay put. Margins between the curves keep the constructions disjoint. For transformers, the first block encodes each context's history in binary so distinct contexts remain distinct even when attention weights converge to uniform, after which the ResNet construction applies directly. The GUFM enforces the zero-mean, fixed-norm, and identical-sample constraints that LayerNorm and the data impose, and Lemma 4.2 shows its CE/MSE optima are fully collapsed.

What would settle it

With dimension $d=4$, two classes, and one sample per class, take an initial feature pair and a GUFM-optimal target pair and search over all smooth curves on the zero-sum sphere for one satisfying the five margin conditions in the proof of Theorem 4.3; an explicit configuration where no such curve exists would break the constructive reduction as written, so the proof would need a different argument.

Watch

Extended reading notes

Core claim

The paper's central claim, Theorem 4.3 with Corollary 4.5 for one-linear-layer blocks and Theorem 4.6 with Corollary 4.7 for two-linear-layer blocks, is that global optima of regularized LayerNorm ResNets and transformers approach the global optima of the corresponding generalized unconstrained features model (GUFM) as the depth $L$ grows, at a rate $\widetilde{O}(L^{-1/2})$ for cross-entropy and mean-squared-error losses, provided the GUFM's optimal loss is positive. Under the assumption that training samples are unique, or that transformer labels are determined by context, the GUFM optima for these two losses are perfectly collapsed: within-class variability vanishes, class means form a maximally separated frame, and features align with the last-layer weight rows. Therefore every global optimum of the deep end-to-end problem is approximately collapsed, with the deviation from perfect collapse bounded by a function of $L$ that tends to zero. The paper also shows that with two linear layers per block and constant regularization this collapse is generically not approached, because feature changes made by two-layer residual blocks scale linearly with the total weight cost. These results give the first end-to-end global-optimality characterization of neural collapse for architectures beyond multilayer perceptrons.

Load-bearing premise

The proof needs the existence of smooth trajectories on the allowed feature sphere that carry each sample to its target one at a time while keeping all other samples outside a fixed margin, and it needs the rank-one ReLU layers used to track them to have non-negative pre-activations on the moving sample so that LayerNorm produces the intended step; the paper states this existence is clear rather than proving it, and the reduction collapses if it fails.

Editorial extensions

If this is right

  • For L-RN1 and L-Tx1 architectures with cross-entropy or mean-squared-error loss, every global optimum is approximately collapsed, and the distance to perfect collapse is upper-bounded by a function of depth that goes to zero.
  • For L-RN2 and L-Tx2 architectures, the same guarantee holds when the regularization on all layers except the last decays faster than $1/\log L$.
  • The end-to-end training objective of deep LayerNorm ResNets and transformers reduces to the corresponding GUFM, so global optima of the GUFM are provably approached by global optima of the full network.
  • For CE and MSE losses the deviation from neural collapse shrinks roughly as $\widetilde{O}(L^{-1/2})$.
  • With two linear layers per block and non-vanishing or uniformly vanishing regularization, neural collapse is generically not reached, so the one-layer-per-block and vanishing-regularization regimes behave qualitatively differently.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper proves a statement about global optima, not about the path gradient descent takes; showing that trained networks actually reach these optima would require a separate argument, and the experiments here only suggest the trend.
  • Because Theorem 4.3 holds for any continuous loss, the same reduction should transfer to other losses whose GUFM optima are known, such as regression or class-imbalanced variants; solving those GUFMs would immediately characterize the end-to-end optima of deep ResNets and transformers.
  • The construction implies that residual depth itself creates a cost-free way to implement feature maps, so increasing depth should push optima toward collapse independently of optimization details; this is a testable prediction beyond the paper's own experiments.
  • The requirement that samples be unique may be relaxable in the ResNet case by a small perturbation of the input data, which would extend collapse guarantees to duplicate training samples that the current Assumption 4.4 excludes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper develops a reduction from deep regularized LayerNorm ResNets and transformers to a generalized unconstrained features model (GUFM). It proves that, under certain architectural and data assumptions, the global optima of the end-to-end training objective converge, in the sense of the Hausdorff-type distance of Theorems 4.3 and 4.6, to the corresponding GUFM optima as the depth grows, and that the approximation tightens with depth. Combined with a GUFM analysis for cross-entropy and mean-squared-error losses, it concludes that neural collapse is globally optimal for these architectures. The authors support the theory with experiments on MNIST, CIFAR-10, and IMDB showing that the NC metrics improve with depth.

Significance. If the proofs are completed, the contribution is substantial: it is the first end-to-end global optimality statement for neural collapse in transformers and a significant extension of the corresponding ResNet results, and it provides a formal justification for the unconstrained-features model in modern architectures. The paper does not merely cite a known UFM result; it constructs explicit networks whose loss approaches the GUFM optimum, and the experiments test the falsifiable prediction that deeper networks collapse more, rather than fitting constants to reproduce a known conclusion. The high-level reduction strategy is elegant and, if the Appendix-A construction is repaired, would be an important step for the field. The main obstacle is the technical completeness of the constructive proof, on which the central theorems rest.

major comments (3)
  1. [Appendix A, proof of Theorem 4.3, penultimate-block construction] The construction of the penultimate blocks defines W_l^j = α_l^j (1 + cm \bar h_j)/\|1 + cm \bar h_j\| \bar h_j^T/√d and b_l^j = -(1-2cm) α_l^j √d /\|1 + cm \bar h_j\| 1, and then claims that after ReLU and LayerNorm the centered update is a positive multiple of \bar h_j. This claim assumes that the pre-activation W_l^j x + b_l^j is non-negative in every coordinate. Since GUFM-optimal features are zero-mean, \bar h_j has negative coordinates, and the paper gives no bound on their magnitude. For a zero-mean vector of norm √d, a negative coordinate can have magnitude up to √(d-1); in that case the term cm\bar h_j^T x \bar h_j on that coordinate is about -cm d√(d-1), which dominates the positive all-ones offset 2cmd for d > 5, so the ReLU would clip exactly the coordinate the proof needs. The displayed formula after the definition of W_l^j and b_l^j is therefore not a proof of the centered update; the trajectory along \bar h_j may break, and the constructed loss need not approach L*_GUFM. Please either prove the non-negativity for a deliberately chosen optimal H (for example, an ETF with bounded negative coordinates) or modify the construction so that ReLU is guaranteed to be inactive on all relevant coordinates.
  2. [Appendix A, proof of Theorem 4.3, first-phase curve construction] The first phase of the construction requires, for each sample, a smooth oriented curve G_{ki} on the zero-sum √d-sphere satisfying five simultaneous conditions: bounded curvature, a margin against all subsequent samples, a unique crossing point with \bar x_{ki}, a segment-wise margin against previously processed samples, and a length bound. The proof dismisses this with the sentence "It is clear that a construction satisfying these properties exists." This assertion is load-bearing: the per-sample block construction and the bound on the total Frobenius-norm cost depend on the simultaneous validity of all these properties, and the statement is not a trivial consequence of finiteness because of the uniform curvature bound and the simultaneous margin constraints. Please supply a rigorous argument, for example a lemma with an explicit curve family or a compactness/topological argument, before the reduction can be considered established.
  3. [Theorem 4.6 and its proof] The proof of Theorem 4.6 states that it follows the proof of Theorem 4.3 with modified weight scalings. Consequently, the two unproved ingredients identified above, the existence of the curve family with the required margin properties and the coordinatewise non-negativity of the ReLU pre-activations, carry over unchanged to the two-linear-layer architectures. In addition, the two-layer construction inserts a second linear layer that is supposed to act as a projection onto the span of the first layer's output; this requires the intermediate ReLU output to be exactly the non-negative vector from the single-layer construction, so the non-negativity issue is not avoided. The statement of Theorem 4.6 should not be considered proven until the underlying construction is repaired.
minor comments (4)
  1. [Appendix A, proof of Lemma 4.2, MSE part] The proof contains the inclusion "MUFM-MSE_0 ⊂ MUFM-MSE_0", which is a tautology; one of the two sets should refer to the solutions of the relaxed problem (13) and the other to the solutions of the constrained GUFM problem (8). Please correct the notation.
  2. [Appendix A, proof of Theorem 4.3, penultimate-block formula] The displayed formula for σ(W_l^j x + b_l^j) in the penultimate-block paragraph appears to be missing the factor 1/√d that multiplies the outer product in the definition of W_l^j. Please verify the algebra, since the LayerNorm scaling depends on the correct constant.
  3. [Section 4, Assumption 4.4 and introduction] The introduction advertises results "under minimal assumptions on the data", but Assumption 4.4 requires all ResNet training samples to be unique. This is a substantive structural requirement, not a minimal one; please rephrase the introduction or soften the claim.
  4. [Appendix A, proof of Lemma 4.2, CE part] The CE part of the proof invokes Theorem 3.1 of [71] and additionally uses the assertion that the norm of the optimal feature vectors is an increasing function of the regularization strength, in order to match the fixed-norm constraint of the GUFM. Since this monotonicity is used to identify the solution sets, it should be stated explicitly as a lemma with either a proof or a precise pointer to the specific statement in [71].

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the core reduction rests on an external UFM optimality theorem and explicit layer constructions; the proof gaps noted by the reader are correctness concerns, not circular reductions.

full rationale

The derivation chain is not circular in the sense defined here. Lemma 4.2 derives collapse of GUFM optima from an external, parameter-free theorem (Zhu et al. [71]) for CE loss and from a direct lower-bound computation for MSE loss; the GUFM constraints (zero-mean features, coincident-feature equivalence for identical contexts) do not encode collapse, so NC is a consequence of the loss rather than an input. Theorems 4.3 and 4.6 prove the end-to-end-to-GUFM reduction by explicit constructions whose Frobenius costs vanish with depth, not by assuming the target conclusion or by fitting parameters to NC metrics. The paper's self-citations (e.g., [23, 48, 49]) appear in related-work or remark contexts and are not load-bearing; the load-bearing citation for CE optimality is to external prior work. The experiments test the falsifiable depth-to-collapse prediction and are not used to set constants. The proof does contain asserted-but-unproved geometric existence statements ('It is clear that a construction satisfying these properties exists') and an unverified ReLU pre-activation non-negativity condition in the penultimate-block step; these are rigor or correctness gaps, not reductions by construction or by self-citation, so under the hard rules they do not raise the circularity score.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests mainly on the GUFM optimality result (borrowed from [71] for CE) and on the depth-splitting construction whose geometric existence is asserted rather than fully proved. There are no fitted parameters and no invented entities.

assumptions (5)
  • domain assumption LayerNorm outputs lie on the zero-sum sphere of radius sqrt(d) and thus satisfy the GUFM feasibility constraints.
    Used throughout to map any (WL, XL) from an end-to-end network into a feasible GUFM point (Section 3, Definition 3.5).
  • domain assumption The global optima of the regularized CE UFM are collapsed (Theorem 3.1 of Zhu et al. [71]).
    Borrowed in Lemma 4.2 to establish NC for the CE GUFM; not re-proved in this paper.
  • ad hoc to paper For generic initial features X1 and GUFM-optimal H, there exist smooth curves on the zero-sum sphere satisfying the five margin conditions (bounded curvature, separation, single crossing point, etc.).
    Assumed in the proof of Theorem 4.3 (Appendix A, before starting the construction); the paper states 'It is clear that a construction satisfying these properties exists' without a formal argument.
  • ad hoc to paper The ReLU pre-activations in the constructed layers are non-negative for the target sample so that the intended movement is preserved after ReLU.
    Relied on implicitly in the weight construction (Eq. 14 in Appendix A); if ReLU clips negative entries, LayerNorm will not produce the intended next point on the curve.
  • domain assumption L*_GUFM > 0 for the losses considered.
    Required in Theorems 4.3 and 4.6 to apply Lemma 4.1; for CE and MSE with K at least 2 and d at least K this holds because the regularization term is positive.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers." pith.science (2026). https://pith.science/paper/CYEJQUMA

@misc{pith2026250515239,
  author       = {Pith},
  title        = {Pith review of: Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYEJQUMA}},
  note         = {Machine review of arXiv:2505.15239}
}
read the original abstract

The empirical emergence of neural collapse -- a surprising symmetry in the feature representations of the training data in the penultimate layer of deep neural networks -- has spurred a line of theoretical research aimed at its understanding. However, existing work focuses on data-agnostic models or, when data structure is taken into account, it remains limited to multi-layer perceptrons. Our paper fills both these gaps by analyzing modern architectures in a data-aware regime: we prove that global optima of deep regularized transformers and residual networks (ResNets) with LayerNorm trained with cross entropy or mean squared error loss are approximately collapsed, and the approximation gets tighter as the depth grows. More generally, we formally reduce any end-to-end large-depth ResNet or transformer training into an equivalent unconstrained features model, thus justifying its wide use in the literature even beyond data-agnostic settings. Our theoretical results are supported by experiments on computer vision and language datasets showing that, as the depth grows, neural collapse indeed becomes more prominent.

Figures

Figures reproduced from arXiv: 2505.15239 by the authors.

Figure 1
Figure 1. log10 of NC1, NC2 and NC3 metrics respectively in the left, middle and right column, as a function of the number of blocks L. First row: L-RN1 on CIFAR10; second row: L-T11 on CIFAR10; third row: pre-LN L-T11 on IMDB; Fourth row: L-RN2 on MNIST with λ ∝ L −1 . 5 Experimental results Our theoretical results suggest an improvement of the NC metrics at the global optima as the depth increases. To empirically verify whe… view at source ↗
Figure 2
Figure 2. MNIST training. log10 of NC1, NC2 and NC3 metrics respectively in the upper, middle and bottom row, as a function of the number of blocks L. The architectures are L-RN1 with λ = 0.005, L-T11 with λ = 0.005, and L-RN2 with λ = 0.0025. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 61 canonical work pages

  1. [1]

    The prevalence of neural collapse in neural multivariate regression

    George Andriopoulos, Zixuan Dong, Li Guo, Zifan Zhao, and Keith Ross. The prevalence of neural collapse in neural multivariate regression. In Conference on Neural Information Processing Systems (NeurIPS), 2024

    work page 2024

  2. [2]

    Intrinsic dimension of data representa- tions in deep neural networks

    Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan. Intrinsic dimension of data representa- tions in deep neural networks. Conference on Neural Information Processing Systems (NeurIPS), 2019

    work page 2019

  3. [3]

    Average gradient outer product as a mechanism for deep neural collapse

    Daniel Beaglehole, Peter Súkeník, Marco Mondelli, and Mikhail Belkin. Average gradient outer product as a mechanism for deep neural collapse. In Conference on Neural Information Processing Systems (NeurIPS), 2024

    work page 2024

  4. [4]

    On the inductive bias of infinite-depth resnets and the bottleneck rank

    Enric Boix-Adsera. On the inductive bias of infinite-depth resnets and the bottleneck rank. arXiv preprint arXiv:2501.19149, 2025

    arXiv 2025

  5. [5]

    Prevalence of simplex compression in adversarial deep neural networks

    Yang Cao, Yanbo Chen, and Weiwei Liu. Prevalence of simplex compression in adversarial deep neural networks. Proceedings of the National Academy of Sciences, 122(17):e2421593122, 2025

    work page 2025

  6. [6]

    Perfectly balanced: Improving transfer and robustness of supervised contrastive learning

    Mayee Chen, Daniel Y Fu, Avanika Narayan, Michael Zhang, Zhao Song, Kayvon Fatahalian, and Christopher Ré. Perfectly balanced: Improving transfer and robustness of supervised contrastive learning. In International Conference on Machine Learning (ICML), 2022

    work page 2022

  7. [7]

    Neural collapse for cross-entropy class-imbalanced learning with unconstrained relu features model

    Hien Dang, Tho Tran Huu, Tan Minh Nguyen, and Nhat Ho. Neural collapse for cross-entropy class-imbalanced learning with unconstrained relu features model. In International Conference on Machine Learning (ICML), 2024

    work page 2024

  8. [8]

    Neural collapse in deep linear network: From balanced to imbalanced data

    Hien Dang, Tan Nguyen, Tho Tran, Hung Tran, and Nhat Ho. Neural collapse in deep linear network: From balanced to imbalanced data. In International Conference on Machine Learning (ICML), 2023

    work page 2023

Show all 76 references
  1. [9]

    Improving self-supervised learning by characterizing idealized representations

    Yann Dubois, Stefano Ermon, Tatsunori B Hashimoto, and Percy S Liang. Improving self-supervised learning by characterizing idealized representations. In Conference on Neural Information Processing Systems (NeurIPS), 2022. 9

    2022

  2. [10]

    Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training

    Cong Fang, Hangfeng He, Qi Long, and Weijie J Su. Exploring deep neural networks via layer-peeled model: Minority collapse in imbalanced training. In Proceedings of the National Academy of Sciences (PNAS), volume 118, 2021

    2021

  3. [11]

    On the implicit bias towards minimal depth of deep neural networks

    Tomer Galanti, Liane Galanti, and Ido Ben-Shaul. On the implicit bias towards minimal depth of deep neural networks. arXiv preprint arXiv:2202.09028, 2022

    2022 arXiv

  4. [12]

    Improved generalization bounds for transfer learning via neural collapse

    Tomer Galanti, András György, and Marcus Hutter. Improved generalization bounds for transfer learning via neural collapse. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML, 2022

    2022

  5. [13]

    The persistence of neural collapse despite low-rank bias: An analytic perspective through unconstrained features

    Connall Garrod and Jonathan P Keating. The persistence of neural collapse despite low-rank bias: An analytic perspective through unconstrained features. arXiv preprint arXiv:2410.23169, 2024

    2024

  6. [14]

    Unifying low dimensional observations in deep learning through the deep linear unconstrained feature model

    Connall Garrod and Jonathan P Keating. Unifying low dimensional observations in deep learning through the deep linear unconstrained feature model. arXiv preprint arXiv:2404.06106, 2024

    2024 arXiv

  7. [15]

    Linking neural collapse and l2 normalization with improved out-of-distribution detection in deep neural networks

    Jarrod Haas, William Yolland, and Bernhard T Rabus. Linking neural collapse and l2 normalization with improved out-of-distribution detection in deep neural networks. Transactions on Machine Learning Research (TMLR), 2022

    2022

  8. [16]

    X. Y . Han, Vardan Papyan, and David L Donoho. Neural collapse under mse loss: Proximity to and dynamics on the central path. In International Conference on Learning Representations (ICLR), 2022

    2022

  9. [17]

    A law of data separation in deep learning

    Hangfeng He and Weijie J Su. A law of data separation in deep learning. Proceedings of the National Academy of Sciences, 120(36), 2023

    2023

  10. [18]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

    2016

  11. [19]

    Beyond unconstrained features: Neural collapse for shallow neural networks with general data

    Wanli Hong and Shuyang Ling. Beyond unconstrained features: Neural collapse for shallow neural networks with general data. arXiv preprint arXiv:2409.01832, 2024

    2024 arXiv

  12. [20]

    Neural collapse for unconstrained feature model under cross-entropy loss with imbalanced data

    Wanli Hong and Shuyang Ling. Neural collapse for unconstrained feature model under cross-entropy loss with imbalanced data. Journal of Machine Learning Research, 25(192):1–48, 2024

    2024

  13. [21]

    https://www.kaggle.com/datasets/lakshmi25npathi/imdb-dataset-of-50k-movie reviews

  14. [22]

    Limitations of neural collapse for understanding generalization in deep learning

    Like Hui, Mikhail Belkin, and Preetum Nakkiran. Limitations of neural collapse for understanding generalization in deep learning. arXiv preprint arXiv:2202.08384, 2022

    2022 arXiv

  15. [23]

    Wide neural networks trained with weight decay provably exhibit neural collapse

    Arthur Jacot, Peter Súkeník, Zihan Wang, and Marco Mondelli. Wide neural networks trained with weight decay provably exhibit neural collapse. In International Conference on Learning Representations (ICLR), 2025

    2025

  16. [24]

    An unconstrained layer-peeled perspective on neural collapse

    Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie J Su. An unconstrained layer-peeled perspective on neural collapse. In International Conference on Learning Representations (ICLR), 2022

    2022

  17. [25]

    Generalized neural collapse for a large number of classes

    Jiachen Jiang, Jinxin Zhou, Peng Wang, Qing Qu, Dustin G Mixon, Chong You, and Zhihui Zhu. Generalized neural collapse for a large number of classes. In Conference on Parsimony and Learning (Recent Spotlight Track), 2023

    2023

  18. [26]

    How does information bottleneck help deep learning? In International Conference on Machine Learning (ICML), 2023

    Kenji Kawaguchi, Zhun Deng, Xu Ji, and Jiaoyang Huang. How does information bottleneck help deep learning? In International Conference on Machine Learning (ICML), 2023

    2023

  19. [27]

    Kernel vs

    Vignesh Kothapalli and Tom Tirer. Kernel vs. kernel: Exploring how the data structure affects neural collapse. In Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  20. [28]

    A neural collapse perspective on feature evolution in graph neural networks

    Vignesh Kothapalli, Tom Tirer, and Joan Bruna. A neural collapse perspective on feature evolution in graph neural networks. In Conference on Neural Information Processing Systems (NeurIPS), volume 36, 2023

    2023

  21. [29]

    Learning multiple layers of features from tiny images

    Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009

    2009

  22. [30]

    The asymmetric maximum margin bias of quasi-homogeneous neural networks

    Daniel Kunin, Atsushi Yamamura, Chao Ma, and Surya Ganguli. The asymmetric maximum margin bias of quasi-homogeneous neural networks. In International Conference on Learning Representations (ICLR), 2022

    2022

  23. [31]

    Gradient-based learning applied to document recognition

    Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998

    1998

  24. [32]

    Principled and efficient transfer learning of deep models via neural collapse

    Xiao Li, Sheng Liu, Jinxin Zhou, Xinyu Lu, Carlos Fernandez-Granda, Zhihui Zhu, and Qing Qu. Principled and efficient transfer learning of deep models via neural collapse. In Conference on Parsimony and Learning (Recent Spotlight Track), 2023

    2023

  25. [33]

    No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier

    Zexi Li, Xinyi Shang, Rui He, Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 10

    2023

  26. [34]

    Inducing neural collapse to a fixed hierarchy-aware frame for reducing mistake severity

    Tong Liang and Jim Davis. Inducing neural collapse to a fixed hierarchy-aware frame for reducing mistake severity. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  27. [35]

    Spurious feature diversification improves out-of-distribution generalization

    Yong Lin, Lu Tan, Yifan Hao, Honam Wong, Hanze Dong, Weizhong Zhang, Yujiu Yang, and Tong Zhang. Spurious feature diversification improves out-of-distribution generalization. In International Conference on Learning Representations (ICLR), 2024

    2024

  28. [36]

    The exploration of neural collapse under imbalanced data

    Haixia Liu. The exploration of neural collapse under imbalanced data. arXiv preprint arXiv:2411.17278, 2024

    2024 arXiv

  29. [37]

    Gen: Pushing the limits of softmax-based out-of-distribution detection

    Xixi Liu, Yaroslava Lochman, and Christopher Zach. Gen: Pushing the limits of softmax-based out-of-distribution detection. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  30. [38]

    Inducing neural collapse in deep long-tailed learning

    Xuantong Liu, Jianfeng Zhang, Tianyang Hu, He Cao, Yuan Yao, and Lujia Pan. Inducing neural collapse in deep long-tailed learning. In International conference on artificial intelligence and statistics, pages 11534–11544. PMLR, 2023

    2023

  31. [39]

    Neural collapse under cross-entropy loss

    Jianfeng Lu and Stefan Steinerberger. Neural collapse under cross-entropy loss. Applied and Computational Harmonic Analysis, 59, 2022

    2022

  32. [40]

    Do we need neural collapse? Learning diverse features for fine-grained and long-tail classification

    Jiawei Ma, Chong You, Sashank J Reddi, Sadeep Jayasumana, Himanshu Jain, Felix Yu, Shih-Fu Chang, and Sanjiv Kumar. Do we need neural collapse? Learning diverse features for fine-grained and long-tail classification. openreview, 2023

    2023

  33. [41]

    The tunnel effect: Building data representations in deep neural networks

    Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Miło´s, and Tomasz Trzcinski. The tunnel effect: Building data representations in deep neural networks. In Conference on Neural Information Processing Systems (NeurIPS), 2023

    2023

  34. [42]

    Neural collapse with unconstrained features

    Dustin G Mixon, Hans Parshall, and Jianzong Pi. Neural collapse with unconstrained features. Sampling Theory, Signal Processing, and Data Analysis, 20(2):11, 2022

    2022

  35. [43]

    Vardan Papyan, X. Y . Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. In Proceedings of the National Academy of Sciences (PNAS), volume 117, 2020

    2020

  36. [44]

    Neural collapse in deep homogeneous classifiers and the role of weight decay

    Akshay Rangamani and Andrzej Banburski-Fahey. Neural collapse in deep homogeneous classifiers and the role of weight decay. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022

    2022

  37. [45]

    Feature learning in deep classifiers through intermediate neural collapse

    Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A Poggio. Feature learning in deep classifiers through intermediate neural collapse. In International Conference on Machine Learning (ICML), 2023

    2023

  38. [46]

    Neural (tangent kernel) collapse

    Mariia Seleznova, Dana Weitzner, Raja Giryes, Gitta Kutyniok, and Hung-Hsu Chou. Neural (tangent kernel) collapse. In Conference on Neural Information Processing Systems (NeurIPS), volume 36, 2023

    2023

  39. [47]

    On the robustness of neural collapse and the neural collapse of robustness

    Jingtong Su, Ya Shi Zhang, Nikolaos Tsilivis, and Julia Kempe. On the robustness of neural collapse and the neural collapse of robustness. Transactions on Machine Learning Research (TMLR), 2024

    2024

  40. [48]

    Peter Súkeník, Marco Mondelli, and Christoph H. Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. In Conference on Neural Information Processing Systems (NeurIPS), 2023

    2023

  41. [49]

    Peter Súkeník, Marco Mondelli, and Christoph H. Lampert. Neural collapse versus low-rank bias: Is deep neural collapse really optimal? Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  42. [50]

    Implicit optimization bias of next-token prediction in linear models

    Christos Thrampoulidis. Implicit optimization bias of next-token prediction in linear models. In Conference on Neural Information Processing Systems (NeurIPS), 2024

    2024

  43. [51]

    Imbalance trouble: Revisiting neural-collapse geometry

    Christos Thrampoulidis, Ganesh Ramachandra Kini, Vala Vakilian, and Tina Behnia. Imbalance trouble: Revisiting neural-collapse geometry. In Conference on Neural Information Processing Systems (NeurIPS), 2022

    2022

  44. [52]

    Extended unconstrained features model for exploring deep neural collapse

    Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Machine Learning (ICML), 2022

    2022

  45. [53]

    Perturbation analysis of neural collapse

    Tom Tirer, Haoxiang Huang, and Jonathan Niles-Weed. Perturbation analysis of neural collapse. In International Conference on Machine Learning (ICML), 2023

    2023

  46. [54]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  47. [55]

    Get the best of both worlds: Improving accuracy and transferability by grassmann class representation

    Haoqi Wang, Zhizhong Li, and Wayne Zhang. Get the best of both worlds: Improving accuracy and transferability by grassmann class representation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  48. [56]

    Understanding deep representation learning via layerwise feature compression and discrimination

    Peng Wang, Xiao Li, Can Yaras, Zhihui Zhu, Laura Balzano, Wei Hu, and Qing Qu. Understanding deep representation learning via layerwise feature compression and discrimination. arXiv preprint arXiv:2311.02960, 2023. 11

    2023 arXiv

  49. [57]

    Linear convergence analysis of neural collapse with unconstrained features

    Peng Wang, Huikang Liu, Can Yaras, Laura Balzano, and Qing Qu. Linear convergence analysis of neural collapse with unconstrained features. In NeurIPS Workshop on Optimization for Machine Learning (OPT), 2022

    2022

  50. [58]

    Progressive feedforward collapse of resnet training

    Sicong Wang, Kuo Gai, and Shihua Zhang. Progressive feedforward collapse of resnet training. arXiv preprint arXiv:2405.00985, 2024

    2024 arXiv

  51. [59]

    How far pre-trained models are from neural collapse on the target dataset informs their transferability

    Zijian Wang, Yadan Luo, Liang Zheng, Zi Huang, and Mahsa Baktashmotlagh. How far pre-trained models are from neural collapse on the target dataset informs their transferability. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  52. [60]

    On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers

    E Weinan and Stephan Wojtowytsch. On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers. In Mathematical and Scientific Machine Learning, 2022

    2022

  53. [61]

    Neural collapse beyond the unconstrainted features model: Landscape, dynamics, and generalization in the mean-field regime

    Diyuan Wu and Marco Mondelli. Neural collapse beyond the unconstrainted features model: Landscape, dynamics, and generalization in the mean-field regime. arXiv preprint arXiv:2501.19104, 2025

    2025 arXiv

  54. [62]

    Linguistic collapse: Neural collapse in (large) language models

    Robert Wu and Vardan Papyan. Linguistic collapse: Neural collapse in (large) language models. arXiv preprint arXiv:2405.17767, 2024

    2024 arXiv

  55. [63]

    Pursuing feature separation based on neural collapse for out-of-distribution detection

    Yingwen Wu, Ruiji Yu, Xinwen Cheng, Zhengbao He, and Xiaolin Huang. Pursuing feature separation based on neural collapse for out-of-distribution detection. CoRR, 2024

    2024

  56. [64]

    Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds

    Mengjia Xu, Akshay Rangamani, Qianli Liao, Tomer Galanti, and Tomaso Poggio. Dynamics in deep classifiers trained with the square loss: Normalization, low rank, neural collapse, and generalization bounds. In Research, volume 6, 2023

    2023

  57. [65]

    Epa: Neural collapse inspired robust out-of- distribution detector

    Jiawei Zhang, Yufan Chen, Cheng Jin, Lei Zhu, and Yuantao Gu. Epa: Neural collapse inspired robust out-of- distribution detector. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6515–6519. IEEE, 2024

    2024

  58. [66]

    Implicit geometry of next-token prediction: From language sparsity patterns to model representations

    Yize Zhao, Tina Behnia, Vala Vakilian, and Christos Thrampoulidis. Implicit geometry of next-token prediction: From language sparsity patterns to model representations. In First Conference on Language Modeling, 2024

    2024

  59. [67]

    Understand- ing imbalanced semantic segmentation through neural collapse

    Zhisheng Zhong, Jiequan Cui, Yibo Yang, Xiaoyang Wu, Xiaojuan Qi, Xiangyu Zhang, and Jiaya Jia. Understand- ing imbalanced semantic segmentation through neural collapse. In Conference on Computer Vision and Pattern Recognition (CVPR), 2023

    2023

  60. [68]

    On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features

    Jinxin Zhou, Xiao Li, Tianyu Ding, Chong You, Qing Qu, and Zhihui Zhu. On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features. In International Conference on Machine Learning (ICML), 2022

    2022

  61. [69]

    Are all losses created equal: A neural collapse perspective

    Jinxin Zhou, Chong You, Xiao Li, Kangning Liu, Sheng Liu, Qing Qu, and Zhihui Zhu. Are all losses created equal: A neural collapse perspective. In Conference on Neural Information Processing Systems (NeurIPS), 2022

    2022

  62. [70]

    Balanced contrastive learning for long-tailed visual recognition

    Jianggang Zhu, Zheng Wang, Jingjing Chen, Yi-Ping Phoebe Chen, and Yu-Gang Jiang. Balanced contrastive learning for long-tailed visual recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022

    2022

  63. [71]

    A geometric analysis of neural collapse with unconstrained features

    Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu. A geometric analysis of neural collapse with unconstrained features. In Conference on Neural Information Processing Systems (NeurIPS), 2021. 12 A Deferred proofs Lemma 4.2. Assume that only ...

    2021

  64. [72]

    The curvature of Gki defined as the Lipschitz constant of the unit-norm oriented tangent function Tki is bounded by B

  65. [73]

    For all (l, j) > (k, i), max x∈Gki xT ljx ≤ d(1 − m) for some m > 0, i.e., all the subsequent points xlj are far enough from the curve Gki

  66. [74]

    Denote ¯Gki as the set of points on Gki between xki and ¯xki

    There is precisely one point ¯xki ∈ Gki such that ¯xT kihki = d(1 − cm), where c >1 is chosen large enough. Denote ¯Gki as the set of points on Gki between xki and ¯xki. Then, we assume that, for all (l, j) < (k, i), max x∈ ¯Gk,i,y∈Gl,j \ ¯Gl,j xT y ≤ d(1 − m)

  67. [75]

    The length of Gki is no more than 2π √ d. 14

  68. [76]

    10cm ≤ (d − max j(ki)̸=j(lp) ¯hT j(ki) ¯hj(lp))/d

    m is chosen small enough s.t. 10cm ≤ (d − max j(ki)̸=j(lp) ¯hT j(ki) ¯hj(lp))/d. It is clear that a construction satisfying these properties exists, since the constants B, c, mare chosen with respect to X, Hand the number of points we consider is finite. We also note that this...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.