{"id":"4458e856-72cd-4766-aeb5-4afd53555020","arxiv_id":"2505.15239","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Neural collapse is globally optimal in deep regularized ResNets and transformers, with the approximation improving as depth grows.","lead":"Proves that global optima of deep regularized ResNets and transformers with LayerNorm are approximately neural-collapsed, and that collapse tightens as depth grows. It is the first formal reduction of end-to-end training of these modern architectures to the simplified unconstrained-features model, and includes experiments on vision and language data.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Penultimate-block ReLU construction in Theorem 4.3 is unverified: zero-mean ETF targets have negative coordinates, so the all-ones offset may not keep pre-activations non-negative; the asserted curve existence is likewise unproved.","rationale":"The high-level strategy is attractive: since a deeper residual network can copy a shallower one, it suffices to construct a sequence of feasible networks with regularization cost tending to zero that realizes any GUFM optimum. The GUFM analysis in Lemma 4.2 is standard, and the reduction would indeed imply approximate neural collapse. I looked for a hidden structural obstruction elsewhere: the transformer step depends on the ResNet construction, so the ResNet construction is the load-bearing part. The curve-existence assertion and the ReLU non-negativity check both live in the proof of Theorem 4.3, and the paper gives no formal argument for either. The reader flagged the same spot, and my reading is consistent: this is a proof gap, not a disagreement with prior consensus or a claim contradicted by the experiments. The experimental section supports the general trend, and the GUFM results are independent value. But the theorem is not machine-checked, and the explicit construction is the only evidence for the upper bound L_{L,1} \\to L*_GUFM; if the ReLU non-negativity or curve-existence check fails, Theorem 4.3 and Corollary 4.5 would need substantial repair or would be false. Therefore the appropriate verdict remains CONDITIONAL, unchanged from the reader's assessment. The concrete test above would settle whether the concern lands: it directly checks the disputed pre-activation signs and the feasibility of the asserted curves.","tokens_in":25649,"tokens_out":9084,"duration_ms":82914,"concrete_test":"Implement the Appendix A penultimate-block formulas for L-RN1 with d=4, K=2, \\bar h_1=(1,1,-1,-1), \\bar h_2=-\\bar h_1, choose x on the zero-sum \\sqrt{d}-sphere with \\bar h_1^T x = d(1-cm), and compute u = W_l^1 x + b_l^1 for c,m satisfying the proof's constraints. If min_i u_i < 0, the ReLU output is not the claimed positive combination of 1 and \\bar h_1, so the LayerNorm update does not equal the step the proof asserts. Repeat with d=8 and a zero-mean target with one large negative coordinate. Independently, run a small feasibility search (N=3, d=4) for a curve satisfying all five listed margin conditions; if none exists, the curve-existence assertion is false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The proof of Theorem 4.3 (Appendix A) rests on an explicit construction whose key step is not verified. After the first N blocks, the j-th penultimate block is supposed to move x_l^j toward \\bar h_j using W_l^j = \\alpha_l^j (1 + cm \\bar h_j)/\\|1 + cm \\bar h_j\\| \\bar h_j^T/\\sqrt{d}, b_l^j = -(1-2cm)\\alpha_l^j \\sqrt{d}/\\|1 + cm \\bar h_j\\| 1. For ReLU+LayerNorm to implement the claimed step along \\bar h_j, the pre-activation W_l^j x + b_l^j must be coordinate-wise non-negative. But GUFM optima are zero-mean, so \\bar h_j has negative coordinates. In the paper's displayed formula, \\sigma(W_l^j x + b_l^j) = \\alpha/\\|1 + cm \\bar h_j\\| ((\\bar h_j^T x - d + 2cmd)1 + cm \\bar h_j^T x \\bar h_j), the all-ones term is O(cmd) while the \\bar h_j term can be O(cmd \\cdot d^{1/2}) on negative coordinates; for d \\ge 4 the ReLU can therefore be inactive exactly where the proof needs the negative part of \\bar h_j. The sentence 'after subtracting the mean ... we are adding \\alpha cm \\bar h_j^T x/\\|1 + m \\bar h_j\\| \\bar h_j' asserts the needed non-negativity rather than proving it. This is not cosmetic: if ReLU zeros those coordinates, the centered update is not a positive multiple of \\bar h_j and the per-sample trajectory breaks, so the constructed network need not approach L*_GUFM. The first phase has the same structure: five curve-margin conditions are dismissed with 'It is clear' and no construction or topological argument is supplied, although the whole reduction uses only this feasible network. Thus the central claim is unproved at exactly this point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper develops a reduction from deep regularized LayerNorm ResNets and transformers to a generalized unconstrained features model (GUFM). It proves that, under certain architectural and data assumptions, the global optima of the end-to-end training objective converge, in the sense of the Hausdorff-type distance of Theorems 4.3 and 4.6, to the corresponding GUFM optima as the depth grows, and that the approximation tightens with depth. Combined with a GUFM analysis for cross-entropy and mean-squared-error losses, it concludes that neural collapse is globally optimal for these architectures. The authors support the theory with experiments on MNIST, CIFAR-10, and IMDB showing that the NC metrics improve with depth.","tokens_in":26087,"tokens_out":12687,"duration_ms":120144,"significance":"If the proofs are completed, the contribution is substantial: it is the first end-to-end global optimality statement for neural collapse in transformers and a significant extension of the corresponding ResNet results, and it provides a formal justification for the unconstrained-features model in modern architectures. The paper does not merely cite a known UFM result; it constructs explicit networks whose loss approaches the GUFM optimum, and the experiments test the falsifiable prediction that deeper networks collapse more, rather than fitting constants to reproduce a known conclusion. The high-level reduction strategy is elegant and, if the Appendix-A construction is repaired, would be an important step for the field. The main obstacle is the technical completeness of the constructive proof, on which the central theorems rest.","major_comments":[{"comment":"The construction of the penultimate blocks defines W_l^j = α_l^j (1 + cm \\bar h_j)/\\|1 + cm \\bar h_j\\| \\bar h_j^T/√d and b_l^j = -(1-2cm) α_l^j √d /\\|1 + cm \\bar h_j\\| 1, and then claims that after ReLU and LayerNorm the centered update is a positive multiple of \\bar h_j. This claim assumes that the pre-activation W_l^j x + b_l^j is non-negative in every coordinate. Since GUFM-optimal features are zero-mean, \\bar h_j has negative coordinates, and the paper gives no bound on their magnitude. For a zero-mean vector of norm √d, a negative coordinate can have magnitude up to √(d-1); in that case the term cm\\bar h_j^T x \\bar h_j on that coordinate is about -cm d√(d-1), which dominates the positive all-ones offset 2cmd for d > 5, so the ReLU would clip exactly the coordinate the proof needs. The displayed formula after the definition of W_l^j and b_l^j is therefore not a proof of the centered update; the trajectory along \\bar h_j may break, and the constructed loss need not approach L*_GUFM. Please either prove the non-negativity for a deliberately chosen optimal H (for example, an ETF with bounded negative coordinates) or modify the construction so that ReLU is guaranteed to be inactive on all relevant coordinates.","section":"Appendix A, proof of Theorem 4.3, penultimate-block construction"},{"comment":"The first phase of the construction requires, for each sample, a smooth oriented curve G_{ki} on the zero-sum √d-sphere satisfying five simultaneous conditions: bounded curvature, a margin against all subsequent samples, a unique crossing point with \\bar x_{ki}, a segment-wise margin against previously processed samples, and a length bound. The proof dismisses this with the sentence \"It is clear that a construction satisfying these properties exists.\" This assertion is load-bearing: the per-sample block construction and the bound on the total Frobenius-norm cost depend on the simultaneous validity of all these properties, and the statement is not a trivial consequence of finiteness because of the uniform curvature bound and the simultaneous margin constraints. Please supply a rigorous argument, for example a lemma with an explicit curve family or a compactness/topological argument, before the reduction can be considered established.","section":"Appendix A, proof of Theorem 4.3, first-phase curve construction"},{"comment":"The proof of Theorem 4.6 states that it follows the proof of Theorem 4.3 with modified weight scalings. Consequently, the two unproved ingredients identified above, the existence of the curve family with the required margin properties and the coordinatewise non-negativity of the ReLU pre-activations, carry over unchanged to the two-linear-layer architectures. In addition, the two-layer construction inserts a second linear layer that is supposed to act as a projection onto the span of the first layer's output; this requires the intermediate ReLU output to be exactly the non-negative vector from the single-layer construction, so the non-negativity issue is not avoided. The statement of Theorem 4.6 should not be considered proven until the underlying construction is repaired.","section":"Theorem 4.6 and its proof"}],"minor_comments":[{"comment":"The proof contains the inclusion \"MUFM-MSE_0 ⊂ MUFM-MSE_0\", which is a tautology; one of the two sets should refer to the solutions of the relaxed problem (13) and the other to the solutions of the constrained GUFM problem (8). Please correct the notation.","section":"Appendix A, proof of Lemma 4.2, MSE part"},{"comment":"The displayed formula for σ(W_l^j x + b_l^j) in the penultimate-block paragraph appears to be missing the factor 1/√d that multiplies the outer product in the definition of W_l^j. Please verify the algebra, since the LayerNorm scaling depends on the correct constant.","section":"Appendix A, proof of Theorem 4.3, penultimate-block formula"},{"comment":"The introduction advertises results \"under minimal assumptions on the data\", but Assumption 4.4 requires all ResNet training samples to be unique. This is a substantive structural requirement, not a minimal one; please rephrase the introduction or soften the claim.","section":"Section 4, Assumption 4.4 and introduction"},{"comment":"The CE part of the proof invokes Theorem 3.1 of [71] and additionally uses the assertion that the norm of the optimal feature vectors is an increasing function of the regularization strength, in order to match the fixed-norm constraint of the GUFM. Since this monotonicity is used to identify the solution sets, it should be stated explicitly as a lemma with either a proof or a precise pointer to the specific statement in [71].","section":"Appendix A, proof of Lemma 4.2, CE part"}],"recommendation":"major_revision","confidential_remarks":"The high-level reduction and the experimental support are genuinely interesting, and I do not see a self-referential or circular structure in the argument. The obstacle is squarely in the constructive proof in Appendix A: the existence of the curve family and the ReLU non-negativity are asserted rather than proved, and the second of these can fail for concrete zero-mean ETF targets. I believe the result is likely repairable, for instance by choosing a convenient optimal H with controlled coordinates or by modifying the per-block construction, but in the current form the central theorems are not rigorously established. The journal should require a repaired appendix before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2505.15239. The headline claim—global optima of deep LayerNorm ResNets and transformers converge to the GUFM, so neural collapse is optimal—is a genuinely new reduction, and if correct it would justify the UFM for modern architectures. But the proof of the main theorem (4.3) has a real, load-bearing gap. The explicit construction in Appendix A does not ensure that the ReLU pre-activations stay non-negative when the target features have negative coordinates, and the earlier curve existence is asserted rather than proved.\n\nWhat's good: the depth-splitting intuition is appealing. Splitting a map across L blocks makes the total Frobenius norm of the weights decay like 1/L, so the regularizer vanishes and the end-to-end objective approaches the GUFM. That is a clever way around the UFM criticism, and the extension to transformers via a zero-attention first block is thought through. The paper also makes a falsifiable prediction—collapse improves with depth—and tests it on vision and language. The authors are honest about what is borrowed: the CE GUFM optimality comes from Zhu et al., not derived here.\n\nThe soft spots are in the proof, and they matter. In the penultimate blocks, W and b are chosen to keep other samples in the negative orthant while moving the target along h_j. But the target's pre-activation has a term proportional to h_j, and since h_j is zero-mean, it has negative coordinates of magnitude up to sqrt(d). The all-ones offset only scales as O(cmd), so for d >= 9 the negative coordinates win and ReLU zeros exactly the components the proof needs. The sentence \"after subtracting the mean ... we are adding ...\" assumes the full vector survives ReLU; it doesn't. The displayed formula also looks off by a factor of sqrt(d) between the 1 and h_j terms, though the problem exists either way. The first phase is similarly shaky: the five curve conditions are dismissed with \"It is clear,\" and no topological argument is supplied. These aren't cosmetic details; the reduction uses this construction to show L*_L -> L*_GUFM. Without it, Theorem 4.3 isn't proven. The O(L^{-1/2}) rate is also stated without derivation, and the language experiments report only converged runs, a mild but real selection bias.\n\nWho should read this: anyone working on neural collapse theory or UFM. The reduction is a good idea that deserves a serious attempt at a correct proof. I'd send it to peer review, but with the expectation of major revision—the authors need to either repair the construction or prove existence by a different route. As it stands, I wouldn't rely on Theorem 4.3.","headline":"Strong idea, shaky proof: the depth-splitting reduction to a GUFM is novel and worth engaging, but the main theorem relies on an asserted construction whose ReLU step does not check out, so the paper needs major repair before it can be trusted.","tokens_in":26599,"tokens_out":8691,"would_cite":false,"duration_ms":73110,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"Deep ResNets and transformers provably collapse at global optima.","keywords":["neural collapse","unconstrained features model","ResNet","transformer","LayerNorm","global optimality","weight regularization","cross-entropy loss"],"falsifier":"With dimension $d=4$, two classes, and one sample per class, take an initial feature pair and a GUFM-optimal target pair and search over all smooth curves on the zero-sum sphere for one satisfying the five margin conditions in the proof of Theorem 4.3; an explicit configuration where no such curve exists would break the constructive reduction as written, so the proof would need a different argument.","tokens_in":25407,"feed_emoji":"🧠","tokens_out":12499,"duration_ms":101167,"temperature":0.7,"pith_summary":"Neural collapse is the empirical tendency of a trained classifier's last-layer features to shrink within each class and for the class means to form a maximally separated simplex. This paper proves that, for deep LayerNorm residual networks and transformers with weight regularization, every global optimum of the end-to-end training problem is approximately collapsed, and the approximation becomes exact as the number of blocks grows. The proof covers both cross-entropy and mean-squared-error losses, and architectures with one or two linear layers per block, provided that in the two-layer case the regularization strength decays with depth. A direct consequence is that the end-to-end objective of a deep-enough ResNet or transformer is equivalent to a generalized unconstrained features model in which the penultimate features are free variables, which justifies using that simpler model to analyze modern architectures.","feed_headline":"Deep ResNets and transformers provably collapse at global optima","feed_subtitle":"As depth grows, all global optima approach neural collapse, validating the widely used unconstrained-features model.","key_machinery":"The load-bearing mechanism is a split-the-map-across-depth construction. In a one-dimensional model, the map $x \\mapsto \\exp(a)x$ can be realized by $L$ residual layers each scaling by $1+a/L$, so the sum of squared Frobenius norms of all weight matrices is $a^2/L$ and vanishes as $L$ grows. The paper generalizes this by assigning each training sample a smooth curve on the zero-sum sphere of radius $\\sqrt{d}$ from its initial feature to its GUFM-optimal target, and building rank-one ReLU layers whose pre-activations are non-negative for the moving sample and negative for all others, so that LayerNorm takes exactly the intended step along the curve while all other samples stay put. Margins between the curves keep the constructions disjoint. For transformers, the first block encodes each context's history in binary so distinct contexts remain distinct even when attention weights converge to uniform, after which the ResNet construction applies directly. The GUFM enforces the zero-mean, fixed-norm, and identical-sample constraints that LayerNorm and the data impose, and Lemma 4.2 shows its CE/MSE optima are fully collapsed.","core_discovery":"The paper's central claim, Theorem 4.3 with Corollary 4.5 for one-linear-layer blocks and Theorem 4.6 with Corollary 4.7 for two-linear-layer blocks, is that global optima of regularized LayerNorm ResNets and transformers approach the global optima of the corresponding generalized unconstrained features model (GUFM) as the depth $L$ grows, at a rate $\\widetilde{O}(L^{-1/2})$ for cross-entropy and mean-squared-error losses, provided the GUFM's optimal loss is positive. Under the assumption that training samples are unique, or that transformer labels are determined by context, the GUFM optima for these two losses are perfectly collapsed: within-class variability vanishes, class means form a maximally separated frame, and features align with the last-layer weight rows. Therefore every global optimum of the deep end-to-end problem is approximately collapsed, with the deviation from perfect collapse bounded by a function of $L$ that tends to zero. The paper also shows that with two linear layers per block and constant regularization this collapse is generically not approached, because feature changes made by two-layer residual blocks scale linearly with the total weight cost. These results give the first end-to-end global-optimality characterization of neural collapse for architectures beyond multilayer perceptrons.","pith_inferences":["The paper proves a statement about global optima, not about the path gradient descent takes; showing that trained networks actually reach these optima would require a separate argument, and the experiments here only suggest the trend.","Because Theorem 4.3 holds for any continuous loss, the same reduction should transfer to other losses whose GUFM optima are known, such as regression or class-imbalanced variants; solving those GUFMs would immediately characterize the end-to-end optima of deep ResNets and transformers.","The construction implies that residual depth itself creates a cost-free way to implement feature maps, so increasing depth should push optima toward collapse independently of optimization details; this is a testable prediction beyond the paper's own experiments.","The requirement that samples be unique may be relaxable in the ResNet case by a small perturbation of the input data, which would extend collapse guarantees to duplicate training samples that the current Assumption 4.4 excludes."],"forward_implications":["For L-RN1 and L-Tx1 architectures with cross-entropy or mean-squared-error loss, every global optimum is approximately collapsed, and the distance to perfect collapse is upper-bounded by a function of depth that goes to zero.","For L-RN2 and L-Tx2 architectures, the same guarantee holds when the regularization on all layers except the last decays faster than $1/\\log L$.","The end-to-end training objective of deep LayerNorm ResNets and transformers reduces to the corresponding GUFM, so global optima of the GUFM are provably approached by global optima of the full network.","For CE and MSE losses the deviation from neural collapse shrinks roughly as $\\widetilde{O}(L^{-1/2})$.","With two linear layers per block and non-vanishing or uniformly vanishing regularization, neural collapse is generically not reached, so the one-layer-per-block and vanishing-regularization regimes behave qualitatively differently."],"supporting_citations":[{"why":"Documents the neural collapse phenomenon and defines the NC1-NC3 metrics used to state the paper's results.","marker":"[43]"},{"why":"Introduces the unconstrained features model that the paper generalizes to the GUFM.","marker":"[42]"},{"why":"Provides the geometric characterization of UFM-CE global optima used in the proof of Lemma 4.2.","marker":"[71]"},{"why":"Supplies the UFM-MSE global-optimum structure that Lemma 4.2 builds on for mean-squared-error loss.","marker":"[68]"},{"why":"Establishes end-to-end neural collapse for wide MLPs with weight decay, the closest prior result that this work extends.","marker":"[23]"},{"why":"Gives the infinite-depth ResNet intuition that splitting a map across layers reduces total weight cost, which the constructive proof uses.","marker":"[4]"}],"fun_headline_variants":["Neural collapse proven optimal in deep ResNets and transformers","Deep networks provably hit neural collapse at global optima","Global optima in deep ResNets and transformers show neural collapse","Depth forces neural collapse in ResNets and transformers","ResNets and transformers: neural collapse is globally optimal"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof needs the existence of smooth trajectories on the allowed feature sphere that carry each sample to its target one at a time while keeping all other samples outside a fixed margin, and it needs the rank-one ReLU layers used to track them to have non-negative pre-activations on the moving sample so that LayerNorm produces the intended step; the paper states this existence is clear rather than proving it, and the reduction collapses if it fails.","fun_headline_variants_meta":{"raw":{"variants":["Neural collapse proven optimal in deep ResNets and transformers","Deep networks provably hit neural collapse at global optima","Global optima in deep ResNets and transformers show neural collapse","Depth forces neural collapse in ResNets and transformers","ResNets and transformers: neural collapse is globally optimal"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1629,"prompt_tokens":986,"completion_tokens":643,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":560}},"tokens_in":602,"tokens_out":643,"duration_ms":5351,"temperature":1.0,"reasoning_tokens":560,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:22:26.793806+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"With dimension $d=4$, two classes, and one sample per class, take an initial feature pair and a GUFM-optimal target pair and search over all smooth curves on the zero-sum sphere for one satisfying the five margin conditions in the proof of Theorem 4.3; an explicit configuration where no such curve exists would break the constructive reduction as written, so the proof would need a different argument.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the neural collapse phenomenon and defines the NC1-NC3 metrics used to state the paper's results."},{"cited_title":"A geometric analysis of neural collapse with unconstrained features","cited_arxiv_id":null,"evidence_quote":"Provides the geometric characterization of UFM-CE global optima used in the proof of Lemma 4.2."},{"cited_title":"On the optimization landscape of neural collapse under MSE loss: Global optimality with unconstrained features","cited_arxiv_id":null,"evidence_quote":"Supplies the UFM-MSE global-optimum structure that Lemma 4.2 builds on for mean-squared-error loss."},{"cited_title":"Wide neural networks trained with weight decay provably exhibit neural collapse","cited_arxiv_id":null,"evidence_quote":"Establishes end-to-end neural collapse for wide MLPs with weight decay, the closest prior result that this work extends."}],"review_version":1}