Pith. sign in

REVIEW 3 major objections 4 minor 8 references

Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalization

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that in classification, generalization is best at an intermediate feature learning strength because too strong feature learning over-aligns the learned predictor to a noisy empirical class mean, while too weak feature learn

desk verdict The empirical optimal-FLS story is solid and worth taking seriously, but the main theorem rests on an orthogonality assumption that the paper's own Gaussian-mixture validation violates, so the advertised proof of the trade-off doesn't cover the regime where it is tested. read the letter →

arxiv 2602.00827 v2 pith:JLB7G6CI submitted 2026-01-31 cs.LG stat.ML

Taesun Yeom , Taehyeok Ha , Jaeho Lee This is my paper
classification cs.LGstat.ML
keywords featurelearningstrengthimplicitbiasgeneralizationtwo-layerReLUnetworksgradientflowover-alignmentover-fittingGaussianmixture
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Feature learning strength (FLS) — the inverse of effective output scaling, controlled by initialization scale or output temperature — is usually assumed to be better when stronger. This paper challenges that: across VGG and ResNet classifiers on CIFAR and synthetic natural-image tasks, an intermediate FLS consistently outperforms both extremes when all models are trained to the same near-perfect training risk; the gain grows as the task's intrinsic dimension grows. The theoretical core is a decomposition of the population excess error of a two-layer ReLU net trained by gradient flow on a Gaussian mixture into two competing terms: over-alignment (the predictor hugs the empirical class-mean direction, which deviates from the true signal in finite samples) and over-fitting (a too-large initialization leaves the model close to a linearized kernel solution that memorizes noise). Theorem 5.6 bounds excess error by OA(α)+OF(α) with OA decreasing and OF increasing in the initialization scale α, yielding a data-dependent optimal FLS. A sympathetic reader would come away convinced that 'stronger feature learning is always better' is false under practical early-stopped training.

What carries the argument

The load-bearing object is the effective predictor ŵ_α(t) — the class-wise sum v_j w_j of the first-layer neurons that have aligned to the positive class after the alignment phase — and its angular alignment Ψ(t) to the empirical class mean x+/||x+||. The α-dependence of Ψ at the stopping time tη,α (Lemmas 5.2 and 5.4) defines the cone H(α) of reachable normalized predictors; Theorem 5.6 then decomposes the excess error into the over-alignment gap OA(α) and the over-fitting gap OF(α) and shows they trade off. The two-phase dynamics (neuron alignment followed by margin maximization) is the mechanism that makes Ψ(α) nearly frozen after the alignment phase, so the optimal FLS is already determ

What would settle it

Train a two-layer ReLU net on a Gaussian mixture with known signal s+, fixed noise σ, and small n; measure the angular alignment Ψ(tη,α) of the effective predictor to the empirical class mean and the test error at the stopping time. The decomposition predicts the over-alignment gap Φ(−v*^T s+/σ)−Φ(−1/σ) should account for most of the error increase at small α. If instead the test error decreases monotonically as α→0 when n is small (or if the optimal α does not shift toward larger α as the empirical mean deviates more from s+), the over-alignment mechanism is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that an optimal feature learning strength exists because the two failure modes pull in opposite directions. In the finite-sample, finite-time setting (training stopped when the training risk reaches a target η), the learned predictor's direction after training is confined to a circular cone H(α) around the empirical class mean x+/||x+||, and the width of that cone shrinks as the initialization scale α shrinks (equivalently, as FLS grows). The excess error over the Bayes classifier splits exactly into OA(α), the error of the best predictor inside that cone versus the signal direction s+, and OF(α), the error of the actually learned predictor versus the cone's best

Load-bearing premise

The whole trade-off rests on Assumption 4.1 (orthogonal separability of the training set): every pair of training samples from different classes must have normalized inner product at least λ>0; the paper notes this does not hold for general datasets, and without it the two-phase dynamics and the alignment point x+/||x+|| that define OA and OF are not guaranteed.

Editorial extensions

If this is right

  • If the claim is right, FLS is a first-class generalization hyperparameter, not a compute-only trade-off: fixing the learning rate and tuning only the output multiplier can recover several percent test accuracy (about 6% on ResNet50/CIFAR-100 in the paper's sweeps).
  • The gap between optimal and default FLS widens as the intrinsic dimensionality of the data grows, so the payoff of FLS tuning increases exactly for the tasks that are hardest.
  • At matched training risk, a larger FLS does not monotonically improve generalization: the paper's bound predicts a U-shaped excess-error curve, and its numerical experiments confirm OA+OF is U-shaped in α.
  • The analysis applies at early stopping (target training loss η), not only at the asymptotic max-margin limit, so the conclusions transfer to the training budgets used in practice.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's mechanism suggests a transferable design rule: since over-alignment is driven by the deviation between the empirical class mean and the true signal, any data-augmentation or regularizer that shrinks that deviation (e.g., label smoothing, mixup) should shift the optimal FLS toward stronger feature learning; this is a testable prediction the paper does not state.
  • By the same logic, temperature scaling in softmax-based classifiers and logit scaling in distillation are FLS dials, so the over-alignment/over-fitting trade-off could explain known temperature-sensitivity results in those settings.
  • A natural extension would be per-class or adaptive alignment penalties that widen H(α) where the empirical mean is unreliable, which might outperform a single global α; the paper's decomposition gives the objective such a scheme would optimize.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper studies how feature learning strength (FLS), controlled by output scaling or initialization scale, affects generalization in classification. Empirically, on CIFAR-10/100 and BigGAN-generated datasets with VGG/ResNet architectures, the authors observe that an intermediate FLS outperforms both very weak and very strong feature learning. Theoretically, they analyze gradient flow for a two-layer ReLU network on a binary Gaussian mixture, and prove an upper bound (Theorem 5.6) on the excess population error. The bound decomposes into an over-alignment term OA(α), which is large when the effective predictor is forced into too narrow a cone around the empirical class mean, and an over-fitting term OF(α), which grows when the predictor deviates from the best predictor in that cone. They argue that this trade-off yields an optimal FLS.

Significance. If the theoretical result is correct, it directly challenges the common intuition that stronger feature learning always improves generalization, and it provides a first principled explanation of an optimal FLS under finite-time training. The empirical study is broad and consistent across architectures and datasets, and the proposed OA/OF decomposition is conceptually appealing. The theory is parameter-free in the sense that no constants are fitted to data. However, the central theorem currently applies to orthogonally separable training sets, and the paper's own Gaussian-mixture validation does not satisfy that condition; as the stress-test note correctly observes, this is a load-bearing mismatch rather than a mere scope limitation. The paper also relies on an informal monotonicity argument for the OF term. These issues can likely be addressed by re-scoping the theory or adding a rigorous treatment, but the current version does not fully establish the claimed Gaussian-mixture trade-off.

major comments (3)
  1. [§4.2, Assumption 4.1; Theorem 5.6]
  2. [§5.2, Lemma 5.4 and Eq. (22)]
  3. [§5.3, Theorem 5.6 assumption of ∥ŵα(tη,α)∥≤1]
minor comments (4)
  1. [General] There are several typos: 'probablity' in Theorem 5.6, 'is is neither' in Section 4.1, and 'approximated by' in Section 4.1. Also, the name 'Orthogonal separability' for a lower bound on cosine similarity is potentially misleading; it is not orthogonality but a margin condition.
  2. [Figure 2 and 3] Blank grids in the heatmaps remove configurations that fail to reach 99% training accuracy for at least one seed. This is a sensible filtering, but it means the empirical claim 'larger FLS hurts generalization' is made only among fitting configurations. Please state this more prominently and consider reporting the number of non-fitting seeds, since non-fitting large-FLS configurations are exactly the ones excluded.
  3. [§A.4] The sentence 'Note that we do not restrict λ here in order to reflect a realistic setup' should be reconciled with the theorem's assumptions. As it stands, Figure 4 is an empirical illustration, not a verification of Theorem 5.6; the paper should say so explicitly.
  4. [Notation] In Theorem 5.6, the vector s appears without a subscript (e.g., 's' in the OF term and in the denominator of Φ terms). Since the data model uses s+ and s−, please clarify that s=s+ in this context. Also define x_min before Lemma 5.4 if it is not already clear.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the OA/OF decomposition is an identity, but the trade-off's content comes from external dynamics and explicit geometry, not from fitted inputs or self-citation.

full rationale

The central decomposition in Eq. (24) is an exact identity: for any predictor wα and any set H(α) containing wα, the excess error splits into inf_H E − E* and E(wα) − inf_H E. Since H(α) in Eq. (25) is defined as the cone of directions whose alignment is at least Ψ(tη,α), and Ψ(tη,α) is by definition the alignment of wα, the learned predictor lies in H(α) by construction. This makes the bookkeeping identity tautological. However, the substantive content of the paper is not circular: the monotonicity of Ψ(tη,α) in α is derived from the two-phase dynamics imported from Min et al. (2024) (Lemmas 5.1, 5.2, 5.4), an external source, and the geometric effect of the cone on Gaussian-mixture error is computed explicitly via Lemma D.4 and the closed form of v*. The over-alignment term decreasing in α is a formal consequence of the cone growing with α, but that growth is itself a derived dynamical fact. The over-fitting term is bounded using standard Rademacher complexity, the stopping-time threshold η, and the constant c(σ,d); no parameter is fitted to data and no target quantity is inserted into the bound to force the U-shape. The numerical Figure 4 computes the exact decomposition from trained models, so the U-shape is not manufactured by construction. The only self-citation, Yeom et al. (2025), appears in a related-work list about initialization controls and is not load-bearing for the theorem. The paper openly acknowledges that Assumption 4.1 (orthogonal separability) does not hold for general datasets and that the numerical validation does not restrict λ (Section A.4); this is a correctness/restrictiveness limitation, not a circular step, because the theorem is explicitly conditional on that assumption. No circular step meeting the required evidence standard was found.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The theory relies on standard prior lemmas and a restrictive data/separability assumption. No free parameters are fitted to data; the central variables (α, Ψ, OA, OF) are defined, not tuned. No new physical or conceptual entities are introduced beyond the named error-decomposition terms.

assumptions (5)
  • domain assumption Orthogonal separability (Assumption 4.1): all distinct training pairs from different classes have cosine similarity ≥ λ > 0.
    Needed for the early-phase ODE to have the interpretable stationary point x+/||x+||; the paper acknowledges it does not hold for general datasets.
  • standard math Two-phase dynamics of two-layer ReLU nets under small initialization (Lemmas 5.1, D.3 from Min et al. 2024).
    Prior published results used as building blocks for the alignment analysis.
  • domain assumption Gaussian mixture data model (Eq. 6) with κ=1 and s+ = -s-.
    Enables closed-form expressions for zero-one error and population logistic risk.
  • ad hoc to paper Small-norm condition ∥ŵα(tη,α)∥ ≤ 1 in Theorem 5.6.
    Required for the Rademacher-based bound; asserted realistic for small α and finite-time training, but not generally guaranteed.
  • standard math Balancedness and sign preservation from initialization (Lemma D.1).
    Standard gradient-flow properties for homogeneous networks (Du et al. 2018, Boursier et al. 2022).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalization." pith.science (2026). https://pith.science/paper/JLB7G6CI

@misc{pith2026260200827,
  author       = {Pith},
  title        = {Pith review of: Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JLB7G6CI}},
  note         = {Machine review of arXiv:2602.00827}
}
abstract

Feature learning strength (FLS), i.e., the inverse of the effective output scaling of a model, plays a critical role in shaping the optimization dynamics of neural nets. While its impact has been extensively studied under the asymptotic regimes -- both in training time and FLS -- existing theory offers limited insight into how FLS affects generalization in practical settings, such as when training is stopped upon reaching a target training risk. In this work, we investigate the impact of FLS on generalization in deep networks under such practical conditions. Through empirical studies, we first uncover the emergence of an $\textit{optimal FLS}$ -- neither too small nor too large -- that yields substantial generalization gains. This finding runs counter to the prevailing intuition that stronger feature learning universally improves generalization. To explain this phenomenon, we develop a theoretical analysis of gradient flow dynamics in two-layer ReLU nets trained with logistic loss, where FLS is controlled via initialization scale. Our main theoretical result establishes the existence of an optimal FLS arising from a trade-off between two competing effects: An excessively large FLS induces an $\textit{over-alignment}$ phenomenon that degrades generalization, while an overly small FLS leads to $\textit{over-fitting}$.

Figures

Figures reproduced from arXiv: 2602.00827 by the authors.

Figure 1
Figure 1. Emergence of an optimal FLS. We empirically observe that, under standard classification setups, stronger feature learning tends to degrade generalization performance of the model when it exceeds a certain threshold, implying the existence of an “optimal FLS” that is neither too large nor too small. 2. Related Work FLS in deep learning. The optimization behavior of neu￾ral nets is highly sensitive to the feature lear… view at source ↗
Figure 2
Figure 2. Emergence of optimal FLS in generalization. Peak test accuracy of various networks trained on CIFAR-100. Blank grids indicate cases where, for at least one of the three seeds, the training accuracy does not exceed 99%. For readability, the learning rate axis is labeled using the pre-normalized values (i.e., η), where k = 6.4 × 10−4 . Further details can be found in Section A.1. ing rate; most similar to our work, Pe… view at source ↗
Figure 3
Figure 3. Optimal FLS is more beneficial for the difficult dataset. The gap of the peak test accuracy of ResNet18 trained on a BigGAN-generated dataset against the best FLS. We have varied the effective dimensionality of the samples generated, to control the task difficulty. Blank grids indicate the cases where, for at least one of the three seeds, the training accuracy does not exceed 99%. For readability, the learning rate … view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Numerical simulation of OA(α), OF(α), and g(α): Note that OA(α) + OF(α) recovers the excess error. and decreasing otherwise. For moderate values of n and d, we have Ψ(tη,α) ≳ p 1/d ≈ 0, since the dominant scaling is governed by d. Moreover, since Ψ(tη,α) remains relati…
Figure 5
Figure 5. Figure 5: Example images from the BigGAN-generated datasets. As the effective dimensionality (i.e., edim) of the input increases, BigGAN produces more diverse images, thereby making the task more difficult. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Best test loss (CIFAR-100). 2 10 2 8 2 6 2 4 2 2 2 0 2 2 Output Multiplier k × 2 0 k × 2 2 k × 2 4 k × 2 6 k × 2 8 k × 2 10 k × 2 12 Learning Rate 86.95 ±0.26 86.52 ±0.59 85.29 ±0.49 81.26 ±0.26 72.29 ±0.32 63.67 ±0.47 56.04 ±0.42 87.79 ±0.23 87.88 ±0.16 87.87 ±0.49 87…
Figure 7
Figure 7. Figure 7: Peak test accuracy (CIFAR-10). 2 10 2 8 2 6 2 4 2 2 2 0 2 2 Output Multiplier k × 2 0 k × 2 2 k × 2 4 k × 2 6 k × 2 8 k × 2 10 k × 2 12 Learning Rate 0.5398 ±0.0165 0.5684 ±0.0132 0.6322 ±0.0135 0.7062 ±0.0289 0.9350 ±0.0119 1.1072 ±0.0088 1.3911 ±0.0153 0.5300 ±0.0125…
Figure 8
Figure 8. Figure 8: Best test loss (CIFAR-10). 13 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: Peak test accuracy for ResNet18 trained on a BigGAN-generated dataset, with varying effective dimensionality. 2 10 2 8 2 6 2 4 2 2 2 0 2 2 Output Multiplier k × 2 0 k × 2 2 k × 2 4 k × 2 6 k × 2 8 k × 2 10 k × 2 12 Learning Rate 0.1190 ±0.0034 0.1361 ±0.0077 0.1855 ±0.…
Figure 10
Figure 10. Figure 10: Best test loss for ResNet18 trained on a BigGAN-generated dataset, with varying effective dimensionality. 2 10 2 8 2 6 2 4 2 2 2 0 2 2 Output Multiplier k × 2 0 k × 2 2 k × 2 4 k × 2 6 k × 2 8 k × 2 10 k × 2 12 Learning Rate 0.80 ±0.08 1.06 ±0.02 2.76 ±0.16 3.73 ±0.04…
Figure 11
Figure 11. Figure 11: Gap of the peak test for VGG-19 trained on a BigGAN-generated dataset, with varying effective dimensionality. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]
Figure 12
Figure 12. Figure 12: Peak test accuracy for VGG19 trained on a BigGAN-generated dataset, with varying effective dimensionality. 2 10 2 8 2 6 2 4 2 2 2 0 2 2 Output Multiplier k × 2 0 k × 2 2 k × 2 4 k × 2 6 k × 2 8 k × 2 10 k × 2 12 Learning Rate 0.5680 ±0.0547 0.1703 ±0.0006 0.1526 ±0.00…
Figure 13
Figure 13. Figure 13: Best test loss for VGG19 trained on a BigGAN-generated dataset, with varying effective dimensionality. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]
Figure 14
Figure 14. Figure 14: , the excess error exhibits a “U-shape,” demonstrating the existence of an optimal FLS. 1.0e-02 2.0e-02 3.0e-02 4.0e-02 5.0e-02 Initialization Scale 0.120 0.125 0.130 0.135 0.140 0.145 0.150 OA+OF OA+OF [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Norm of the effective predictor. 1.0e-02 2.0e-02 3.0e-02 4.0e-02 5.0e-02 Initialization Scale 0.130 0.132 0.134 0.136 0.138 0.140 0.142 0.144 OA+OF OA+OF [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Excess error across different initializations scale (η = 0.01). 16 [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]
Figure 17
Figure 17. Figure 17: Phase 2 results. Following Section 5.2, this section presents the alignment results obtained in Phase 2. For completeness, we recall Lemma 5.4: Lemma E.7 (Lemma 5.4 in the paper). Let β := (λxmin) 2/(32xmax) and t2 = O(log(1/α)/n). Then, for any risk threshold η > 0, …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

8 extracted references · 3 linked inside Pith

  1. [1]

    Then, by Min et al. (2024, Lemma 10), we have d dt ψj(t)− 1−ψ 2 j (t) ∥x+∥ ≤2nx max max i |f(x i;W(t),v(t))|.(39) This yields d dt ψj(t)≥ 1−ψ 2 j (t) ∥x+∥ −2nxmax max i |f(x i;W(t),v(t))|(40) ≥ 1−ψ 2 j (t) ∥x+∥ −να∥x+∥,(41) =∥x +∥ 1−να−ψ 2 j (t) ,(42) where Equation (41) is due to Lemma D.3. Now, we consider two cases: •Case 1.We haveψ j(t)< √1−ναfor allt...

    2024

  2. [2]

    (2024, Appendix D.2), andβ:= (λx min)2/(32xmax)

    (75) ≤x maxn (t2 −t α) ˆL(tα) + 1 β log ˆL(t2) η ! .(76) Here, Equation (74) is due to monotonic decreasing property of the training risk and the results from Min et al. (2024, Appendix D.2), andβ:= (λx min)2/(32xmax). 21 Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalization C.4. Proof of Theorem 5.6 For the proof, we wil...

    2024

  3. [3]

    S., Lucchi, A., Pascanu, R., et al

    Masarczyk, W., Ostaszewski, M., Cheng, T. S., Lucchi, A., Pascanu, R., et al. Unpacking softmax: How temperature drives representation collapse, compression, and general- ization.arXiv preprint arXiv:2506.01562,

    arXiv

  4. [5]

    U-shape,

    edim=32edim=64edim=128 Figure 5.Example images from the BigGAN-generated datasets.As the effective dimensionality (i.e.,edim) of the input increases, BigGAN produces more diverse images, thereby making the task more difficult. 12 Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalization A.2. Experiments on CIFARs In this subs...

  5. [7]

    (2022).)sign(v j(t)) = sign(vj(0))

    •(Sign preservation, from Boursier et al. (2022).)sign(v j(t)) = sign(vj(0)). Proof.See each paper for the proof. Lemma D.2.Letϕ(t)be a differentiable function satisfying ˙ϕ(t)≥b c2 −ϕ 2(t) for someb, c >0. Then, we have: d dt arctanh (ϕ(t)/c)≥bc.(127) Proof.First, note that the derivative of thearctanh(x)is1/(1−x 2). Then, by the chain rule, d dt arctanh...

    2022

  6. [8]

    AUXILIARYLEMMATA Before start, we define Π∥ to be a orthogonal projection onto s and Π⊥ to be the orthogonal projection onto its complement

    29 Over-Alignment vs Over-Fitting: The Role of Feature Learning Strength in Generalization E.1.1. AUXILIARYLEMMATA Before start, we define Π∥ to be a orthogonal projection onto s and Π⊥ to be the orthogonal projection onto its complement. Next, from the auxiliary lemmata, let us define An,δ := s max{0,(d−1)−2 p (d−1) log(1/δ)} n , B n,δ := √ d+ p 2 log(1/...

    2000

  7. [2018]

    Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity.arXiv preprint arXiv:2106.15933,

    Jacot, A., Ged, F., S ¸ims ¸ek, B., Hongler, C., and Gabriel, F. Saddle-to-saddle dynamics in deep linear networks: Small initialization training, symmetry, and sparsity.arXiv preprint arXiv:2106.15933,

    arXiv

  8. [2020]

    Gradient de- scent quantizes relu network features.arXiv preprint arXiv:1803.08367,

    Maennel, H., Bousquet, O., and Gelly, S. Gradient de- scent quantizes relu network features.arXiv preprint arXiv:1803.08367,

    arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.