Pith. sign in

REVIEW 1 major objections 5 minor 38 references

From Logistic Regression to the Perceptron Algorithm: Exploring Gradient Descent with Large Step Sizes

T0 review · 1 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read In the infinite-step-size limit, logistic regression with gradient descent becomes the batch perceptron algorithm, and this limit explains why arbitrarily large steps still solve linearly separable classification problems.

desk verdict LR+GD with infinite step size is exactly the batch perceptron; new, clean proof, plus a solid suboptimality lower bound and a normalized variant. read the letter →

arxiv 2412.08424 v1 pith:NLTMMNO4 submitted 2024-12-11 cs.LG

Alexander Tyurin This is my paper · ORCID
classification cs.LG MSC 68T0590C25
keywords logisticregressiongradientdescentlargestepsizesperceptronalgorithmedgeofstabilityiterationcomplexityseparabledatalosslandscapemetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to explain a paradox: logistic regression trained by gradient descent (LR+GD) still solves linearly separable classification problems when the step size is arbitrarily large, even though classical optimization theory says GD should diverge past the smoothness threshold. The central claim is that in the limit γ→∞, the rescaled LR+GD iterates θ_t/γ converge to the iterates of a batch version of the perceptron algorithm, so LR+GD inherits the classic perceptron convergence bound. It also claims that in this regime the logistic loss and gradient norm become misleading: larger steps raise the loss while speeding up the search for a separator. Finally, it argues that the batch perceptron's dependence on the number of samples n is a real weakness, proves an Ω(n) lower bound, and introduces Normalized LR+GD, a simple adaptive rescaling, to recover an n-free rate.

What carries the argument

The load-bearing object is the rescaling θ_t/γ together with the sigmoid-to-threshold convergence: (1 + exp(γ y_i a_i^T (θ_t/γ)))^{-1} → 1 if y_i a_i^T θ̂_t < 0 and → 0 otherwise. Because each batch perceptron iterate is a nonnegative combination of samples, Assumption 3.1 ensures no sample sits exactly on the decision boundary, so the thresholds are well defined. This pointwise limit is what lets one transfer the classical perceptron analysis to LR+GD, and its failure mode—samples on the boundary—is exactly the technical assumption the paper needs.

What would settle it

Run LR+GD on a linearly separable dataset that violates Assumption 3.1 (e.g., a dataset where a batch perceptron iterate passes exactly through a sample) with a sequence of step sizes γ_k → ∞; if the rescaled iterates θ_t/γ_k still match the batch perceptron for all t, the assumption is unnecessary, while if they deviate, the theorem's stated scope fails. A second check: on the paper's lower-bound dataset with n samples, count iterations to perfect accuracy for LR+GD with very large γ and compare with Ω(n); a matching linear dependence supports the claim, and a sublinear count would refute it.

Watch

Extended reading notes

Core claim

The paper's central discovery is a limit identity: under a mild non-degeneracy condition, for every fixed t, θ_t/γ → θ̂_t as γ→∞, where θ_t is the t-th LR+GD iterate from θ0=0 and θ̂_t is the t-th iterate of the batch perceptron algorithm. The mechanism is that the logistic sigmoid factors (1+exp(y_i a_i^T θ_t))^{-1} become hard thresholds 1[y_i a_i^T θ̂_t < 0] in the limit, so each gradient step converges to the batch perceptron update θ̂_t + (1/n)∑_{i∈S_t} y_i a_i. Because the classical perceptron bound guarantees separation in at most nR²/µ² steps, this gives a direct explanation of large-step-size success and identifies the effective iteration complexity of LR+GD in that regime. The paper further shows that this rate is suboptimal in n, proves an Ω(n) lower bound on a specific dataset, and introduces Normalized LR+GD, whose step is rescaled by the inverse average sigmoid, to recover the n-free rate R²/µ² while keeping the connection to the perceptron.

Load-bearing premise

The whole limit story rests on the non-degeneracy assumption that no training sample ever lies exactly on a decision boundary of a batch perceptron iterate, and on the unproven uniformity of the pointwise limit θ_t/γ → θ̂_t, which is needed to conclude that LR+GD finds a separator within the perceptron's nR²/µ² iteration bound.

Editorial extensions

If this is right

  • For any separable dataset satisfying Assumption 3.1, LR+GD with γ→∞ solves the classification problem in at most nR²/µ² iterations.
  • Larger step sizes in LR+GD drive the logistic loss and gradient norm up while speeding up the search for a separator, so loss and gradient values are unreliable indicators of progress toward a solution.
  • The dependence on n in the iteration bound is essential for LR+GD with γ→∞: there are separable datasets with constant margin and radius on which it needs Ω(n) iterations.
  • Normalized LR+GD, which rescales the gradient step by the inverse average sigmoid, solves the problem in at most R²/µ² + 2 log(2n−1)/(γµ²) iterations, improving on LR+GD by a factor of n as γ→∞.
  • The max-margin implicit bias of small-step LR+GD does not carry over to the γ→∞ limit, since the batch perceptron does not generally return a max-margin separator.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: the same thresholding mechanism should apply to any smooth margin loss whose gradient approaches a hard indicator at large arguments, so other losses may inherit perceptron-like dynamics at large step sizes.
  • Extension: if loss and gradient values are unreliable on separable problems, existing benchmark comparisons of optimizers may be ranking methods by the wrong metric; reporting iterations-to-perfect-accuracy could reorder those rankings.
  • Extension: a formal measure-theoretic statement that degenerate datasets (Assumption 3.1 failures) have probability zero under any continuous data distribution would strengthen the paper's 'almost all datasets' claim without changing the argument.
  • Extension: on imbalanced data, where the paper observes normalized variants winning, an adaptive per-class normalization of the step might yield further iteration-count gains; this is directly testable with the paper's experimental setup.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 5 minor

Summary. The paper analyzes gradient descent on logistic loss (LR+GD) for linearly separable data and shows that, as the step size γ tends to infinity, the normalized iterates θ_t/γ converge pointwise to the iterates of a batch perceptron (Theorem 3.2). This gives a simple mechanistic explanation for the empirically observed ability of LR+GD with very large step sizes to solve classification, and it identifies the limiting iteration count as nR²/μ². The paper further shows that logistic loss and gradient norms can be misleading indicators of progress (Theorem 4.1), proves a lower bound of Ω(n) iterations for the batch perceptron on a worst-case dataset (Theorem 5.1), and proposes Normalized LR+GD, with an iteration bound of R²/μ² + O(log n/(γμ²)) (Theorem 6.1). Numerical experiments on CIFAR-10, FashionMNIST, EuroSAT, and MNIST support the claimed alignment between large-step LR+GD and the batch perceptron.

Significance. If correct, the paper provides a clean and novel reduction of large-step-size logistic regression to the perceptron, connecting edge-of-stability behavior to a classical algorithm. The main theorem has a short, self-contained proof; the proof of Theorem 6.1 is a transparent potential argument; and the classical perceptron bound is reproved in the appendix. The paper also makes a useful conceptual point about evaluating optimization methods for classification by iteration count rather than by function values. No fitted parameters enter the theoretical claims, and the experiments are described in enough detail to be reproduced. The result should be of interest to the machine-learning theory community.

major comments (1)
  1. [Section 5.1, proof of Theorem 5.1] The case analysis misidentifies which class is misclassified. After θ̂1 = (0.25, (n−2)/(2n)), the misclassified sample is the class +1 sample (0.5,−1), not the class −1 sample; the displayed update +(1/n)(0.5,−1) is y_i a_i for the class +1 sample. The later stopping conditions are likewise written for the wrong class. The recurrence for θ̂_t is correct, and the Ω(n) bound follows once the labels are corrected, but as written the proof is not internally consistent.
minor comments (5)
  1. [Section 3, after Theorems 3.2 and 3.3] The transfer from pointwise convergence to a statement about finite γ is not spelled out. Since the batch perceptron terminates at a finite τ, one only needs the pointwise limit at t=τ plus the fact that θ̂_τ has positive margin on every sample; no uniformity in t is required. Please add this one-sentence argument to remove any appearance of a gap.
  2. [Section 6, paragraph before Theorem 6.1] The claimed limit '∇f(θ_t) → 1/|S_t| ∑_{i∈S_t} y_i a_i' is incorrect. From (3), ∇f(θ_t) → −(1/n)∑_{i∈S_t} y_i a_i and β_t → |S_t|, so the quantity that converges to 1/|S_t|∑_{i∈S_t} y_i a_i is −β_t∇f(θ_t), not ∇f(θ_t). Please correct the sign and normalization in this motivating statement.
  3. [Theorem 4.1 statement] The assumption 'θ1 = 0' does not match the proof, where θ0=0 and θ1 is computed explicitly. Please change it to θ0=0.
  4. [Theorem 5.1 proof, final paragraph] The displayed stopping condition involving 0.5×0.25(...) + (−1)×(n−2k)/(2n) > 0 is the margin of the positive sample, not of the negative sample; please correct the attribution or the expression.
  5. [Assumption 3.1] Consider noting explicitly that for θ̂_t = (1/n)∑_{i=1}^n (0.5+k_i)y_i a_i, the condition y_j a_j^T θ̂_t ≠ 0 is exactly the non-intersection of the hyperplane with (0.5+k_1, ..., 0.5+k_n), which will help readers connect the assumption to the proof.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the perceptron-limit reduction is derived from the explicit gradient formula, and the auxiliary convergence proofs are self-contained.

full rationale

The central claim (Theorem 3.2) is a first-principles limit computation. The paper writes the exact LR+GD update as theta_{t+1}/gamma = theta_t/gamma + (1/n) sum_i [1/(1+exp(gamma * y_i a_i^T (theta_t/gamma)))] y_i a_i, and since each sigmoid factor converges to 1 or 0 according to the sign of y_i a_i^T hat_theta_t, the update converges to the Batch Perceptron update. The limit is proved for each fixed t and does not presuppose the conclusion; Assumption 3.1 only excludes boundary ties, and the paper notes it holds for almost all datasets. The transfer to finite large gamma is not circular: for any dataset, the Batch Perceptron terminates at some finite tau <= nR^2/mu^2, and because the limit is pointwise at each of the finitely many steps 1,...,tau, a single finite gamma threshold suffices. The perceptron iteration bound (Theorem 3.3) is reproved from scratch in Appendix B using the margin assumption, not imported from the authors' own work. The Normalized LR+GD result (Theorem 6.1) is proven directly with a potential-function argument in Appendix C, and the lower bound for Batch Perceptron (Theorem 5.1) is an explicit worst-case construction. No fitted parameter is relabeled as a prediction, no load-bearing premise is justified only by self-citation, and no known result is merely renamed. Therefore the derivation chain is self-contained and no circular step is present.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The central limit theorem depends on the separability assumption and a non-degeneracy condition; Normalized LR+GD is a new algorithm but not an invented entity. No free parameters are fitted to data; γ is an explicit step size hyperparameter.

assumptions (2)
  • domain assumption Assumption 1.1: the dataset is linearly separable with margin μ > 0.
    The entire analysis is restricted to separable data; the perceptron and margin arguments require μ > 0.
  • ad hoc to paper Assumption 3.1: no data point lies exactly on a batch perceptron decision boundary at any iterate, i.e., the hyperplane in the statement avoids all points (0.5+k_1,...,0.5+k_n).
    Introduced solely to make the sigmoid factors converge to hard thresholds in the proof of Theorem 3.2; the paper argues it holds for almost all datasets.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Logistic Regression to the Perceptron Algorithm: Exploring Gradient Descent with Large Step Sizes." pith.science (2026). https://pith.science/paper/NLTMMNO4

@misc{pith2026241208424,
  author       = {Pith},
  title        = {Pith review of: From Logistic Regression to the Perceptron Algorithm: Exploring Gradient Descent with Large Step Sizes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NLTMMNO4}},
  note         = {Machine review of arXiv:2412.08424}
}
abstract

We focus on the classification problem with a separable dataset, one of the most important and classical problems from machine learning. The standard approach to this task is logistic regression with gradient descent (LR+GD). Recent studies have observed that LR+GD can find a solution with arbitrarily large step sizes, defying conventional optimization theory. Our work investigates this phenomenon and makes three interconnected key observations about LR+GD with large step sizes. First, we find a remarkably simple explanation of why LR+GD with large step sizes solves the classification problem: LR+GD reduces to a batch version of the celebrated perceptron algorithm when the step size $\gamma \to \infty.$ Second, we observe that larger step sizes lead LR+GD to higher logistic losses when it tends to the perceptron algorithm, but larger step sizes also lead to faster convergence to a solution for the classification problem, meaning that logistic loss is an unreliable metric of the proximity to a solution. Surprisingly, high loss values can actually indicate faster convergence. Third, since the convergence rate in terms of loss function values of LR+GD is unreliable, we examine the iteration complexity required by LR+GD with large step sizes to solve the classification problem and prove that this complexity is suboptimal. To address this, we propose a new method, Normalized LR+GD - based on the connection between LR+GD and the perceptron algorithm - with much better theoretical guarantees.

Figures

Figures reproduced from arXiv: 2412.08424 by the authors.

Figure 1
Figure 1. Illustration on a subset of CIFAR-10 dataset (Krizhevsky, Hinton et al. 2009) with 5 000 samples and two classes. We run LR+GD with various step sizes. Note that there is no randomness involved in the process. Oscilla￾tion is a natural behavior of LR+GD with separable data and large step sizes. When we test this fact, Theorem 3.2, with numerical ex￾periments (see [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. We show that LR+GD with large step sizes aligns with Batch Perceptron on CIFAR-10. 0 1000 2000 3000 4000 5000 # of iterations 0.90 0.92 0.94 0.96 0.98 1.00 Accuracy LR+GD Step: 100.0 LR+GD Step: 10.0 LR+GD Step: 1.0 LR+GD Step: 0.1 LR+GD Step: 0.01 LR+GD Step: 0.001 0 1000 2000 3000 4000 5000 # of iterations 0.90 0.92 0.94 0.96 0.98 1.00 Accuracy Batch Perceptron [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. We show that LR+GD with large step sizes aligns with Batch Perceptron on FashionMNIST. Theorem 3.3 and Theorem 3.2 explain why LR+GD solves (1) with γ → ∞. Notice that the convergence rate (4) does not degenerate when γ → ∞. Crucially, we provide the convergence guarantees for the task (1), not for the task (2). The latter is merely a proxy problem. In practice, what matters is how fast we find a separator rather th… view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Accuracy, function values, and the norm of gradients of the logistic loss ( [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Accuracy, function values, and the norm of gradients on [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Comparison of perceptron algorithms. 0 100 200 300 400 500 600 # of iterations 0.98 0.99 1.00 1.01 Accuracy Batch Perceptron Normalized Batch Perceptron (a) Fashion MNIST 0 2000 4000 6000 8000 10000 # of iterations 0.98 0.99 1.00 1.01 Accuracy Batch Perceptron Normaliz…
Figure 7
Figure 7. Figure 7: Comparison of perceptron algorithms on imbalanced data (see details in Section E). On two of three datasets Nor￾malized Batch Perceptron converges to Accuracy=1.0 faster than Batch Perceptron. iterations if ˆθ0 = 0. One can see that Perceptron and Normalized Batch Per￾…
Figure 8
Figure 8. Figure 8: We show that LR+GD with large step sizes aligns with Batch Perceptron on EuroSAT. 0 50000 100000 150000 200000 250000 300000 350000 400000 # of iterations 0.80 0.85 0.90 0.95 1.00 Accuracy LR+GD Step: 100.0 LR+GD Step: 10.0 LR+GD Step: 1.0 LR+GD Step: 0.1 LR+GD Step: 0…
Figure 9
Figure 9. Figure 9: Accuracy, function values, and the norm of gradients of the logistic loss ( [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: We show that LR+GD with large step sizes aligns with Batch Perceptron on MNIST. 0 200 400 600 800 1000 # of iterations 0.98 0.99 1.00 1.01 Accuracy LR+GD Step: 100.0 LR+GD Step: 10.0 LR+GD Step: 1.0 LR+GD Step: 0.1 LR+GD Step: 0.01 LR+GD Step: 0.001 0 200 400 600 800 …
Figure 11
Figure 11. Figure 11: Accuracy, function values, and the norm of gradients of the logistic loss ( [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: In these plots we repeat the experiments from Figure [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Accuracy, function values, and the norm of gradients of the logistic loss ( [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 16
Figure 16. Figure 16: In these plots we repeat the experiments from Figure [PITH_FULL_IMAGE:figures/full_fig_p014_16.png]
Figure 17
Figure 17. Figure 17: Accuracy, function values, and the norm of gradients of the logistic loss ( [PITH_FULL_IMAGE:figures/full_fig_p014_17.png]
Figure 14
Figure 14. Figure 14: In these plots we repeat the experiments from Figure [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]
Figure 15
Figure 15. Figure 15: Accuracy, function values, and the norm of gradients of the logistic loss ( [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]
Figure 18
Figure 18. Figure 18: In these plots we repeat the experiments from Figure [PITH_FULL_IMAGE:figures/full_fig_p015_18.png]
Figure 19
Figure 19. Figure 19: Accuracy, function values, and the norm of gradients of the logistic loss ( [PITH_FULL_IMAGE:figures/full_fig_p015_19.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 23 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    T.; Suarez, F.; and Zhang, Y

    Ahn, K.; Bubeck, S.; Chewi, S.; Lee, Y. T.; Suarez, F.; and Zhang, Y. 2024. Learning threshold neurons via edge of stability. Advances in Neural Information Processing Systems, 36

    2024

  4. [4]

    Ahn, K.; Zhang, J.; and Sra, S. 2022. Understanding the unstable convergence of gradient descent. In International Conference on Machine Learning, 247--257. PMLR

    work page 2022

  5. [5]

    M.; and Nasrabadi, N

    Bishop, C. M.; and Nasrabadi, N. M. 2006. Pattern recognition and machine learning, volume 4. Springer

    2006

  6. [6]

    Block, H.-D. 1962. The perceptron: A model for brain functioning. i. Reviews of Modern Physics, 34(1): 123

    work page 1962

  7. [7]

    B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al

    Brown, T. B.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  8. [8]

    C.; Hinder, O.; and Sidford, A

    Carmon, Y.; Duchi, J. C.; Hinder, O.; and Sidford, A. 2020. Lower bounds for finding stationary points I. Mathematical Programming, 184(1): 71--120

    work page 2020

Show all 38 references
  1. [9]

    Chen, L.; and Bruna, J. 2022. On gradient descent convergence beyond the edge of stability. arXiv preprint arXiv:2206.04172, 3

    2022 arXiv

  2. [10]

    Z.; and Talwalkar, A

    Cohen, J.; Kaur, S.; Li, Y.; Kolter, J. Z.; and Talwalkar, A. 2021. Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability. In International Conference on Learning Representations

    2021

  3. [11]

    Cortes, C.; and Vapnik, V. 1995. Support-vector networks. Machine learning, 20: 273--297

    1995

  4. [12]

    Damian, A.; Nichani, E.; and Lee, J. D. 2023. Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability. In The Eleventh International Conference on Learning Representations

    2023

  5. [13]

    Duda, R.; Hart, P.; and G.Stork, D. 2001. Pattern Classification. Wiley Interscience

    2001

  6. [14]

    Even, M.; Pesme, S.; Gunasekar, S.; and Flammarion, N. 2023. (S)GD over Diagonal Linear Networks: Implicit bias, Large Stepsizes and Edge of Stability. Advances in Neural Information Processing Systems, 36: 29406--29448

    2023

  7. [15]

    Helber, P.; Bischke, B.; Dengel, A.; and Borth, D. 2019. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7): 2217--2226

    2019

  8. [16]

    Ji, Z.; and Telgarsky, M. 2018. Risk and parameter convergence of logistic regression. arXiv preprint arXiv:1803.07300

    2018 arXiv

  9. [17]

    Johnson, R.; and Zhang, T. 2013. Accelerating stochastic gradient descent using predictive variance reduction. Advances in Neural Information Processing Systems, 26

    2013

  10. [18]

    S.; Soudry, D.; and Carmon, Y

    Kreisler, I.; Nacson, M. S.; Soudry, D.; and Carmon, Y. 2023. Gradient descent monotonically decreases the sharpness of gradient flow solutions in scalar networks and beyond. In International Conference on Machine Learning, 17684--17744. PMLR

    2023

  11. [19]

    Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto

    2009

  12. [20]

    LeCun, Y.; Cortes, C.; and Burges, C. 2010. MNIST handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2

    2010

  13. [21]

    Lewkowycz, A.; Bahri, Y.; Dyer, E.; Sohl-Dickstein, J.; and Gur-Ari, G. 2020. The large learning rate phase of deep learning: the catapult mechanism. arXiv preprint arXiv:2003.02218

    2020 arXiv

  14. [22]

    C.; and Nocedal, J

    Liu, D. C.; and Nocedal, J. 1989. On the limited memory BFGS method for large scale optimization. Mathematical programming, 45(1-3): 503--528

    1989

  15. [23]

    Lyu, K.; Li, Z.; and Arora, S. 2022. Understanding the generalization benefit of normalization layers: Sharpness reduction. Advances in Neural Information Processing Systems, 35: 34689--34708

    2022

  16. [24]

    Ma, C.; Kunin, D.; Wu, L.; and Ying, L. 2022. Beyond the quadratic approximation: the multiscale structure of neural network loss landscapes. arXiv preprint arXiv:2204.11326

    2022 arXiv

  17. [25]

    Y.; Orvieto, A.; Cao, D

    Meng, S. Y.; Orvieto, A.; Cao, D. Y.; and De Sa, C. 2024. Gradient Descent on Logistic Regression with Non-Separable Data and Large Step Sizes. arXiv preprint arXiv:2406.05033

    2024 arXiv

  18. [26]

    Nesterov, Y. 1983. A Method for Solving a Convex Programming Problem with Convergence Rate O (1/k**2). In Soviet Mathematics. Doklady, volume 27, 367--372

    1983

  19. [27]

    Nesterov, Y. 2018. Lectures on convex optimization, volume 137. Springer

    2018

  20. [28]

    Novikoff, A. B. 1962. On convergence proofs on perceptrons. In Proceedings of the Symposium on the Mathematical Theory of Automata, volume 12, 615--622. New York, NY

    1962

  21. [29]

    Orabona, F. 2024. A Minimizer Far, Far Away . https://parameterfree.com/2024/02/14/a-minimizer-far-far-away/

    2024

  22. [30]

    Robbins, H.; and Monro, S. 1951. A stochastic approximation method. The Annals of Mathematical Statistics, 400--407

    1951

  23. [31]

    Schmidt, M.; Le Roux, N.; and Bach, F. 2017. Minimizing finite sums with the stochastic average gradient. Mathematical Programming, 162: 83--112

    2017

  24. [32]

    Song, M.; and Yun, C. 2023. Trajectory alignment: understanding the edge of stability phenomenon via bifurcation theory. arXiv preprint arXiv:2307.04204

    2023 arXiv

  25. [33]

    S.; Gunasekar, S.; and Srebro, N

    Soudry, D.; Hoffer, E.; Nacson, M. S.; Gunasekar, S.; and Srebro, N. 2018. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1): 2822--2878

    2018

  26. [34]

    Wang, Z.; Li, Z.; and Li, J. 2022. Analyzing sharpness along GD trajectory: Progressive sharpening and edge of stability. Advances in Neural Information Processing Systems, 35: 9983--9994

    2022

  27. [35]

    L.; Telgarsky, M.; and Yu, B

    Wu, J.; Bartlett, P. L.; Telgarsky, M.; and Yu, B. 2024. Large Stepsize Gradient Descent for Logistic Loss: Non-Monotonicity of the Loss Improves Optimization Efficiency. In Conference on Learning Theory

    2024

  28. [36]

    Wu, J.; Braverman, V.; and Lee, J. D. 2024. Implicit bias of gradient descent for logistic regression at the edge of stability. Advances in Neural Information Processing Systems, 36

    2024

  29. [37]

    Xiao, H.; Rasul, K.; and Vollgraf, R. 2017. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747

    2017 arXiv

  30. [38]

    Zhu, X.; Wang, Z.; Wang, X.; Zhou, M.; and Ge, R. 2022. Understanding edge-of-stability training dynamics with a minimalist example. arXiv preprint arXiv:2210.03294

    2022 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.