Pith. sign in

REVIEW 4 major objections 4 minor 19 references

Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Pith's one-line read This paper claims that every deep-network training objective is both generalized-convex and generalized-smooth, making the optimal learning rate of its generalized gradient descent exactly 1.

desk verdict A coherent framework undone by a false central premise: no translation-invariant Legendre energies can bound Taylor remainders of DNN objectives on unbounded parameter spaces, so the convergence theorems do not apply as stated. read the letter →

arxiv 2608.09523 v1 pith:SG5S3644 submitted 2026-08-10 cs.LG

classification cs.LG MSC 90C2690C2568T07
keywords generalizedconvexitysmoothnessLegendrefunctionsconjugatedualitygradientenergyDNNtrainingtheorySGDconvergenceanalysisJacobiansingularvalues
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's project is to explain why stochastic gradient descent reliably trains deep neural networks whose objectives are neither convex nor smooth in the classical sense. It replaces the quadratic energy that underlies strong convexity and Lipschitz smoothness with a broad family of Legendre energy functions, defining $H(\psi)$-convexity and $H(\Psi)$-smoothness through upper and lower bounds on the first-order Taylor remainder $S_G(\mu,\nu)$. The central claim is that every DNN training objective is both $H(\Psi)$-smooth and $H(\psi)$-convex, that generalized GD then has optimal learning rate exactly 1, and that the empirical risk is governed jointly by the gradient energy $\Psi^*(\nabla G(\theta))$ and the extreme singular values of the network Jacobian. If these claims hold, the framework unifies convex and non-convex, smooth and non-smooth objectives in one duality, and it would offer a principled account of why plain SGD finds low-risk solutions on real networks.

What carries the argument

The central object is the energy function: a Legendre function $\Psi$ with radial representation $\Psi(\mu)=\Psi_\circ(\|\mu\|)$ and its unique minimum at the origin, whose convex conjugate $\Psi^*$ is again an energy function. The generalized properties are defined by sandwiching the first-order Taylor remainder $S_G(\mu,\nu)=G(\mu)-G(\nu)-\langle\nabla G(\nu),\mu-\nu\rangle$ between $\psi(\mu-\nu)$ and $\Psi(\mu-\nu)$; the constant terms $\psi(0)$ and $\Psi(0)$ carry the degree of non-convexity and non-smoothness. The argument's engine is conjugate duality: the generalized GD update $\theta_{k+1}:=\theta_k-\alpha(\nabla G(\theta_k))^*_{\Psi^*}$ uses $\nabla\Psi^*$ to map the gradient to a dual direction, and the gradient energy $\Psi^*(\nabla G(\theta))$ measures progress. The Fenchel–Young identity turns the dual update into a descent inequality that makes $\alpha=1$ optimal, while the chain rule turns the composite objective's bounds into statements about the Jacobian's upper and lower induced norms.

What would settle it

A concrete way to settle the claim: take a small network, such as a two-parameter ReLU network with squared loss on a fixed dataset, and test the two inequalities $S_G(\mu,\nu)\leq \Psi(\mu-\nu)$ and $S_G(\mu,\nu)\geq \psi(\mu-\nu)$ on a dense grid of parameter pairs spanning unbounded directions. Because $\psi$ and $\Psi$ are fixed convex radial functions, a single pair where the remainder violates either inequality for every admissible energy candidate refutes Proposition 3.3 for that objective, and with it the universality that Theorems 4.1, 4.2, and 5.1 assume. A cheaper probe is to move along a direction of negative curvature and measure whether $S_G(\mu,\nu)$ grows negatively without bound; if it does while $\psi(0)$ must stay near zero, no convex radial lower bound can hold, and the framework's applicability to that network collapses.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that classical strong convexity and Lipschitz smoothness are two halves of one duality: both bound how fast a function can vary through the quadratic term $\|\cdot\|_2^2$, and convex conjugation converts one half into the other. Replacing that quadratic with a Legendre energy function $\Psi$ (radial, minimized at the origin) yields generalized properties: $G$ is $H(\Psi)$-smooth when $S_G(\mu,\nu) \leq \Psi(\mu-\nu)$ and $H(\psi)$-convex when $S_G(\mu,\nu) \geq \psi(\mu-\nu)$. The paper proves the duality that $G$ is $H(\Psi)$-smooth if and only if its convex conjugate $G^*$ is $H(\Psi^*)$-convex, and proves Theorem 3.1, which traps the suboptimality gap $G(\mu)-G^*$ between $\Psi^*(\nabla G(\mu))$ and $\psi^*(\nabla G(\mu))$. Generalized GD and SGD move along the conjugate-dual direction of the gradient, which yields a monotone descent inequality and forces the optimal learning rate to be $\alpha=1$. For the composite structure of DNN training, the paper shows that convergence reduces to jointly decreasing the gradient energy and controlling the induced norm (extreme singular values) of the Jacobian, with explicit risk bounds for MSE, Softmax CrossEntropy, and Fenchel–Young losses, and architecture-level consequences for width, initialization, and skip connections.

Load-bearing premise

The load-bearing premise is Proposition 3.3: for every neural-network training objective there exist global convex Legendre energies $\Psi$ and $\psi$ such that the Taylor remainder obeys $\psi(\mu-\nu)\leq S_G(\mu,\nu)\leq \Psi(\mu-\nu)$ for every pair of parameter values, with $\psi(0)$ and $\Psi(0)$ close enough to zero to keep the bounds useful. The paper asserts this without construction, and the heuristic in Section 3.4 of increasing the constants $a$, $r$, and $c$ cannot certify a convex lower bound on a non-convex remainder over an unbounded parameter space; every convergence theorem inherits its applicability from this assumption.

Editorial extensions

If this is right

  • Generalized GD with step size $\alpha=1$ is monotonically descending in the sense $G(\theta_{k+1})\leq G(\theta_k)-\Psi^*(\nabla G(\theta_k))$, reaching gradient energy below $\Psi^*(\varepsilon)$ in $O(1/\Psi^*(\varepsilon))$ iterations.
  • With quadratic energies the framework recovers classical GD and SGD: optimal learning rate $1/L$, the classical condition number $L/\sigma$, and the standard $O(\kappa\log(1/\varepsilon))$ strongly-convex rates.
  • For SGD, the achievable gradient-energy floor is set by $(n-m)M/m$ and the suboptimality by the model capacity risk $R_G(s,m)$, predicting that batch size trades convergence speed against attainable solution quality, consistent with batch-size scheduling in practice.
  • For ERM with MSE or Softmax CrossEntropy, the empirical risk is sandwiched between gradient energy scaled by the squared extreme singular values of the Jacobian, e.g. $\|\nabla_\theta L\|_2^2/(2\sigma_{\max}^2)\leq L(\theta,z)\leq \|\nabla_\theta L\|_2^2/(2\sigma_{\min}^2)$.
  • Architectural design is reinterpreted as spectral control: over-parameterization and random initialization shrink $\sigma_{\max}/\sigma_{\min}$ of the Jacobian, while skip connections prevent its extreme singular values from decaying with depth, keeping the risk bounds tight in deep networks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The framework's usefulness will hinge on how tightly $\psi(0)$ and $\Psi(0)$ can be pinned: overly loose choices make the bounds vacuous, and the paper's own conjecture that finding the globally optimal $\Psi$ is NP-hard suggests that usable constants for real networks will come from architecture-specific construction, not from global computation.
  • Because classical GD is generalized GD only when the update direction is the gradient, the theory suggests interpreting adaptive optimizers such as Adam as empirical approximations of the conjugate-dual direction; if so, learning-rate schedules in practice could be understood as surrogate dual steps.
  • A natural testable extension is to connect the Jacobian spectral bounds to flat-minima geometry: minima with large $\sigma_{\min}(J)$ would automatically have tight risk bounds, giving a spectral characterization of generalization-favoring flatness that the paper does not pursue.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper proposes a generalized notion of convexity and smoothness based on Legendre energy functions: H(Ψ)-smoothness and H(ψ)-convexity (Definition 3.2). It then defines generalized GD and SGD through conjugate directions (Definitions 4.1 and 4.2), claims that generalized GD has an optimal learning rate of exactly 1 (Theorem 4.1), and derives convergence rates expressed in terms of gradient energy. A composite optimization formulation is used to argue that DNN training is governed by two quantities: gradient energy and the induced norm of the network Jacobian (Theorem 5.1, Corollaries 5.2 and 5.3). The empirical section reports correlations between training risk and the components of the proposed bounds across many architectures, datasets, optimizers, and losses. The central enabling result is Proposition 3.3, which asserts without proof that every DNN training objective is both H(Ψ)-smooth and H(ψ)-convex in the parameters.

Significance. If Proposition 3.3 were valid, the framework would indeed be broad: it would unify convex and nonconvex, smooth and nonsmooth objectives, and the duality in Proposition 3.2 is a mathematically natural extension of classical strong convexity/Lipschitz smoothness. The experimental campaign is extensive, covering multiple architectures, datasets, optimizers, and loss functions, and the appendix supplies detailed proofs for many auxiliary statements. However, the paper's central premise is asserted rather than proved, and it fails for non-quadratic objectives on unbounded parameter spaces. Since Theorems 4.1, 4.2, and 5.1 all inherit their applicability from Proposition 3.3, the convergence and optimal-learning-rate claims do not apply to general DNN objectives as stated. The claimed optimal learning rate of 1 is also a normalization built into the definition of the update direction rather than a fitted or predicted constant.

major comments (4)
  1. [Section 3.2 (Proposition 3.3) and Section 3.4] Proposition 3.3 is stated without proof and is not true for non-quadratic objectives on unbounded domains. Definition 3.2 requires the Taylor remainder S_G(µ,ν) to be bounded above and below by functions of the difference µ−ν only. For G(w)=w^4, S_G(µ,ν)=(µ−ν)(µ^3+µ^2ν+µν^2−3ν^3); fixing δ=µ−ν and letting ν grow shows S_G grows like 6ν^2δ, which is unbounded. No finite radial energy function Ψ(δ) can upper-bound this remainder, and no convex radial ψ(δ) can lower-bound it when the remainder is negative and unbounded below. The heuristic in Section 3.4 of increasing a_Ψ, r_Ψ, and c_Ψ cannot repair this. ReLU network objectives contain similar multiplicative interactions, so the proposition fails in exactly the regime the paper targets, and Theorems 4.1, 4.2, and 5.1 do not apply to general DNN objectives as stated.
  2. [Definition 4.1 and Theorem 4.1] The claim that the optimal learning rate of generalized GD is exactly 1 is a normalization choice rather than a discovered property. The update direction in Eq. (14) is ∇Ψ*(∇G(θ_k)); when Ψ(·)=L/2‖·‖^2, this direction is ∇G(θ_k)/L, so α=1 corresponds to the classical step size 1/L. Corollary 4.1 confirms this explicitly. Thus the statement that the optimal learning rate is identically 1 is built into the definition of the conjugate direction, and it does not eliminate learning-rate selection for classical GD as the abstract suggests.
  3. [Theorems 4.1/4.2, Sections 3.4 and 4.2] The convergence rates are expressed in terms of Ψ and ψ, which are chosen after the fact to majorize or minorize the Taylor remainder. Because any valid Ψ can be scaled (Proposition 3.2), the rates O(1/Ψ*(ε)) are not intrinsic properties of the objective; Section 3.4 concedes that computing a tight Ψ for a DNN is NP-hard. The rates therefore do not yield quantitative a priori predictions for DNN training. The empirical section only reports local Pearson correlations between the risk and the bound components, not the actual rate constants or the strict inequalities claimed in Corollary 5.3.
  4. [Theorem 4.2 and Definition 4.3] The gradient correlation factor M defined in Eq. (25) is a maximum over the actual optimization trajectory, and the paper acknowledges it is generally intractable. The iteration bounds in Theorem 4.2 are expressed in terms of M, so they cannot be evaluated or empirically verified without an additional a priori bound on M. This is a load-bearing gap in the SGD convergence claims, not merely a presentation issue.
minor comments (4)
  1. [Figures 2–18] The rendered figures contain unicode substitution artifacts such as "/uni00000013/uni00000018" in axis labels and legends, making several plots unreadable; they need to be regenerated.
  2. [Section 6.3.2] The experiments validate the framework by measuring local Pearson correlations between log risk and bound components rather than checking whether the strict inequalities in Corollary 5.3 hold; the text should state this limitation explicitly.
  3. [Theorem 5.1 / Corollary 5.3] The upper bounds involve ‖J_θh_θ(z)^+‖ and divisions by σ_min(J_θf_θ(z)); these are finite only under a full-column-rank condition on the Jacobian, which is not stated in Definition 5.3. When σ_min=0 the bounds are vacuous, so the conditions under which the bounds are non-trivial should be made explicit.
  4. [Section 2.2] Setting the surrogate gradient to zero at non-differentiable points makes the statement 'it ensures that the assumption of zero gradient at the global optimum is satisfied' a convention, not a property of the objective; this should be clarified because a nonsmooth global minimizer may have nonzero subgradients.

Circularity Check

3 steps flagged · score 8.0 of 10

DNN-wide H(Ψ)-smoothness and H(ψ)-convexity are asserted by choosing the energy functions after the fact, and the 'optimal learning rate of exactly 1' is a renormalization of the step into the conjugate direction, so the central convergence claims reduce to the definitions.

  1. self definitional [Section 3.2, heuristic paragraph before Proposition 3.3 (and Proposition 3.3 itself)]
    "More generally, for a general function G(µ), the inequality S_G(µ,ν)≤(a_Ψ∥µ−ν∥_2)^{r_Ψ}+c_Ψ can be satisfied by increasing a_Ψ, r_Ψ, and c_Ψ, so that G(µ) is H(Ψ)-smooth. The construction of ψ under H(ψ)-convexity is analogous. ... From this, we obtain the following proposition: Proposition 3.3 The objective function corresponding to neural network training is both H(Ψ)-smooth and H(ψ)-convex with respect to the parameters."

    H(Ψ)-smoothness and H(ψ)-convexity (Definition 3.2) are defined by the existence of energy functions bounding the first-order Taylor remainder S_G. The proposition that every DNN objective has these properties is not derived from the architecture or loss; it is obtained directly from the assertion that the required inequality 'can be satisfied by increasing a_Ψ, r_Ψ, and c_Ψ'. That is, the property is imposed by choosing the bounding energy rather than established over the unbounded parameter domain. Since Theorems 4.1, 4.2, and 5.1 all assume Proposition 3.3, the convergence conclusions inherit a definitional input in place of an independent premise.

  2. renaming known result [Section 4.1.1 Definition 4.1, Section 4.1.2 Theorem 4.1 and Proposition 4.2]
    "The generalized GD method with constant step size α then follows the update rule θ_{k+1} := θ_k − α(∇G(θ_k))^*_{Ψ^*}, where (∇G(θ_k))^*_{Ψ^*} := ∇Ψ^*(∇G(θ_k)). ... Theorem 4.1: The optimal learning rate is given by α=1. ... Proposition 4.2: If a time-varying learning rate is employed, the classical GD is equivalent to generalized GD."

    The update direction is defined as the conjugate direction ∇Ψ^*(∇G(θ)), and the step size α multiplies this direction. For the quadratic energy Ψ(·) = (L/2)∥·∥^2, ∇Ψ^*(g) = g/L, so generalized GD with α=1 is exactly classical GD with step 1/L. The 'optimal learning rate of exactly 1' is therefore the unit of a reparametrized step, not an independently determined constant: the Lipschitz/curvature information has been absorbed into Ψ. Corollary 4.1 makes the reduction explicit by recovering α=1/L for classical GD. Consequently the headline prediction is a normalization choice, and the experiments in Section 6.2 verify it on objectives for which Φ was precomputed as a tight bound (Lemma 6.1).

1 more flagged steps
  1. fitted input called prediction [Section 3.4 Proposition 3.4 and Section 4.1.2 Theorem 4.1]
    "Proposition 3.4: For (ψ,Ψ) to be useful in optimization analysis, the following two construction principles should be satisfied: 1. ψ(0) and Ψ(0) should be as close to 0 as possible; 2. The pair (Ψ, ψ) should provide tight upper and lower bounds for the remainder S_G(θ,η). ... Theorem 4.1: The number of iterations required ... is bounded by T=O(1/Ψ^*(ε))."

    The convergence complexity is stated in terms of the gradient energy Ψ^*, and Ψ is the same function that the paper instructs us to construct so that it tightly bounds S_G. Hence the rate T=O(1/Ψ^*(ε)) is a restatement of the defining smoothness inequality, not an independent quantitative prediction about DNN training. For DNNs the paper concedes that the globally optimal Ψ is 'extremely challenging' to compute and conjecturally NP-hard, so the bound cannot be evaluated without reintroducing the very energy function whose existence Proposition 3.3 asserted by construction.

full rationale

The paper's central universality claim, Proposition 3.3, is not derived from the structure of neural network objectives; it is obtained by declaring that the defining inequality 'can be satisfied by increasing a_Ψ, r_Ψ, and c_Ψ'. Since H(Ψ)-smoothness and H(ψ)-convexity are defined by existence of such energy functions, this makes every DNN objective smooth and convex within the framework by construction rather than by proof. All downstream convergence theorems (4.1, 4.2, 5.1) inherit this definitional input. Separately, the 'optimal learning rate of exactly 1' is a reparametrization of the classical 1/L step: the conjugate direction ∇Ψ^*(∇G) absorbs the curvature constant, so α=1 is the unit of a rescaled coordinate, as Proposition 4.2 and Corollary 4.1 explicitly show. The convergence rates are also tautological with respect to the chosen Ψ: Theorem 4.1 states T=O(1/Ψ^*(ε)) where Ψ is the same post-hoc chosen majorant of the Taylor remainder. The paper itself acknowledges that globally optimal Ψ is extremely challenging to compute and conjecturally NP-hard, so the bounds cannot be evaluated independently for DNNs. The self-citations to Qi et al. (2025a,b) occur only in the related-work and motivation discussion and are not load-bearing, so they do not raise the score further. The Fenchel–Young duality identities and the Jacobian-spectral inequalities are mathematically real conditional statements, which is why the score is 8 rather than 10, but the paper's advertised claims about DNN trainability and the universal learning rate are forced by definition rather than established by independent content.

Assumptions & free parameters 4 free parameters · 5 assumptions · 2 invented entities

The theory is conditional on a pair (ψ,Ψ) that is never constructed for a real network. Two additional unmeasured quantities, the gradient correlation factor M and the model capacity risk R_G, enter the SGD and composite bounds. This means the quantitative convergence claims are not directly checkable on DNNs. The Jacobian-column independence assumption in Proposition 5.1 is another strong postulate with no evidence.

free parameters (4)
  • Energy functions Ψ and ψ for DNN objectives = unspecified for real networks
    Every theorem is conditional on a chosen pair (ψ,Ψ). For DNNs the paper never constructs or computes them; Section 3.4 only gives qualitative principles, and the paper admits computing the globally optimal Ψ is NP-hard.
  • Gradient correlation factor M = not estimated
    Definition 4.3 introduces M to bound off-batch loss changes. Theorem 4.2 rates depend on M, but the paper provides no estimator or measurement procedure for it.
  • Model capacity risk R_G(s,m) = not estimated
    Definition 4.4 abstracts the gap between full-set and batch-optimal losses. It appears in the SGD and composite bounds, but there is no method to compute it for a given network and dataset.
  • Jacobian column radius ϵ in Proposition 5.1 = not related to initialization
    Proposition 5.1 assumes columns of the Jacobian are drawn uniformly from a ball of radius ϵ. The value of ϵ is never connected to initialization scale, width, or depth.
assumptions (5)
  • ad hoc to paper Every DNN objective is H(Ψ)-smooth and H(ψ)-convex for some Legendre energy pair.
    Proposition 3.3 asserts this without proof. The scaling heuristic in Section 3.4 does not establish existence of convex Ψ and ψ globally, especially for nonconvex objectives with negative curvature.
  • ad hoc to paper Approximate gradient independence: columns of the Jacobian of a randomly initialized DNN are i.i.d. uniform in a ball.
    Section 5.3.2, before Proposition 5.1, introduces this as an idealized assumption. It is not verified empirically or theoretically for trained networks.
  • domain assumption Surrogate gradient is set to zero at all non-differentiable points.
    Section 2.2 states this convention for ReLU-type networks. It ensures the global minimum has zero gradient but changes the definition of stationarity.
  • domain assumption The model can fit any single sample, i.e., L(z)* = 0 for all z.
    Corollary 5.3 relies on this to simplify the risk bounds. It is reasonable for overparameterized models with MSE, but for Softmax CrossEntropy the infimum is approached, not attained, with finite logits.
  • domain assumption Input space X is finite.
    Section 2.1 assumes |X| is finite, aligning with empirical risk but limiting the scope of the theoretical statements.
invented entities (2)
  • Gradient correlation factor M
    purpose: Bounds the change in off-batch loss after a mini-batch update and absorbs architecture and data effects into the SGD rate.
    Defined in Definition 4.3 but never measured or estimated for any network or dataset in the experimental section.
  • Model capacity risk R_G(s,m)
    purpose: Abstracts the gap between the optimal full-set loss and expected optimal batch loss, attributing the gap to limited model capacity.
    Defined in Definition 4.4 and used in Theorem 4.2 and Theorem 5.1, but no procedure is given to compute it for a concrete model.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks." pith.science (2026). https://pith.science/paper/SG5S3644

@misc{pith2026260809523,
  author       = {Pith},
  title        = {Pith review of: Generalized Convexity and Smoothness via Conjugate Duality: Optimization Theory for Deep Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SG5S3644}},
  note         = {Machine review of arXiv:2608.09523}
}
abstract

Deep neural network (DNN) training with stochastic gradient descent (SGD) and its variants achieves strong empirical performance, yet classical optimization theory does not fully explain this success. This limitation arises because conventional analyses rely on assumptions such as differentiability, convexity, or smoothness, which are often violated by DNN objectives. In this paper, we establish a unified optimization framework for DNN training by generalizing classical convexity and smoothness through Legendre functions and convex conjugation. Specifically, we introduce $\mathcal{H}(\psi)$-convexity and $\mathcal{H}(\Psi)$-smoothness, which unify convex and non-convex as well as smooth and non-smooth objectives within a single formalism and reveal a natural duality between generalized smoothness and convexity. Building on these generalized properties, we introduce generalized gradient descent (GD) and generalized SGD through convex conjugation. We theoretically prove that generalized GD admits an optimal learning rate of exactly $1$, and derive rigorous gradient-energy-based convergence rates for both proposed optimizers. We further reformulate DNN training as a composite optimization problem, demonstrating that its convergence relies on jointly reducing the gradient energy and controlling the induced norm of the network Jacobian. To characterize the practical influences of network architectures and training configurations, we introduce the gradient correlation factor and model capacity risk, and quantitatively analyze how architectural designs, batch size, and model capacity shape training convergence. Extensive experiments across diverse network architectures, datasets, optimizers, and loss functions validate our theoretical bounds and demonstrate precise alignment between our theoretical predictions and empirical training dynamics.

Figures

Figures reproduced from arXiv: 2608.09523 by the authors.

Figure 1
Figure 1. An overview of the logical relationships among the key definitions and conclusions. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Convergence curves for all r ∈ [1.1, 2.0] under fixed α. Optimal convergence, in terms of both speed and stability, is achieved at α = 1, corresponding to the theoretically optimal learning rate. Deviations from α = 1 reduce convergence speed, stagnation appears at α = 2, and divergence occurs for α > 2, which is consistent with Proposition 4.2. in Proposition 4.2. Complementing these findings, [PITH_FULL_IMAGE:fig… view at source ↗
Figure 3
Figure 3. Convergence curves for fixed r under different values of α. For every tested r, α = 1 yields the best performance, verifying the universality of the optimal learning rate. 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 Iteration k -40 -30 -20 -10 0 10 l o g 2|| F ( k )||2 2 Convergence with optimal learning rate =1 r=1.1 r=1.2 r=1.3 r=1.4 r=1.5 r=1.6 r=1.7 r=1.8 r=1.9 r=2.0 [PITH_FULL_IMAGE:figures/full_fig_p029_3.png] view at source ↗
Figures from the paper (14 more)
Figure 4
Figure 4. Figure 4: Convergence speed under α = 1, the optimal learning rate. Larger r leads to faster convergence. 6.3.1 Experimental Designs To comprehensively monitor the training dynamics and verify the effectiveness of the the￾oretical bounds, the following three core quantities are …
Figure 5
Figure 5. Figure 5: Model architectures and configuration parameters. This figure illustrates the model [PITH_FULL_IMAGE:figures/full_fig_p031_5.png]
Figure 6
Figure 6. Figure 6: Training dynamics of the custom models on MNIST. Left: risk and its upper/lower [PITH_FULL_IMAGE:figures/full_fig_p033_6.png]
Figure 7
Figure 7. Figure 7: Training dynamics of the custom models under the false-label setting. Left: risk and [PITH_FULL_IMAGE:figures/full_fig_p034_7.png]
Figure 8
Figure 8. Figure 8: Training dynamics of LeNet on CIFAR-10 and CIFAR-100. [PITH_FULL_IMAGE:figures/full_fig_p035_8.png]
Figure 9
Figure 9. Figure 9: Training dynamics of ResNet-18 on CIFAR-10 and CIFAR-100. [PITH_FULL_IMAGE:figures/full_fig_p035_9.png]
Figure 10
Figure 10. Figure 10: Training dynamics of Vision Transformer (ViT) on CIFAR-10 and CIFAR-100. [PITH_FULL_IMAGE:figures/full_fig_p036_10.png]
Figure 11
Figure 11. Figure 11: Training dynamics of ResNet-18 on CIFAR-10 under different optimization con [PITH_FULL_IMAGE:figures/full_fig_p037_11.png]
Figure 12
Figure 12. Figure 12: Training dynamics of ResNet-18 on CIFAR-10 using different PyTorch built-in loss [PITH_FULL_IMAGE:figures/full_fig_p038_12.png]
Figure 13
Figure 13. Figure 13: Training dynamics of model a on MNIST using the custom loss family [PITH_FULL_IMAGE:figures/full_fig_p039_13.png]
Figure 14
Figure 14. Figure 14: Training dynamics on text classification datasets (TREC, SST-2). This figure [PITH_FULL_IMAGE:figures/full_fig_p040_14.png]
Figure 15
Figure 15. Figure 15: Training dynamics under different model scales (ResNet variants). This figure [PITH_FULL_IMAGE:figures/full_fig_p041_15.png]
Figure 16
Figure 16. Figure 16: Variation of the extreme singular values with model width. [PITH_FULL_IMAGE:figures/full_fig_p043_16.png]
Figure 17
Figure 17. Figure 17: Variation of the extreme singular values with model depth. [PITH_FULL_IMAGE:figures/full_fig_p043_17.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 6 canonical work pages

  1. [11]

    N. S. Keskar and R. Socher. Improving generalization performance by switching from adam to SGD.CoRR, abs/1712.07628,

  2. [12]

    doi: 10.3115/V1/ D14-1181. M. Kohler. On the rate of convergence of an over-parametrized deep neural network regres- sion estimate learned by gradient descent.IEEE Trans. Inf. Theory, 72(3):1777–1797,

  3. [14]

    H. Li, J. Qian, Y. Tian, A. Rakhlin, and A. Jadbabaie. Convex and non-convex optimization under generalized smoothness. InProc. Adv. Neural Inf. Process. Syst. (NeurIPS), New Orleans, LA, USA, 2023a. Nips. H. Li, J. Qian, Y. Tian, A. Rakhlin, and A. Jadbabaie. Convex and non-convex optimization under generalized smoothness. InNeurIPS, 2023b. 81 Qi X. Li a...

  4. [15]

    Reisizadeh, H

    A. Reisizadeh, H. Li, S. Das, and A. Jadbabaie. Variance-reduced clipping for non-convex optimization.arXiv preprint, abs/2303.00883,

  5. [16]

    doi: 10.1016/ j.spa.2019.06.003

    ISSN 0304-4149. doi: 10.1016/ j.spa.2019.06.003. R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Potts. Recursive deep models for semantic compositionality over a sentiment treebank. InProc. Conf. Empirical Methods Nat. Lang. Process. (EMNLP), pages 1631–1642. ACL,

  6. [18]

    N. Vyas, Y. Bansal, and P. Nakkiran. Empirical limitations of the NTK for understanding scaling laws in deep learning.Trans. Mach. Learn. Res., 2023,

  7. [1998]

    doi: 10.1109/5. 726791. S. Lee, C. He, and S. Avestimehr. Achieving small-batch accuracy with large-batch scala- bility via hessian-aware learning rate adjustment.Neural Networks, 158:1–14,

  8. [2001]

    doi: 10.1109/9780470544976.ch9. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), pages 770–778, Las Vegas, NV, USA,

Show all 19 references
  1. [2006]

    doi: 10.1109/TIT.2005.864431. A. Cutkosky and F. Orabona. Momentum-based variance reduction in non-convex SGD. InProc. Adv. Neural Inf. Process. Syst. (NeurIPS), pages 15210–15219, Vancouver, BC, Canada,

  2. [2009]

    Karhadkar, M

    K. Karhadkar, M. Murray, H. Tseran, and G. Mont´ ufar. Mildly overparameterized relu networks have a favorable loss landscape.Trans. Mach. Learn. Res., 2024,

  3. [2013]

    doi: 10.18653/V1/D13-1170. T. Str¨ omberg. A note on the differentiability of conjugate functions.Archiv der Mathematik, 93:481–485,

  4. [2014]

    doi: 10.1017/CBO9780511804441

    ISBN 978-0-521-83378-3. doi: 10.1017/CBO9780511804441. Y. Carmon, J. C. Duchi, O. Hinder, and A. Sidford. Lower bounds for finding stationary points I.Math. Program., 184(1):71–120,

  5. [2016]

    doi: 10.1109/CVPR.2016.90

    IEEE Computer Society. doi: 10.1109/CVPR.2016.90. H. Hu. Chinese-text-classification-pytorch. GitHub Repository,https://github.com/ kentaroy47/vision-transformers-cifar10, Mar

  6. [2018]

    doi: 10.1137/16M1080173. S. P. Boyd and L. Vandenberghe.Convex Optimization. Cambridge University Press,

  7. [2019]

    nips. H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.CoRR, abs/1708.07747,

  8. [2020]

    doi: 10.1007/S10107-019-01406-Y. S. Chen, Z. Li, and Y. Chi. Escaping saddle points in heterogeneous federated learning via distributed SGD with communication compression. In S. Dasgupta, S. Mandt, and Y. Li, editors,Proc. Mach. Learn. Res. (AISTATS), Proceedings of Machine Le...

  9. [2021]

    doi: 10.1093/imanum/drz055. S. Kakade, S. Shalev-Shwartz, A. Tewari, et al. On the duality of strong convexity and strong smoothness: Learning applications and matrix regularization.Unpublished Manuscript, http://ttic. uchicago. edu/shai/papers/KakadeShalevTewari09. pdf, 2(1):35,

  10. [2023]

    doi: 10.1007/S10107-022-01822-7. J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization.arXiv preprint,

  11. [2024]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Trans- formers for image recognition at scale.arXiv preprint, 2020.arXiv:2010.11929. A. C. Duzgun, S. Jelas...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.