Pith. sign in

REVIEW 3 major objections 5 minor 35 references

Curvature in the Looking-Glass: Optimal Methods to Exploit Curvature of Expectation in the Loss Landscape

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ReLU crossings create a 'gradient glass' that the Hessian cannot see, and a modified quasi-Newton step exploits it.

desk verdict The gradient glass idea is genuinely interesting, but Theorem 4's 3/2 bound is postulated rather than proven, so the paper's main theoretical support for Alice is currently a hole. read the letter →

arxiv 2411.16914 v1 pith:HW5BVY32 submitted 2024-11-25 cs.LG stat.ML

Jed A. Duersch , Tommie A. Catanach , Alexander Safonov , Jeremy Wendt This is my paper · ORCID
classification cs.LGstat.ML MSC 68T0790C5365K05
keywords gradientglassReLUboundarieslosslandscapecurvatureofexpectationquasi-NewtonmethodsNesterovaccelerationvariationsdiagonalestimation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the Hessian is not always the right second-order object for describing loss curvature in deep ReLU networks, because each ReLU creates a parameter boundary whose crossing adds a pseudorandom jump to the gradient. The accumulated effect of many such jumps is a 'gradient glass' whose density $\rho$ can be estimated from ordinary gradient evaluations, and the paper derives an upper bound on expected loss increase of the form $\mathbb{E}[\Delta L] \le \sqrt{2/(3\pi)}\,\rho^{1/2\,T}|\delta|^{3/2}$, a 3/2 power law rather than the quadratic law a smooth Hessian would give. From this bound the paper constructs an optimal diagonal estimator with Rademacher perturbations and a modified quasi-Newton step that combines glass density with the Hessian diagonal. The resulting algorithm, Alice, demonstrates improved loss reduction on ResNet, Vision Transformer, and transformer language-model benchmarks. A sympathetic reader would care because this reframes curvature in deep learning: what matters near gradient discontinuities is the curvature of expectation, not the pointwise Hessian.

What carries the argument

The gradient-glass density matrix $\rho$ and the 3/2-power expectation bound of Theorem 4 are the load-bearing machinery: they replace the pointwise Hessian with a local density of gradient variations that arises as the continuum limit of many ReLU boundaries. Theorems 2 and 3 supply the optimal kernel and the Rademacher sample distribution that make $\rho$ and the Hessian diagonal $h$ computable from ordinary backpropagation, and Theorem 5 assembles both terms into the modified Hessian $\bar h$ that defines the optimal displacement. Theorem 6 closes the loop by fixing the Nesterov coefficients that make this approximation exactly track the true gradient's linear dependence.

What would settle it

Measure the exponent $p$ in Equation (2) on a deep network with smooth activations (e.g., tanh or sinusoidal); if $p=2$ at all parameter subsets and training epochs, the glass term is an artifact of ReLU boundaries and the Hessian suffices. Alternatively, on a small ReLU MLP, estimate $\mathbb{E}[\Delta L(\delta)]$ empirically for many Rademacher perturbations at several scales and check whether the data respects the 3/2-power bound with a single coordinate-independent $\rho$; a systematic violation would refute the floor construction.

Watch

Extended reading notes

Core claim

The paper's central claim is that the Hessian alone frequently fails to describe loss curvature in deep ReLU networks, because ReLU activations generate a dense family of parameter-space half-space boundaries; crossing one changes the gradient by a pseudorandom vector, so the expected change in loss under a displacement $\delta$ is governed by a 3/2 power law, $\mathbb{E}[\Delta L(\delta)] \le \sqrt{2/(3\pi)}\,\rho^{1/2\,T}|\delta|^{3/2}$, with $\rho$ the local density of gradient variations. The paper shows that this density, together with the locally averaged Hessian diagonal, can be estimated optimally from three gradient evaluations per step using Rademacher-distributed perturbations and the kernel $\kappa_i^*(\delta_i)=c^{-1}\delta_i/(\delta_i^2+\omega_i^2)$, and that the resulting curvature yields the modified quasi-Newton step $\delta = -g * \bar h^{-1}$, where $\bar h = \hat h + h + \sqrt{\hat h*(\hat h+2h)}+\varepsilon$ and $\hat h = 3\rho*(4\pi|g|+\varepsilon)^{-1}$. It also proves that with Nesterov coefficients $\varphi=1-\beta_1$ and $\omega=1$, the running gradient average exactly cancels unknown linear dependencies while damping errors by $\beta_1$.

Load-bearing premise

The load-bearing premise is the 'local floor' assertion in Theorem 4: that loss reductions caused by small parameter changes cannot go below a coordinate-independent floor, so each loss trajectory can be reflected through zero; this floor is imposed by taking absolute values, not derived from ReLU dynamics.

Editorial extensions

If this is right

  • Optimizers for ReLU networks should estimate and use the glass density $\rho$ alongside the Hessian diagonal, since the 3/2-power bound shortens safe step lengths near discontinuities and prevents quasi-Newton overshoot.
  • The exponent $p$ in Equation (2) becomes a measurable diagnostic: values between 1 and 2 indicate the landscape is in the glass regime, while $p=2$ signals a smooth (Hessian-dominated) region.
  • Estimating the diagonal of a linear operator from matrix-vector products is solved optimally by Rademacher perturbations and the kernel of Theorem 2; this recipe applies to any local average of a Hessian, not only to ReLU networks.
  • The Nesterov coefficients $\varphi=1-\beta_1$, $\omega=1$ convert the running gradient average into an exact tracker of hidden linear dependencies, so Alice-style updates can be extended to any momentum-based optimizer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gradient glass dominates early training, sensitivity-based pruning criteria built on Hessian diagonals (such as magnitude times the second derivative) may mis-rank parameters; reweighting by $\rho$ could make pruning and quantization more accurate.
  • The local-floor assumption is the most fragile link: if loss decreases are correlated across coordinates rather than reflected independently, the bound may shift toward a quadratic law; comparing fitted exponents across activation functions (ReLU vs tanh vs GELU) would test the regime.
  • The framework implies a testable robustness prediction: for a fixed parameter perturbation radius, layers with higher glass density should show larger loss jumps, so measuring $\rho$ per layer could predict which layers need smaller quantization step sizes.
  • Alice's quick-step mode reuses a stale $\rho$ for several gradient steps; an adaptive refresh rule driven by the rate of gradient change would be a natural next step that the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that the Hessian is not always a faithful local model of loss curvature in deep ReLU networks, because ReLU boundaries create many small gradient discontinuities. These discontinuities are modeled as a "gradient glass": a dense collection of pseudorandom gradient perturbations whose local density can be estimated from ordinary gradient evaluations. The paper derives a bound on expected loss increase that scales as the 3/2 power of the displacement (Theorem 4), an optimal quasi-Newton step that combines Hessian and glass terms (Theorem 5), an optimal diagonal-estimation kernel and perturbation density (Theorems 2 and 3), and exactness properties for Nesterov-accelerated updates (Theorem 6). It then presents the Alice optimizer and experiments on ResNet18, a ViT, and a Tensorised Transformer showing competitive or improved early loss reduction compared with Adam, SGD-M, and AdaHessian.

Significance. If the theoretical claims hold, the paper offers a genuinely new way of thinking about non-smooth loss landscapes: instead of ignoring ReLU-induced derivative discontinuities or treating them as noise, it proposes an explicit density model and exploits it in optimization. The optimal-kernel and optimal-density results (Theorems 2 and 3) are clean variational derivations, and Theorem 6 is a neat algebraic identity relating Nesterov coefficients to momentum. The authors also provide code and reproducible experimental infrastructure with min/median/max reporting. However, the central mathematical support for the 3/2 power law and for the Alice step is currently incomplete, because the key derivation in Theorem 4 rests on an unsupported absolute-value mapping and Theorem 5 converts an expectation bound into a pointwise majorizer. The empirical results are suggestive, but they do not by themselves establish the theoretical framework.

major comments (3)
  1. [Section 3, Theorem 4 and Appendix A, proof of Theorem 4] The proof of Theorem 4 "enforces the local floor" by mapping each realized loss trajectory to its absolute value, i.e. ∆L(δ_k) = |∆(δ_k)|, and then uses the half-normal expectation to obtain Equation (7). This mapping is not a consequence of the condition ∆L(δ_k) ≥ 0, nor is it derived from ReLU dynamics. For any fixed variance σ² and any target C > 0, there exists a nonnegative random variable Y with Var(Y) = σ² and E[Y] > C, for example Y = 0 with probability 1−p and Y = M with probability p where p(1−p)M² = σ² and pM > C. Thus the nonnegativity constraint alone cannot justify the Gaussian absolute-value calculation. Since Equation (7) is the sole basis for the claimed 3/2 power law, this is a load-bearing gap.
  2. [Section 3, Theorem 5] Equation (7) is an expectation over the random glass perturbations, but the proof of Theorem 5 treats the bound as a deterministic pointwise majorizer: it writes L(µ + δ) ≤ L(µ) + δᵀ(g + ½h∗δ) + sqrt(2/(3π)) ρᵀ|δ|^{3/2} and minimizes that expression. An expectation bound does not imply a pathwise inequality for every realization, unless the glass randomness is explicitly framed as a distribution over losses and the step is defined as minimizing a stochastic majorizer. Without such a framing, Equations (8) and (9) do not follow from the preceding theorem, and the claimed optimality of the Alice update is not established.
  3. [Section 3, Theorem 1 and the diagonal approximation] The paper passes from v(δ) ≤ R|δ| to v ≈ ρ ∗ |δ| with ρ = diag(R) by asserting that this is "a simple and reasonable approximation." This diagonal approximation is not proved or quantitatively bounded; it is a modeling assumption. Because Theorem 4 and Theorem 5 both depend on the diagonal form v ≈ ρ ∗ |δ|, this assumption is load-bearing. At minimum, the paper should provide an empirical validation of the diagonal approximation against off-diagonal contributions, or a bound controlling the error introduced by neglecting off-diagonal terms.
minor comments (5)
  1. [Algorithm 2, line 7] The full topography update in Algorithm 2 sets parameters to µ before evaluating g(0), but the text and Equations (17)–(21) say that evaluations are anchored to ν rather than µ. This appears to be a typo (θ ← ν), and it should be corrected because the running averages otherwise mix two different evaluation centers.
  2. [Equation (2) and Figure 1] The notation p(P) should explicitly state that P is a subset of coordinates and that the sums run over i ∈ P; currently the definition is compressed into the sentence following the equation.
  3. [Theorem 2 statement] The condition P_{j≠i} M_{ij}² = ω_i² m_i² requires ω_i to be known before applying the optimal kernel, but the theorem statement does not discuss how ω_i is obtained in practice; the later remark that ω_i² ≤ 1 for diagonally dominant matrices is useful and should be highlighted earlier.
  4. [Related Work] There is a typo: "showing how that the largest Hessian eigenvalues" should read "showing how the largest Hessian eigenvalues."
  5. [Appendix C and Tables] The reported AdaHessian range on ResNet18 (78.41% to 91.50%) is much wider than the other methods; a sentence explaining whether this is due to learning-rate sensitivity or to the specific architecture would help the reader interpret Figure 6.

Circularity Check

1 steps flagged · score 6.0 of 10

Theorem 4's 3/2 loss bound is imposed by defining the floored loss as |X|; the power-law 'prediction' is the half-normal expectation by construction.

  1. self definitional [Section 3, Theorem 4, and Appendix A, Proof of Theorem 4 (Equation (7))]
    "We enforce the local floor ∆L(δk) ≥ 0 by mapping every virtual loss trajectory that could be realized by ∆(δk) to a bounded trajectory ∆L(δk) = |∆(δk)|. The stated result follows from Z |x|dN (x | 0, λ2) = q 2 π λ2"

    The proof stipulates, rather than derives, the distribution of the floored loss: it replaces the realized loss change by |X| for X ~ N(0, ρ|δ|^3/3). Equation (7) is then literally E|X|, so the 3/2 exponent and the prefactor sqrt(2/(3π)) are inserted by the mapping itself. The stated premise (a local floor with ∆L ≥ 0) does not imply a folded Gaussian: a nonnegative variable with the same variance can have an arbitrarily large mean, so the bound is not a consequence of the floor. The power-law prediction, and the subsequent optimal Alice step in Theorem 5 that is derived from this bound, therefore reduce to the imposed absolute-value construction.

full rationale

The paper's empirical components are self-contained and not circular: the power-law measurements in Figure 1, the diagonal estimation theory in Theorems 2-3, and the Alice benchmark comparisons against Adam, SGD-M, and AdaHessian are independent external evidence. There is no load-bearing self-citation; the only self-citation (Duersch 2024) concerns variational inference and is not used to justify the glass model. The circular content is concentrated in Theorem 4, where the claimed 3/2 curvature law is manufactured by mapping the Gaussian virtual loss to its absolute value, so the theorem's conclusion is equivalent to the folded-Gaussian assumption imposed in the proof rather than being derived from ReLU dynamics or from the local-floor premise alone. Because Theorem 5's optimal quasi-Newton modification is obtained by minimizing that same bound, the theoretical justification of Alice inherits this circular step. Separately, Equation (7) is an expectation bound, and Theorem 5 applies it as a deterministic pointwise majorizer, which is an additional unsupported inference; this is a validity concern rather than circularity. Overall, the analytic 'prediction' reduces by construction, giving partial circularity, while the optimizer's empirical success provides independent support.

Assumptions & free parameters 6 free parameters · 6 assumptions · 2 invented entities

The central claims rest on several modeling assumptions: uniform ReLU pre-activations, pseudorandom independence, a coordinate-wise local floor, and a diagonal dominance bound. These are not empirically validated or derived from first principles. The algorithm also has several tuned hyperparameters (lambda, beta's, step bounds, Nesterov coefficients) that affect performance, though they are not fitted to the theory itself.

free parameters (6)
  • psi (interval width)
    Theorem 1 assumes pre-activations of ReLUs in S_psi are uniformly distributed in [-psi, psi]; psi is a modeling parameter, not measured or estimated, and the density matrix R depends on it.
  • lambda (perturbation scale) = 0.001 to 0.005, tuned per benchmark
    Alice's gradient evaluation perturbation scale; selected from a small set per problem, affecting the estimates of h and rho.
  • beta1, beta2 (moving average coefficients) = 0.9, 0.999
    Exponential moving average coefficients in Alice, inherited from Adam-like averaging; set by convention rather than fitted to the theory.
  • lambda_min, lambda_max (step bounds) = e.g., lambda_max 0.002 to 0.01
    Exploitation bounds in Algorithm 3; tuned per benchmark and essential to Alice's stability and performance.
  • phi, omega (Nesterov coefficients) = phi = 0.1 or 1.0, omega = 1.0
    Tuned per benchmark; Theorem 6 fixes phi = 1 - beta1 and omega = 1 for exactness in a linear model, but experiments vary phi.
  • omega_i^2 (off-diagonal scaling in Theorem 2) = bounded by 1 for diagonally dominant matrices
    Theorem 2 requires knowledge of omega_i^2 to choose the optimal kernel; the paper only provides a conservative bound, not an estimate.
assumptions (6)
  • domain assumption ReLU pre-activations are uniformly distributed in [-psi, psi] within a neighborhood of the current parameters
    The probability of crossing a ReLU threshold is taken as |delta^T gamma_hat|/(2 psi), which assumes uniformity; entered in Theorem 1.
  • domain assumption Pre-activation gradients gamma_hat(k) are independent zero-mean pseudorandom variables
    This lets the paper sum variances instead of covariances; stated in Theorem 1 after the equation.
  • ad hoc to paper A local floor enforces Delta L(delta_k) >= 0 for each coordinate, and loss trajectories can be mapped to |Delta(delta_k)|
    The 3/2 bound in Theorem 4 follows from this asserted floor and the absolute-value mapping; not derived from the ReLU model.
  • ad hoc to paper The diagonal approximation v approx rho * |delta| is adequate
    After Theorem 1 the paper states the diagonal term is a 'simple and reasonable approximation', ignoring off-diagonal R terms.
  • domain assumption For diagonally dominant matrices, omega_i^2 <= 1
    Used to choose a conservative kernel in Theorem 2 when omega_i^2 is not known; stated without proof in the main text.
  • domain assumption The true gradient is linear with unknown H through the projected optimum
    Theorem 6's exactness result relies on this linear model holding through the step; in real losses it is approximate.
invented entities (2)
  • Gradient glass (and glass density matrix R)
    purpose: Conceptual construct to model the aggregate effect of ReLU boundary crossings on gradient variations and loss topography.
    The paper provides an estimation procedure for rho from gradients, but no independent external validation that the glass framework is the correct description beyond the paper's own experiments.
  • Local floor
    purpose: Postulated lower bound on loss reduction that generates the 3/2 power law.
    Used to derive Theorem 4 and the Alice step; no independent evidence or falsifiable prediction is made.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Curvature in the Looking-Glass: Optimal Methods to Exploit Curvature of Expectation in the Loss Landscape." pith.science (2026). https://pith.science/paper/HW5BVY32

@misc{pith2026241116914,
  author       = {Pith},
  title        = {Pith review of: Curvature in the Looking-Glass: Optimal Methods to Exploit Curvature of Expectation in the Loss Landscape},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HW5BVY32}},
  note         = {Machine review of arXiv:2411.16914}
}
read the original abstract

Harnessing the local topography of the loss landscape is a central challenge in advanced optimization tasks. By accounting for the effect of potential parameter changes, we can alter the model more efficiently. Contrary to standard assumptions, we find that the Hessian does not always approximate loss curvature well, particularly near gradient discontinuities, which commonly arise in deep learning architectures. We present a new conceptual framework to understand how curvature of expected changes in loss emerges in architectures with many rectified linear units. Each ReLU creates a parameter boundary that, when crossed, induces a pseudorandom gradient perturbation. Our derivations show how these discontinuities combine to form a glass-like structure, similar to amorphous solids that contain microscopic domains of strong, but random, atomic alignment. By estimating the density of the resulting gradient variations, we can bound how the loss may change with parameter movement. Our analysis includes the optimal kernel and sample distribution for approximating glass density from ordinary gradient evaluations. We also derive the optimal modification to quasi-Newton steps that incorporate both glass and Hessian terms, as well as certain exactness properties that are possible with Nesterov-accelerated gradient updates. Our algorithm, Alice, tests these techniques to determine which curvature terms are most impactful for training a given architecture and dataset. Additional safeguards enforce stable exploitation through step bounds that expand on the functionality of Adam. These theoretical and experimental tools lay groundwork to improve future efforts (e.g., pruning and quantization) by providing new insight into the loss landscape.

Figures

Figures reproduced from arXiv: 2411.16914 by the authors.

Figure 1
Figure 1. Exponential dependence of gradient variations on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of gradient glass in 2D (top). Gray lines [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. ReLU extrapolation. Hessian fitting cannot ac [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Alice training with various curvature terms: a) full [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Training effect of increased exploitation bound. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Optimization method comparisions. 64 downsampled Imagenet (Deng et al. 2009; Chrabaszcz, Loshchilov, and Hutter 2017). The first column shows full QN steps, and the second employs the NAQ coefficients in Equation (15). Both NAQ and curvature using glass terms yield the…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 21 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Becker, S.; Le Cun, Y.; et al. 1988. Improving the convergence of back-propagation learning with second order methods. In Proceedings of the 1988 connectionist models summer school, 29--37

    work page 1988

  4. [4]

    E.; and Nocedal, J

    Bottou, L.; Curtis, F. E.; and Nocedal, J. 2018. Optimization methods for large-scale machine learning. SIAM review, 60(2): 223--311

    2018

  5. [5]

    B.; and LeCun, Y

    Choromanska, A.; Henaff, M.; Mathieu, M.; Arous, G. B.; and LeCun, Y. 2015. The loss surfaces of multilayer networks. In Artificial intelligence and statistics, 192--204. PMLR

    work page 2015

  6. [6]

    Chrabaszcz, P.; Loshchilov, I.; and Hutter, F. 2017. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv:1707.08819

    arXiv 2017

  7. [7]

    N.; Pascanu, R.; Gulcehre, C.; Cho, K.; Ganguli, S.; and Bengio, Y

    Dauphin, Y. N.; Pascanu, R.; Gulcehre, C.; Cho, K.; Ganguli, S.; and Bengio, Y. 2014. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Advances in neural information processing systems, 27

    work page 2014

  8. [8]

    Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248--255. Ieee

    2009

Show all 35 references
  1. [9]

    Duersch, J. A. 2024. Projective Integral Updates for High-Dimensional Variational Inference. SIAM/ASA Journal on Uncertainty Quantification, 12(1): 69--100

    2024

  2. [10]

    Frumkin, N.; Gope, D.; and Marculescu, D. 2023. Jumping through local minima: Quantization in the loss landscape of vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 16978--16988

    2023

  3. [11]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

    2016

  4. [12]

    Hemati, S.; Zhang, G.; Estiri, A.; and Chen, X. 2023. Understanding hessian alignment for domain generalization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 19004--19014

    2023

  5. [13]

    Indrapriyadarsini, S.; Mahboubi, S.; Ninomiya, H.; and Asai, H. 2020. A stochastic quasi-newton method with nesterov’s accelerated gradient. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, W \"u rzburg, Germany, September 16--20, ...

    2020

  6. [14]

    Kaur, S.; Cohen, J.; and Lipton, Z. C. 2023. On the maximum hessian eigenvalue and generalization. In Proceedings of Machine Learning Research, 51--65. PMLR

    2023

  7. [15]

    P.; and Ba, J

    Kingma, D. P.; and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    2014 arXiv

  8. [16]

    Kong, Y.; Liu, L.; Chen, H.; Kacprzyk, J.; and Tao, D. 2023. Overcoming catastrophic forgetting in continual learning by exploring eigenvalues of Hessian matrix. IEEE Transactions on Neural Networks and Learning Systems

    2023

  9. [17]

    Krizhevsky, A.; and Hinton, G. 2009. Learning multiple layers of features from tiny images. Technical report, Citeseer

    2009

  10. [18]

    LeCun, Y.; Denker, J.; and Solla, S. 1989. Optimal brain damage. Advances in neural information processing systems, 2

    1989

  11. [19]

    Lee, J.; Yoo, J.; and Kwak, N. 2023. SHOT: suppressing the hessian along the optimization trajectory for gradient-based meta-learning. Advances in Neural Information Processing Systems, 36: 61450--61465

    2023

  12. [20]

    Lee, S.; He, C.; and Avestimehr, S. 2023. Achieving small-batch accuracy with large-batch scalability via Hessian-aware learning rate adjustment. Neural Networks, 158: 1--14

    2023

  13. [21]

    Li, L.; and Spratling, M. 2023. Understanding and combating robust overfitting via input loss landscape analysis and regularization. Pattern Recognition, 136: 109229

    2023

  14. [22]

    Li, X.-C.; Li, L.; and Zhan, D.-C. 2024. Visualizing, Rethinking, and Mining the Loss Landscape of Deep Neural Networks. arXiv preprint arXiv:2405.12493

    2024 arXiv

  15. [23]

    A.; Pascanu, R.; and Dabney, W

    Lyle, C.; Zheng, Z.; Nikishin, E.; Pires, B. A.; Pascanu, R.; and Dabney, W. 2023. Understanding plasticity in neural networks. In International Conference on Machine Learning, 23190--23211. PMLR

    2023

  16. [24]

    Ma, X.; Zhang, P.; Zhang, S.; Duan, N.; Hou, Y.; Zhou, M.; and Song, D. 2019. A tensorized transformer for language modeling. Advances in neural information processing systems, 32

    2019

  17. [25]

    Merity, S.; Xiong, C.; Bradbury, J.; and Socher, R. 2016. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843

    2016 arXiv

  18. [26]

    Nair, V.; and Hinton, G. E. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), 807--814

    2010

  19. [27]

    Ninomiya, H. 2017. A novel quasi-Newton-based optimization for neural network training incorporating Nesterov's accelerated gradient. Nonlinear Theory and Its Applications, IEICE, 8(4): 289--301

    2017

  20. [28]

    Nocedal, J.; and Wright, S. J. 2006. Theory of constrained optimization. Numerical optimization, 304--354

    2006

  21. [29]

    Parisi, G. 2007. Course 3 Mean field theory of spin glasses: Statics and dynamics. Les Houches, 85: 131--178

    2007

  22. [30]

    Ruder, S. 2016. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747

    2016 arXiv

  23. [31]

    K.; and Qin, A

    Sen, M.; Mohan, C. K.; and Qin, A. K. 2023. Federated Optimization with Linear-Time Approximated Hessian Diagonal. In International Conference on Pattern Recognition and Machine Intelligence, 106--113. Springer

    2023

  24. [32]

    Shen, Z.; Zhang, M.; Zhao, H.; Yi, S.; and Li, H. 2021. Efficient attention: Attention with linear complexities. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, 3531--3539

    2021

  25. [33]

    Spigler, S.; Geiger, M.; d’Ascoli, S.; Sagun, L.; Biroli, G.; and Wyart, M. 2019. A jamming transition from under-to over-parametrization affects generalization in deep learning. Journal of Physics A: Mathematical and Theoretical, 52(47): 474001

    2019

  26. [34]

    Yao, Z.; Gholami, A.; Keutzer, K.; and Mahoney, M. W. 2020. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE international conference on big data (Big data), 581--590. IEEE

    2020

  27. [35]

    Yao, Z.; Gholami, A.; Shen, S.; Mustafa, M.; Keutzer, K.; and Mahoney, M. 2021. Adahessian: An adaptive second order optimizer for machine learning. In proceedings of the AAAI conference on artificial intelligence, volume 35, 10665--10673

    2021

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.