REVIEW 4 major objections 4 minor 8 references
How Many Factors Influence Minima in SGD?
T0 review · 4 major / 4 minor · reviewed 2026-08-27 · deepseek-v4-flash
Pith's one-line read This paper reports ResNet-56/CIFAR-10 experiments that confirm the four-factor gradient-covariance formula for SGD minima and find the Hessian-trace formula overestimates expected loss by 2.4 to 11.8 times.
desk verdict A clear review of two SGD-minima formulas with a new but weakly tested numerical comparison; the trace discrepancy does not by itself confirm the four-factor formula. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the trace of the gradient covariance matrix $\operatorname{tr}[\sigma^2(X(\infty))]$ at the critical point, which enters the expected-excess-loss formula as $\frac{\delta}{4m}\operatorname{tr}[\sigma^2]$. The theoretical machinery behind it is the continuous-time limit of SGD as an Ornstein-Uhlenbeck process: the normalized fluctuation $V(t) = (m/\delta)^{1/2}[X^m_\delta(t) - X(t)]$ converges to an OU process whose stationary covariance $\Gamma(\infty)$ solves the algebraic Riccati equation $\Gamma(\infty)Hf(X(\infty)) + Hf(X(\infty))\Gamma(\infty) = \sigma^2(X(\infty))$, and the loss identity follows from that covariance. The numerical comparison contrasts this with the special-case formula $\frac{\delta}{4m}\operatorname{Tr}(H)$ obtained when $\sigma^2 = H$.
What would settle it
Compute the same traces after training for substantially longer, for example 200 or 300 epochs with the same decay schedule, and check whether the ratio Tr(H)/tr[σ²] and the alignment of learning curves under fixed BS/LR remain stable; if the ratio drifts or the curves separate, the asymptotic claim is not supported.
Extended reading notes
Core claim
The central claim is that the expected excess loss of SGD around a local minimum is governed by the trace of the gradient covariance $\operatorname{tr}[\sigma^2(X(\infty))]$ at the critical point, with $\mathbb{E}[f(X^m_\delta(t))] = f(X(t)) + \frac{\delta}{4m}\operatorname{tr}[\sigma^2(X(\infty))] + o(\delta/m)$, as derived in Wang (2019). The paper's numerical experiments with ResNet-56 on CIFAR-10 reproduce the prediction that keeping batch size and learning rate in the same ratio keeps the learning dynamics and expected loss essentially unchanged, and they quantify the gap between this covariance trace and the Hessian trace: the Hessian trace is 2.4 to 11.8 times larger. From this the authors conclude that the three-factor formula of Jastrzebski et al. (2018), which uses $\operatorname{Tr}(H)$, is a special case that does not survive outside the regime where gradient covariance equals the Hessian.
Load-bearing premise
The numerical test assumes that after 125 epochs of training, with learning-rate decays at epochs 60 and 100, the ResNet-56 is close enough to a local minimum with a positive-definite Hessian that the asymptotic stationary-distribution formulas of Result 3.2 apply, but the paper never checks convergence to stationarity or that the computed traces at the final weights equal the infinite-time limits.
Editorial extensions
If this is right
- If the four-factor formula is correct, the expected minimum loss depends on learning rate and batch size only through their ratio, so rescaling both by the same factor leaves the SGD stationary distribution and expected loss unchanged.
- The Hessian trace is not a reliable proxy for gradient covariance in deep networks; using it overestimates expected loss by a factor that grows with learning rate.
- The three-factor formula of Jastrzebski et al. (2018) is a limiting case: it becomes exact only when the gradient covariance equals the Hessian, a condition that the numerical results show is violated in practice.
- The numerical finding that $\operatorname{tr}[\sigma^2]$ decreases with learning rate while $\operatorname{Tr}(H)$ increases suggests that the two quantities respond to training hyperparameters in opposite directions.
Reading between the lines
- If the covariance trace is the right object, then flatness measures for generalization should be based on gradient covariance (or Fisher information) rather than the Hessian trace, since the Hessian trace can be inflated by directions of high curvature that SGD noise never explores.
- Because the ratio of Hessian to covariance trace grows with learning rate, the discrepancy between the two formulas may partly explain why larger learning rates are observed to find flatter minima: the covariance trace shrinks while the Hessian trace grows.
- One could turn the comparison into a sharper test by using the loss curvature at a true stationary point from a much longer run, or by evaluating the covariance trace on the noise during training rather than at the final weights; the paper's estimates are point-in-time.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reviews two SDE-based analyses of SGD minima: Jastrzebski et al. (2018), which predicts that expected loss near a minimum is proportional to (δ/m)Tr(H) under the assumption that gradient covariance equals the Hessian, and Wang (2019), which gives expected excess loss proportional to (δ/m)tr[σ²] without that assumption. The authors train ResNet-56 on CIFAR-10 under KLD and gradient-log losses, vary batch size and learning rate, and compare accuracy curves and the two trace quantities. From these experiments they conclude that the four-factor formula of Wang (2019) is confirmed and that the three-factor formula of Jastrzebski et al. (2018) may not hold beyond the considered special case.
Significance. The question is substantive: whether SGD minima are characterized by (δ/m)Tr(H) or by the four-factor (δ/m)tr[σ²] relationship matters for interpreting wide minima and for learning-rate/batch-size transfer. The paper is useful as a concise comparison of the two theoretical frameworks, and it correctly notes that (12) reduces to (6) only under the covariance-equals-Hessian assumption. A particular strength is that the theoretical formulas are stated explicitly and the comparison does not involve fitted free parameters. However, because the experiments do not measure the quantities that the formulas actually predict—expected excess loss and expected squared gradient norm—the empirical contribution is at present only suggestive. The significance of the paper depends on whether the authors add direct validation of the predicted loss quantities.
major comments (4)
- [§5.1, §5.2, Figs. 2 and 3] Equations [8](4.45) and [8](4.46) describe the expected training loss and expected squared gradient norm near a minimum, respectively; Figures 2 and 3 plot training and validation accuracy only. The observation that accuracy curves coincide when the BS/LR ratio is held constant is consistent with the SDE scaling in (9), which is shared by both competing formulas, and it therefore cannot distinguish between the tr[σ²] term in (12) and the Tr(H) term in (6). To confirm these equations, the authors need to report measured training loss (or its excess over f(θ*)) and measured squared gradient norm, matched against the predicted values.
- [§5.3, Table 1] The central conclusion of Section 5.3 is inferred from comparing two right-hand-side traces, tr[σ²(X(∞))] and Tr(H), rather than from the quantity that both (12) and (6) actually predict, namely expected excess loss. A trace ratio of 2.4 to 11.8 shows that the two formulas would give different predictions, but without an observed excess-loss value (or a comparison of predicted versus observed loss) it does not show that (12) is correct and (6) is not.
- [§4, Result 3.2] Result 3.2 is an asymptotic statement: it requires t→∞ with fixed small δ and fixed batch size m, convergence of the gradient flow to a local minimizer with positive definite Hessian, and the limiting stationary covariance. The experiments train for only 125 epochs with the learning rate reduced by a factor of 10 at epochs 60 and 100, and the paper performs no stationarity or convergence diagnostics for the iterates or the trace estimates. If the final weights are not near the stationary distribution, the numbers reported in Table 1 cannot be interpreted as tr[σ²(X(∞))] and Tr(Hf(X(∞))), and the comparison is inconclusive.
- [Table 1] The trace values are averaged over five runs but no standard errors, confidence intervals, or run-to-run spreads are reported, and the text does not describe how tr[σ²(X(∞))] and Tr(H) are computed from TensorFlow/Keras (e.g., which loss gradients are evaluated, at which weights, and by what estimator). Without this information the reported magnitude differences cannot be assessed for statistical significance.
minor comments (4)
- [§4, paragraph 1] The text contains the typo "neutral network model setup"; it should read "neural network model setup."
- [Table 1] The experiment numbering is inconsistent: rows 3 and 4 are both labeled "4" (with BS/LR 200/0.5 and 2,500/0.08); please renumber.
- [Equation (3)] The displayed equation writes the noise term as δ√m ε with ε having covariance C(θ); from the preceding variance calculation Σ(θ)=C(θ)/m, the correct factor should be δ/√m (or the covariance of ε should be adjusted). Please check and correct.
- [Section 5.2, title] The section title refers to "Equation [8](4.46) for Gradient Log Loss," but the text and Figure 3 report accuracy curves rather than the squared gradient norm; either measure the squared gradient norm or adjust the title and prose to reflect what is actually plotted.
Circularity Check
No significant circularity: the numerical study is an external check, not a fit, and the self-citation to Wang (2019) is prior published work rather than a definitional reduction.
full rationale
The paper's theoretical formulas are explicitly reviewed and cited from Wang (2019) and Jastrzebski et al. (2018), not derived from the present paper's data. The numerical experiments are external to those derivations: Figures 2 and 3 compare accuracy trajectories of ResNet-56 on CIFAR-10 under fixed BS/LR ratios, and Table 1 reports traces computed from trained networks. No parameter used in the compared formulas is fitted to the outcome being claimed; in particular, tr[sigma^2] and Tr(H) are computed quantities, not fit parameters, and the paper never measures expected excess loss to fit either formula. The main weakness, that Figures 2 and 3 only test the BS/LR scaling shared by both candidate formulas and Table 1 compares traces without measuring the loss predicted by Equations [8](4.45) and [4](9), is a correctness or test-selectivity concern, not a circularity: the evidence may be underdetermining, but it is not equivalent to the inputs by construction. The self-citation to Wang (2019) is present and the confirming language is tied to that work, but the cited result is a published external theoretical result with stated assumptions and its own proofs, and the present paper offers independent numerical evidence. Thus no circular step can be exhibited, and the paper should not receive a circularity penalty beyond, at most, ordinary self-citation weight.
Assumptions & free parameters
assumptions (4)
- domain assumption Wang (2019) Result 3.2: under smoothness, gradient CLT, and positive-definite Hessian at the limit point, the expected loss and gradient norm satisfy equations (12) and (13).
- domain assumption The trained network is near a local minimum and the SGD process has reached its stationary distribution by the end of 125 epochs.
- domain assumption The gradient covariance and Hessian can be estimated reliably from the finite trained model and mini-batch gradient samples.
- domain assumption Training and validation accuracy curves are a valid proxy for the expected loss and gradient-norm quantities in Equations [8](4.45) and [8](4.46).
Cite this review
Pith. "Pith review of How Many Factors Influence Minima in SGD?." pith.science (2026). https://pith.science/paper/H6TQI36A
@misc{pith2026200911858,
author = {Pith},
title = {Pith review of: How Many Factors Influence Minima in SGD?},
year = {2026},
howpublished = {\url{https://pith.science/paper/H6TQI36A}},
note = {Machine review of arXiv:2009.11858}
}
read the original abstract
Stochastic gradient descent (SGD) is often applied to train Deep Neural Networks (DNNs), and research efforts have been devoted to investigate the convergent dynamics of SGD and minima found by SGD. The influencing factors identified in the literature include learning rate, batch size, Hessian, and gradient covariance, and stochastic differential equations are used to model SGD and establish the relationships among these factors for characterizing minima found by SGD. It has been found that the ratio of batch size to learning rate is a main factor in highlighting the underlying SGD dynamics; however, the influence of other important factors such as the Hessian and gradient covariance is not entirely agreed upon. This paper describes the factors and relationships in the recent literature and presents numerical findings on the relationships. In particular, it confirms the four-factor and general relationship results obtained in Wang (2019), while the three-factor and associated relationship results found in Jastrz\c{e}bski et al. (2018) may not hold beyond the considered special case.
Figures
Reference graph
Works this paper leans on
-
[8]
Y. Wang, Asymptotic Analysis via Stochastic Differential Equations of Gradient Descent Algorithms in Statistical and Computational Paradigms , November 12, 2019. arXiv:1711.09514v5 [stat.ML]. A version co-authored with Shang Wu will be published in Journal of Machine Learning Research 21, 2020. 10
work page Pith review arXiv 2019
-
[1]
A. A. Brown and M. C. Bartholomew-Biggs, Some Effective Methods for Unconstrained Optimization Based on the Solution of Systems of Ordinary Differential Equations , Journal of Optimization Theory 9 and Applications, 62(2):211-224, 1989
work page 1989
-
[2]
D. J. Foster, A. Sekhari, O. Shamir, N. Srebro, K. Sridharan, B. Woodworth, The Complexity of Making the Gradient Small in Stochastic Convex Optimization , Proceedings of Machine Learning Research 99, 1-27 (2019)
work page 2019
-
[3]
C. W. Gardiner, Stochastic Methods: A Handbook for the Natural and Social Sciences , Springer, 4th edition. 2009
work page 2009
-
[4]
Three Factors Influencing Minima in SGD
S. Jastrz¸ ebski, Z. Kenton, D. Arpit, N. Ballas, A. Fischer, Y. Bengio, and A. Storkey. Three Factors Influencing Minima in SGD , September 14, 2018. arXiv: 1711.04623v3 [cs.LG]
work page Pith review arXiv 2018
-
[5]
K. Kawaguchi, Deep Learning Without Poor Local Minima, In Advances In Neural Information Processing Systems, pages 586-594, 2016
work page 2016
-
[6]
Y. Nesterov, Gradient Methods for Minimizing Composite Functions , Mathematical Programming, 140(1):125-161, 2013
work page 2013
-
[7]
An overview of gradient descent optimization algorithms
S. Ruder, An Overview of Gradient Descent Optimization Algorithms , arXiv:1609.04747v1, 2016 [cs.LG]
work page Pith review arXiv 2016
Reviewed August 27, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.