{"id":"0993c81a-6cfa-4e8e-b2cf-25efd8b03418","arxiv_id":"2509.03594","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A geometrically derived optimizer that rescales gradients by 1/(1+ξ||∇L||²) is competitive with AdamW and Muon, with one RMS-based variant showing slight average improvement.","lead":"This paper derives a new optimizer for neural networks from the geometry of the loss landscape and tests it against standard methods. A variant using an RMSprop-style preconditioner shows slight average gains, and the framework ties together gradient clipping, learning rate scheduling, and weight decay.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Curvature-adaptivity claim conflates gradient norm with curvature; Eq. 13 is a global gradient-norm rescaling, so the central theoretical selling point is unsupported.","rationale":"The algebraic derivation and available code support the empirical claims at the level stated, and the paper honestly hedges the neural-network gains. The load-bearing concern is the interpretation of Eq. 13 as curvature-adaptive: a simple quadratic/saddle experiment can decide this. If the curvature wording is withdrawn, the optimizer remains a legitimate normalized-gradient method and the empirical comparisons are not invalidated. The reader's weakest assumption identifies exactly this issue, so the conditional verdict stands without adjustment.","tokens_in":18799,"tokens_out":6718,"duration_ms":77576,"concrete_test":"Analytical/2D check on a separable quadratic L = 0.5 a x² + 0.5 b y² with a ≫ b, no momentum, fixed ξ. Record per-coordinate updates under Eq. 13. If the mechanism were curvature-adaptive, steps along the high-curvature x-axis should be preferentially reduced relative to y; in fact, δx/δy = (a x)/(b y), identical to unmodified GD, with only one global scalar η/(1 + ξ(a²x² + b²y²)) multiplying both. Then evaluate L = 0.5 a x² − 0.5 b y² near the saddle; ∇L ≈ 0 gives r_t ≈ 1 while Hessian eigenvalues are large. This settles whether the denominator responds to curvature or merely to gradient norm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that Eq. 13 \"automatically decreases the effective learning rate in regions of high curvature\" is not supported by the derivation. Eq. 7 defines the pull-back metric as g_ij = γ_ij + ∂_i L ∂_j L, and the denominator in Eq. 13 is 1 + ξ Σ_k (∂_k L)^2. This depends only on first derivatives (the slope of the loss graph), not on second derivatives or any curvature invariant. On L = 0.5 a θ², the scaling factor is largest where |θ| is large and ≈1 at the minimum, i.e., the step is least reduced where the curvature is constant and most reduced where the loss is steep but curvature is not changing. At a saddle point, ∇L = 0 gives no adaptation even when Hessian eigenvalues are large. More generally, Eq. 13 is ordinary gradient descent multiplied by a single global scalar; it cannot distinguish high-curvature directions from low-curvature directions. Thus the \"induced metric\" contributes no curvature information beyond a global per-step gradient-norm clip. This does not disprove the empirical usefulness of the algorithm as a normalized-SGD variant, but it undercuts the abstract's principal theoretical justification. (Additionally, the unit vertical metric component in Eq. 5 makes the clipping threshold dependent on the arbitrary loss normalization: L → cL changes the denominator to 1 + ξ c² Σ(∂L)^2 unless ξ is rescaled.)","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new class of optimizers derived from the Riemannian pull-back metric obtained by embedding the loss function as a graph over parameter space. For an ambient metric with identity horizontal part, the Sherman–Morrison formula gives the inverse metric and leads to the update δθ_i = -η ∂L/∂θ_i / (1 + ξ Σ_k (∂L/∂θ_k)^2) (Eq. 13). Variants using a log-loss embedding and an RMSprop-based ambient metric are also proposed. The paper benchmarks these optimizers against SGD, Adam, AdamW, and Muon on five low-dimensional test functions, an MLP regression task, MNIST MLP classification, ResNet-18 on CIFAR-10, and a TinyShakespeare transformer language task. Claims include automatic step-size reduction in regions of high curvature, natural appearance of decoupled weight decay and a scheduled learning rate, and slight empirical improvement of the RMS variant over Adam/AdamW.","tokens_in":19281,"tokens_out":6298,"duration_ms":69499,"significance":"The manuscript's strengths are its transparent algebraic derivation, a relatively broad set of benchmarks, and the provision of JAX/PyTorch code. If the geometric framing were fully established, the connection between loss-landscape visualization and optimization would be a useful perspective, and the algorithms are indeed cheap (O(N) per step). However, the central theoretical claim that the method adapts to curvature is not supported by the equations, and the reported empirical gains are smaller than the run-to-run variability. The framework may still be of interest as a smooth, global gradient-norm clipping scheme, but the current text overstates what has been rigorously shown.","major_comments":[{"comment":"The statement that the effective learning rate is 'automatically decreased in regions of high curvature' is not supported. The prefactor in Eq. (13) is 1/(1+ξΣ_k(∂L/∂θ_k)^2), which depends only on first derivatives and is a single global scalar for all coordinates. On L = 0.5 a θ², the denominator is largest far from the minimum even though the curvature a is constant; at a saddle point ∇L = 0, so no adaptation occurs despite possibly large Hessian eigenvalues. Eq. (13) is therefore a global gradient-norm rescaling, not a curvature-adaptive preconditioner. The text should be reworded to 'large-gradient' or 'steep-slope' regions, and any curvature claim should be either removed or separately motivated by additional analysis.","section":"Abstract, §2 Eq. (13)"},{"comment":"The pseudocode writes the weight-decay update as θ_t ← θ_{t−1} − η r_t m̂_t + λθ_{t−1}, which adds λθ_{t−1}. The shipped code in the appendix (lines 125, 208, 308) implements updates = -lr * metric_scale * m / (...) - lr * weight_decay * p, i.e., subtraction. Since the experiments are run with the code, the pseudocode does not describe the evaluated method. This discrepancy must be fixed and the sign/direction of weight decay clarified before the results are reproducible from the paper alone.","section":"Algorithm 1 line 12, Algorithm 2 line 13, Appendix listing"},{"comment":"The vertical metric component is set to 1 with the statement that its value 'can be absorbed into the normalisation of the loss function.' This is only partly true for the update: under L → cL, the denominator in Eq. (13) becomes 1 + ξ c² Σ(∂L)^2 unless ξ is rescaled. Thus the effective clipping threshold depends on the arbitrary normalization of the loss, and the claimed 'automatic' adaptivity is not scale-invariant. The paper should state explicitly that ξ absorbs the loss scale, which weakens the geometric interpretation of the clipping scale.","section":"§2 Eq. (5), Eq. (13)"},{"comment":"The conclusion that one variant 'demonstrated slight improvement on average over Adam and AdamW' is not supported by the reported statistics. For CIFAR-10, SGD RMS has mean max validation accuracy 0.8263 ± 0.0217 versus Adam's 0.8260 ± 0.0210; for TinyShakespeare, SGD RMS has min validation perplexity 4.4328 ± 0.0437 versus AdamW's 4.4372 ± 0.0397. These differences are far smaller than one standard deviation, no significance or paired comparisons are provided, and selecting the best 50 of 200–500 Bayesian-sweep runs can inflate apparent differences. Please report paired significance tests or honest error bars, and either qualify the claim as 'competitive' or provide stronger statistical evidence for 'improvement'.","section":"§3.3–3.5, Tables 3–4"}],"minor_comments":[{"comment":"Including f(L) on the right-hand side of the gradient-flow equation is introduced as 'the author's choice, and not fundamental.' This is a substantial modeling choice; a geometric derivation or at least a more principled justification would strengthen the paper.","section":"§2 Eq. (6)"},{"comment":"The pseudocode applies bias correction of momentum inside γ^{-1}, while the code applies it after forming the tree_map update. Please clarify the exact ordering in the implementation so the pseudocode matches the code.","section":"Algorithm 1 line 11, Appendix"},{"comment":"The symbol l_i is used for both γ^{ij} l_j and ∂L/∂θ_i. This is confusing; use a different notation for the raised-index object.","section":"§2 before Eq. (8)"},{"comment":"In batched training, the EMA decay β is a second new hyperparameter; the sentence should be qualified to the non-batched case or to the case where β is fixed.","section":"§2 bullet 'Only one new hyperparameter'"},{"comment":"The text claims custom optimizers were 'typically' fastest, but on Rastrigin SGD is fastest. The main text acknowledges this; the figure captions and summary should be more consistent about wall-time versus iteration counts.","section":"§3.1"},{"comment":"The protocol for choosing the 'best 50' runs from the hyperparameter sweeps should be stated (e.g., based on final validation metric, or best validation during training?). This is important for interpreting the distributions.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's core algorithm is essentially normalized SGD with a smooth global gradient-norm clipping; the geometric framing is attractive but, as written, the curvature interpretation is not supported by Eq. (13). The empirical claims are weaker than the summary suggests. Both issues are fixable without changing the algorithm, and the pseudocode/code mismatch is an author-side bug that must be corrected. I do not recommend rejection, but the manuscript needs substantive revision before it can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper derives a new optimizer from the pull-back metric of the loss landscape, and the algebra is correct, but the headline theoretical claim—that the effective learning rate decreases in regions of high curvature—does not survive contact with the equations. Eq. 13 is a global rescaling by 1/(1 + ξ Σ (∂L/∂θ)²), which depends on the squared gradient norm, not on any second derivative. At a saddle point the gradient vanishes and there is no adaptation. So the 'curvature' story is not supported.\n\nWhat the paper does well: it takes a geometric perspective seriously enough to derive concrete update rules, it ships code and runs careful hyperparameter sweeps over multiple tasks, and it is honest about the size of the empirical gains—'slight improvement' is the paper's own phrase. The Sherman-Morrison step is clean, and the two variants (log-loss and RMS) are genuinely new functional forms. The observation that AdamW and clipped SGD emerge as limit cases is a nice unifying frame.\n\nThe soft spots are real. First, the curvature claim is load-bearing and wrong; the denominator is a gradient-norm term, so the method is better described as smooth gradient clipping with a per-step global scale. Second, the vertical metric component in Eq. 5 is set to 1 by fiat, making the effective threshold dependent on the arbitrary normalization of the loss; ξ absorbs that, but then the 'geometry' is not fixing the scale. Third, Algorithm 1 line 12 writes +λθ but the shipped code uses -lr·λ·p; the code is right, the pseudocode is wrong. Fourth, the benchmark gains are small and not significance-tested; some variants (SGD Metric on CIFAR-10) are clearly worse, and the log-loss variant is wildly task-dependent.\n\nWho is this for? Someone thinking about geometric interpretations of optimizers, or looking for a new normalized-gradient variant to test. It deserves a serious referee: the derivation is sound, the experiments are reproducible, and the flaws are fixable—but the abstract and conclusion need to stop claiming curvature adaptation. My bottom line: conditional accept after revision.","headline":"The geometry is neat and the code ships, but the central curvature-adaptivity claim doesn't hold up; treat this as a well-tested normalized-gradient variant, not a curvature-adaptive method.","tokens_in":19619,"tokens_out":3727,"would_cite":false,"duration_ms":37815,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C26","53B20"],"pacs":[],"model":"deepseek-v4-flash","headline":"Using the loss landscape's induced Riemannian metric as a gradient preconditioner yields optimizers that shrink steps in steep regions and, in one RMSprop-based variant, slightly beat Adam and AdamW on average.","keywords":["loss landscape","induced metric","pull-back metric","gradient preconditioning","smoothed gradient clipping","adaptive optimizers","decoupled weight decay","learning rate scheduling"],"falsifier":"Train on a strictly linear loss L(θ)=a·θ, whose curvature is exactly zero but whose gradient norm ‖a‖ is large. Equation 13 still multiplies the update by 1/(1+ξ‖a‖²), so the claimed curvature-adaptive shrinkage is visibly just gradient-norm clipping. The converse check is a sharply curved quadratic centered near zero with a small gradient: the formula gives almost no shrinkage, contradicting the claim that high curvature triggers smaller steps.","tokens_in":18761,"feed_emoji":"📉","tokens_out":7832,"duration_ms":77209,"temperature":0.7,"pith_summary":"This paper tries to show that the Riemannian metric a loss landscape inherits when drawn in one extra dimension—the same geometry implicit in 3D loss-landscape visualizations—can be used directly to build practical optimizers. Pulling that metric back to parameter space and inverting it yields an update rule in which every gradient is divided by 1 plus a scaled squared gradient norm, so step sizes shrink automatically in steep, high-gradient regions. The paper argues this is a smoothed form of gradient clipping and that decoupled weight decay and, for a log-loss embedding, scheduled learning rates emerge naturally from the geometry. Across low-dimensional hard benchmarks, regression, MNIST, CIFAR-10, and TinyShakespeare, one variant built on the RMSprop metric performed slightly better on average than Adam and AdamW, at Adam-like computational cost. A sympathetic reader would take the central claim as: loss-landscape visualization geometry is not just illustration but a valid preconditioning framework.","feed_headline":"Loss plots' hidden metric trains models, beating Adam on average","feed_subtitle":"Pull the landscape's induced metric into gradients: steps shrink where gradients grow, at Adam-level cost.","key_machinery":"The induced (pull-back) metric g_ij = γ_ij + l_i l_j on the loss-surface graph, inverted by a rank-one update of γ^{-1}. It does the work: the denominator 1 + Σ γ^{kl} l_k l_l is a scalar computed by one dot product, converting gradient magnitude into a per-step shrinkage factor while preserving the descent direction set by the ambient geometry.","core_discovery":"Embed the loss surface as the graph L = f(L(θ)) in an ambient space with coordinates (θ, L) and a metric that is block-diagonal, with any chosen parameter metric γ in the horizontal block and 1 in the vertical block. Pulling this metric back to parameter space gives g_ij = γ_ij + (∂L/∂θ_i)(∂L/∂θ_j). Inverting via the rank-one update formula for a matrix plus an outer product and using the inverse metric to precondition gradient descent yields δθ_i = −η Σ_j γ^{ij} (∂L/∂θ_j) / (1 + Σ_{k,l} γ^{kl} (∂L/∂θ_k)(∂L/∂θ_l)). With diagonal γ this is the base gradient divided by one plus a scaled squared gradient norm, so the step shrinks where the gradient is large without rotating the direction of ste","pith_inferences":["Because the shrinkage factor is global, one outlier gradient component can suppress the whole update; a per-layer or per-block normalization of Σ l_k² would be a natural testable extension.","The paper's claimed tie between gradient norm and curvature can be checked directly: on a linear loss (zero curvature, large gradient) the update still shrinks by 1/(1+ξ‖∇L‖²), so in that regime the effect is indistinguishable from plain gradient clipping.","The log-loss variant's erratic transfer between low and high dimensions hints that loss-scale dynamics, not geometry alone, drive its success; testing on losses with a lower bound or on reinforcement-learning returns would separate those effects.","If the geometric derivation is the real source of the gains, embedding the loss into more than one extra dimension—for instance, one vertical coordinate per loss term—should change optimizer behavior in predictable ways; this is directly testable and could yield multi-objective optimizers."],"forward_implications":["Updates follow Eq. 13: divide the gradient by 1 + ξ Σ (∂L/∂θ_i)², so a single scalar dot product converts gradient magnitude into automatic step-size reduction at O(N) cost per step, matching Adam.","The framework is a wrapper: replacing γ with the metric implied by any preconditioner such as RMSprop or Muon induces an optimizer with the same shrinkage property, and standard SGD and AdamW are recovered as ξ→0 limits.","Decoupled weight decay is the geometrically natural regularizer, so no ad-hoc coupling is needed.","The log-loss embedding f(L) = ln L produces a learning-rate-schedule-like behavior with warm-up and decay phases, tying scheduling to the loss magnitude itself.","Empirically, the RMS-metric variant achieved the best average validation performance on the regression and TinyShakespeare tasks and the best single-run accuracy on CIFAR-10, while the log-loss variant was the only optimizer to solve all tested low-dimensional benchmark functions."],"supporting_citations":[{"why":"Supplies the loss-landscape visualization geometry that the paper treats as an implicit metric.","marker":"[1]"},{"why":"Gives the rank-one inverse formula that converts the pull-back metric into a one-dot-product update.","marker":"[25]"},{"why":"Provides the RMSprop diagonal metric adopted in the best-performing variant.","marker":"[24]"},{"why":"Defines decoupled weight decay, which the geometry singles out and which serves as a baseline.","marker":"[5]"},{"why":"Introduces gradient clipping, the behavior the denominator of Eq. 13 reproduces smoothly.","marker":"[2]"},{"why":"Establishes natural-gradient preconditioning, the main existing geometric alternative the paper contrasts.","marker":"[22]"},{"why":"Modern preconditioning baseline (Muon) and an example of a metric γ the framework can wrap.","marker":"[6]"},{"why":"Supplies the Bayesian hyperparameter-search methodology used for fair benchmark comparisons.","marker":"[32]"}],"fun_headline_variants":["Loss landscape's metric trains nets, outpaces Adam in tests","New optimizer uses loss geometry to auto-shrink big gradient steps","Induced metric from loss plots yields smoother gradient clipping","Geometric optimizer from loss embedding matches Adam speed, better results","Loss embedding metric drives optimizer with curvature-adaptive steps"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The load-bearing premise is that the squared gradient norm measures local curvature, together with the choice that the ambient metric's vertical component is 1 (which fixes the loss scale); if the gradient is large on a flat but sloped surface, the 'automatic' shrinkage is actually controlled by the free parameter ξ rather than by geometry.","fun_headline_variants_meta":{"raw":{"variants":["Loss landscape's metric trains nets, outpaces Adam in tests","New optimizer uses loss geometry to auto-shrink big gradient steps","Induced metric from loss plots yields smoother gradient clipping","Geometric optimizer from loss embedding matches Adam speed, better results","Loss embedding metric drives optimizer with curvature-adaptive steps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000955,"raw_usage":{"total_tokens":3925,"prompt_tokens":776,"completion_tokens":3149,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":3066}},"tokens_in":520,"tokens_out":3149,"duration_ms":26651,"temperature":1.0,"reasoning_tokens":3066,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:49:42.868608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train on a strictly linear loss L(θ)=a·θ, whose curvature is exactly zero but whose gradient norm ‖a‖ is large. Equation 13 still multiplies the update by 1/(1+ξ‖a‖²), so the claimed curvature-adaptive shrinkage is visibly just gradient-norm clipping. The converse check is a sharply curved quadratic centered near zero with a small gradient: the formula gives almost no shrinkage, contradicting the claim that high curvature triggers smaller steps.","supporting_citations":[{"cited_title":"Visualizing the loss landscape of neural nets,","cited_arxiv_id":null,"evidence_quote":"Supplies the loss-landscape visualization geometry that the paper treats as an implicit metric."},{"cited_title":"Adjustment of an inverse matrix corresponding to a change in one element of a given matrix,","cited_arxiv_id":null,"evidence_quote":"Gives the rank-one inverse formula that converts the pull-back metric into a one-dot-product update."},{"cited_title":"Neural networks for machine learning","cited_arxiv_id":null,"evidence_quote":"Provides the RMSprop diagonal metric adopted in the best-performing variant."},{"cited_title":"On the difficulty of training recurrent neural networks,","cited_arxiv_id":null,"evidence_quote":"Introduces gradient clipping, the behavior the denominator of Eq. 13 reproduces smoothly."},{"cited_title":"Natural gradient works efficiently in learning,","cited_arxiv_id":null,"evidence_quote":"Establishes natural-gradient preconditioning, the main existing geometric alternative the paper contrasts."},{"cited_title":"Muon: An optimizer for hidden layers in neural networks,","cited_arxiv_id":null,"evidence_quote":"Modern preconditioning baseline (Muon) and an example of a metric γ the framework can wrap."},{"cited_title":"Taking the human out of the loop: A review of bayesian optimization,","cited_arxiv_id":null,"evidence_quote":"Supplies the Bayesian hyperparameter-search methodology used for fair benchmark comparisons."}],"review_version":1}