{"id":"f5cb0eee-b4f2-4470-9d61-65e74ec9a54e","arxiv_id":"2411.16914","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"This paper introduces a 'gradient glass' model of ReLU-induced loss landscape non-smoothness and an optimizer, Alice, that combines Hessian and glass curvature, yielding modest gains over Adam on several benchmarks.","lead":"A new framework models the jagged, non-smooth changes in neural network loss caused by ReLU boundaries as a 'gradient glass', and it derives an optimizer, Alice, that uses both Hessian and glass curvature to take better steps. The paper reports modest accuracy improvements over Adam on several benchmarks, and argues the Hessian alone misses crucial loss topography.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 4's 3/2 bound is not derived: it substitutes |X| for a lower-bounded loss change, and Theorem 5 then treats an expectation bound as a pointwise majorizer, so the Alice step rests on an unsupported mapping.","rationale":"The paper's central claim is that ReLU-induced gradient discontinuities create a 'glass' term that must be added to the Hessian to explain loss topography and to build better optimizers. Theorems 1-3 give a plausible construction and are mostly self-contained; Theorem 6 is exact algebra. The hinge is Theorem 4, because without the 3/2 bound there is no justification for the Alice modified Hessian in Equations (8)-(9) and no theoretical basis for preferring the glass density rho over the Hessian diagonal h. The reader identified this hinge, and I agree. I would sharpen the objection: the proof does not merely rely on an unproven 'local floor'; it makes a specific false inference from boundedness to the half-normal expectation. A bounded random variable with fixed variance can have arbitrarily large mean, so the floor alone cannot yield the coefficient sqrt(2/(3 pi)). In addition, the transition from an expectation bound to a pointwise upper bound in Theorem 5 requires an explicit stochastic optimization framework; as written, the optimality claim is unsupported. The empirical results are modest and do not isolate the theoretical mechanism from generic adaptive scaling. The paper has useful components, but the central theoretical claim needs either a real derivation from ReLU dynamics or an explicit stochastic majorization setup. Therefore I see no reason to change the reader's CONDITIONAL verdict.","tokens_in":14987,"tokens_out":9343,"duration_ms":98182,"concrete_test":"Analytical check of the proof's inference: let X ~ N(0, sigma^2) be the 'virtual' loss change and consider all nonnegative Y with Var(Y) = sigma^2. Show that sup E[Y] is infinite by taking Y = 0 with probability 1-p and Y = M with probability p, with p(1-p)M^2 = sigma^2; then E[Y] = sigma sqrt(p/(1-p)) diverges as p -> 1. For example, with sigma = 1 and p = 0.99, E[Y] is about 9.95, far exceeding sqrt(2/pi) ~ 0.798 used in Equation (7). This settles that a floor plus fixed variance does not produce the half-normal expectation, so Theorem 4 as written is invalid and Theorem 5's step derivation lacks its required majorizer.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3, Theorem 4 is the foundation for the claimed 3/2 power law and for the Alice update in Theorem 5. The proof models the coordinate loss change as a zero-mean normal with variance rho_k |delta_k|^3 / 3 and then 'enforces the local floor Delta L >= 0' by replacing the realized trajectory with its absolute value. The reported bound is simply E|X| for a Gaussian X. This step is not a consequence of ReLU dynamics. A nonnegativity constraint plus fixed variance does not imply a half-normal expectation: for any fixed sigma^2 and any C > 0 there is a nonnegative Y with Var(Y) = sigma^2 and E[Y] > C (take Y = 0 with probability 1-p and Y = M with probability p, where p(1-p)M^2 = sigma^2 and pM > C). So the local floor alone cannot justify Equation (7). Moreover, Equation (7) is an expectation bound, yet Theorem 5 converts it into the deterministic inequality L(mu + delta) <= L(mu) + delta^T(g + 1/2 h * delta) + sqrt(2/(3 pi)) rho^T |delta|^3 and minimizes that expression to obtain Equations (8)-(9). Unless the glass randomness is framed explicitly as an expectation over a distribution of losses and the step is minimizing a stochastic majorizer, the claimed optimal modification to quasi-Newton steps is not established. The empirical improvements of Alice are therefore not backed by the stated theory.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that the Hessian is not always a faithful local model of loss curvature in deep ReLU networks, because ReLU boundaries create many small gradient discontinuities. These discontinuities are modeled as a \"gradient glass\": a dense collection of pseudorandom gradient perturbations whose local density can be estimated from ordinary gradient evaluations. The paper derives a bound on expected loss increase that scales as the 3/2 power of the displacement (Theorem 4), an optimal quasi-Newton step that combines Hessian and glass terms (Theorem 5), an optimal diagonal-estimation kernel and perturbation density (Theorems 2 and 3), and exactness properties for Nesterov-accelerated updates (Theorem 6). It then presents the Alice optimizer and experiments on ResNet18, a ViT, and a Tensorised Transformer showing competitive or improved early loss reduction compared with Adam, SGD-M, and AdaHessian.","tokens_in":124,"tokens_out":4691,"duration_ms":108375,"significance":"If the theoretical claims hold, the paper offers a genuinely new way of thinking about non-smooth loss landscapes: instead of ignoring ReLU-induced derivative discontinuities or treating them as noise, it proposes an explicit density model and exploits it in optimization. The optimal-kernel and optimal-density results (Theorems 2 and 3) are clean variational derivations, and Theorem 6 is a neat algebraic identity relating Nesterov coefficients to momentum. The authors also provide code and reproducible experimental infrastructure with min/median/max reporting. However, the central mathematical support for the 3/2 power law and for the Alice step is currently incomplete, because the key derivation in Theorem 4 rests on an unsupported absolute-value mapping and Theorem 5 converts an expectation bound into a pointwise majorizer. The empirical results are suggestive, but they do not by themselves establish the theoretical framework.","major_comments":[{"comment":"The proof of Theorem 4 \"enforces the local floor\" by mapping each realized loss trajectory to its absolute value, i.e. ∆L(δ_k) = |∆(δ_k)|, and then uses the half-normal expectation to obtain Equation (7). This mapping is not a consequence of the condition ∆L(δ_k) ≥ 0, nor is it derived from ReLU dynamics. For any fixed variance σ² and any target C > 0, there exists a nonnegative random variable Y with Var(Y) = σ² and E[Y] > C, for example Y = 0 with probability 1−p and Y = M with probability p where p(1−p)M² = σ² and pM > C. Thus the nonnegativity constraint alone cannot justify the Gaussian absolute-value calculation. Since Equation (7) is the sole basis for the claimed 3/2 power law, this is a load-bearing gap.","section":"Section 3, Theorem 4 and Appendix A, proof of Theorem 4"},{"comment":"Equation (7) is an expectation over the random glass perturbations, but the proof of Theorem 5 treats the bound as a deterministic pointwise majorizer: it writes L(µ + δ) ≤ L(µ) + δᵀ(g + ½h∗δ) + sqrt(2/(3π)) ρᵀ|δ|^{3/2} and minimizes that expression. An expectation bound does not imply a pathwise inequality for every realization, unless the glass randomness is explicitly framed as a distribution over losses and the step is defined as minimizing a stochastic majorizer. Without such a framing, Equations (8) and (9) do not follow from the preceding theorem, and the claimed optimality of the Alice update is not established.","section":"Section 3, Theorem 5"},{"comment":"The paper passes from v(δ) ≤ R|δ| to v ≈ ρ ∗ |δ| with ρ = diag(R) by asserting that this is \"a simple and reasonable approximation.\" This diagonal approximation is not proved or quantitatively bounded; it is a modeling assumption. Because Theorem 4 and Theorem 5 both depend on the diagonal form v ≈ ρ ∗ |δ|, this assumption is load-bearing. At minimum, the paper should provide an empirical validation of the diagonal approximation against off-diagonal contributions, or a bound controlling the error introduced by neglecting off-diagonal terms.","section":"Section 3, Theorem 1 and the diagonal approximation"}],"minor_comments":[{"comment":"The full topography update in Algorithm 2 sets parameters to µ before evaluating g(0), but the text and Equations (17)–(21) say that evaluations are anchored to ν rather than µ. This appears to be a typo (θ ← ν), and it should be corrected because the running averages otherwise mix two different evaluation centers.","section":"Algorithm 2, line 7"},{"comment":"The notation p(P) should explicitly state that P is a subset of coordinates and that the sums run over i ∈ P; currently the definition is compressed into the sentence following the equation.","section":"Equation (2) and Figure 1"},{"comment":"The condition P_{j≠i} M_{ij}² = ω_i² m_i² requires ω_i to be known before applying the optimal kernel, but the theorem statement does not discuss how ω_i is obtained in practice; the later remark that ω_i² ≤ 1 for diagonally dominant matrices is useful and should be highlighted earlier.","section":"Theorem 2 statement"},{"comment":"There is a typo: \"showing how that the largest Hessian eigenvalues\" should read \"showing how the largest Hessian eigenvalues.\"","section":"Related Work"},{"comment":"The reported AdaHessian range on ResNet18 (78.41% to 91.50%) is much wider than the other methods; a sentence explaining whether this is due to learning-rate sensitivity or to the specific architecture would help the reader interpret Figure 6.","section":"Appendix C and Tables"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the journal's scope and the authors are transparent about their experimental setup. The main issue is not novelty or presentation but the derivation gap in Theorem 4 and the expectation-to-pointwise step in Theorem 5. If the authors can replace the absolute-value mapping with a principled stochastic model—or downgrade the claim from a rigorous bound to a heuristic—the paper could be publishable after substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the gradient glass framework is a real conceptual contribution: the idea that ReLU boundaries create a pseudorandom gradient perturbation field, with a density matrix R that captures sub-linear gradient variations, deserves attention. Second, the central bound that makes the Alice step 'optimal' is not actually derived. The stress-test note is correct. In Theorem 4, the proof models the coordinate loss change as zero-mean normal, then 'enforces the local floor' by mapping every trajectory to its absolute value. That is how they get E|X| = sqrt(2/(3π)) rho^{1/2}|delta|^{3/2}. But nonnegativity plus fixed variance does not imply a half-normal expectation; a nonnegative random variable with the same variance can have arbitrarily large mean. So Equation (7) is a modeling postulate, not a theorem. The subsequent leap in Theorem 5, from an expectation bound to a deterministic majorizer used to derive the modified quasi-Newton step, compounds the problem. The Alice update might be a useful heuristic, but the optimality claims are not supported.\n\nWhat is genuinely new and done well: Theorem 1's ReLU-to-glass derivation is plausible and novel; Theorems 2 and 3 are clean, even if the Rademacher optimality is likely known in the randomized linear algebra literature; Theorem 6 is a neat exactness result. The experiments are honestly presented, with min/median/max over seeds, and the consistent improvements over Adam on ResNet18 and the ViT are real but modest. The paper openly admits the Tensorized Transformer results are mixed. I also credit them for flagging the local floor assumption rather than hiding it.\n\nSoft spots beyond Theorem 4: the pseudorandom assumption on ReLU pre-activations is plausible but not directly tested; the local floor is not empirically validated; there is no code release despite the repo link; and the AdaHessian baseline looks poorly tuned (78-91% accuracy on ResNet18), which weakens the comparison. These are minor relative to Theorem 4.\n\nWho this is for: researchers working on loss-landscape analysis and practical optimizers. It deserves a serious referee because the framework is novel and the empirical direction is useful, but the current form is not acceptable as-is. I would send it to peer review with a request for major revision: either justify the floor from network dynamics, or honestly reframe the 3/2 bound as a heuristic assumption and soften the optimality claims. A code release would help substantially. My verdict is more skeptical than the reader's 'conditional'—the central theory is load-bearing and it's a hole—but the paper is a solid basis for revision.","headline":"The gradient glass idea is genuinely interesting, but Theorem 4's 3/2 bound is postulated rather than proven, so the paper's main theoretical support for Alice is currently a hole.","tokens_in":15928,"tokens_out":2267,"would_cite":false,"duration_ms":23355,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07","90C53","65K05"],"pacs":[],"model":"deepseek-v4-flash","headline":"ReLU crossings create a 'gradient glass' that the Hessian cannot see, and a modified quasi-Newton step exploits it.","keywords":["gradient glass","ReLU boundaries","loss landscape","curvature of expectation","quasi-Newton methods","Nesterov acceleration","gradient variations","diagonal estimation"],"falsifier":"Measure the exponent $p$ in Equation (2) on a deep network with smooth activations (e.g., tanh or sinusoidal); if $p=2$ at all parameter subsets and training epochs, the glass term is an artifact of ReLU boundaries and the Hessian suffices. Alternatively, on a small ReLU MLP, estimate $\\mathbb{E}[\\Delta L(\\delta)]$ empirically for many Rademacher perturbations at several scales and check whether the data respects the 3/2-power bound with a single coordinate-independent $\\rho$; a systematic violation would refute the floor construction.","tokens_in":14734,"feed_emoji":"🧊","tokens_out":9481,"duration_ms":72838,"temperature":0.7,"pith_summary":"This paper argues that the Hessian is not always the right second-order object for describing loss curvature in deep ReLU networks, because each ReLU creates a parameter boundary whose crossing adds a pseudorandom jump to the gradient. The accumulated effect of many such jumps is a 'gradient glass' whose density $\\rho$ can be estimated from ordinary gradient evaluations, and the paper derives an upper bound on expected loss increase of the form $\\mathbb{E}[\\Delta L] \\le \\sqrt{2/(3\\pi)}\\,\\rho^{1/2\\,T}|\\delta|^{3/2}$, a 3/2 power law rather than the quadratic law a smooth Hessian would give. From this bound the paper constructs an optimal diagonal estimator with Rademacher perturbations and a modified quasi-Newton step that combines glass density with the Hessian diagonal. The resulting algorithm, Alice, demonstrates improved loss reduction on ResNet, Vision Transformer, and transformer language-model benchmarks. A sympathetic reader would care because this reframes curvature in deep learning: what matters near gradient discontinuities is the curvature of expectation, not the pointwise Hessian.","feed_headline":"ReLU boundaries create a 'gradient glass' that beats the Hessian","feed_subtitle":"A 3/2-power bound on loss change yields a modified quasi-Newton step that lifts accuracy on deep networks.","key_machinery":"The gradient-glass density matrix $\\rho$ and the 3/2-power expectation bound of Theorem 4 are the load-bearing machinery: they replace the pointwise Hessian with a local density of gradient variations that arises as the continuum limit of many ReLU boundaries. Theorems 2 and 3 supply the optimal kernel and the Rademacher sample distribution that make $\\rho$ and the Hessian diagonal $h$ computable from ordinary backpropagation, and Theorem 5 assembles both terms into the modified Hessian $\\bar h$ that defines the optimal displacement. Theorem 6 closes the loop by fixing the Nesterov coefficients that make this approximation exactly track the true gradient's linear dependence.","core_discovery":"The paper's central claim is that the Hessian alone frequently fails to describe loss curvature in deep ReLU networks, because ReLU activations generate a dense family of parameter-space half-space boundaries; crossing one changes the gradient by a pseudorandom vector, so the expected change in loss under a displacement $\\delta$ is governed by a 3/2 power law, $\\mathbb{E}[\\Delta L(\\delta)] \\le \\sqrt{2/(3\\pi)}\\,\\rho^{1/2\\,T}|\\delta|^{3/2}$, with $\\rho$ the local density of gradient variations. The paper shows that this density, together with the locally averaged Hessian diagonal, can be estimated optimally from three gradient evaluations per step using Rademacher-distributed perturbations and the kernel $\\kappa_i^*(\\delta_i)=c^{-1}\\delta_i/(\\delta_i^2+\\omega_i^2)$, and that the resulting curvature yields the modified quasi-Newton step $\\delta = -g * \\bar h^{-1}$, where $\\bar h = \\hat h + h + \\sqrt{\\hat h*(\\hat h+2h)}+\\varepsilon$ and $\\hat h = 3\\rho*(4\\pi|g|+\\varepsilon)^{-1}$. It also proves that with Nesterov coefficients $\\varphi=1-\\beta_1$ and $\\omega=1$, the running gradient average exactly cancels unknown linear dependencies while damping errors by $\\beta_1$.","pith_inferences":["If the gradient glass dominates early training, sensitivity-based pruning criteria built on Hessian diagonals (such as magnitude times the second derivative) may mis-rank parameters; reweighting by $\\rho$ could make pruning and quantization more accurate.","The local-floor assumption is the most fragile link: if loss decreases are correlated across coordinates rather than reflected independently, the bound may shift toward a quadratic law; comparing fitted exponents across activation functions (ReLU vs tanh vs GELU) would test the regime.","The framework implies a testable robustness prediction: for a fixed parameter perturbation radius, layers with higher glass density should show larger loss jumps, so measuring $\\rho$ per layer could predict which layers need smaller quantization step sizes.","Alice's quick-step mode reuses a stale $\\rho$ for several gradient steps; an adaptive refresh rule driven by the rate of gradient change would be a natural next step that the paper does not explore."],"forward_implications":["Optimizers for ReLU networks should estimate and use the glass density $\\rho$ alongside the Hessian diagonal, since the 3/2-power bound shortens safe step lengths near discontinuities and prevents quasi-Newton overshoot.","The exponent $p$ in Equation (2) becomes a measurable diagnostic: values between 1 and 2 indicate the landscape is in the glass regime, while $p=2$ signals a smooth (Hessian-dominated) region.","Estimating the diagonal of a linear operator from matrix-vector products is solved optimally by Rademacher perturbations and the kernel of Theorem 2; this recipe applies to any local average of a Hessian, not only to ReLU networks.","The Nesterov coefficients $\\varphi=1-\\beta_1$, $\\omega=1$ convert the running gradient average into an exact tracker of hidden linear dependencies, so Alice-style updates can be extended to any momentum-based optimizer."],"supporting_citations":[{"why":"Defines the ReLU activation whose derivative discontinuities generate the gradient glass.","marker":"Nair and Hinton (2010)"},{"why":"Provides the ResNet18 architecture used in the power-law experiment that motivates the framework.","marker":"He et al. (2016)"},{"why":"Supplies the CIFAR-10 dataset for the key gradient-variation measurements.","marker":"Krizhevsky and Hinton (2009)"},{"why":"Early work on second-order (Hessian diagonal) optimization that the paper extends.","marker":"Becker, Le Cun et al. (1988)"},{"why":"Uses diagonal Hessian for parameter importance (Optimal Brain Damage), a target application for glass-aware curvature.","marker":"LeCun, Denker, and Solla (1989)"},{"why":"Overviews second-order optimization and justifies using approximate Hessians, grounding the paper's methodological stance.","marker":"Bottou, Curtis, and Nocedal (2018)"},{"why":"Provides the Adam optimizer whose momentum coefficient and step bounds Alice adapts.","marker":"Kingma and Ba (2014)"},{"why":"AdaHessian is the curvature-based baseline Alice is compared against.","marker":"Yao et al. (2021)"},{"why":"Reports v-shaped loss structure along random trajectories, cited as empirical support for the local-floor predicate in Theorem 4.","marker":"Li, Li, and Zhan (2024)"},{"why":"Proposed Nesterov accelerated quasi-Newton methods, which Theorem 6 refines with exactness coefficients.","marker":"Ninomiya (2017)"}],"fun_headline_variants":["ReLU boundaries create a gradient glass that outperforms the Hessian","Optimal kernel from ReLU glass yields better quasi-Newton steps","Alice exploits curvature glass to beat Hessian in deep learning","3/2-power loss bound from gradient glass improves optimization","Curvature of expectation: ReLU glass surpasses Hessian estimates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is the 'local floor' assertion in Theorem 4: that loss reductions caused by small parameter changes cannot go below a coordinate-independent floor, so each loss trajectory can be reflected through zero; this floor is imposed by taking absolute values, not derived from ReLU dynamics.","fun_headline_variants_meta":{"raw":{"variants":["ReLU boundaries create a gradient glass that outperforms the Hessian","Optimal kernel from ReLU glass yields better quasi-Newton steps","Alice exploits curvature glass to beat Hessian in deep learning","3/2-power loss bound from gradient glass improves optimization","Curvature of expectation: ReLU glass surpasses Hessian estimates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000357,"raw_usage":{"total_tokens":2022,"prompt_tokens":1121,"completion_tokens":901,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":737,"completion_tokens_details":{"reasoning_tokens":813}},"tokens_in":737,"tokens_out":901,"duration_ms":8427,"temperature":1.0,"reasoning_tokens":813,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:45:05.581518+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the exponent $p$ in Equation (2) on a deep network with smooth activations (e.g., tanh or sinusoidal); if $p=2$ at all parameter subsets and training epochs, the glass term is an artifact of ReLU boundaries and the Hessian suffices. Alternatively, on a small ReLU MLP, estimate $\\mathbb{E}[\\Delta L(\\delta)]$ empirically for many Rademacher perturbations at several scales and check whether the data respects the 3/2-power bound with a single coordinate-independent $\\rho$; a systematic violation would refute the floor construction.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the ReLU activation whose derivative discontinuities generate the gradient glass."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CIFAR-10 dataset for the key gradient-variation measurements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Early work on second-order (Hessian diagonal) optimization that the paper extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Proposed Nesterov accelerated quasi-Newton methods, which Theorem 6 refines with exactness coefficients."}],"review_version":1}