{"id":"591a0805-24d0-4486-8f13-7c63fb7fd6ce","arxiv_id":"2504.16020","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AlphaGrad normalizes gradients layer-wise and applies tanh(alpha times the normalized gradient), reporting advantages over Adam in on-policy PPO but instability in off-policy DQN, with convergence to stationarity proven.","lead":"AlphaGrad is a simple optimizer that normalizes each layer's gradient and squashes it through a tanh curve, controlled by one parameter alpha. It claims better stability than Adam in some reinforcement learning tasks while using less memory, but the evidence is limited to a few benchmarks with no published code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AlphaGrad's PPO superiority claim rests on post hoc alpha selection and unseeded single-run curves; without seed-level CIs the 2x reward gap is unverified.","rationale":"The reader's weakest_assumption identifies the empirical basis of the central PPO claim as the load-bearing premise. That is the right concern. AlphaGrad's algorithmic idea is simple, the deterministic non-convex convergence analysis is plausible (though weak, as the reader notes), and the memory-efficiency claim follows structurally. What would make the paper's central contribution true is that AlphaGrad actually and reliably outperforms Adam in on-policy PPO. The reported evidence does not establish this: no code, no seeds, no error bars, alpha selected post hoc on the same benchmark, and the paper explicitly acknowledges a limited budget and context-dependence. The theoretical results are deterministic and do not apply to stochastic PPO gradients, so they cannot rescue the empirical superiority claim. Thus the strongest conclusion supported by the paper is that AlphaGrad is a plausible optimizer with a formal but weak stationarity guarantee and promising but unverified initial RL results. This matches the CONDITIONAL verdict; the requested additional evidence (code, seeds, error bars, a validation-based alpha selection protocol) would settle the concern. No stronger attack—such as a mathematical error in the convergence proof—survives scrutiny: Lemma 2 and Theorem 2 are internally consistent, and the convex-rate vacuousness near vanishing gradients is explicitly acknowledged. Therefore the reader's conditional assessment is unchanged.","tokens_in":18609,"tokens_out":13212,"duration_ms":139810,"concrete_test":"Re-run the Sec. 6.3 comparison under CleanRL defaults with at least 10 independent seeds per condition (Adam; AlphaGrad α ∈ {98, 196, 294}), reporting mean final return with 95% bootstrap confidence intervals, and pre-specify the α selection rule on a validation split before the final comparison. The headline PPO claim survives only if the confidence interval for the chosen AlphaGrad configuration's advantage over Adam excludes zero; otherwise the reported 2x gap is attributable to selection or seed noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result—that AlphaGrad 'achieves substantially superior performance' over Adam in on-policy PPO (Sec. 6.3)—depends entirely on the assumption that the reported HalfCheetah-v5 curves are representative. The manuscript reports no seeds, no error bars, and no code, and it selects alpha values α∈{98,196,294} by grid search on the same benchmark; the α=98 result, 'more than double Adam's reward,' is the best of that search. The abstract itself concedes the performance profile is 'highly context-dependent' and requires 'careful α tuning.' Because PPO returns are noisy, a single-run comparison against a default CleanRL Adam baseline can produce a large gap through seed luck or post hoc selection. The convergence theorems (Thm 1–2) are deterministic and do not cover stochastic RL, so they cannot substitute for repeated-seed empirical evidence. The reported Hopper/TD3 stability result suffers the same limitation: α is tuned on the evaluation environment, making the observed smoothness partly a fitted outcome rather than a prediction.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AlphaGrad, an optimizer that layer-wise L2-normalizes gradients and then applies a smooth tanh nonlinearity, g' = tanh(alpha * g/(||g||_2 + epsilon)), with alpha acting as a single steepness parameter. The manuscript claims three contributions: (1) the algorithm formulation with a heuristic alpha proportional to sqrt(d_L); (2) a deterministic convergence analysis in convex and non-convex settings; and (3) empirical results on CartPole/DQN, Hopper/TD3, and HalfCheetah/PPO, comparing AlphaGrad against Adam. The reported results are context-dependent: instability in DQN, stability with competitive performance in TD3 when alpha is tuned, and a large PPO advantage for alpha=98. The theoretical sections are self-contained, and the authors explicitly acknowledge several limitations, including the nonstandard convergence measure h(N_t) = N_t^2/(N_t + epsilon) and the possibility that the convex rate becomes vacuous when gamma_min approaches zero.","tokens_in":18705,"tokens_out":6583,"duration_ms":63825,"significance":"If the empirical claims were reproducible, AlphaGrad would be an attractive stateless alternative to Adam for on-policy RL and for memory-constrained training, and the paper's transparent discussion of the theory's limitations is a genuine strength. The proofs of Lemma 2 and Theorems 1-2 are standard and appear correct under the stated assumptions, and the alpha scaling heuristic is clearly motivated. However, the headline empirical claims rest on single-run curves with post hoc alpha selection, and the convergence analysis applies to global gradient normalization rather than the layer-wise normalization used in the implemented algorithm. The empirical contribution therefore needs substantially stronger evidence before the paper's central claims can be accepted.","major_comments":[{"comment":"The central empirical claim that AlphaGrad 'achieves substantially superior performance' over Adam in PPO is based on single-run learning curves with no seeds, no confidence intervals, and alpha values (98, 196, 294) selected by a grid search on the same HalfCheetah-v5 benchmark. Because PPO returns are highly variable, the reported more-than-double reward gap for alpha=98 is not statistically supported and could be due to seed luck or post hoc selection. The same concern applies to the TD3 stability claim in Section 6.2. The paper should report multiple seeds (at least five) with median/interquartile ranges, and should either pre-register the alpha grid or use a separate validation environment for alpha selection before evaluating on the final benchmark.","section":"Section 6.3, Figure 3"},{"comment":"The convergence analysis considers the full parameter vector x and normalizes the complete gradient by its global L2 norm, x_{t+1} = x_t - eta * tanh(alpha * grad f(x_t)/||grad f(x_t)||). The algorithm in Section 2, however, normalizes each layer or tensor independently (Eq. (2)) and uses a layer-specific alpha_L. Consequently, Theorems 1 and 2 do not directly establish convergence for the layer-wise AlphaGrad variant actually evaluated in the experiments. A block-wise or tensor-wise analysis is needed, or the theory must be explicitly restricted to the single-tensor case; as written, the claim of a formal convergence guarantee for the proposed optimizer is not fully supported.","section":"Section 3, Eqs. (8)-(10)"}],"minor_comments":[{"comment":"The figures do not indicate the number of seeds or show any measure of variance; please add per-seed curves or median/interquartile ranges, and state the seed protocol in Section 5.1.","section":"Figures 1-3"},{"comment":"The claim that tensor-wise normalization leads to improved convergence on a 'private regression dataset (not shown in this paper)' is unverifiable and should be removed or replaced with a citable, documented experiment.","section":"Section 5.3"},{"comment":"The notation is inconsistent: Eq. (2) uses subscript p for the normalized gradient, while Eqs. (1) and (3) use subscript L for the layer; please harmonize the notation.","section":"Section 2, Eq. (2)"},{"comment":"The abstract's 'extensive empirical evaluation' overstates the scope, as only three environments are tested, and Section 7, titled 'Further Empirical Validation', contains no new experiments; consider renaming Section 7 to 'Future Work'.","section":"Abstract and Section 7"},{"comment":"The paper describes AlphaGrad as 'conditionally stateless' but the momentum variant stores one variable per parameter; the memory comparison with Adam should state that momentum-enabled AlphaGrad uses half the per-parameter state of Adam, while the stateless regime is memory-free.","section":"Section 2, Eqs. (5)-(6)"},{"comment":"No code or configuration files are provided; given the strong dependence of the results on alpha and the 'reduced learning rate' mentioned in Section 6.2, releasing the exact CleanRL configurations is necessary for reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The main issues are the gap between the headline empirical claims and the evidence, and the mismatch between the theory's global-normalization setting and the algorithm's layer-wise normalization. Neither problem appears unfixable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: AlphaGrad is a simple, novel update rule — normalize each tensor's gradient, then apply tanh(α·ĝ) — that interpolates between normalized gradient descent and sign updates. The paper is honest about its own limits: the author states up front that performance is context-dependent and α requires careful tuning, and the convergence analysis acknowledges that the non-convex rate controls N²/(N+ε) rather than ‖g‖², and that the convex rate degrades near the optimum. That candor is real.\n\nWhat is new: the update rule itself, plus the α ≈ k√d heuristic. Neither is a breakthrough, but the rule is not in the cited literature. The theory is standard — L-smooth analysis with a tanh alignment lemma — and the proofs check out. The empirical section covers DQN, TD3, and PPO and reports failure (DQN instability) as well as success, which is more than many optimizer papers do.\n\nSoft spots: the evidence for the headline claims is thin. No code, no seeds, no error bars; the curves look like single runs. The PPO result (α=98 reaching >3000 vs Adam ~1500) is the best of a three-point grid searched on the same benchmark, so it is a fitted outcome, not a prediction. The TD3 stability claim rests on α values chosen after seeing the evaluation environment. The theory is deterministic and does not cover stochastic RL, so it cannot validate the RL comparisons. The speculative Section 7 (sin(tanh(...)) variants, LLM scaling) is clearly marked as untested and reads as future-work filler rather than evidence.\n\nIn proportion: the author does not overclaim. The abstract explicitly says “highly context-dependent” and “careful α tuning.” The paper is what it says it is: a proposal with preliminary evidence. But the phrase “substantially superior performance in on-policy PPO” is a headline that the current evidence does not support.\n\nWho it is for: someone working on memory-efficient optimizers or RL training who can quickly re-run with proper seeds. The theory is a minor note; the empirical claim needs verification.\n\nRecommendation: send it to peer review, not desk reject. A good referee will ask for code, seeds, error bars, and a predetermined α-selection protocol, or at least all grid values reported. If the PPO gap survives repeated seeds, it is a useful result. If not, the paper still stands as a modest, honest contribution. It deserves serious referee time.","headline":"A genuinely new but simple optimizer with sound, self-aware theory and unverified empirical claims: single runs, no code, alpha chosen post hoc on the test bed.","tokens_in":19339,"tokens_out":2516,"would_cite":false,"duration_ms":23931,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AlphaGrad more than doubles PPO reward over Adam, the paper says","keywords":["AlphaGrad","gradient normalization","tanh clipping","memory-efficient optimizer","reinforcement learning","PPO","non-convex convergence","Adam comparison"],"falsifier":"Run AlphaGrad with $\\alpha=98$ against Adam on HalfCheetah-v5 with PPO across at least ten independent seeds and compare the full distributions of final returns; if Adam's median falls within the spread of AlphaGrad's runs, the paper's headline empirical claim fails. A complementary check repeats the Hopper-v4 TD3 comparison with $\\alpha\\in[100,250]$ over multiple seeds to see whether the reported smoothness survives seed variance.","tokens_in":18289,"feed_emoji":"⚙️","tokens_out":8448,"duration_ms":73144,"temperature":0.7,"pith_summary":"This paper introduces AlphaGrad, an optimizer that updates each weight with $\\tanh(\\alpha \\tilde{g})$, where $\\tilde{g}$ is the layer's gradient divided by its L2 norm. The author is trying to establish that this one-knob, nearly stateless rule can deliver the scale-invariance of adaptive methods without storing per-parameter moments, and that on reinforcement learning it can be a stronger choice than Adam: substantially larger final reward in on-policy PPO, competitive peak reward with smoother training in TD3, though unstable in DQN. It also proves, under standard smoothness assumptions, convergence to stationary points at an $O(1/\\sqrt{T})$ rate. A sympathetic reader would care because, if true, AlphaGrad is a memory-lean alternative to Adam for on-policy training and a concrete demonstration that per-parameter adaptivity is not required for strong RL optimization.","feed_headline":"AlphaGrad more than doubles PPO reward over Adam, the paper says","feed_subtitle":"A single alpha knob shapes normalized-tanh gradient steps; empirical tuning decides when it wins.","key_machinery":"The load-bearing object is the normalized-gradient-tanh map $g' = \\tanh(\\alpha \\cdot g/(\\|g\\|_2+\\epsilon))$, applied per parameter tensor. The L2 normalization removes layer-scale differences, the tanh keeps every coordinate in $(-1,1)$, and $\\alpha$ sets how much of the normalized vector saturates into sign-like behavior. The convergence proof is carried by an alignment inequality, $\\langle g_t, g'_t\\rangle \\ge \\tanh(\\alpha)\\,\\|g_t\\|_2^2/(\\|g_t\\|_2+\\epsilon)$, proven by concavity of $\\phi(x)=\\tanh(\\alpha x)-x\\tanh(\\alpha)$ on $[0,1]$; substituting that inequality into the L-smoothness descent inequality makes the error terms telescope into the stated rates. The paper also proposes the dimensionality heuristic $\\alpha\\approx k\\sqrt{d}$ for choosing the steepness per layer, while emphasizing that empirical tuning typically overrides it.","core_discovery":"The paper's central claim is that the map $g' = \\tanh(\\alpha\\, g/(\\|g\\|_2+\\epsilon))$, applied tensor-wise, is a complete optimizer: it enforces scale invariance, bounds every update component in $(-1,1)$, and interpolates smoothly between normalized gradient descent ($\\alpha\\to 0$) and sign-based updates ($\\alpha\\to\\infty$). On benchmarks, the paper reports AlphaGrad at $\\alpha=98$ reaching rewards above 3000 on HalfCheetah-v5 with PPO, more than double Adam's plateau around 1500, with higher explained variance; on Hopper-v4 with TD3, $\\alpha\\in[100,250]$ gives Adam-competitive peaks with markedly smoother learning curves; on CartPole-v1 with DQN, AlphaGrad is unstable. The convergence analysis proves stationarity in the smooth non-convex case with the $O(1/\\sqrt{T})$ bound stated for the quantity $\\|g_t\\|_2^2/(\\|g_t\\|_2+\\epsilon)$, plus a convex average-iterate bound whose rate degrades as gradients vanish near the optimum.","pith_inferences":["Extending the paper, the bounded $(-1,1)$ update makes AlphaGrad a natural candidate for fixed-point or low-precision training; a direct test would be training a quantized network with AlphaGrad and checking whether the boundedness removes the need for gradient scaling.","A testable prediction outside the paper: adding Adam-style momentum or a short EMA of normalized gradients to AlphaGrad should recover DQN stability while keeping most of the PPO gain, isolating statelessness as the cause of the DQN failure.","Because the PPO gain is far larger than the formal convergence bound predicts, the mechanism is likely interaction with the clipped surrogate objective rather than stationarity; comparing AlphaGrad with signSGD under PPO would separate tanh shaping from sign-like saturation.","The paper's alpha-as-directional-filter examples suggest that a saturation-ratio scheduler, maintaining a target fraction of saturated coordinates, is a plausible alpha-tuning replacement that could remove the per-benchmark grid search."],"forward_implications":["Training with AlphaGrad removes per-parameter first and second moment buffers, so memory per weight drops to zero beyond the weights themselves, with only an optional single momentum scalar per parameter.","In on-policy PPO, the paper's result implies a stateless, bounded update can take larger effective trust-region steps than Adam while keeping value-function learning accurate, since reported explained variance is higher.","In off-policy DQN, the direct corollary is that stateless reactivity without temporal smoothing amplifies TD-loss spikes; the paper says alpha scheduling or added momentum would be needed.","For TD3-style actor-critic training, tuned AlphaGrad implies stability can be separated from peak performance: learning curves can be smooth and monotonic while matching Adam's ceiling.","The non-convex theorem means the optimizer is guaranteed to reach stationarity in deterministic smooth problems, though the stationarity measure is the modified quantity rather than the squared gradient norm itself."],"supporting_citations":[{"why":"The adaptive baseline AlphaGrad is designed to replace; supplies the per-parameter moment state and hyperparameter complexity that motivate the method.","marker":"[4]"},{"why":"The on-policy PPO algorithm used in the main performance claim on HalfCheetah-v5.","marker":"[12]"},{"why":"The TD3 actor-critic algorithm used in the stability-competitive Hopper-v4 result.","marker":"[11]"},{"why":"The off-policy DQN algorithm where AlphaGrad's instability is observed and analyzed.","marker":"[3]"},{"why":"Supplies the single-file RL implementations and default hyperparameters used in all benchmark comparisons.","marker":"[29]"},{"why":"SignSGD is the limit case AlphaGrad approaches as alpha grows, framing the smooth interpolation argument.","marker":"[10]"},{"why":"The standard smooth convex optimization template on which the paper's Theorem 1 convergence proof is built.","marker":"[16]"},{"why":"The standard non-convex convergence template used for Theorem 2's stationarity guarantee.","marker":"[21]"},{"why":"The high-dimensional Gaussian marginal heuristic behind the alpha proportional to sqrt(d) scaling guideline.","marker":"[14]"}],"fun_headline_variants":["AlphaGrad: tanh-normalized optimizer doubles PPO reward vs Adam","Single alpha knob: AlphaGrad stabilizes PPO, doubles reward over Adam","AlphaGrad's tanh trick: PPO rewards 2x Adam with one parameter","Memory-light optimizer AlphaGrad shines in PPO, flops in DQN","AlphaGrad: scale-invariant tanh steps, big PPO wins, careful tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The empirical claims assume that the training curves shown, from single runs with alpha chosen on the same benchmark it is then measured on, represent typical optimizer behavior; if the runs are noisy or the alpha selection is post hoc, the reported advantages over Adam do not generalize.","fun_headline_variants_meta":{"raw":{"variants":["AlphaGrad: tanh-normalized optimizer doubles PPO reward vs Adam","Single alpha knob: AlphaGrad stabilizes PPO, doubles reward over Adam","AlphaGrad's tanh trick: PPO rewards 2x Adam with one parameter","Memory-light optimizer AlphaGrad shines in PPO, flops in DQN","AlphaGrad: scale-invariant tanh steps, big PPO wins, careful tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000458,"raw_usage":{"total_tokens":2319,"prompt_tokens":993,"completion_tokens":1326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":1216}},"tokens_in":609,"tokens_out":1326,"duration_ms":9615,"temperature":1.0,"reasoning_tokens":1216,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:11:41.654795+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run AlphaGrad with $\\alpha=98$ against Adam on HalfCheetah-v5 with PPO across at least ten independent seeds and compare the full distributions of final returns; if Adam's median falls within the spread of AlphaGrad's runs, the paper's headline empirical claim fails. A complementary check repeats the Hopper-v4 TD3 comparison with $\\alpha\\in[100,250]$ over multiple seeds to see whether the reported smoothness survives seed variance.","supporting_citations":[{"cited_title":"A., Veness, J., Bellemare, M","cited_arxiv_id":null,"evidence_quote":"The off-policy DQN algorithm where AlphaGrad's instability is observed and analyzed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the single-file RL implementations and default hyperparameters used in all benchmark comparisons."},{"cited_title":"(2004).Introductory Lectures on Convex Optimization: A Basic Course","cited_arxiv_id":null,"evidence_quote":"The standard smooth convex optimization template on which the paper's Theorem 1 convergence proof is built."},{"cited_title":"and Lan, G","cited_arxiv_id":null,"evidence_quote":"The standard non-convex convergence template used for Theorem 2's stationarity guarantee."},{"cited_title":"(2018).High-Dimensional Probability: An Introduction with Applications in Data Science","cited_arxiv_id":null,"evidence_quote":"The high-dimensional Gaussian marginal heuristic behind the alpha proportional to sqrt(d) scaling guideline."}],"review_version":1}