{"id":"e438ed65-eb7f-49cc-b032-40bb86a9be6b","arxiv_id":"2505.15514","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":10,"one_line_summary":"AM-PPO modulates GAE advantages with a feedback-controlled tanh gate and reports improved reward trajectories on MuJoCo benchmarks, tested once per configuration.","lead":"AM-PPO is a proposed modification to PPO that reshapes reinforcement learning advantage signals with an adaptive tanh gate before using them in policy and value updates. The paper reports higher final rewards on MuJoCo control tasks, but the evidence rests on single-seed runs with no released code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central empirical claim rests on single-seed runs with no error bars; the reported reward advantages and clipping reductions may be artifacts of run-to-run variance rather than AM-PPO's mechanism.","rationale":"The reader's verdict is REJECT with high confidence, and the identified weakest assumption is the single-seed evaluation. After independently reviewing the paper, I agree that this is the most load-bearing concern. The entire contribution is an empirical claim of superiority, and the evidence consists of one run per configuration. No error bars, no repeated seeds, no statistical significance test, and no released code to independently verify or extend the runs. If the result is a lucky seed, the abstract's claims are unsupported. I considered other potential concerns, such as the biased value targets in Eq. 10 or the lack of ablations isolating the adaptive controller, but those are secondary: even a perfect mechanism would fail to be established by the current evidence. The paper honestly states its computational limits and lacks a formal proof, but honesty does not substitute for the missing multi-seed statistics. The verdict of REJECT is appropriate, and no adjustment is needed. The concrete test of running 10 seeds per condition would either confirm the claim or expose it as variance.","tokens_in":10905,"tokens_out":4337,"duration_ms":42112,"concrete_test":"Re-run the Swimmer-v4, Hopper-v4, Ant-v4, and Humanoid-v4 comparisons with at least 10 random seeds per algorithm-optimizer configuration, and report the full distribution of final rewards and DynAG clipping fractions. If the 95% confidence intervals of final reward overlap for any environment, or if the DynAG clipping reduction is not consistently observed across seeds, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Abstract, Section 4.4) is empirical: AM-PPO achieves superior reward trajectories and reduces DynAG clipping. The evidence for this is Section 4.1, which states 'All experiments were run with a set seed' — a single seed per configuration, with no error bars, no repeated trials, and no statistical tests. The appendix further admits resource constraints and promises code but does not release it. In RL, single-seed comparisons are known to be unreliable: different seeds can reverse performance rankings, especially over 1M timesteps on MuJoCo tasks where variance is high. The reported 'consistent' advantage and the 'markedly suppressed' clipping fraction could arise from one favorable seed. This is load-bearing because if a different seed reverses the reward ordering, the central claim is false. The paper itself cannot distinguish signal from noise. While the method description is detailed and the design is heuristically motivated, the absence of multi-seed evidence means the empirical premise is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AM-PPO, an extension of PPO that adaptively modulates GAE advantage estimates before they are used in both the policy and value-function losses. The modulation consists of an EMA-driven adaptive scaler αA, a tanh-based gate, and a saturation-feedback controller, together producing a non-linear, batch-wise transformed advantage signal. The authors give a qualitative motivation for the design, describe the algorithm in detail, and report experiments on four MuJoCo/Gymnasium environments comparing PPO with AM-PPO under Adam and under the author's DynAG optimizer. The central claim is that AM-PPO improves reward trajectories, sustains learning progress, and reduces clipping required by adaptive optimizers.","tokens_in":11076,"tokens_out":4806,"duration_ms":47094,"significance":"If the claims were supported, the idea of using a learned/adaptive, saturation-targeted tanh gate on advantages, applied consistently to both actor and critic, would be a useful and potentially generalizable signal-conditioning technique. The paper's formulation is explicit and self-contained, and the authors are candid about the heuristic nature of the theory. However, the significance is currently limited by the evidence: the empirical evaluation is the sole support for the central claims, and that evaluation is preliminary in ways the manuscript itself acknowledges.","major_comments":[{"comment":"The central empirical claim rests on a single seed per configuration, with no error bars, no repeated trials, and no statistical tests. The paper states 'All experiments were run with a set seed' (Section 4.1), and the abstract claims 'superior reward trajectories' and 'sustained learning progression.' In high-variance continuous-control benchmarks, single-seed comparisons can easily reverse rankings, so the reported advantages over PPO could be due to a favorable seed. This is a load-bearing issue because the method has no formal convergence or optimality guarantees, and the paper's own Section 3 explicitly says no formal proof is presented.","section":"Section 4.1, Figure 2, Section 4.4"},{"comment":"The claim that AM-PPO 'significantly reduces the clipping required by adaptive optimizers' is supported only by the author's own DynAG optimizer, on two main-text environments (Swimmer-v4 and Hopper-v4), again without error bars. The paper itself lists multi-seed robustness, hyperparameter sensitivity, and longer training as future work (Appendix B), and the Ant-v4/Humanoid-v4 ablations are described as preliminary with a '2 million step limitation.' These self-identified gaps undercut the strength of the abstract's claims and need to be addressed with actual experiments before the paper can be accepted.","section":"Section 4.4, Appendix B, Appendix A"},{"comment":"The value-function target Vtarget,j = Amod,j + Vφold(sj) is a non-linear, batch-dependent transformation of an already estimated advantage, not an unbiased estimate of the return. Because this target is used to train the critic that subsequently produces the GAE advantages in Eq. (1)-(2), the method's closed-loop behavior is not justified by the usual PPO/GAE bias-variance argument. The paper provides only a heuristic discussion of why the modulated target might help; given the lack of a formal analysis, the empirical validation must be strong enough to show the modified critic target does not introduce harmful bias, and the current single-seed experiments do not provide that evidence.","section":"Section 3.2, Eq. (10)"}],"minor_comments":[{"comment":"The sentence 'The sign of the modulated advantage Amod is determined by the sign of this gate Mgate' is imprecise; since αA and the L2 norm are positive, Eq. (5) implies the sign of Amod is the sign of Araw. Please rephrase.","section":"Section 2.2, Eq. (5)"},{"comment":"The statement 'All experiments were run with a set seed and achieved similar rewards, but PPO-AM constantly achieved a higher final reward' is internally ambiguous; specify whether the same seed was used across algorithms and what 'similar rewards' means.","section":"Section 4.1"},{"comment":"The paper lacks a table of final mean rewards or other quantitative summaries; the plots are hard to read precisely, and the claimed 'consistently higher' performance should be accompanied by numeric values and, ideally, confidence intervals.","section":"Section 4, Figure 2"},{"comment":"There is a typographical error in the definition of Mgate: the opening parenthesis before κshared is unbalanced ('Mgate = ( κshared · tanh(ZA,mb)').","section":"Section 2.2, Eq. (5)"},{"comment":"DynAG is cited only to a GitHub repository, and Hybrid GRPO is a self-cited preprint; the paper should describe the DynAG baseline sufficiently in the text or use a standard implementation so readers can reproduce the clipping-fraction comparison.","section":"References [3], [13]"},{"comment":"The code is only promised ('will be made available'); for reproducibility, the revision should include the repository URL and a version/commit identifier, or the code should be released with the paper.","section":"Appendix C"},{"comment":"The phrase 'Proof of this connection can be observed across both environments tested' is too strong for observational training curves; 'consistent with' or 'suggestive of' would be more appropriate.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is honest about its limitations and the method description is clear, but the empirical evidence is far below the bar for the strength of the abstract's claims. I would not reject outright because the central issues are fixable in principle: multi-seed experiments with error bars, a sensitivity analysis for the new hyperparameters, and a more careful discussion of the value-target bias. I would also make code release a condition of acceptance, since the DynAG-based comparison and the exact training curves are hard to evaluate otherwise."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: AM-PPO is a clearly specified, well-written heuristic extension of PPO, but the paper's central empirical claim is currently unverified. The method description is detailed and the author is refreshingly honest about what is missing — no formal proof, single GPU, limited compute. That honesty doesn't repair the evidence.\n\nWhat's actually new: a concrete modulation formula that L2-normalizes raw GAE advantages, passes them through a tanh gate scaled by an adaptively controlled alpha, and uses the modulated signal for both policy and value targets. The controller targeting a saturation probability is a specific, reproducible design. The equations are self-consistent and the hyperparameters are listed. For someone wanting to try this, the paper is enough to reimplement it. That is real credit.\n\nThe soft spot is the load-bearing one: every reported result is a single seed. Section 4.1 says 'All experiments were run with a set seed.' No error bars, no repeated runs, no statistical test. On MuJoCo tasks, seed-to-seed variance can reverse rankings, so the claimed 'superior reward trajectories' and 'markedly suppressed' DynAG clipping may be one lucky draw. The appendix compounds this by admitting the experiments were compute-limited and code is only promised, not released. The theory section also explicitly states there is no formal proof; it's qualitative intuition, which is fine as motivation but not as support for the empirical claims.\n\nThe citation pattern is mostly fine. Standard PPO and GAE are cited properly. The self-citations to DynAG and Hybrid GRPO are relevant since the method builds on those ideas. That's not by itself a flaw.\n\nVerdict: reject under current evidence, not because the idea is bad, but because the paper cannot distinguish signal from noise. It's a solid write-up of a proposal, not a demonstration. Who is it for? Researchers working on PPO variants who might want to test this on their own benchmarks with proper seeds. It doesn't deserve a serious referee until the author or someone else supplies multi-seed results and releases the code.\n\nRecommendation: desk reject with encouragement to resubmit after adding at least 5–10 seeds per configuration, error bars, and a public repository.","headline":"Clearly specified heuristic extension of PPO, but the load-bearing empirical claim rests on single-seed runs with no error bars; useful as a proposal, not as evidence.","tokens_in":11643,"tokens_out":1855,"would_cite":false,"duration_ms":16620,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AM-PPO adaptively reshapes PPO's advantage estimates with a tanh-based controller and reports higher rewards, sustained learning, and reduced clipping on continuous-control benchmarks.","keywords":["proximal policy optimization","advantage modulation","generalized advantage estimation","adaptive scaling","continuous control","reinforcement learning","tanh gating","clipping reduction"],"falsifier":"Rerun the PPO and AM-PPO configurations from the paper on Swimmer-v4 and Hopper-v4 with at least 10 seeds, keeping the stated hyperparameters, and compare the final-reward distributions and the DynAG clipping-fraction curves; if AM-PPO's mean final reward falls inside the PPO seed noise or the clipping suppression fails to reproduce, the paper's central empirical claim is not supported.","tokens_in":10646,"feed_emoji":"🤖","tokens_out":10172,"duration_ms":79028,"temperature":0.7,"pith_summary":"AM-PPO is a proposed change to Proximal Policy Optimization: before advantage estimates drive policy and value updates, they are adaptively reshaped by a tanh-based gate whose scaling factor is set by a controller tracking the batch's L2 norm, standard deviation, and how often the transformed signal saturates. The same modulated advantages are used to build the value-function target, so the actor and critic learn from one consistently conditioned signal. The paper reports that on Swimmer-v4, Hopper-v4, Ant-v4, and Humanoid-v4 this produces higher final rewards, a more gradual and continued reward growth instead of early spikes and plateaus, and a marked drop in the clipping fraction of an adaptive optimizer. The motivation is that raw advantage signals are noisy, non-Gaussian, and poorly scaled, and that adaptive nonlinear conditioning of the signal itself matters as much as the optimizer that consumes it. If the effect is real, the paper has identified a broadly applicable way to condition learning signals, not just a PPO-specific patch.","feed_headline":"Alpha-modulated PPO beats plain PPO on MuJoCo continuous control","feed_subtitle":"Adaptive tanh gating of advantage estimates sustains reward growth and cuts optimizer clipping in tests.","key_machinery":"The machinery is an adaptive advantage conditioner built around a feedback-controlled scaler and a tanh gate. A controller keeps exponential moving averages of an adaptive scaling factor $\\alpha_{A,\\mathrm{ema}}$ and of the observed saturation ratio $s_{\\mathrm{prev},A,\\mathrm{ema}}$; once per iteration it computes a target $\\hat\\alpha_A = \\kappa_{\\mathrm{shared}}\\frac{N_A+\\epsilon_A}{\\sigma_A}\\left(\\frac{p^{\\star}_A}{s_{\\mathrm{prev},A,\\mathrm{ema}}+\\epsilon_A}\\right)^{\\eta_A}$, clamps and smooths it, and observes what fraction of $|Z_A|$ exceeds the saturation threshold $\\tau_A$. During PPO update epochs, the frozen controller state sets $Z_A = \\alpha_A \\tilde{A}^{\\mathrm{raw}}$ for each minibatch, and the gate $\\kappa_{\\mathrm{shared}}\\tanh(Z_A)$ multiplies the raw advantage magnitude. In short, the controller continuously retunes how hard the tanh gate compresses or amplifies advantages, making the standard PPO update statistics-driven rather than fixed.","core_discovery":"The central claim is that AM-PPO's $\\alpha$-modulation mechanism improves PPO's learning dynamics. Starting from raw generalized-advantage estimates $A^{\\mathrm{raw}}_t$, the method normalizes by the batch's L2 norm, scales by an adaptively updated $\\alpha_A$, and computes $A^{\\mathrm{mod}} = |A^{\\mathrm{raw}}| \\odot (\\kappa_{\\mathrm{shared}}\\,\\tanh(\\alpha_A \\tilde{A}^{\\mathrm{raw}}))$. The gate term is bounded by $\\kappa_{\\mathrm{shared}}$, and the magnitude of each modulated advantage is the raw magnitude times this bounded gate. Both the clipped policy objective and the value target $V^{\\mathrm{target}} = A^{\\mathrm{mod}} + V_{\\phi_{\\mathrm{old}}}$ use the transformed signal, so policy and value learning share the same conditioning. The paper argues this stabilizes gradients and conditions the policy-gradient landscape, and it cites as empirical support higher final rewards, sustained reward growth, altered entropy behavior, and suppressed clipping by the adaptive optimizer.","pith_inferences":["A direct testable prediction of the saturation-feedback design is that disabling the feedback branch (setting $\\eta_A = 0$ or freezing the saturation state) should remove the entropy-rebound and clipping-suppression effects; the paper does not report that control experiment, but it follows from the controller equations.","Because the modulation statistics are computed per minibatch, the method's behavior should shift with minibatch size and rollout length; varying those hyperparameters is a cheap way to test whether the reported benefit is due to the modulation or to incidental scaling changes.","If the entropy rebound really is triggered by plateaued advantage statistics, the same controller could be used as an explicit exploration scheduler for other on-policy algorithms, decoupling exploration from hand-tuned entropy coefficients.","The decisive generalization test is beyond MuJoCo continuous control; a discrete-action environment would be a direct next experiment because the modulation pipeline drops into the PPO loss unchanged."],"forward_implications":["On the tested continuous-control environments, AM-PPO reports higher final rewards than standard PPO under both Adam and the DynAG optimizer, with reward trajectories that keep climbing rather than plateauing early.","The internal clipping fraction of the DynAG optimizer is much lower when it consumes AM-PPO's modulated advantages, indicating that the modulated signal is better scaled and needs fewer aggressive update corrections.","AM-PPO changes exploration behavior: policy entropy is consistently lower on Swimmer-v4 and shows a rebounding increase during Hopper-v4 reward plateaus, hinting at adaptive exploration tied to learning stagnation.","Because the value function is trained on targets built from modulated advantages, value-loss dynamics differ from standard PPO; the paper observes higher value loss on Hopper-v4 even while rewards improve.","The alpha-modulation principle is proposed as a general signal-conditioning device, with sketched extensions to reward modulation, value-function-free Monte Carlo returns, and Q-learning or TD-error streams."],"supporting_citations":[{"why":"Defines the PPO clipped surrogate objective that AM-PPO builds on and that serves as the baseline in every comparison.","marker":"[1]"},{"why":"Supplies Generalized Advantage Estimation, the source of the raw advantage signals that AM-PPO then modulates.","marker":"[2]"},{"why":"Describes the DynAG optimizer whose clipping fraction is the paper's key diagnostic and the adaptive-optimizer comparison point.","marker":"[3]"},{"why":"Provides the Gymnasium continuous-control environments used for all reported experiments.","marker":"[10]"},{"why":"Provides the CleanRL implementation framework in which both PPO and AM-PPO were implemented and compared.","marker":"[11]"}],"fun_headline_variants":["AM-PPO's adaptive tanh gating cuts clipping and lifts rewards","Adaptive alpha modulation makes PPO learn more consistently","PPO with alpha-modulated advantages earns higher rewards","Tanh-scaled advantages reduce clipping in PPO training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every experimental comparison rests on runs with a single fixed seed per environment, so the reported differences in rewards and clipping could be seed-specific rather than caused by the modulation.","fun_headline_variants_meta":{"raw":{"variants":["AM-PPO's adaptive tanh gating cuts clipping and lifts rewards","Adaptive alpha modulation makes PPO learn more consistently","PPO with alpha-modulated advantages earns higher rewards","Tanh-scaled advantages reduce clipping in PPO training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000515,"raw_usage":{"total_tokens":2515,"prompt_tokens":975,"completion_tokens":1540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":591,"completion_tokens_details":{"reasoning_tokens":1471}},"tokens_in":591,"tokens_out":1540,"duration_ms":10184,"temperature":1.0,"reasoning_tokens":1471,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:15:42.455739+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the PPO and AM-PPO configurations from the paper on Swimmer-v4 and Hopper-v4 with at least 10 seeds, keeping the stated hyperparameters, and compare the final-reward distributions and the DynAG clipping-fraction curves; if AM-PPO's mean final reward falls inside the PPO seed noise or the clipping suppression fails to reproduce, the paper's central empirical claim is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the DynAG optimizer whose clipping fraction is the paper's key diagnostic and the adaptive-optimizer comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Gymnasium continuous-control environments used for all reported experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CleanRL implementation framework in which both PPO and AM-PPO were implemented and compared."}],"review_version":1}