{"id":"2ae94bcb-a149-4787-b1f8-ba0e5da1c8b9","arxiv_id":"2510.10759","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"ROGER's online threshold-normalized gain adaptation yields near-zero constraint violations and higher primary reward than fixed-gain and adaptive baselines in quadruped and MuJoCo locomotion, including one-hour real-world learning without falls.","lead":"ROGER automatically adjusts how much a robot's learning algorithm weights forward-progress rewards versus safety penalties as the robot interacts with its environment. The authors report near-zero constraint violations and higher forward speed than tuned baselines in a simulated 60-kg quadruped, plus one-hour real-world walking learned from scratch without falls.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Appendix B's guarantee is invalid even granting dR_t/dt≥0: Eq. S4 overclaims a primary-reward guarantee; R0 can decrease while R_t increases.","rationale":"The reader identified the unproven dR_t/dt≥0 assumption as the weakest point. I agree that this is important, but the problem is more severe: even if that assumption were granted, Appendix B's proof of a primary-reward guarantee is mathematically invalid. The paper's central empirical claims—near-zero violations in simulation, higher primary reward, and successful real-world learning—are supported by the reported experiments and do not depend entirely on the theory. However, the overclaimed theoretical guarantee is load-bearing for the paper's advertised safety assurance and for its statement that ROGER is 'proven' stable and reward-increasing. A conditional accept remains appropriate: the empirical core is credible, but the authors must correct or temper the Appendix B guarantee and clarify that constraint satisfaction is an empirical property, not a proven one.","tokens_in":18014,"tokens_out":9161,"duration_ms":76793,"concrete_test":"Independently re-derive Appendix B from Eq. S1 and dR_t/dt≥0 without treating the differential inequality as an ODE. Then test the simple trajectory R0(t)=1-0.1t, R1(t)=0.8-0.1t over t∈[0,6]: R_t increases monotonically from -0.152 to 0.376, so dR_t/dt>0, yet R0 falls from 1.0 to 0.4. If this trajectory is admitted by the stated assumptions, Eq. S4's claimed guarantee fails. A correct derivation should either exclude such trajectories explicitly or retract the guarantee language.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is not just the unproven dR_t/dt≥0 premise, but the internal derivation in Appendix B. Even granting dR_t/dt≥0, the 'Learning Analysis' does not follow. From Eq. S1, dR_t = (1-R1^2)dR0 - R1(2R0+3R1)dR1 ≥0 yields the differential inequality dR0 ≥ [R1(2R0+3R1)/(1-R1^2)] dR1, but the paper then solves this as if it were an ODE with R1 as the independent variable and integrates to obtain Eq. S4, replacing the integration constant c with R0t0 without justification. This step is invalid. A concrete counterexample: take R0 from 1.0 to 0.4 while R1 goes from 0.8 to 0.2. Then R_t = R0 - R0R1^2 - R1^3 increases from -0.152 to 0.376, so dR_t/dt>0, yet the primary reward R0 decreases. Thus Eq. S4's claim that ∫dR0 ≥ R0t0, and hence that the primary reward is guaranteed to increase, is false even under the paper's own monotonic-total-reward assumption. The stability analysis inherits the same unproven premise. Because the paper advertises 'ensuring constraint satisfaction throughout learning' and relies on Appendix B for its theoretical backing, this is a genuine load-bearing problem for the central claim, though the empirical results themselves remain credible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ROGER, an online reward-gain adaptation mechanism for constrained reinforcement learning in robot locomotion. ROGER adjusts the primary-reward and penalty gains at each timestep based on the ratio of estimated penalties to user-specified constraint thresholds, shrinking the primary gain and growing penalty gains near constraint boundaries. The authors claim that ROGER ensures constraint satisfaction during training, improves primary reward over fixed-gain and adaptive baselines, and reduces reward-tuning burden. They support this with simulated and physical experiments on a 60-kg Unitree B1 quadruped, simulated MuJoCo locomotion benchmarks (hopper, Ant, Half Cheetah, Walker2D), and a real-world learning demonstration with no falls in under an hour. A Lyapunov-style analysis in Appendix B is offered as theoretical backing.","tokens_in":18362,"tokens_out":2687,"duration_ms":23587,"significance":"If the empirical claims hold, ROGER would be a practically useful and simple add-on for real-world continual locomotion learning, where constraint violations can be destructive and reward-shaping is labor-intensive. The strengths of the paper are its extensive empirical evaluation: multiple seeds/repetitions, statistical tests, hyperparameter sensitivity analyses, a real-world hardware demonstration, and a clear comparison against fixed-gain, CBF, PDO, CRPO, and OL-AUX baselines. The authors also provide code and videos. However, the paper's theoretical contribution—the claimed guarantee of constraint satisfaction and monotone primary-reward increase in Appendix B—is not internally sound. The derivation contains a load-bearing logical error, and the main premise (dR_t/dt ≥ 0) is neither proven nor consistent with the authors' own exploration experiments. Therefore the central advertised claim of 'ensuring constraint satisfaction throughout learning' is not backed by the theory, and the paper currently overstates what is rigorously established.","major_comments":[{"comment":"The learning analysis is invalid as written. Starting from dR_t ≥ 0 and the simplified total reward R_t = R0 − R0 R1² − R1³, the authors obtain the differential inequality dR0 ≥ [R1(2R0 + 3R1)/(1 − R1²)] dR1. They then solve the associated equality as if it were an ODE, integrate, and identify the constant of integration with R0t0. This step is unjustified: an inequality cannot be integrated as an ODE, and the constant cannot be set to R0t0 without proof. The sketch's claim is actually false: for R0 decreasing from 1.0 to 0.4 while R1 decreases from 0.8 to 0.2, R_t = R0 − R0R1² − R1³ increases from −0.152 to 0.376, so dR_t/dt > 0, yet the primary reward R0 decreases. Thus Eq. (S4), and the derived guarantee that ∫dR0 ≥ R0t0, does not follow. This undermines the theoretical basis for the 'expected primary reward guaranteed to increase' and 'partial stability' claims. The authors should ei","section":"Appendix B, Eq. (S2)–(S4)"},{"comment":"The Lyapunov and learning analyses both rely on the assertion 'Since the learning rule maximizes R_t, it follows that Ṙ_t ≥ 0.' This is not generally true for AGOL or PPO with exploration, and the paper's own Figure S8a shows that excessive learning rates lead to constraint violations, i.e., R_t need not be monotone. The premise is thus unproven and empirically violated in the paper's own experiments. At minimum, the theorem statements in Appendix B must be explicitly conditional on this assumption, and the paper should not state without qualification that ROGER 'ensures constraint satisfaction' or 'guarantees an increase in primary reward' (Section IV, Abstract).","section":"Appendix B, premise dR_t/dt ≥ 0"},{"comment":"There are arithmetic errors in the reported violation probabilities. The text states that ROGER exhibited 'one in 50,000 timesteps' and then equates this to 'an estimated violation probability of around 2×10⁻¹²%.' 1/50,000 = 2×10⁻⁵, which is 2×10⁻³%, not 2×10⁻¹²%. Similarly, the fixed-gain penalty is credited with 'three violations in 25,000 timesteps' but an estimated violation probability of 2×10⁻¹². These numbers differ by many orders of magnitude. This is not a presentation typo only: the quantitative comparison of violation rates is a central empirical claim. The authors should correct these values and restate the corresponding statistical comparisons.","section":"Section III.A.1, violation probabilities"},{"comment":"The paper's abstract and Section II state that ROGER 'ensures constraint satisfaction throughout learning' and that the adaptation strategy is 'proven to be partially stable.' Given the issues above, the theoretical support for these claims is not present. The empirical evidence does show very low violation counts in the quadruped experiments, but the theory does not establish a guarantee. I recommend that the authors rephrase these claims as empirical findings—for example, 'ROGER achieved near-zero violations in these experiments'—and clearly separate what is proven from what is observed.","section":"Section II / Conclusion, 'ensuring constraint satisfaction'"}],"minor_comments":[{"comment":"Two near-identical paragraphs begin with 'Recent techniques proposed in early 2025 build on these approaches.' The duplicate should be removed.","section":"Section I.B (intro)"},{"comment":"Figure 3's caption is identical to the paragraph in Section II, and the full caption is repeated in the figure. Consider shortening the caption to its descriptive part only.","section":"Figures 1 and 3 captions"},{"comment":"The notation is generally clear, but the paper should state explicitly that Δ_t and r_λit are recomputed only at update intervals, not necessarily every timestep, and how the statistical estimate R̃_it is computed over the stored trajectory window.","section":"Equations (2)–(5)"},{"comment":"The YouTube links have extra spaces in the text (e.g., 'Cqu7vLT Piw?si=...'). This is a typesetting issue but should be fixed for the final version.","section":"Section III.A.1, 'videos'"}],"recommendation":"major_revision","confidential_remarks":"The paper has a strong empirical core and would be a valuable addition if the theoretical claims are scaled back to match the actual evidence. The Appendix B derivation is not merely missing a step; it contains an invalid inference that produces a false statement. The authors can likely fix this by downgrading the guarantees to empirical observations and re-running the statistical check on the corrected violation rate. If the flawed theory is removed and the claims are recalibrated, the contribution remains significant for the real-world robot learning community. I would not accept it in its current form because the abstract and conclusions advertise a guarantee that the manuscript does not establish."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ROGER's adaptive gain rule is a genuine new heuristic, and the empirical work—especially the physical quadruped learning to walk in under an hour—is credible. But the Appendix B guarantee does not hold up, and the paper leans on it more than it should.\n\nWhat is new: the continuous, threshold-normalized gain update (Eqs. 2-5) is distinct from PDO's dual ascent, CRPO's switching, and OL-AUX's gradient-based weighting. The simulations use multiple seeds and significance tests, the MuJoCo B1 is a reasonable platform, and the real-robot result is a real existence proof: 300 episodes, no falls, roughly 0.3 m/s by episode 250. The added compute is about 0.5 ms per update, so the cost is negligible. The hopper and Walker2D results showing torque and orientation reductions are a decent stress test beyond the stated assumptions.\n\nSoft spots, in order of severity. First, the Appendix B 'learning analysis' is invalid. The paper asserts dR_t/dt >= 0 without proof; but even granting that, the step from differential inequality (S2) to integral claim (S4) does not follow. The authors solve an equivalent ODE and then set the integration constant c to R0t0 with no justification—the constant is fixed by the initial values of both R0 and R1, not by R0t0. Concrete counterexample: R0 from 1.0 to 0.4 while R1 goes from 0.8 to 0.2 increases R_t (so dR_t/dt > 0 holds) while the primary reward decreases. So the advertised 'guaranteed increase in R0' is false even under the paper's own monotonicity assumption. The stability argument inherits the same unproven premise, and the honest limitation section (gentle system and learning dynamics, proprioceptive constraints only) is in tension with the main text's 'ensuring constraint satisfaction' language. The empirical near-zero-violation numbers can stand as empirical claims; the 'guarantee' language cannot.\n\nSecond, the reported violation probabilities are wrong by orders of magnitude: one in 50,000 timesteps is 2e-3 percent, not 2e-12, and three in 25,000 is 1.2e-4, not 2e-12. Since these numbers are quoted as central evidence, they need correction.\n\nMinor: the intro has a duplicated paragraph, a sentence about the proof appears twice around Figure 3, and the physical experiment has no baseline comparison—acceptable for a demo but worth stating.\n\nOverall: this is a solid subfield contribution, honestly scoped in the limitations, with a method worth citing and an empirical core worth refereeing. A serious reviewer should require the authors to fix or remove the theory section and the arithmetic before publication. Not a desk reject. I would take it, and I would bring it to the group.","headline":"ROGER's adaptive gain rule is genuinely new and the real-robot results are credible, but the Appendix B 'guarantee' does not hold up even on the paper's own assumptions.","tokens_in":18852,"tokens_out":5264,"would_cite":true,"duration_ms":45406,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that by adapting reward and penalty gains online based on how close a robot is to violating physical constraints, a robot can learn to walk from scratch in the real world without falling and without tedious offline gain tu","keywords":["constrained reinforcement learning","reward shaping","robot locomotion","online gain adaptation","constraint satisfaction","quadruped robot","real-world RL"],"falsifier":"Train the same 60-kg quadruped with ROGER at four times the nominal learning rate and count how often body roll or pitch exceeds the 10-degree threshold; the paper's own appendix reports that failures appear under this condition. A cleaner test: inject a controlled perturbation that forces the total reward to decrease sharply mid-episode and check whether ROGER's gains reallocate before the constraint is crossed; if violations occur, the near-zero-violation claim holds only under the gentle-dynamics condition stated in the paper.","tokens_in":17871,"feed_emoji":"🤖","tokens_out":7202,"duration_ms":59590,"temperature":0.7,"pith_summary":"The paper proposes ROGER, a rule that adjusts the weighting between a robot's positive reward and its constraint penalties during learning, based on how close the measured behavior is to user-set safety thresholds. The paper tries to establish that this online adjustment removes the need for offline reward-gain tuning while keeping constraint violations near zero throughout training, something fixed-weight and existing adaptive-weight methods do not reliably achieve. If the claim is right, it matters because safe-in-training learning is the main barrier to teaching heavy robots to walk directly in the real world, where a fall can damage the machine or its surroundings.","feed_headline":"Adaptive gains let a robot learn to walk without falling","feed_subtitle":"It shifts reward weight toward whichever constraint is near violation, and the robot stays upright.","key_machinery":"The central object is the gain-allocation rule ROGER. It computes an urgency signal from the squared ratios of estimated constraint penalties to their thresholds; the primary reward gain is set to one minus that signal, and the signal is distributed among the penalty gains according to each penalty's share of the total urgency. This ratio-based reallocation is what makes the same reward function act performance-oriented in safe states and safety-oriented near constraints, with the only user-set hyperparameters being intuitive threshold values such as a maximum allowable body tilt.","core_discovery":"The central claim is that a single gain-allocation rule, ROGER, can replace manual reward tuning and still satisfy constraints during learning. At every timestep the rule lowers the primary reward gain and raises the penalty gains in proportion to the squared ratio of each estimated penalty to its threshold; near a threshold the primary reward is almost ignored and the offending constraint dominates, while far from thresholds the primary reward dominates and performance is prioritized. In a simulated 60-kg quadruped, this yields one constraint violation in 50,000 timesteps and a final forward speed of 0.6 m/s, 50% higher than careful fixed-gain and reward-switching baselines; on the physical","pith_inferences":["The success of ROGER suggests a general design principle for safe RL: make reward weights explicit functions of constraint proximity rather than tuned constants. A natural next test would be applying the same ratio-allocation rule to manipulation tasks with force or position limits.","Because the rule reacts to statistical estimates of penalties, its safety margin depends on how well those estimates track the true state. An untested extension would be to feed the rule with deliberately noisy or biased sensor estimates to map the boundary of the safety margin.","The paper leaves global-position and other exteroceptive constraints as future work; the same logic could plausibly handle them by treating accumulated position error as a penalty channel with a threshold, but the accumulating nature of such errors is exactly where the current proof's assumptions would be stressed.","If the total-reward-nondecreasing assumption is violated by non-smooth or high-learning-rate updates, ROGER's guarantee weakens; a stress-test variant that deliberately perturbs learning with occasional large decreases would show whether the practical margin covers what the theory does not."],"forward_implications":["Real-world learning from a standstill becomes practical for heavy robots: exploration near a safety boundary is countered immediately by gain reallocation, so the robot need not fall to learn.","Reward tuning reduces to setting constraint tolerances (e.g., 10 degrees of tilt) rather than grid-searching dozens of weighting constants, cutting both human time and the risk of picking unsafe weights.","Constraint satisfaction does not have to come at the cost of task reward in safe regions: the final primary reward matches the unconstrained objective while keeping near-zero violations.","ROGER is an add-on to the reward function, so it can be combined with different control architectures and learning rules rather than replacing them.","The same rule generalizes to more dynamic simulated tasks, where it improves distance and reduces torque and orientation deviation relative to default weights, even though the theoretical safety guarantee is weaker there."],"fun_headline_variants":["Auto-tuned reward gains let a robot learn to walk in an hour","ROGER adapts reward weights, cutting robot falls during learning","One rule auto-adjusts gains for safe, fast robot locomotion learning","Reward gain adaptation achieves near-zero constraint violations","Learned walking without falling: adaptive gains replace manual tuning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The safety guarantee rests on the assumption that the total reward never decreases during learning and that both the robot's dynamics and the learning process are gentle enough for that to hold; when exploration or a high learning rate makes the reward drop abruptly, ROGER can violate constraints, which the paper itself demonstrates for excessive learning rates.","fun_headline_variants_meta":{"raw":{"variants":["Auto-tuned reward gains let a robot learn to walk in an hour","ROGER adapts reward weights, cutting robot falls during learning","One rule auto-adjusts gains for safe, fast robot locomotion learning","Reward gain adaptation achieves near-zero constraint violations","Learned walking without falling: adaptive gains replace manual tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000224,"raw_usage":{"total_tokens":1313,"prompt_tokens":775,"completion_tokens":538,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":466}},"tokens_in":519,"tokens_out":538,"duration_ms":5544,"temperature":1.0,"reasoning_tokens":466,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:14:45.043271+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 60-kg quadruped with ROGER at four times the nominal learning rate and count how often body roll or pitch exceeds the 10-degree threshold; the paper's own appendix reports that failures appear under this condition. A cleaner test: inject a controlled perturbation that forces the total reward to decrease sharply mid-episode and check whether ROGER's gains reallocate before the constraint is crossed; if violations occur, the near-zero-violation claim holds only under the gentle-dynamics condition stated in the paper.","supporting_citations":[],"review_version":1}