{"id":"454e6b70-3b2c-4f1b-b67d-d74ccdf2e989","arxiv_id":"2505.00488","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A two-phase RL framework with a nominal policy and a GRF-rewarded adaptive policy improves payload adaptation for quadrupedal locomotion across varied terrains.","lead":"This paper trains a quadruped robot with a two-stage reinforcement learning controller: a basic walking policy plus an adaptive correction policy for changing payloads. The authors report improved height and speed tracking on flat ground, slopes, and stairs when carrying up to 10 kg.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The adaptive policy's GRF reward encodes the exact payload mass in its force threshold, so 'unknown load adaptation' is not yet separated from training-time load information; an ablation is needed.","rationale":"The reader's weakest assumption identifies the same core issue: the GRF reward uses the exact payload mass m_p in the force threshold, which is available during training but not at deployment. My stress-test agrees that this is the most load-bearing weakness, and adds the observation that the reward and the policy observation share the same Jacobian-based force estimator. This makes the concern sharper: the policy is not merely trained with a reward that knows m_p; it is trained to act on estimated forces that the reward also uses to define the desired behavior. If the force estimator is inaccurate in the real world, the learned correction could be tuned to estimator artifacts rather than to true load changes. The paper gives qualitative figures and hardware demonstrations, which are real evidence, but it does not quantify tracking errors, report error bars, or compare against more than one baseline. These gaps reinforce the reader's CONDITIONAL verdict rather than changing it. The proposed ablation directly tests whether the m_p term in the reward is necessary for the observed adaptation, which is the minimal experiment that would settle whether the 'unknown load' claim is justified.","tokens_in":7410,"tokens_out":3472,"duration_ms":42913,"concrete_test":"Retrain the Phase-2 adaptive policy with the GRF reward threshold fixed to m_r*g (or to a constant independent of m_p), keeping all other rewards, observations, and training settings identical. Then run the same simulation payload profiles (0-4 kg with 4 s resampling) and report height and velocity RMSE with error bars over at least five seeds. If the adaptive policy's advantage over the baseline shrinks or disappears, the training-time m_p term is load-bearing and the 'unknown load' claim should be weakened. As a secondary check, evaluate the original policy on payloads outside the training distribution (e.g., 6-8 kg) and with injected noise in the foot-force estimator, to see whether adaptation transfers without the exact threshold and with imperfect force feedback.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II.E defines the GRF tracking reward as r_GRF = 0.75*(h > h_cmd) + 0.50*(h < h_cmd)*1(sum_i |f_i| > (m_r + m_p)g). This threshold contains the true payload mass m_p at training time, even though the deployed adaptive policy never observes m_p. That is not by itself a deployment leak, but it means the learned corrective policy is optimized against a reward that exactly encodes the payload-dependent force level required. The adaptive observation is (obs, f), where f comes from the same Jacobian-based estimator used to compute the reward, so the policy could learn to match estimated force magnitudes to a threshold supplied by the reward rather than to infer load from the robot's dynamics. If that estimated force is biased or noisy on hardware, the corrective behavior may be tuned to a simulation artifact. The paper provides no ablation that removes m_p from the reward, no analysis of how the policy would behave if the force estimator were inaccurate, and no quantitative comparison between estimated and ground-truth foot forces. Without such evidence, the central claim of 'unknown load adaptation' is not fully established: the controller may be adapting to a training-time reward signal that already knows the payload mass.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an adaptive reinforcement learning framework for quadruped locomotion under unknown payloads and varying terrains. A Phase-1 nominal policy is trained for baseline locomotion, and a Phase-2 adaptive policy is trained to output corrective actions using an augmented observation that includes estimated foot forces. Training is done in Isaac Gym on a Unitree Go1 model, and deployment is demonstrated on a physical Go1 with static and dynamic payloads. The central claim is that the adaptive controller consistently outperforms a DreamWaQ-based baseline in tracking body height and velocity commands on flat ground, slopes, and stairs.","tokens_in":7635,"tokens_out":2919,"duration_ms":34276,"significance":"If the empirical claims hold, the two-phase training scheme is a practical alternative to explicit payload estimation and to conservative domain randomization, with a hardware demonstration on a real quadruped. The paper also correctly identifies a relevant gap: model-based payload adaptation often relies on predefined gaits or trajectory generators. However, the evidence provided is mostly qualitative raw time series, with no quantitative summary statistics, error bars, or ablations, so the strength of the central claim is not yet commensurate with the level of the assertion. The paper does not ship code or reproducible training configurations, which limits independent verification.","major_comments":[{"comment":"The central empirical claim that the adaptive controller 'consistently outperformed' the baseline is supported only by qualitative descriptions and raw time-series plots (Figures 4, 5, and 6). No means, standard deviations, error bars, RMSE/MAE values, or success rates are reported for height or velocity tracking, and no trial counts are given for the hardware experiments. This is load-bearing: without aggregated quantitative results, the reader cannot assess the statistical significance or the magnitude of the claimed improvement.","section":"III.A and III.B"},{"comment":"The GRF tracking reward uses the exact payload mass in the threshold (m_r + m_p)g. Although the deployed policy does not observe m_p, the training-time reward provides a precise payload-dependent force target, and the policy's observation includes the same estimated foot forces that appear in the reward. The claim of 'unknown load adaptation' is therefore not fully separated from training-time access to the payload. The authors should provide an ablation that removes m_p from the reward (e.g., a fixed or mass-independent threshold), an analysis of robustness to force-estimator bias or noise, and a quantitative comparison between estimated and ground-truth foot forces. Without such evidence, the paper does not establish that the learned corrective behavior is a genuine load-adaptation mechanism rather than a reward-shaped response to a known force threshold.","section":"II.E, GRF tracking reward"},{"comment":"The baseline is described only as 'DreamWaQ, with the base mass randomized in the range [0, 10] kg.' No details are given about the baseline's training configuration, architecture, reward weights, observation space, or training budget, making it impossible to judge whether the comparison is fair. In addition, the paper does not compare the adaptive policy against its own Phase-1 nominal policy without the corrective term, which would be a more direct ablation of the proposed contribution. The claim of consistent superiority is also weakened by the absence of quantitative results for slopes in the main text and by the lack of hardware plots for slopes and stairs.","section":"III.A, baseline comparison"}],"minor_comments":[{"comment":"The observation vector is written as o_t = [w_t, g_t, c_t, θ_t, θ̇_t, a_{t-1}], but the text uses 'wt' and 'gt' in the description; please use consistent notation for the angular velocity and gravity vector.","section":"II.D, observations"},{"comment":"The caption refers to 'flat red segments' in the curves, but the figure itself is not included in the text and the meaning of the red segments is not defined. Please clarify what the red color indicates and describe the failure mode explicitly.","section":"III.A, Figure 5"},{"comment":"The hardware section reports qualitative observations for slopes and stairs but does not show any plots or metrics for those terrains. Given the abstract's claim of 'across all settings,' quantitative hardware results for slopes and stairs should be included or the scope stated more precisely.","section":"III.B, hardware experiments"},{"comment":"The reference list contains several self-citations and recent arXiv preprints, but no code or data availability statement is given. For an empirical RL paper with a project website, releasing training configurations and evaluation scripts would greatly improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope for the robotics audience, but it currently reads as an early-stage empirical report. The absence of quantitative metrics and ablations, plus the unresolved question of how much the GRF reward's payload-dependent threshold drives the result, are the main barriers. A revision that adds these analyses could make the paper publishable; without them, the central claim is not fully established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on 2505.00488. The genuinely new piece is the two-phase training: a nominal locomotion policy plus a separate adaptive policy that gets foot-force observations and a GRF-tracking reward, trained after the nominal policy is frozen. That staged setup is a real departure from DreamWaQ and RMA, and the hardware results on a Go1 with up to 10 kg payload are a point in its favor. The idea that a corrective policy can be learned without explicit payload estimation is well motivated, and the paper does show the adaptive action norm tracking payload changes on hardware.\n\nThe soft spots are mostly about evidence quality, not the idea. First, the GRF reward in Section II.E uses the true payload mass in the threshold: 0.75*(h > h_cmd) + 0.50*(h < h_cmd)*1(sum_i |f_i| > (m_r + m_p)g). That is a privileged training signal. It is not a deployment leak—the policy never sees m_p—but it means the adaptive policy is optimized against a reward that knows the payload, and the paper's claim of 'unknown load adaptation' needs an ablation that removes m_p from the reward (e.g., a fixed threshold or an estimated-mass term) to show the policy is not just matching a training-time target. That is the main substantive issue.\n\nSecond, the results are reported as raw time series. There are no error bars, no aggregated statistics, no ablations of the reward weights or the adaptive observation, and the baseline is one DreamWaQ variant. Slopes get a sentence and no plot. The simulation claims about 'consistently outperformed' would be much stronger with a table of tracking errors across conditions. Code and data release would also help.\n\nNone of this invalidates the core approach. The two-phase adaptive policy is sensible, the hardware experiments are non-trivial, and the qualitative evidence is directionally consistent. The citation pattern looks fine; [12] is a self-citation but not load-bearing.\n\nWho is this for? Robotics researchers working on legged locomotion and RL adaptation. They will find the staged training idea worth reading, but they will want the ablations before building on it. I would send it to peer review: the method is novel enough and the hardware evidence is real, but the authors need to tighten the evaluation and address the reward-privilege question. Not a desk reject, but it will need solid revision.","headline":"A useful two-phase adaptive RL idea with real hardware evidence, but the GRF reward's training-time use of the true payload mass and the lack of quantitative ablations keep the central claim from being fully established.","tokens_in":8174,"tokens_out":2782,"would_cite":false,"duration_ms":29225,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A two-policy RL controller lets a quadruped carry changing payloads across terrain without explicit load estimation.","keywords":["quadrupedal locomotion","reinforcement learning","adaptive control","payload adaptation","ground reaction force tracking","terrain robustness","Unitree Go1","Proximal Policy Optimization"],"falsifier":"Retrain the adaptive policy with the GRF tracking reward threshold set to robot mass only, $(m_r + 0)g$, instead of $(m_r + m_p)g$, then deploy under a 6 kg payload: if height tracking error grows to the baseline's level, the reported adaptation was enabled by training-time knowledge of the true payload rather than by proprioceptive adaptation.","tokens_in":1416,"feed_emoji":"🦾","tokens_out":2615,"duration_ms":71076,"temperature":0.7,"pith_summary":"The paper tries to establish that adding a second, adaptive policy on top of a normal locomotion policy lets a quadruped keep tracking body height and velocity commands as payloads change on flat ground, slopes, and stairs. The adaptive policy learns corrective joint actions that generate more ground reaction force when the payload increases, without explicit payload estimation, gait design, or manual tuning. In large-scale simulation and on a physical Unitree Go1 quadruped carrying up to 10 kg, the adaptive controller consistently outperforms the baseline in command tracking. If true, this means learned corrective actions, rather than model-based parameter updates, can handle unknown and changing loads in real time.","feed_headline":"Adaptive RL keeps a Go1 upright under payloads up to 10 kg","feed_subtitle":"A corrective policy improves height and velocity tracking on flat ground, slopes, and stairs.","key_machinery":"The load-bearing mechanism is the two-phase PPO training scheme plus a corrective action $\\Delta a_t$ produced by an adaptive policy $\\pi_\\phi$ that receives the nominal observations augmented with estimated foot forces. Foot forces are estimated from joint torques via the Jacobian pseudoinverse $f = (J(\\theta)^T)^\\dagger \\tau$, and the adaptive policy is rewarded by a GRF tracking term $r_{\\mathrm{GRF}}$ that encourages total foot forces above $(m_r + m_p)g$ whenever height is below command. The combined action $a_t + \\Delta a_t$ is applied to the environment, and the adaptive reward weights are set so that the corrective policy focuses on height stabilization and ground reaction force generation while the nominal policy keeps velocity tracking.","core_discovery":"The central claim is that an adaptive policy, trained in a second phase alongside a frozen nominal policy, produces corrective actions whose magnitude tracks the payload: when the body height drops below the commanded height, the adaptive policy increases the estimated foot forces and restores tracking. The paper validates this by comparing the adaptive controller against a baseline on flat ground, stairs, and slopes, and by showing on hardware that the baseline becomes unstable or stalls under high payloads while the adaptive controller continues. The authors claim this works without explicit payload parameter estimation and without predefined gait schedules, relying instead on the corrective policy's learned response to observed state and estimated foot forces.","pith_inferences":["The GRF tracking reward uses the true payload mass $m_p$ in its threshold during training, so the decisive control experiment would be to retrain the adaptive policy with a payload-invariant threshold; the paper does not report this ablation.","Since foot forces are estimated from commanded torques through a Jacobian pseudoinverse, the policy's corrective behavior may degrade on soft or slippery ground where the commanded torque-to-force mapping is unreliable; that setting is outside the reported tests.","The corrective-policy architecture is agnostic to what causes the tracking error, so the same two-phase design could be trained for external pushes, uneven step heights, or actuator degradation without changing the framework.","A testable extension is to provide the latent context vector from the encoder to the adaptive policy and train the two policies end-to-end, allowing the nominal policy's state estimate to change rather than only offsetting actions."],"forward_implications":["On flat ground, the adaptive controller reduces body-height tracking error that otherwise grows with payload, while velocity tracking stays comparable to the baseline.","On stairs, the baseline controller stalls under high payloads, while the adaptive controller maintains forward progression and lower height and velocity tracking errors.","On hardware, the adaptive controller carries progressive loads up to 10 kg with stable locomotion, whereas the baseline shows foot scuffing and instability at higher payloads.","The norm of the corrective action rises and falls with each payload change, suggesting the policy acts as an implicit load detector without being explicitly told to intervene.","Because no gait schedule or manual gain tuning is used, the same two-phase recipe could apply to other command-tracking tasks beyond height and velocity."],"supporting_citations":[{"why":"Supplies the rapid motor adaptation formulation that motivates learning an implicit context from observation history.","marker":"[7]"},{"why":"Provides the actuator-network pre-training used to track Go1 joint angles in simulation and the multi-behavior reward idea.","marker":"[8]"},{"why":"Supplies the encoder loss (CE Net) and serves as the baseline controller against which the adaptive policy is compared.","marker":"[9]"},{"why":"Establishes the massively parallel GPU simulation training setup the paper uses for large-scale policy training.","marker":"[10]"},{"why":"Provides the Proximal Policy Optimization objective used to update both the nominal and adaptive policies.","marker":"[17]"},{"why":"Supplies the actuator dynamics model that makes simulated joint behavior match the physical quadruped.","marker":"[18]"}],"fun_headline_variants":["Adaptive policy handles unknown payloads on rough terrain","Quadruped learns to carry unknown loads without predefined gaits","Corrective RL restores stability under varying payloads on slopes and stairs","Unitree Go1 stays on track with adaptive corrections for unknown load"],"cache_read_input_tokens":10368,"weakest_assumption_plain":"The training reward for corrective actions uses the true payload weight in its force threshold, even though the deployed controller never sees the payload; if the learned response actually relies on that hidden information, the claim of unknown-load adaptation is much weaker.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive policy handles unknown payloads on rough terrain","Quadruped learns to carry unknown loads without predefined gaits","Corrective RL restores stability under varying payloads on slopes and stairs","Unitree Go1 stays on track with adaptive corrections for unknown load"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1393,"prompt_tokens":842,"completion_tokens":551,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":479}},"tokens_in":458,"tokens_out":551,"duration_ms":5733,"temperature":1.0,"reasoning_tokens":479,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:40:03.741202+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the adaptive policy with the GRF tracking reward threshold set to robot mass only, $(m_r + 0)g$, instead of $(m_r + m_p)g$, then deploy under a 6 kg payload: if height tracking error grows to the baseline's level, the reported adaptation was enabled by training-time knowledge of the true payload rather than by proprioceptive adaptation.","supporting_citations":[{"cited_title":"Dreamwaq: Learning robust quadrupedal locomotion with implicit terrain imagination via deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the encoder loss (CE Net) and serves as the baseline controller against which the adaptive policy is compared."}],"review_version":1}