{"id":"64325bc7-b402-4bfa-a04a-ca1da9dde78e","arxiv_id":"2508.10423","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Limb-as-agent MAPPO with a shared global critic speeds up humanoid walking policy training and improves gait smoothness over single-agent PPO in simulation and on hardware.","lead":"This paper trains a humanoid robot to walk by treating each limb as an independent learning agent that shares a common critic, rather than training the whole body as one agent. The approach reportedly converges faster and yields smoother gaits than single-agent reinforcement learning, and it transfers to a real robot.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MASH vs. single-agent PPO baseline is confounded by undisclosed information asymmetry: the baseline's observation/critic setup is unspecified while MASH's critic uses privileged global state.","rationale":"The reader's weakest assumption correctly identifies the information asymmetry between MASH and the baseline. MASH's critic uses privileged global information (external forces, friction, mass, contact masks) while the baseline's setup is entirely undisclosed. This is the most load-bearing concern because the paper's headline contribution—that MARL decomposition improves single-humanoid locomotion—rests entirely on the comparison against this baseline. Without controlling for the critic's privileged information, the observed gains could be a trivial consequence of giving the critic more information, not of the multi-agent formulation. The paper also lacks error bars and multiple seeds, which compounds the uncertainty, but the information asymmetry is the primary confound because it directly affects the internal validity of the experimental comparison. The concrete test proposed—running a single-agent PPO with the same privileged critic and observations—would settle whether the multi-agent design itself is responsible. The reader's CONDITIONAL verdict is appropriate: the idea is plausible and the hardware demo is encouraging, but the evidence is insufficient without this control. I therefore recommend no change to the reader's verdict.","tokens_in":12526,"tokens_out":5295,"duration_ms":59014,"concrete_test":"Train a single-agent PPO baseline with the same reward function, the same 106-dimensional privileged critic input as MASH (including f_t, τ_t, friction, mass, contact masks), and an actor observation that is the full concatenation of all limb observations (64-dim for bipedal, 116-dim for full-body) with the same one-hot IDs and timing signals. Use the same policy network architecture, learning rate, and training length (3000 episodes, 48-step episodes), and run at least 5 random seeds with reported mean ± standard error. If this controlled baseline closes the gap in TConv, final reward, and Table II metrics, the MASH advantage is explained by information/architecture rather than the multi-agent decomposition. If MASH still converges faster and achieves better coordination, the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that the multi-agent decomposition (limbs as independent agents) accelerates convergence and improves coordination over single-agent PPO. The only evidence is a comparison against a 'Single-agent PPO' baseline, but Section V-A3 does not specify the baseline's observation space, critic inputs, network architecture, or hyperparameters. Meanwhile, MASH's critic (Section IV-A) receives a privileged 106-dimensional global state including external disturbance forces f_t, external torques τ_t, friction coefficient μ_t, body mass m_t, and stance/contact masks. If the baseline is trained with actor-only observations and a critic lacking this privileged information, the observed improvements in Fig. 4 and Table II could stem entirely from asymmetric information rather than from the MARL formulation. The paper attributes the gains to the multi-agent design, but the experiment cannot distinguish that explanation from a single-agent PPO that uses the same privileged critic and observation preprocessing. This is a load-bearing confound because it directly undermines the causal claim that treating limbs as agents is better.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MASH, a cooperative-heterogeneous MARL framework for a single humanoid robot. Each limb (two legs, two arms) is treated as an independent agent, with parameter sharing across symmetric limbs and a centralized global critic under the CTDE paradigm. The method is evaluated in Isaac Gym against a single-agent PPO baseline on bipedal and arm-swing walking, with metrics for convergence time, action smoothness, torso stability, and limb coordination, followed by a real-world deployment demonstrating a periodic knee trajectory. The central claim is that modeling the locomotion problem as cooperative MARL accelerates training convergence and improves whole-body coordination relative to conventional single-agent RL.","tokens_in":12816,"tokens_out":3950,"duration_ms":49733,"significance":"If the empirical comparison were clean, the paper would make a useful contribution by extending the recent idea of applying MARL to a single legged robot from quadrupeds (MASQ) to humanoids, with hardware validation. The real-world deployment with domain randomization is a concrete strength. However, the current evidence does not yet support the central claim: the single-agent PPO baseline is underspecified and likely lacks the privileged global information available to MASH's critic, the results are single-seed, and several evaluation metrics overlap with the reward terms used for training. These are fixable within the scope of a revision, so the paper is worth pursuing, but the comparison must be made rigorous before the claimed advantage can be accepted.","major_comments":[{"comment":"The single-agent PPO baseline is described only as 'conventional single-agent PPO'; its observation space, critic inputs, network architecture, and hyperparameters are not given. Meanwhile, MASH's critic takes a 106-dimensional privileged global state that includes external disturbance forces f_t, external torques τ_t, friction coefficient μ_t, body mass m_t, and stance/contact masks. If the baseline uses actor-only observations and a critic without these privileged quantities, the improvements in Fig. 4 and Table II may be entirely due to information asymmetry rather than the MARL decomposition. This is the load-bearing comparison for the paper's central claim. Please specify the baseline completely and add an ablation: single-agent PPO with the same privileged critic input and the same observation preprocessing as MASH's actor, plus full hyperparameter tables for both methods.","section":"§V-A3 and §IV-A"},{"comment":"All results appear to come from a single training run per condition. RL locomotion is high-variance, and the reported convergence-time differences (~1306 vs ~1661 and ~1017 vs ~1238 iterations) are modest; without multiple seeds and error bars these differences cannot be distinguished from stochasticity. Report at least five independent seeds with mean ± std (or box plots) for convergence time and deployment metrics, and state the number of seeds in the figure captions and table notes.","section":"§V-B, Fig. 4, Table II"},{"comment":"The evaluation metrics are not independent of the training objective. T_conv is defined directly from the training reward curve, S_action matches the 'Action Smoothness 1' reward term (Table I), S_torso combines quantities penalized by the Orientation and Base Height rewards, and C_limb is shaped by the temporal director of Eq. (12). Table II may therefore be measuring reward-shaping effects rather than genuine behavioral superiority. Please report task-level metrics that are not directly part of the reward, such as forward walking speed versus command, energy consumption, push-recovery success rate, or a holdout validation curve, and define convergence using such a validation measure rather than the training return.","section":"§V-A2 and Table I"},{"comment":"The agent decomposition is not fully specified. The text states that each limb is an independent agent, but the actor network receives concatenated observations from 'two agents' for the legs and 'two agents' for the arms, and the critic outputs a value V_t ∈ R^4. Equations (10)–(11) show shared policies across left/right limbs, so it is unclear whether there are four agents with two shared policies, or two groups. This matters for interpreting the MARL contribution and for reproducing the baseline comparison. Clarify the exact number of policies, how agent IDs are used, and what the four value outputs represent. Also state how the single-agent PPO baseline's action space and actor/critic sizes were matched.","section":"§IV-A and §IV-B"}],"minor_comments":[{"comment":"The sentence 'To enhance sim-to-real transfer, we employ domain randomization.' is duplicated verbatim in the same paragraph.","section":"§IV-C"},{"comment":"The text says S_action is the squared L2 norm of the second-order difference, but Eq. (13) uses (a_{i,t+1} - a_{i,t})^2, which is a first-order difference. Please align the definition and formula.","section":"§V-A2, Eq. (13)"},{"comment":"The y-axis label 'Referece position' is misspelled; also clarify the units and time/step axis for the hip pitch trajectory.","section":"Fig. 5"},{"comment":"The episode length is stated as 48 steps; please state the control frequency and physical time per episode, since 48 steps may be less than one full stride and affects the interpretation of the gait metrics.","section":"§V-A3"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the hardware validation is a positive element, but the central comparison is currently confounded by an underspecified baseline. The authors should also distinguish their contribution from the closely related MASQ work [24] more explicitly, since the present paper is essentially the humanoid analogue; a clear comparison of the algorithmic differences would help the editor and reviewers assess novelty."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a clean, readable paper that applies an existing idea — treating each limb of a legged robot as an independent agent trained with MAPPO and a shared global critic — to a humanoid, adding two arm agents. The authors cite MASQ [24] and do not oversell conceptual novelty. The real contribution is an empirical demonstration on a physical humanoid, with a smooth gait. The writing is clear and the MDP formulation is standard.\n\nWhere it needs work: the central evidence, Figure 4 and Table II, compares against a single-agent PPO baseline whose observation space, critic structure, and hyperparameters are not described. MASH's critic gets a privileged 106-dimensional global state (external forces, friction, mass, contact masks). If the baseline is vanilla PPO with local observations, then the comparison confounds the multi-agent decomposition with asymmetric information. A single-agent PPO with the same privileged critic and the same temporal director / shared context might perform just as well. This is not a minor detail; it directly undercuts the causal claim that treating limbs as agents helps. The paper needs an ablation that controls for the critic information.\n\nAlso missing: error bars, multiple seeds, and ablations. The evaluation metrics overlap with reward terms (action smoothness, torso stability, limb coordination are shaped in Table I), so improvement on those metrics is partly expected. Convergence time is defined from the same reward curve used for training, which is a measure-construction issue rather than circular reasoning, but it would be stronger to report wall-clock time or sample complexity.\n\nNone of this suggests the method is wrong. The hardware result shows the policy transfers and walks stably, which is worth something. But the paper's claim of superiority over single-agent RL is not yet supported.\n\nWho is this for? Researchers working on per-limb decomposition for humanoid control, and people thinking about CTDE for single-robot problems. A referee can usefully push for a controlled baseline, a proper single-agent asymmetric-critic variant, and multiple seeds. I would send it to review, but I would expect heavy revision.","headline":"Plausible extension of limb-as-agent MARL to humanoids, with a nice hardware demo, but the central comparison to single-agent PPO is confounded by an unspecified baseline and a privileged critic.","tokens_in":13253,"tokens_out":2766,"would_cite":false,"duration_ms":30434,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Treating each limb as an independent learning agent produces faster, smoother humanoid walking than training the whole body as one agent.","keywords":["humanoid locomotion","multi-agent reinforcement learning","MAPPO","centralized training decentralized execution","cooperative agents","sim-to-real transfer","whole-body control","gait coordination"],"falsifier":"Train the single-agent PPO baseline with the same 106-dimensional privileged observations used by MASH's critic (and the same reward), then compare convergence and deployment metrics; if the gap collapses, the multi-agent design is not the cause. A second check is to strip MASH's global critic down to per-limb local critics and see whether the coordination metrics hold.","tokens_in":12487,"feed_emoji":"🤖","tokens_out":3682,"duration_ms":39436,"temperature":0.7,"pith_summary":"Most humanoid locomotion policies are trained as a single reinforcement-learning agent that controls the whole body. MASH instead treats the two legs and two arms as four independent agents that share a global critic, and is trained with multi-agent PPO under centralized training with decentralized execution. The paper claims this accelerates convergence, raises asymptotic reward, and improves limb coordination compared with single-agent PPO, and that the resulting policy transfers to a physical robot. The point, if right, is that decomposing one robot into cooperating limb-agents can be a better inductive bias for whole-body control than monolithic policies.","feed_headline":"Four limb-agents beat one whole-body agent at walking","feed_subtitle":"Treating each leg and arm as a cooperative RL agent speeds training, smooths the gait, and transfers to real hardware.","key_machinery":"The load-bearing object is the decPOMDP reformulation of a single body: each limb is an agent, with shared-parameter actor networks across symmetric limbs, a global critic under centralized training with decentralized execution (CTDE), and a temporal director that injects gait phase $T_i(t) = \\sin(2\\pi(kt + \\Delta_i))$. The shared actors exploit left–right symmetry; the global critic provides a cooperative value signal; the phase offsets synchronize stepping and arm swing.","core_discovery":"The central claim is that the MDP for a single humanoid can be reformulated as a decPOMDP whose agents are the limbs, and that MAPPO with a shared global critic learns a better walking policy than single-agent PPO. Each leg agent observes joint positions, velocities, previous action, timing phase, torso orientation, and its own one-hot ID (32 dimensions); each arm agent observes a 26-dimensional analogue; and the critic sees a 106-dimensional global state including forces, friction, mass, and contact masks. Two shared-parameter actors—one for legs, one for arms—are trained with a shared reward and a temporal director $T_i(t) = \\sin(2\\pi(kt + \\Delta_i))$ that encodes gait phase. Experiments r","pith_inferences":["The reported advantage may partly reflect information asymmetry: the single-agent PPO baseline likely lacked the global critic's privileged state, so the gain could come from extra observations rather than from multi-agent credit assignment.","A cleaner test of the cooperation story is to ablate the global critic: if per-limb local critics perform as well, the symmetry and parameter sharing may be doing the work, not the centralized value function.","The temporal director is effectively a gait rhythm prior; with more limbs or DOF, learning the phase offsets rather than hand-setting them could be a natural extension.","If the limb-decomposition result holds across robots, it suggests whole-body loco-manipulation—where arms already have distinct roles—could inherit the same architecture without retraining from scratch."],"forward_implications":["MARL becomes a viable paradigm for single-robot control, not just multi-robot teams.","Shared-parameter actors with a phase-offset temporal director can encode gait symmetry and inter-limb coordination directly into the policy architecture.","Domain randomization combined with MAPPO supports sim-to-real transfer for humanoid walking.","Faster convergence means lower sample complexity for high-dimensional humanoid control tasks.","The same decomposition could apply to other robot configurations, as the paper states as future work."],"supporting_citations":[{"why":"Supplies the PPO algorithm used as both the single-agent baseline and the foundation of MAPPO.","marker":"[43]"},{"why":"Provides the multi-agent PPO objective that MASH optimizes for the limb agents.","marker":"[45]"},{"why":"Isaac Gym is the GPU-accelerated simulation environment where MASH is trained.","marker":"[47]"},{"why":"Prior work applying MARL to a single quadruped robot, the direct precedent MASH extends to humanoids.","marker":"[24]"},{"why":"Establishes the centralized-training-with-decentralized-execution paradigm that MASH adopts.","marker":"[46]"},{"why":"Supplies the decPOMDP formalization used to model the single humanoid as a multi-agent system.","marker":"[44]"}],"fun_headline_variants":["Limb-agents walk better than one-body agent","Cooperative limbs beat single-agent humanoid control","Each limb as an RL agent speeds robot walking","MASH: multi-agent limbs outperform single-agent walking"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The single-agent PPO baseline is given the same reward and equivalent information, so the reported gains come from the multi-agent decomposition rather than from MASH's critic seeing privileged global state.","fun_headline_variants_meta":{"raw":{"variants":["Limb-agents walk better than one-body agent","Cooperative limbs beat single-agent humanoid control","Each limb as an RL agent speeds robot walking","MASH: multi-agent limbs outperform single-agent walking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000152,"raw_usage":{"total_tokens":1019,"prompt_tokens":699,"completion_tokens":320,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":259}},"tokens_in":443,"tokens_out":320,"duration_ms":5075,"temperature":1.0,"reasoning_tokens":259,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:26:24.789348+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the single-agent PPO baseline with the same 106-dimensional privileged observations used by MASH's critic (and the same reward), then compare convergence and deployment metrics; if the gap collapses, the multi-agent design is not the cause. A second check is to strip MASH's global critic down to per-limb local critics and see whether the coordination metrics hold.","supporting_citations":[{"cited_title":"The surprising effectiveness of ppo in cooperative multi-agent games,","cited_arxiv_id":null,"evidence_quote":"Provides the multi-agent PPO objective that MASH optimizes for the limb agents."},{"cited_title":"Isaac gym: High performance gpu based physics simulation for robot learning,","cited_arxiv_id":null,"evidence_quote":"Isaac Gym is the GPU-accelerated simulation environment where MASH is trained."},{"cited_title":"MASQ: Multi-Agent Reinforcement Learning for Single Quadruped Robot Locomotion","cited_arxiv_id":"2408.13759","evidence_quote":"Prior work applying MARL to a single quadruped robot, the direct precedent MASH extends to humanoids."},{"cited_title":"Stabilising experience replay for deep multi-agent rein- forcement learning,","cited_arxiv_id":null,"evidence_quote":"Establishes the centralized-training-with-decentralized-execution paradigm that MASH adopts."},{"cited_title":"Pomdps for robotic tasks with mixed observability","cited_arxiv_id":null,"evidence_quote":"Supplies the decPOMDP formalization used to model the single humanoid as a multi-agent system."}],"review_version":1}