{"id":"0e545638-2b48-4e54-a4af-1b6c7115cba4","arxiv_id":"2505.13549","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TD-GRPC trains a humanoid locomotion policy by combining softmax group-relative Q-value advantages with an MPPI-action matching loss inside TD-MPC, reporting improved sample efficiency on eight of ten HumanoidBench tasks.","lead":"TD-GRPC adds a group-relative policy constraint to the TD-MPC robot-learning framework and tests it on ten simulated locomotion tasks for the Unitree H1-2 humanoid. The authors report faster convergence and more stable gaits than TD-MPC2, TD-M(PC)2, and SAC in most tasks, but not in crawling or stair climbing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The implemented objective is not the announced trust-region constraint: Eq. 9 replaces KL with behavioral cloning, and Algorithm 1's line 20 is action clipping, not Eq. 4. Reported gains may come from a different mechanism.","rationale":"The reader's weakest-assumption analysis identifies exactly the point on which the paper's stated mechanism hinges. The abstract and contributions promise a trust-region constraint in latent policy space; Eq. 9 is the only place the paper defines the full policy objective, and it replaces the announced constraint with a behavioral-cloning term. Algorithm 1 commits a further mismatch by labeling a hard clipping operation as Eq. 4, and no KL is ever computed because no old policy snapshot exists. This is not a stylistic inconsistency: if the training objective in the code is the pseudocode or Eq. 9, then the experiments are evidence about a different algorithm, and the title's 'Group Relative Policy Constraint' is not what is being evaluated. The empirical curves and Table II could still be valid for that different algorithm, which is why the appropriate disposition is CONDITIONAL rather than REJECT: release the exact objective, verify the trust region is actually enforced (or drop that claim), and add multi-seed and ablation statistics. I would not change the reader's verdict; the condition requested is necessary and sufficient. The variance-bound argument after Eq. 5 is also unproved, but it is secondary to the objective mismatch and would not by itself block the empirical claim.","tokens_in":12487,"tokens_out":6893,"duration_ms":74232,"concrete_test":"Obtain the released training code and instrument three measurements: (i) store a snapshot of pi_theta before each update and compute DKL(pi_theta_new || pi_theta_old) for every update; (ii) ablate the three candidate objectives--(A) the exact Lagrangian loss max{DKL - epsilon, 0} with a saved old policy, (B) the beta log mu(a|s) term from Eq. 9, and (C) the threshold clipping in Algorithm 1 line 20--keeping all other hyperparameters and seeds fixed; (iii) report the realized KL and return curves for each variant over at least 5 seeds. If variant B or C reproduces the reported performance while variant A does not, or if the realized KL exceeds epsilon by an order of magnitude under the published scheme, then the central claim that the trust-region constraint produces the gains is refuted and the paper must be rewritten around the actual mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central methodological claim is that TD-GRPC stabilizes off-policy MBRL by imposing a trust-region constraint DKL(pi||pi_old)<=epsilon in latent policy space. In the actual training objective this constraint is not present. Eq. 9 writes L_pi = (1/G) sum_i A_i log pi_theta(a_i|s) + beta log mu(a|s), and the sentence \"The second term of Eq. 9 imposes a residual-style regularization equivalent to the trust-region\" is the only justification. Maximizing log mu(a|s) does not enforce DKL(pi||pi_old)<=epsilon; it is a behavioral-cloning term toward the planner prior mu, with no stored old policy and no KL computation in the loop. Algorithm 1's line 20, labeled (Eq. 4), computes epsilon=(b_hat_i - mu_G)/sigma_G and then clips b_hat_i, which is a hard action-normalization operation, not the Lagrangian penalty max{DKL - epsilon, 0} of Eq. 4. Since no snapshot of pi_old is maintained and the pseudocode's L_KL_i is never defined, the implemented loss is ambiguous and cannot be mapped to the trust-region method advertised in the title and abstract. The theoretical support (variance bound in Eq. 6 and the \"lower bounded\" expression before Eq. 4) is asserted without proof and does not repair this gap. If the algorithm that produced Table II and Fig. 3 is exactly the pseudocode, then the experiments evaluate a different objective; if it is Eq. 9 as written, the trust-region constraint is absent. Either way, the central causal claim that trust-region policy constraints drive the stability improvements is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TD-GRPC, an extension of the TD-MPC framework for humanoid locomotion. It combines GRPO-style group-relative softmax advantages with a policy constraint that the authors describe as a trust-region regularization in latent policy space. The method is evaluated on 10 locomotion tasks from HumanoidBench on the 26-DoF Unitree H1-2 humanoid, with comparisons against SAC, TD-MPC2, and TD-M(PC)2. The authors report faster convergence for TD-GRPC on most tasks, qualitative behavioral improvements, and a success/failure summary in Table II.","tokens_in":12861,"tokens_out":7872,"duration_ms":78898,"significance":"If the claimed mechanism were established, this would be a valuable contribution: a sample-efficient model-based RL method with stable off-policy updates for high-dimensional humanoid control, validated across a broad task suite. The empirical scope is a genuine strength, and the qualitative behavioral analysis (e.g., forward vs. backward walking, sitting stability) is informative. However, the central mechanism advertised in the title and abstract—an explicit trust-region constraint—is not actually implemented in the stated objective, and the variance-reduction argument that supports the method is asserted rather than proved. These issues are load-bearing: without resolving them, the reader cannot tell whether the reported gains come from the named mechanism, from behavioral cloning toward the planner distribution, or from the action clipping in Algorithm 1.","major_comments":[{"comment":"The implemented policy objective is not the advertised trust-region constraint. Equation (4) defines LPC = max{DKL(pi||pi_old) - epsilon, 0}, but Eq. (9) writes the policy constraint term as beta log mu(a|s) and asserts, without derivation, that this is \"equivalent to the trust-region.\" Maximizing log mu(a|s) is a behavioral-cloning term toward the planner prior mu; it does not constrain DKL(pi||pi_old), because no snapshot of pi_old is stored and no KL divergence is computed in the loop. Algorithm 1, lines 19-20, labels an operation as \"(Eq. 4)\" but computes epsilon = (b_hat_i - mu_G)/sigma_G and clips the sampled action b_hat_i; this is action normalization/clipping, not the Lagrangian penalty of Eq. (4). Line 24 also uses L_KL_i, which is never defined. As a result, the manuscript does not establish that the experiments evaluate the trust-region mechanism named in the title and abstract; either the objective is Eq. (9) without a trust region, or the pseudocode is incomplete. This mismatch is load-bearing for the central stability claim.","section":"Section III.B, Eqs. (4), (9), and Algorithm 1"},{"comment":"The variance-reduction claim is asserted, not proved. The assumption that ||grad_theta log pi_theta(a|s)|| is constant C and that rewards/values are bounded does not by itself imply Var[grad L_softmax] <= Var[grad L_std-norm]; the variance of a weighted gradient sum depends on the covariance between the advantage weights and the log-policy gradients, and Eq. (7) only confines the norm of the score, not the variance of the resulting gradient. Since the paper's second contribution (\"theoretical insights\") rests on this inequality, a proof or a precise reference is needed; otherwise the statement should be labeled as a heuristic.","section":"Section III.B.2, Eqs. (6)-(7)"},{"comment":"The quantitative claims of faster convergence and significant improvement are not supported by statistics: the paper does not report the number of seeds, error bars, or standard deviations, and Section IV.A lists qualitative observations without thresholds. The solution markers in Table II also appear inconsistent with the text: hurdling is marked solved although Section IV.A.9 only reports clearing one track, and balance is marked ● for TD-GRPC although Section IV.A.6 says all methods are approximately similar and all struggle. Please report repeated-run statistics with explicit success criteria and reconcile Table II with the narrative.","section":"Section IV, Fig. 3 and Table II"}],"minor_comments":[{"comment":"z_t is defined both as h_theta(s_t) and as d_theta(s_{t-1}, a_{t-1}); the second definition should presumably be z_{t+1} = d_theta(z_t, a_t), matching Eq. (10a).","section":"Eq. (2)"},{"comment":"The formula for sigma_t^2 contains mu_j where mu_t is intended, and the text does not define how the top-k trajectories are selected; please clarify.","section":"Eq. (3)"},{"comment":"The sentence \"where µ_k denotes the behavior policy at kth iteration from the buffer D obtained from Eq. 3\" is disconnected from Eq. (8), which contains no µ_k; please revise or delete.","section":"Section III.B.2, after Eq. (8)"},{"comment":"The symbol epsilon is overloaded: it denotes the trust-region threshold in Eq. (4) and the normalized action value on line 19; rename one of them.","section":"Algorithm 1, line 19"},{"comment":"The caption and panel labels are difficult to read (e.g., \"TD-GRPC/ go to the wall...\" in Fig. 4a); please add a legend and clean the annotations.","section":"Fig. 4"},{"comment":"The paper does not report the values of beta, tau, and epsilon used in training, the policy/model architecture, or the inference-time CEM parameters, and no code is provided; given the ambiguity in Algorithm 1, a reproducibility appendix would be important.","section":"Section IV.A"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern that the implemented objective replaces the trust region with behavioral cloning is, in my reading, correct and is the main reason for major revision. The empirical study is broad, but the missing seeds and loose success criteria make it hard to interpret the quantitative claims. I would not reject outright because the method as implemented (GRPO-style advantage weighting plus behavioral cloning toward the planner) could still be a useful contribution if the authors reframe the claims and add the missing analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful empirical package wrapped around a mechanism the paper never actually specifies. The comparison against TD-M(PC)2 on HumanoidBench's H1-2 is worth a serious look, but the trust-region story in the title and abstract is not the loss in Eq. 9 nor the operation in Algorithm 1.\n\nWhat is genuinely new is the combination: GRPO-style softmax group advantages with an MPPI-action matching term inside the TD-MPC actor update, evaluated on a wider and heavier H1-2 suite than most prior work. The qualitative section has real value—the backward-walking observation and the arm-locking variation are the kind of detail that helps the legged-RL community. The paper also cites and directly extends TD-M(PC)2, which is the right baseline. Citation patterns look normal.\n\nThe soft spots are real and central. Eq. 4 announces a KL trust region max{DKL(pi||pi_old)-epsilon, 0}. Equation 9 replaces that with beta log mu(a|s), and the text calls this 'equivalent' with no proof. Maximizing log mu is a behavioral-cloning term toward the planner prior; it is not DKL(pi||pi_old) and it does not require keeping an old policy. Algorithm 1 does not repair the gap: line 20 is an action threshold with epsilon=(b_hat_i-mu_G)/sigma_G, which is not the Lagrangian penalty of Eq. 4. As written, the implemented objective is ambiguous. If the experiments used Eq. 9, the trust-region constraint named in the title is absent; if they used something else, the paper does not say what. The variance-reduction claims in Eqs. 6–7 are asserted, not derived. And the experiments report single learning curves with no seeds or error bars; Table II's checkmarks are subjective. These are fixable, but they are load-bearing.\n\nWho this is for: people working on model-based RL for legged robots. Value: the task coverage and qualitative comparisons are useful; the algorithm description is not yet. I would send it to a serious referee with a clear request to rewrite the objective and pseudocode consistently, add multi-seed statistics, and either prove or remove the variance claim. A grounded revision could become a solid contribution.","headline":"Plausible empirical gains on H1-2, but the advertised trust-region mechanism is not actually in the equations or pseudocode; worth refereeing if the authors clarify and add multi-seed evidence.","tokens_in":13420,"tokens_out":4458,"would_cite":false,"duration_ms":54759,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A group-relative trust-region objective stabilizes humanoid locomotion learning, enabling forward walking, running, and sitting on a 26-DoF robot.","keywords":["humanoid locomotion","model-based reinforcement learning","temporal difference learning","group relative policy optimization","policy constraints","trust-region optimization","latent-space planning","TD-MPC"],"falsifier":"During TD-GRPC training on the walking task, measure the actual KL divergence between the updated policy and the planner's prior action distribution at each update; if the KL drifts well above the announced threshold while the behavioral-cloning term stays small, then the named trust-region mechanism is not what stabilizes training and the performance gain must come from something else. A second check is to replace the behavioral-cloning term with an explicit KL penalty of the same strength and compare convergence curves.","tokens_in":12258,"feed_emoji":"🤖","tokens_out":7732,"duration_ms":73368,"temperature":0.7,"pith_summary":"This paper claims that model-based reinforcement learning for humanoid locomotion can be made more stable and sample-efficient by adding two linked mechanisms to the TD-MPC recipe: group-relative ranking of sampled actions, and an explicit policy constraint that keeps the learned policy close to the planner's action distribution. If correct, this matters because high-dimensional humanoid control is brittle under off-policy updates, and the method reportedly lets a 26-DoF robot stand, walk, run, and sit with faster convergence than prior model-based baselines while leaving the planner itself unchanged. The contribution is an extension rather than a new theory: absolute value targets are replaced with softmax group-relative advantages, and a residual-style regularization term is added in latent space.","feed_headline":"Group-relative trust region makes a humanoid walk, run, sit","feed_subtitle":"A policy constraint plus group-relative ranking fixes forward gaits on a 26-DoF humanoid.","key_machinery":"The load-bearing machinery is the TD-GRPC policy objective, $\\mathcal{L}_\\pi = \\frac{1}{G}\\sum_i A_i(q)\\log\\pi_\\theta(a_i|s) + \\beta\\log\\mu(a|s)$, together with group-relative advantage scores $A_i(q)$ defined by a softmax over sampled actions' predicted Q-values. The first term replaces absolute advantage estimates with bounded relative weights, lowering gradient variance; the second term enforces closeness to the planner-induced policy distribution as a stand-in for a KL trust-region constraint. The surrounding latent dynamics, encoder, reward, and value networks are trained with a TD-MPC-style model objective, and inference still uses a sampling-based planner, so the constraint acts on policy learning only.","core_discovery":"The central discovery is that combining relative action ranking with a policy constraint in latent space stabilizes off-policy TD learning for humanoid control. Concretely, TD-GRPC samples G actions per latent state, scores them with a softmax over predicted Q-values, and uses those bounded scores as policy-gradient weights; it then adds a term that pulls the policy toward the behavior distribution behind the planner's rollouts, which the paper treats as a residual-style trust-region constraint. Because the planner itself is left unchanged, the constraint shapes only policy learning. In experiments on a 26-DoF H1-2 humanoid, this recipe solves standing, walking, running, and sitting that prior baselines fail or only partially solve, converges faster on most tasks, and produces forward gaits where comparison policies walk backward.","pith_inferences":["If the asserted equivalence between $\\beta\\log\\mu(a|s)$ and a KL trust-region holds, the same 'constraint without replanning' recipe could be layered onto other latent-space planners beyond the TD-MPC family, since the design deliberately keeps the planner untouched.","The backward-walking failure of the comparison policies only on the heavier H1-2 suggests those baselines inherit body-mass-specific pose priors; a direct test would be to train them with a forward-velocity reward term and see whether the gap persists.","The arm-locking experiment implies that upper-body pose constraints, not just policy constraints, could be a cheap source of stability; this is a finding the paper reports but does not turn into a training objective.","Because the softmax advantage weights are bounded, the method might transfer to other off-policy actor-critic algorithms as a drop-in variance-reduction trick, but that extension is not demonstrated here."],"forward_implications":["On the 26-DoF H1-2, TD-GRPC solves stand, walk, run, and sit, while prior baselines fail or only partially solve them.","TD-GRPC converges faster than prior TD-MPC-style baselines on most locomotion tasks in the benchmark, using the same planning horizon and buffer size.","The learned policies walk and run forward on the heavier H1-2, where comparison policies walk backward or freeze, indicating the constraint fixes a pose and direction pathology, not just return.","TD-GRPC can clear one hurdle and briefly balance on a ball-board, tasks where the comparison methods produce no physically meaningful behavior.","The harder tasks—stair climbing, full balancing, and proper crawling—remain unsolved by all methods, including TD-GRPC."],"supporting_citations":[{"why":"Supplies the latent world model and actor-critic objective that TD-GRPC extends.","marker":"[11]"},{"why":"The closest baseline; already adds a policy constraint to TD-MPC, isolating the contribution of group-relative ranking.","marker":"[12]"},{"why":"Provides the group-relative policy optimization objective that the paper adapts into Eq. 8.","marker":"[32]"},{"why":"Defines the locomotion task suite and the 26-DoF H1-2 simulation environment used for evaluation.","marker":"[33]"},{"why":"Introduces the TD-MPC planner-plus-TD-learning framework, including latent MPPI rollouts.","marker":"[26]"},{"why":"Supplies the planner sub-optimality theorem used to motivate constraining policy divergence.","marker":"[10]"},{"why":"Provides the value-error bound (Lemma 3.1) used to justify residual policy learning.","marker":"[41]"},{"why":"Cited as the trust-region method behind the equivalence of the residual regularization term.","marker":"[43]"}],"fun_headline_variants":["Latent trust region yields stable humanoid walking and running","Group-relative policy constraint improves humanoid control","TD-GRPC: robust humanoid locomotion without planner changes","Policy constraint and relative ranking fix humanoid gaits"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the extra term pushing the policy to imitate the planner's action distribution is truly the same as keeping the policy inside a trust region around that distribution; the paper asserts this equivalence without proof.","fun_headline_variants_meta":{"raw":{"variants":["Latent trust region yields stable humanoid walking and running","Group-relative policy constraint improves humanoid control","TD-GRPC: robust humanoid locomotion without planner changes","Policy constraint and relative ranking fix humanoid gaits"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000295,"raw_usage":{"total_tokens":1726,"prompt_tokens":971,"completion_tokens":755,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":690}},"tokens_in":587,"tokens_out":755,"duration_ms":7215,"temperature":1.0,"reasoning_tokens":690,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:28:46.793883+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"During TD-GRPC training on the walking task, measure the actual KL divergence between the updated policy and the planner's prior action distribution at each update; if the KL drifts well above the announced threshold while the behavioral-cloning term stays small, then the named trust-region mechanism is not what stabilizes training and the performance gain must come from something else. A second check is to replace the behavioral-cloning term with an explicit KL penalty of the same strength and compare convergence curves.","supporting_citations":[{"cited_title":"An upper bound on the loss from approximate optimal-value functions,","cited_arxiv_id":null,"evidence_quote":"Provides the value-error bound (Lemma 3.1) used to justify residual policy learning."},{"cited_title":"Trust region policy optimization,","cited_arxiv_id":null,"evidence_quote":"Cited as the trust-region method behind the equivalence of the residual regularization term."}],"review_version":1}