Pith. sign in

REVIEW 3 major objections 6 minor 47 references

TD-GRPC: Temporal Difference Learning with Group Relative Policy Constraint for Humanoid Locomotion

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A group-relative trust-region objective stabilizes humanoid locomotion learning, enabling forward walking, running, and sitting on a 26-DoF robot.

desk verdict Plausible empirical gains on H1-2, but the advertised trust-region mechanism is not actually in the equations or pseudocode; worth refereeing if the authors clarify and add multi-seed evidence. read the letter →

arxiv 2505.13549 v1 pith:C3Y3NMN2 submitted 2025-05-19 cs.RO

Khang Nguyen , Khai Nguyen , An T. Le , Jan Peters , Manfred Huber , Ngo Anh Vien , Minh Nhat Vu This is my paper · ORCID
classification cs.RO
keywords humanoidlocomotionmodel-basedreinforcementlearningtemporaldifferencegrouprelativepolicyoptimizationconstraintstrust-regionlatent-spaceplanningTD-MPC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that model-based reinforcement learning for humanoid locomotion can be made more stable and sample-efficient by adding two linked mechanisms to the TD-MPC recipe: group-relative ranking of sampled actions, and an explicit policy constraint that keeps the learned policy close to the planner's action distribution. If correct, this matters because high-dimensional humanoid control is brittle under off-policy updates, and the method reportedly lets a 26-DoF robot stand, walk, run, and sit with faster convergence than prior model-based baselines while leaving the planner itself unchanged. The contribution is an extension rather than a new theory: absolute value targets are replaced with softmax group-relative advantages, and a residual-style regularization term is added in latent space.

What carries the argument

The load-bearing machinery is the TD-GRPC policy objective, $\mathcal{L}_\pi = \frac{1}{G}\sum_i A_i(q)\log\pi_\theta(a_i|s) + \beta\log\mu(a|s)$, together with group-relative advantage scores $A_i(q)$ defined by a softmax over sampled actions' predicted Q-values. The first term replaces absolute advantage estimates with bounded relative weights, lowering gradient variance; the second term enforces closeness to the planner-induced policy distribution as a stand-in for a KL trust-region constraint. The surrounding latent dynamics, encoder, reward, and value networks are trained with a TD-MPC-style model objective, and inference still uses a sampling-based planner, so the constraint acts on policy learning only.

What would settle it

During TD-GRPC training on the walking task, measure the actual KL divergence between the updated policy and the planner's prior action distribution at each update; if the KL drifts well above the announced threshold while the behavioral-cloning term stays small, then the named trust-region mechanism is not what stabilizes training and the performance gain must come from something else. A second check is to replace the behavioral-cloning term with an explicit KL penalty of the same strength and compare convergence curves.

Watch

Extended reading notes

Core claim

The central discovery is that combining relative action ranking with a policy constraint in latent space stabilizes off-policy TD learning for humanoid control. Concretely, TD-GRPC samples G actions per latent state, scores them with a softmax over predicted Q-values, and uses those bounded scores as policy-gradient weights; it then adds a term that pulls the policy toward the behavior distribution behind the planner's rollouts, which the paper treats as a residual-style trust-region constraint. Because the planner itself is left unchanged, the constraint shapes only policy learning. In experiments on a 26-DoF H1-2 humanoid, this recipe solves standing, walking, running, and sitting that prior baselines fail or only partially solve, converges faster on most tasks, and produces forward gaits where comparison policies walk backward.

Load-bearing premise

The load-bearing premise is that the extra term pushing the policy to imitate the planner's action distribution is truly the same as keeping the policy inside a trust region around that distribution; the paper asserts this equivalence without proof.

Editorial extensions

If this is right

  • On the 26-DoF H1-2, TD-GRPC solves stand, walk, run, and sit, while prior baselines fail or only partially solve them.
  • TD-GRPC converges faster than prior TD-MPC-style baselines on most locomotion tasks in the benchmark, using the same planning horizon and buffer size.
  • The learned policies walk and run forward on the heavier H1-2, where comparison policies walk backward or freeze, indicating the constraint fixes a pose and direction pathology, not just return.
  • TD-GRPC can clear one hurdle and briefly balance on a ball-board, tasks where the comparison methods produce no physically meaningful behavior.
  • The harder tasks—stair climbing, full balancing, and proper crawling—remain unsolved by all methods, including TD-GRPC.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the asserted equivalence between $\beta\log\mu(a|s)$ and a KL trust-region holds, the same 'constraint without replanning' recipe could be layered onto other latent-space planners beyond the TD-MPC family, since the design deliberately keeps the planner untouched.
  • The backward-walking failure of the comparison policies only on the heavier H1-2 suggests those baselines inherit body-mass-specific pose priors; a direct test would be to train them with a forward-velocity reward term and see whether the gap persists.
  • The arm-locking experiment implies that upper-body pose constraints, not just policy constraints, could be a cheap source of stability; this is a finding the paper reports but does not turn into a training objective.
  • Because the softmax advantage weights are bounded, the method might transfer to other off-policy actor-critic algorithms as a drop-in variance-reduction trick, but that extension is not demonstrated here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes TD-GRPC, an extension of the TD-MPC framework for humanoid locomotion. It combines GRPO-style group-relative softmax advantages with a policy constraint that the authors describe as a trust-region regularization in latent policy space. The method is evaluated on 10 locomotion tasks from HumanoidBench on the 26-DoF Unitree H1-2 humanoid, with comparisons against SAC, TD-MPC2, and TD-M(PC)2. The authors report faster convergence for TD-GRPC on most tasks, qualitative behavioral improvements, and a success/failure summary in Table II.

Significance. If the claimed mechanism were established, this would be a valuable contribution: a sample-efficient model-based RL method with stable off-policy updates for high-dimensional humanoid control, validated across a broad task suite. The empirical scope is a genuine strength, and the qualitative behavioral analysis (e.g., forward vs. backward walking, sitting stability) is informative. However, the central mechanism advertised in the title and abstract—an explicit trust-region constraint—is not actually implemented in the stated objective, and the variance-reduction argument that supports the method is asserted rather than proved. These issues are load-bearing: without resolving them, the reader cannot tell whether the reported gains come from the named mechanism, from behavioral cloning toward the planner distribution, or from the action clipping in Algorithm 1.

major comments (3)
  1. [Section III.B, Eqs. (4), (9), and Algorithm 1] The implemented policy objective is not the advertised trust-region constraint. Equation (4) defines LPC = max{DKL(pi||pi_old) - epsilon, 0}, but Eq. (9) writes the policy constraint term as beta log mu(a|s) and asserts, without derivation, that this is "equivalent to the trust-region." Maximizing log mu(a|s) is a behavioral-cloning term toward the planner prior mu; it does not constrain DKL(pi||pi_old), because no snapshot of pi_old is stored and no KL divergence is computed in the loop. Algorithm 1, lines 19-20, labels an operation as "(Eq. 4)" but computes epsilon = (b_hat_i - mu_G)/sigma_G and clips the sampled action b_hat_i; this is action normalization/clipping, not the Lagrangian penalty of Eq. (4). Line 24 also uses L_KL_i, which is never defined. As a result, the manuscript does not establish that the experiments evaluate the trust-region mechanism named in the title and abstract; either the objective is Eq. (9) without a trust region, or the pseudocode is incomplete. This mismatch is load-bearing for the central stability claim.
  2. [Section III.B.2, Eqs. (6)-(7)] The variance-reduction claim is asserted, not proved. The assumption that ||grad_theta log pi_theta(a|s)|| is constant C and that rewards/values are bounded does not by itself imply Var[grad L_softmax] <= Var[grad L_std-norm]; the variance of a weighted gradient sum depends on the covariance between the advantage weights and the log-policy gradients, and Eq. (7) only confines the norm of the score, not the variance of the resulting gradient. Since the paper's second contribution ("theoretical insights") rests on this inequality, a proof or a precise reference is needed; otherwise the statement should be labeled as a heuristic.
  3. [Section IV, Fig. 3 and Table II] The quantitative claims of faster convergence and significant improvement are not supported by statistics: the paper does not report the number of seeds, error bars, or standard deviations, and Section IV.A lists qualitative observations without thresholds. The solution markers in Table II also appear inconsistent with the text: hurdling is marked solved although Section IV.A.9 only reports clearing one track, and balance is marked ● for TD-GRPC although Section IV.A.6 says all methods are approximately similar and all struggle. Please report repeated-run statistics with explicit success criteria and reconcile Table II with the narrative.
minor comments (6)
  1. [Eq. (2)] z_t is defined both as h_theta(s_t) and as d_theta(s_{t-1}, a_{t-1}); the second definition should presumably be z_{t+1} = d_theta(z_t, a_t), matching Eq. (10a).
  2. [Eq. (3)] The formula for sigma_t^2 contains mu_j where mu_t is intended, and the text does not define how the top-k trajectories are selected; please clarify.
  3. [Section III.B.2, after Eq. (8)] The sentence "where µ_k denotes the behavior policy at kth iteration from the buffer D obtained from Eq. 3" is disconnected from Eq. (8), which contains no µ_k; please revise or delete.
  4. [Algorithm 1, line 19] The symbol epsilon is overloaded: it denotes the trust-region threshold in Eq. (4) and the normalized action value on line 19; rename one of them.
  5. [Fig. 4] The caption and panel labels are difficult to read (e.g., "TD-GRPC/ go to the wall..." in Fig. 4a); please add a legend and clean the annotations.
  6. [Section IV.A] The paper does not report the values of beta, tau, and epsilon used in training, the policy/model architecture, or the inference-time CEM parameters, and no code is provided; given the ambiguity in Algorithm 1, a reproducibility appendix would be important.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation is self-contained; the Eq. 9 vs Eq. 4 mismatch is an implementation/mechanism gap, not a circular reduction.

full rationale

I walked the derivation chain from Eq. 4 through Eq. 9 and Algorithm 1, and inspected all references. No load-bearing step reduces to its own input. The paper does not fit a parameter and then call a closely related quantity a prediction; no self-citation supplies a uniqueness theorem or a forced ansatz; and the empirical comparison against SAC, TD-MPC2, and TD-M(PC)2 on HumanoidBench is an external benchmark, not an internal consistency check. The strongest potential concern is that the announced trust-region constraint DKL(pi||pi_old)<=epsilon in Eq. 4 is replaced in Eq. 9 by beta log mu(a|s), with the sentence 'The second term of Eq. 9 imposes a residual-style regularization equivalent to the trust-region' as the only bridge, and Algorithm 1's line 20 labeled '(Eq. 4)' actually performs normalization/clipping rather than the Lagrangian penalty. That is a genuine correctness/mechanism gap: log mu(a|s) is not derived from, equal to, or defined in terms of DKL(pi||pi_old), so the advertised mechanism may not be what was evaluated. But the absence of a derivation is not a circular derivation. The quantities are distinct, and the claim of equivalence is an unsupported assertion rather than a tautology, so this belongs in correctness risk rather than in the circularity score. No circular step meets the quoted-evidence bar, so the score is 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on standard RL assumptions (bounded rewards, discounted MDP), on external performance bounds (Singh-Yee, LOOP), on the simulator's fidelity, and on an unverified gradient-norm assumption used to motivate the softmax advantage. The implemented policy constraint also depends on an unproved equivalence between a KL trust region and a behavioral-cloning term. No new physical or conceptual entities are introduced beyond the algorithmic components.

free parameters (5)
  • beta = not reported
    Weight on the MPPI-action matching term in Eq. 9; the text gives no value or tuning procedure.
  • tau = not reported
    Temperature in softmax advantages (Eq. 5) and trajectory weighting (Eq. 3); no value stated.
  • epsilon = not reported
    Trust-region threshold in Eq. 4; no value or adaptation rule stated.
  • G = 3
    Number of action groups in Eq. 8; stated in Sec. IV-A but with no sensitivity analysis.
  • H = 3
    MPPI planning horizon; stated in Sec. IV-A.
assumptions (4)
  • domain assumption HumanoidBench simulator dynamics P provide a faithful model of the Unitree H1-2 for the claims made.
    All validation is in simulation; if the simulator is inaccurate the qualitative behavior claims do not transfer. Invoked throughout Sec. IV.
  • standard math Bounded rewards and discount factor gamma in (0,1] with Rmax ensure Qmax = Rmax/(1-gamma).
    Used in the performance bound derivation following Lemma 3.1 of Singh and Yee and Theorem 1 of LOOP.
  • ad hoc to paper The gradient norm ||grad_theta log pi_theta(a|s)|| is constant C and rewards/values are bounded, supporting Eq. 6.
    Assumed without justification solely to motivate the softmax advantage variance bound; no verification for the MLP policies used.
  • standard math Lemma 3.1 of Singh and Yee and Theorem 1 of LOOP apply to the planner-induced policy mu_k.
    External bounds imported to justify the trust-region constraint; the paper does not verify their conditions in this setting beyond stating them.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TD-GRPC: Temporal Difference Learning with Group Relative Policy Constraint for Humanoid Locomotion." pith.science (2026). https://pith.science/paper/C3Y3NMN2

@misc{pith2026250513549,
  author       = {Pith},
  title        = {Pith review of: TD-GRPC: Temporal Difference Learning with Group Relative Policy Constraint for Humanoid Locomotion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/C3Y3NMN2}},
  note         = {Machine review of arXiv:2505.13549}
}
read the original abstract

Robot learning in high-dimensional control settings, such as humanoid locomotion, presents persistent challenges for reinforcement learning (RL) algorithms due to unstable dynamics, complex contact interactions, and sensitivity to distributional shifts during training. Model-based methods, \textit{e.g.}, Temporal-Difference Model Predictive Control (TD-MPC), have demonstrated promising results by combining short-horizon planning with value-based learning, enabling efficient solutions for basic locomotion tasks. However, these approaches remain ineffective in addressing policy mismatch and instability introduced by off-policy updates. Thus, in this work, we introduce Temporal-Difference Group Relative Policy Constraint (TD-GRPC), an extension of the TD-MPC framework that unifies Group Relative Policy Optimization (GRPO) with explicit Policy Constraints (PC). TD-GRPC applies a trust-region constraint in the latent policy space to maintain consistency between the planning priors and learned rollouts, while leveraging group-relative ranking to assess and preserve the physical feasibility of candidate trajectories. Unlike prior methods, TD-GRPC achieves robust motions without modifying the underlying planner, enabling flexible planning and policy learning. We validate our method across a locomotion task suite ranging from basic walking to highly dynamic movements on the 26-DoF Unitree H1-2 humanoid robot. Through simulation results, TD-GRPC demonstrates its improvements in stability and policy robustness with sampling efficiency while training for complex humanoid control tasks.

Figures

Figures reproduced from arXiv: 2505.13549 by the authors.

Figure 1
Figure 1. Locomotion Tasks Performed by the Unitree H1-2 Humanoid with TD-GRPC: [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of TD-GRPC for Humanoid Locomotion: [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Episode Returns of TD-GRPC and Baselines on H1–2 in Humanoid Locomotion Tasks: [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Behavioral Analysis of H1-2 in Humanoid Locomotion Tasks: [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 17 canonical work pages

  1. [1]

    Learning humanoid locomotion with transformers,

    I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath, “Learning humanoid locomotion with transformers,” CoRR, 2023

    2023

  2. [2]

    Real-world humanoid locomotion with reinforcement learning,

    ——, “Real-world humanoid locomotion with reinforcement learning,” Science Robotics, vol. 9, no. 89, p. eadi9579, 2024

    2024

  3. [3]

    Neural network dynamics for model-based deep reinforcement learning with model- free fine-tuning,

    A. Nagabandi, G. Kahn, R. S. Fearing, and S. Levine, “Neural network dynamics for model-based deep reinforcement learning with model- free fine-tuning,” in IEEE ICRA, 2018

    work page 2018

  4. [4]

    Deep rein- forcement learning in a handful of trials using probabilistic dynamics models,

    K. Chua, R. Calandra, R. McAllister, and S. Levine, “Deep rein- forcement learning in a handful of trials using probabilistic dynamics models,” NeurIPS, 2018

    work page 2018

  5. [5]

    Model-based offline planning,

    A. Argenson and G. Dulac-Arnold, “Model-based offline planning,” arXiv preprint arXiv:2008.05556 , 2020

    arXiv 2008

  6. [6]

    Model predictive control,

    B. Kouvaritakis and M. Cannon, “Model predictive control,” Switzer- land: Springer International Publishing, vol. 38, no. 13-56, p. 7, 2016

    work page 2016

  7. [7]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” in ICML, 2018

    2018

  8. [8]

    Mastering atari, go, chess and shogi by planning with a learned model,

    J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel et al. , “Mastering atari, go, chess and shogi by planning with a learned model,” Nature, vol. 588, no. 7839, pp. 604–609, 2020

    2020

Show all 47 references
  1. [9]

    Mastering atari games with limited data,

    W. Ye, S. Liu, T. Kurutach, P. Abbeel, and Y . Gao, “Mastering atari games with limited data,” NeurIPS, 2021

    2021

  2. [10]

    Learning off-policy with online planning,

    H. Sikchi, W. Zhou, and D. Held, “Learning off-policy with online planning,” in CoRL, 2022

    2022

  3. [11]

    Td-mpc2: Scalable, robust world models for continuous control,

    N. Hansen, H. Su, and X. Wang, “Td-mpc2: Scalable, robust world models for continuous control,” arXiv preprint arXiv:2310.16828 , 2023

    2023 arXiv

  4. [12]

    Improving temporal differ- ence mpc through policy constraint,

    H. Lin, P. Wang, J. Schneider, and G. Shi, “Improving temporal differ- ence mpc through policy constraint,”arXiv preprint arXiv:2502.03550, 2025

    2025 arXiv

  5. [13]

    Dyna, an integrated architecture for learning, planning, and reacting,

    R. S. Sutton, “Dyna, an integrated architecture for learning, planning, and reacting,” ACM Sigart Bulletin , vol. 2, no. 4, pp. 160–163, 1991

    1991

  6. [14]

    When to trust your model: Model-based policy optimization,

    M. Janner, J. Fu, M. Zhang, and S. Levine, “When to trust your model: Model-based policy optimization,” NeurIPS, 2019

    2019

  7. [15]

    Model-based reinforcement learning for atari,

    L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Camp- bell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine et al., “Model-based reinforcement learning for atari,” arXiv preprint arXiv:1903.00374, 2019

    1903 arXiv

  8. [16]

    Pay attention to what and where? inter- pretable feature extractor in vision-based deep reinforcement learning,

    T. Pham and A. Cangelosi, “Pay attention to what and where? inter- pretable feature extractor in vision-based deep reinforcement learning,” in IJCNN, 2025

    2025

  9. [17]

    Learning agile and dynamic motor skills for legged robots,

    J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V . Tsounis, V . Koltun, and M. Hutter, “Learning agile and dynamic motor skills for legged robots,” Science Robotics, vol. 4, no. 26, p. eaau5872, 2019

    2019

  10. [18]

    Learning quadrupedal locomotion over challenging terrain,

    J. Lee, J. Hwangbo, L. Wellhausen, V . Koltun, and M. Hutter, “Learning quadrupedal locomotion over challenging terrain,” Science robotics, vol. 5, no. 47, p. eabc5986, 2020

    2020

  11. [19]

    Learning robust perceptive locomotion for quadrupedal robots in the wild,

    T. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V . Koltun, and M. Hutter, “Learning robust perceptive locomotion for quadrupedal robots in the wild,” Science robotics, vol. 7, no. 62, p. eabk2822, 2022

    2022

  12. [20]

    Dream to con- trol: Learning behaviors by latent imagination,

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi, “Dream to con- trol: Learning behaviors by latent imagination,” arXiv preprint arXiv:1912.01603, 2019

    1912 arXiv

  13. [21]

    Mastering atari with discrete world models,

    D. Hafner, T. Lillicrap, M. Norouzi, and J. Ba, “Mastering atari with discrete world models,” arXiv preprint arXiv:2010.02193 , 2020

    2010 arXiv

  14. [22]

    Mastering diverse domains through world models,

    D. Hafner, J. Pasukonis, J. Ba, and T. Lillicrap, “Mastering diverse domains through world models,” arXiv preprint arXiv:2301.04104 , 2023

    2023 arXiv

  15. [23]

    Recurrent world models facilitate policy evolution,

    D. Ha and J. Schmidhuber, “Recurrent world models facilitate policy evolution,” NeurIPS, 2018

    2018

  16. [24]

    Objective mismatch in model-based reinforcement learning,

    N. Lambert, B. Amos, O. Yadan, and R. Calandra, “Objective mismatch in model-based reinforcement learning,” arXiv preprint arXiv:2002.04523, 2020

    2002 arXiv

  17. [25]

    On the feasibility of cross-task transfer with model-based reinforcement learning,

    Y . Xu, N. Hansen, Z. Wang, Y .-C. Chan, H. Su, and Z. Tu, “On the feasibility of cross-task transfer with model-based reinforcement learning,” arXiv preprint arXiv:2210.10763 , 2022

    2022 arXiv

  18. [26]

    Temporal difference learning for model predictive control,

    N. Hansen, X. Wang, and H. Su, “Temporal difference learning for model predictive control,” arXiv preprint arXiv:2203.04955 , 2022

    2022 arXiv

  19. [27]

    Model-augmented actor-critic: Backpropagating through paths,

    I. Clavera, V . Fu, and P. Abbeel, “Model-augmented actor-critic: Backpropagating through paths,” arXiv preprint arXiv:2005.08068 , 2020

    2005 arXiv

  20. [28]

    Off-policy deep reinforcement learning without exploration,

    S. Fujimoto, D. Meger, and D. Precup, “Off-policy deep reinforcement learning without exploration,” in ICML, 2019

    2019

  21. [29]

    Offline reinforcement learning with implicit q-learning,

    I. Kostrikov, A. Nair, and S. Levine, “Offline reinforcement learning with implicit q-learning,” arXiv preprint arXiv:2110.06169 , 2021

    2021 arXiv

  22. [30]

    A distributional per- spective on reinforcement learning,

    M. G. Bellemare, W. Dabney, and R. Munos, “A distributional per- spective on reinforcement learning,” in ICML, 2017

    2017

  23. [31]

    Offline reinforcement learning: Tutorial, review, and perspectives on open problems,

    S. Levine, A. Kumar, G. Tucker, and J. Fu, “Offline reinforcement learning: Tutorial, review, and perspectives on open problems,” arXiv preprint arXiv:2005.01643, 2020

    2005 arXiv

  24. [32]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models,

    Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, X. Bi, H. Zhang, M. Zhang, Y . Li, Y . Wuet al., “Deepseekmath: Pushing the limits of mathematical reasoning in open language models,”arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  25. [33]

    Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation,

    C. Sferrazza, D.-M. Huang, X. Lin, Y . Lee, and P. Abbeel, “Humanoid- bench: Simulated humanoid benchmark for whole-body locomotion and manipulation,” arXiv preprint arXiv:2403.10506 , 2024

    2024 arXiv

  26. [34]

    Reinforcement learning for humanoid robotics,

    J. Peters, S. Vijayakumar, and S. Schaal, “Reinforcement learning for humanoid robotics,” in IEEE-RAS Humanoids, 2003

    2003

  27. [35]

    Stabilizing off- policy q-learning via bootstrapping error reduction,

    A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine, “Stabilizing off- policy q-learning via bootstrapping error reduction,” NeurIPS, vol. 32, 2019

    2019

  28. [36]

    Conservative q- learning for offline reinforcement learning,

    A. Kumar, A. Zhou, G. Tucker, and S. Levine, “Conservative q- learning for offline reinforcement learning,” NeurIPS, vol. 33, pp. 1179–1191, 2020

    2020

  29. [37]

    A minimalist approach to offline reinforce- ment learning,

    S. Fujimoto and S. S. Gu, “A minimalist approach to offline reinforce- ment learning,” NeurIPS, vol. 34, pp. 20 132–20 145, 2021

    2021

  30. [38]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning,

    X. B. Peng, A. Kumar, G. Zhang, and S. Levine, “Advantage-weighted regression: Simple and scalable off-policy reinforcement learning,” arXiv preprint arXiv:1910.00177 , 2019

    1910 arXiv

  31. [39]

    Extreme q-learning: Maxent rl without entropy,

    D. Garg, J. Hejna, M. Geist, and S. Ermon, “Extreme q-learning: Maxent rl without entropy,” arXiv preprint arXiv:2301.02328 , 2023

    2023 arXiv

  32. [40]

    Aggressive driving with model predictive path integral control,

    G. Williams, P. Drews, B. Goldfain, J. M. Rehg, and E. A. Theodorou, “Aggressive driving with model predictive path integral control,” in IEEE ICRA, 2016

    2016

  33. [41]

    An upper bound on the loss from approximate optimal-value functions,

    S. P. Singh and R. C. Yee, “An upper bound on the loss from approximate optimal-value functions,” Machine Learning, vol. 16, pp. 227–233, 1994

    1994

  34. [42]

    Relative entropy policy search,

    J. Peters, K. Mulling, and Y . Altun, “Relative entropy policy search,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 24, no. 1, 2010, pp. 1607–1612

    2010

  35. [43]

    Trust region policy optimization,

    J. Schulman, S. Levine, P. Abbeel, M. Jordan, and P. Moritz, “Trust region policy optimization,” in ICML, 2015

    2015

  36. [44]

    Awac: Accelerating online reinforcement learning with offline datasets,

    A. Nair, A. Gupta, M. Dalal, and S. Levine, “Awac: Accelerating online reinforcement learning with offline datasets,” arXiv preprint arXiv:2006.09359, 2020

    2006 arXiv

  37. [45]

    Optimization of computer simulation models with rare events,

    R. Y . Rubinstein, “Optimization of computer simulation models with rare events,”European Journal of Operational Research, vol. 99, no. 1, pp. 89–112, 1997

    1997

  38. [46]

    Maximum entropy inverse reinforcement learning

    B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey et al., “Maximum entropy inverse reinforcement learning.” in Aaai, vol. 8. Chicago, IL, USA, 2008, pp. 1433–1438

    2008

  39. [47]

    Deepmind control suite,

    Y . Tassa, Y . Doron, A. Muldal, T. Erez, Y . Li, D. d. L. Casas, D. Budden, A. Abdolmaleki, J. Merel, A. Lefrancq et al., “Deepmind control suite,” arXiv preprint arXiv:1801.00690 , 2018

    2018 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.