Pith. sign in

REVIEW 4 major objections 4 minor 44 references

PEGrad claims that energy minimization in RL robot control can be made non-conflicting with task performance by projecting energy gradients onto the orthogonal complement of the task-reward gradient, yielding comparable returns with 64% low

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

PEGrad projects energy-minimization gradients orthogonal to task-reward gradients in RL, achieving 64% torque reduction in simulation and reduced battery draw on a Unitree Go2 without sacrificing task reward.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection PEGrad is a clean, well-tested gradient-surgery trick for energy-efficient RL control, but the 'non-conflicting' claim is stronger than the evidence, especially the paper's own Limitations section. the 4 major comments →

arxiv 2509.01765 v1 pith:H7F353VD submitted 2025-09-01 cs.RO

Non-conflicting Energy Minimization in Reinforcement Learning based Robot Control

Skand Peri , Akhil Perincherry , Bikram Pandit , Stefan Lee This is my paper
classification cs.RO
keywords energy-efficient locomotionreinforcement learninggradient projectionmulti-objective optimizationPEGradSim2Realquadruped robottorque minimization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that an RL policy can be made energy-efficient without the usual trade-off tuning: instead of adding an energy penalty to the reward with a weight λ, update the policy with the energy gradient projected orthogonal to the task-reward gradient. On six DM-Control and four HumanoidBench tasks trained with SAC, this yields policies with returns comparable to the unconstrained baseline while cutting average applied torque by about 64%, and on four of six tasks it lands beyond the Pareto front of λ-weighted baselines. On a real Unitree Go2 trained with AMP-style priors and PPO, the same projection cuts battery current by roughly 20% on walking compared with the best manually tuned λ baseline. The method is hyperparameter-free, adds a second backward pass, and is claimed to work with any policy-gradient algorithm. The payoff: energy-efficient controllers without per-task λ sweeps, better battery life, and a principled alternative to reward shaping.

Core claim

The paper introduces PEGrad (Projecting Energy Gradients), a gradient-level modification for policy-gradient RL that treats energy minimization as a subordinate objective to task reward. At each update, the energy gradient gE is projected onto the subspace orthogonal to the task-reward gradient gR, giving gE⊥R = gE − (gR·gE / gR·gR) gR, and the policy step is taken as −α gR − β gE⊥R with an adaptive β that caps the energy step's norm at the reward step's norm. The central claim is that this update reduces energy expenditure while leaving task performance unchanged to first order, because a step orthogonal to the reward gradient moves along an approximate level-set of the reward loss. The pap

What carries the argument

The central object is the orthogonal projection operator P⊥_R(gE) = gE − (gR·gE / gR·gR) gR, applied to the energy gradient before each policy update, combined with the adaptive scaling β = α·min(1, ‖gR‖₂/‖gE⊥R‖₂). This projection keeps energy-minimizing parameter movement inside the null space of the first-order reward change, so the policy is pushed toward lower energy while the task-reward loss is, to first order, untouched.

Load-bearing premise

The load-bearing premise is that a parameter step exactly orthogonal to the current task-reward gradient leaves task performance unchanged; this is a first-order, local statement inside a non-stationary RL objective, and the paper's own Section 6 shows that over training it can drift into behaviors (maximizing base height, dragging legs, hopping) that are unsafe or undesirable when no style reward constrains them.

What would settle it

Run PEGrad training past the reported 1.5–2M steps on a fixed task and track task return: if return visibly decays while torque stays low, the no-conflict guarantee fails over long horizons. Alternatively, construct a minimal LQR-style task whose reward gradient and energy gradient are always exactly opposite; PEGrad should then either fail to reduce energy or degrade reward, revealing that the projection only promises non-conflict when the two objectives are not strictly adversarial.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Energy savings without reward reweighting: PEGrad removes the λ search from the reward function, so the same code applies to new embodiments and tasks with no per-task tuning.
  • On 4 of 6 DM-Control tasks, PEGrad policies dominate the manually tuned Pareto front: higher or equal return at lower torque.
  • Sample-efficiency gains appear on some humanoid tasks (h1-run, h1-walk), suggesting energy reduction can act as a useful training signal, not just a constraint.
  • Sim2Real: PEGrad policies transfer to the Unitree Go2, reducing battery current by about 20% on walking compared with the tuned λ=0.0002 AMP+PPO baseline, and outperform the factory controller.
  • PCGrad-style conditional projection without adaptive scaling over-optimizes energy and collapses task performance, motivating PEGrad's always-orthogonal projection plus bounded β.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's no-conflict claim is only first-order and local; over many updates the policy can drift along the reward level-set, which explains the unsafe behaviors (hopping, leg-dragging, base-height maximization) the authors observed without style rewards. A natural extension is to combine PEGrad with a safety or style constraint that bounds this drift.
  • Because the projection treats energy as subordinate to reward, it generalizes to any ordered pair of objectives; the authors hint at recursive PEGrad over multiple priorities, which could be tested for smoothness, joint-wear, or actuator-temperature objectives.
  • The energy proxy matters: the paper uses sum of absolute torques because current draw is roughly proportional to torque; on embodiments with heterogeneous motors or gear ratios, including per-joint coefficients is a direct, testable refinement.
  • PEGrad could be combined with constrained-RL or barrier methods to guarantee that level-set drift never leaves a safe set, turning the empirical 64% savings into a certified bound rather than an average.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes PEGrad, a multi-objective RL method that adds a separate energy-consumption critic and modifies policy-gradient updates by projecting the energy gradient orthogonal to the task-reward gradient (Eq. 6), with an adaptive rescaling (Eq. 7). The authors claim this minimizes energy 'without conflicting with task performance' and is hyperparameter-free. They evaluate on six DM-Control and four HumanoidBench tasks with SAC, report ~64% average torque reduction at comparable returns, compare against λ-scalarized and PCGrad baselines, and deploy AMP+PPO policies on a Unitree Go2, reporting lower current and torque than factory and tuned baselines. The core projection is simple and clearly described, and the empirical results are reported with seeds and confidence intervals.

Significance. If substantiated, this is a practically valuable contribution: a simple, algorithm-agnostic gradient surgery that removes energy-reward weight tuning. The paper's strengths are its clear formulation, use of two independent critics, experiments with seeds and CIs, and real-robot validation. The evidence is consistent with a first-order energy reduction on top of task reward, but the 'non-conflicting' guarantee is only local and is contradicted by the paper's own Section 6, which reports unsafe or unwanted behaviors (maximizing base height, leg dragging, hopping) when style rewards are absent. Thus the current significance is conditional; with honest reframing and additional style-constrained evaluation it could become a useful practical tool.

major comments (4)
  1. [§3.2, Eq. (5)–(6)] The claim that PEGrad minimizes energy 'without conflicting with task performance' rests on a first-order argument: a step orthogonal to gR leaves LR unchanged to first order. This is only valid for an infinitesimal update to a fixed scalar loss. In actual training, LR is non-stationary (critics, replay buffer, and policy distribution change), and accumulated updates can move the policy to a different behavior class with the same or higher scalar reward. Section 6 provides direct evidence: without style rewards, PEGrad policies maximized base height, dragged legs, and hopped. These are task-violating or unsafe behaviors, not 'non-conflicting' energy savings. The manuscript should either restrict the claim to 'energy minimization that does not reduce the scalar task reward' with explicit caveats, or add behavioral-metric analysis showing the learned gaits remain acceptable.
  2. [§4.1, Figs. 2–3] The simulation evaluation is conducted in DM-Control and HumanoidBench, which do not include style rewards. Section 6 shows that, in exactly this setting, PEGrad can discover qualitatively undesired low-torque gaits. Therefore the headline ~64% average torque reduction and the Pareto-front plots may conflate genuine energy efficiency with reward exploitation. The authors need to either (a) report per-task qualitative or behavioral checks (e.g., gait videos, base height, foot-contact patterns) to rule out such artifacts in the reported tasks, or (b) train with style penalties and show the energy reduction persists. Without this, the central quantitative claim is confounded.
  3. [§4.2, Table 1; §8.3] The real-robot experiments use AMP with an explicit style reward and additional task/penalty terms; PEGrad is applied on top of this pipeline. Thus Table 1 demonstrates that PEGrad can reduce energy when style is separately enforced, not that PEGrad alone is non-conflicting. This is a weaker claim than the abstract. In addition, for standing, PEGrad's current draw (2.533±0.022 mA) is not clearly better than the tuned baseline (2.389±0.170). The authors should report task performance (e.g., velocity tracking or distance completed) and trial counts for all conditions, and avoid overstating 'comparable' without statistical support.
  4. [§3.2, Eq. (7)] The adaptive scaling β is a hand-designed formula with no derivation; the paper gives only an 'empirical justification.' This matters because the 'hyperparameter-free' claim is one of the main contributions. The authors should show sensitivity to the form of β (e.g., min(1, ||gR||/||gE⊥R||) vs. alternative norms) or explicitly present β as a heuristic design choice rather than a parameter-free consequence. If performance depends strongly on this formula, the method is not hyperparameter-free in any meaningful sense.
minor comments (4)
  1. [Abstract and Conclusion] The Abstract says 'reduction of 64% energy usage' while the Conclusion says 'up to 64%'; clarify whether this is an average over the ten tasks and report per-task numbers.
  2. [Figs. 2–3] The legend labels render as '∏=0⊿001' and similar; the lambda notation needs to be fixed for readability.
  3. [References] Reference [33] is cited for IsaacLab but the reference is the Orbit system; update the citation to the correct IsaacLab/Orbit source.
  4. [§1] Typo: 'IssacLab' should be 'IsaacLab'.

Circularity Check

0 steps flagged

No significant circularity: the energy-gradient projection is derived from a first-order Taylor expansion and is not fitted to the reported results.

full rationale

The paper's derivation chain is self-contained rather than circular. The update rule in Eq. 6 follows directly from the first-order Taylor approximation in Eq. 5 and an orthogonal projection of the energy gradient onto the reward gradient; it does not define the projection in terms of the measured torque reductions or returns. The energy critic Q^e is learned from environment samples like any standard critic, and the adaptive scaling in Eq. 7 is a hand-specified rule with an empirical justification, not a parameter fitted to the 64% energy-saving number. There are no load-bearing self-citations: the cited gradient-surgery methods (PCGrad, CAGrad, GradNorm, etc.) are external prior works, and no uniqueness theorem or author-imported assumption is used to force the method. The limitations in Section 6 describe behavioral failures when style rewards are absent; this is a validity/generalization concern about the 'non-conflict' claim, but it does not make the derivation circular, since those empirical outcomes are not encoded as inputs into the projection step. The central claim therefore has independent empirical content and no significant circularity.

Axiom & Free-Parameter Ledger

1 free parameters · 3 axioms · 0 invented entities

The method introduces no new physical entities. The central new ingredient is a gradient-transformation rule (Eq. 6-7) plus a learned energy critic; both are computational constructs rather than axioms. The load-bearing assumptions are mathematical and modeling choices listed above.

free parameters (1)
  • Adaptive energy-gradient scaling β = α * min(1, ||gR|| / ||gE⊥R||)
    Ad hoc formula introduced in Eq. 7 to limit the energy update's norm; justified by saying it 'works well across multiple settings' but not derived from the objective.
axioms (3)
  • standard math First-order Taylor expansion of the policy loss is valid over update steps (Eq. 5).
    Basis for projecting energy gradient orthogonally to task gradient; only valid locally.
  • domain assumption Sum of absolute joint torques is a suitable proxy for battery energy consumption.
    Used in all experiments; paper acknowledges it ignores motor/gear heterogeneity.
  • standard math Policy gradient estimates for task and energy are unbiased with finite variance.
    Needed for the projection to be meaningful; standard in policy gradient methods.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Non-conflicting Energy Minimization in Reinforcement Learning based Robot Control." pith.science (2026). https://pith.science/paper/H7F353VD

@misc{pith2026250901765,
  author       = {Pith},
  title        = {Pith review of: Non-conflicting Energy Minimization in Reinforcement Learning based Robot Control},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H7F353VD}},
  note         = {Machine review of arXiv:2509.01765}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Efficient robot control often requires balancing task performance with energy expenditure. A common approach in reinforcement learning (RL) is to penalize energy use directly as part of the reward function. This requires carefully tuning weight terms to avoid undesirable trade-offs where energy minimization harms task success. In this work, we propose a hyperparameter-free gradient optimization method to minimize energy expenditure without conflicting with task performance. Inspired by recent works in multitask learning, our method applies policy gradient projection between task and energy objectives to derive policy updates that minimize energy expenditure in ways that do not impact task performance. We evaluate this technique on standard locomotion benchmarks of DM-Control and HumanoidBench and demonstrate a reduction of 64% energy usage while maintaining comparable task performance. Further, we conduct experiments on a Unitree GO2 quadruped showcasing Sim2Real transfer of energy efficient policies. Our method is easy to implement in standard RL pipelines with minimal code changes, is applicable to any policy gradient method, and offers a principled alternative to reward shaping for energy efficient control policies.

Figures

Figures reproduced from arXiv: 2509.01765 by Akhil Perincherry, Bikram Pandit, Skand Peri, Stefan Lee.

Figure 1
Figure 1. Figure 1: RL control policies often optimize a weighted combination of task reward and energy [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: DMControl Suite Results: We show results on six tasks from DMControl suite [11]. Low applied torque and higher returns are better.Across all tasks, PEGrad achieves high task perfor￾mant policies that are also energy-efficient. For 4 out of 6 tasks, PEGrad achieves results beyond the Pareto front identified by adjusting λ. For the experiments on DM-Control suite and HumanoidBench we use SAC as the base RL a… view at source ↗
Figure 3
Figure 3. Figure 3: HumanoidBench Results: We show results on four tasks from HumanoidBench suite [12]. Low applied torque and higher returns are better. Across all tasks, PEGrad achieves highly performant policies that are also energy-efficient with PEGrad. Further, energy minimization also improves sample-efficiency on h1-run-v0 and h1-walk-v0 tasks. All results shown are run for 1.5-2M steps and run for 3 seeds. Shaded are… view at source ↗
Figure 4
Figure 4. Figure 4: a) Environments – We consider 4 locomotion environments – (i) DMControl-Dog, (ii) DMControl-Quadruped, (iii) HumanoidBench-H1 Humanoid, and (iv) Unitree Go2. We test PEGrad on 10 simulated tasks with DMControl and HumanoidBench environments and conduct a Sim2Real evaluation of standing and walking tasks with Unitree Go2 quadruped. b) Policy Encoder – We choose a SimBa-like [43] architecture that has shown … view at source ↗
Figure 5
Figure 5. Figure 5: We show results on HumanoidBench’s sit simple and walk tasks with mechanical power (τ.ω) as the energy formulation. We observe that PEGrad continues to achieve energy￾efficient and task-performant policies even with a different formulation. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Real world setup. (left) Go2 in Lab-setting: We consider a rubber mat flooring terrain on which the dataset for AMP has been collected. For the results in [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 31 canonical work pages · 2 internal anchors

  1. [1]

    Go2 developer documentation, 2024

    Unitree Robotics. Go2 developer documentation, 2024. URL https://support.unitree. com/home/en/developer/about_Go2. Accessed: 2025-04-24

    work page 2024

  2. [2]

    Hwangbo, J

    J. Hwangbo, J. Lee, A. Dosovitskiy, C. S. Bellicoso, V . Tsounis, V . Koltun, and M. Hutter. Learning agile and dynamic motor skills for legged robots. In Science Robotics, 2019

    work page 2019

  3. [3]

    J. Lee, J. Hwangbo, L. Wellhausen, V . Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. In Science Robotics, 2020

    work page 2020

  4. [4]

    Long et al

    X. Long et al. Robust terrain-adaptive locomotion for legged robots via h-infinity reinforce- ment learning. In IEEE Robotics and Automation Letters, 2023

    work page 2023

  5. [5]

    Y . Yang, T. Zhang, E. Coumans, J. Tan, and B. Boots. Fast and efficient locomotion via learned gait transitions. Proceedings of the Conference on Robot Learning, 2022

    work page 2022

  6. [6]

    Kumar, Z

    A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. In Robotics: Science and Systems (RSS), 2021

    work page 2021

  7. [7]

    Sim-to-Real Learning of All Common Bipedal Gaits via Periodic Reward Composition

    J. Siekmann, Y . Godse, A. Fern, and J. W. Hurst. Sim-to-real learning of all common bipedal gaits via periodic reward composition. arXiv preprint arXiv:2011.01387, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2011

  8. [8]

    van Marum, A

    B. van Marum, A. Shrestha, H. Duan, P. Dugar, J. Dao, and A. Fern. Revisiting reward design and evaluation for robust humanoid standing and walking. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2024

    work page 2024

  9. [9]

    Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: Learning a unified policy for manipulation and locomotion. In Proceedings of The 6th Conference on Robot Learning , Proceedings of Machine Learning Research, 2023. 9

    work page 2023

  10. [10]

    J. Tan, T. Zhang, E. Coumans, A. Iscen, Y . Bai, D. Hafner, S. Bohez, and V . Vanhoucke. Sim- to-real: Learning agile locomotion for quadruped robots. arXiv preprint arXiv:1804.10332 , 2018

    Pith/arXiv arXiv 2018

  11. [11]

    Tunyasuvunakool, A

    S. Tunyasuvunakool, A. Muldal, Y . Doron, S. Liu, S. Bohez, J. Merel, T. Erez, T. Lillicrap, N. Heess, and Y . Tassa. dm control: Software and tasks for continuous control. Software Impacts, 2020

    work page 2020

  12. [12]

    Sferrazza, D.-M

    C. Sferrazza, D.-M. Huang, X. Lin, Y . Lee, and P. Abbeel. Humanoidbench: Simu- lated humanoid benchmark for whole-body locomotion and manipulation. arXiv preprint arXiv:2403.10506, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    W. Chen, X. Zhang, B. Lin, X. Lin, H. Zhao, Q. Zhang, and J. T. Kwok. Gradient-based multi-objective deep learning: Algorithms, theories, applications, and beyond. arXiv preprint arXiv:2501.10945, 2025

    Pith/arXiv arXiv 2025

  14. [14]

    Sener and V

    O. Sener and V . Koltun. Multi-task learning as multi-objective optimization. Advances in neural information processing systems, 31, 2018

    work page 2018

  15. [15]

    D ´esid´eri

    J.-A. D ´esid´eri. Multiple-gradient descent algorithm (mgda) for multiobjective opti- mization. Comptes Rendus Mathematique , 350:313–318, 2012. URL https://api. semanticscholar.org/CorpusID:120459561

    work page 2012

  16. [16]

    T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi- task learning. In Advances in Neural Information Processing Systems, 2020

    work page 2020

  17. [17]

    B. Liu, X. Liu, X. Jin, P. Stone, and Q. Liu. Conflict-averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, 34:18878–18890, 2021

    work page 2021

  18. [18]

    Z. Chen, V . Badrinarayanan, C.-Y . Lee, and A. Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. InInternational conference on machine learning, pages 794–803. PMLR, 2018

    work page 2018

  19. [19]

    Y . Du, W. M. Czarnecki, S. M. Jayakumar, M. Farajtabar, R. Pascanu, and B. Lak- shminarayanan. Adapting auxiliary losses using gradient similarity. arXiv preprint arXiv:1812.02224, 2018

    Pith/arXiv arXiv 2018

  20. [20]

    Z. Chen, J. Ngiam, Y . Huang, T. Luong, H. Kretzschmar, Y . Chai, and D. Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. Advances in Neural Information Processing Systems, 33:2039–2050, 2020

    work page 2039

  21. [21]

    P. Xu, X. Shang, V . B. Zordan, and I. Karamouzas. Composite motion learning with task control. ACM Transactions on Graphics (TOG), 2023

    work page 2023

  22. [22]

    R. Nai, J. You, L. Cao, H. Cui, S. Zhang, H. Xu, and Y . Gao. Fine-tuning hard-to-simulate objectives for quadruped locomotion: A case study on total power saving. arXiv preprint arXiv:2502.10956, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Z. Fu, A. Kumar, J. Malik, and D. Pathak. Minimizing energy consumption leads to the emer- gence of gaits in legged robots. arXiv preprint arXiv:2111.01674, 2021

    Pith/arXiv arXiv 2021

  24. [24]

    Mahankali, C.-C

    S. Mahankali, C.-C. Lee, G. B. Margolis, Z.-W. Hong, and P. Agrawal. Maximizing quadruped velocity by minimizing energy. In 2024 IEEE International Conference on Robotics and Au- tomation (ICRA). IEEE, 2024

    work page 2024

  25. [25]

    C. F. Hayes, R. R ˘adulescu, E. Bargiacchi, J. K ¨allstr¨om, M. Macfarlane, M. Reymond, T. Ver- straeten, L. M. Zintgraf, R. Dazeley, F. Heintz, et al. A practical guide to multi-objective reinforcement learning and planning. Autonomous Agents and Multi-Agent Systems, 36(1):26, 2022. 10

    work page 2022

  26. [26]

    V . R. Konda and J. N. Tsitsiklis. Actor-critic algorithms. Technical report, MIT, 1999. Appears in NeurIPS 1999 workshop-style proceedings

    work page 1999

  27. [27]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning , 2018. URL https://proceedings.mlr.press/v80/ haarnoja18b.html

    work page 2018

  28. [28]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  29. [29]

    Todorov, T

    E. Todorov, T. Erez, and Y . Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2012. doi:10.1109/IROS.2012.6386109

    arXiv 2012

  30. [30]

    Moens and contributors

    V . Moens and contributors. Leanrl: Turbo-implementations of cleanrl scripts. https://github.com/pytorch-labs/leanrl, 2024. URL https://github.com/ pytorch-labs/leanrl. GitHub repository. Fork of CleanRL, optimized for PyTorch 2.0 features

    work page 2024

  31. [31]

    Huang, R

    S. Huang, R. F. J. Dossa, C. Ye, J. Braga, D. Chakraborty, K. Mehta, and J. G. Ara´ujo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.Journal of Machine Learning Research, 2022. URL http://jmlr.org/papers/v23/21-1342.html

    work page 2022

  32. [32]

    W.-C. Tseng. Weichengtseng/pytorch-pcgrad, 2020. URL https://github.com/ WeiChengTseng/Pytorch-PCGrad.git

    work page 2020

  33. [33]

    Mittal, C

    M. Mittal, C. Yu, Q. Yu, J. Liu, N. Rudin, D. Hoeller, J. L. Yuan, R. Singh, Y . Guo, H. Mazhar, A. Mandlekar, B. Babich, G. State, M. Hutter, and A. Garg. Orbit: A unified simulation framework for interactive robot learning environments.IEEE Robotics and Automation Letters, 8(6):3740–3747, 2023. doi:10.1109/LRA.2023.3270034

    arXiv 2023

  34. [34]

    Rudin, D

    N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of the 5th Conference on Robot Learning, Proceedings of Machine Learning Research. PMLR, 2022. URL https://proceedings. mlr.press/v164/rudin22a.html

    work page 2022

  35. [35]

    X. B. Peng, Z. Ma, P. Abbeel, S. Levine, and A. Kanazawa. Amp: adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics, 40(4):1–20, July

    work page

  36. [36]

    Y . Wang, Z. Jiang, and J. Chen. Learning robust, agile, natural legged locomotion skills in the wild. arXiv preprint arXiv:2304.10888, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Think on your feet: Seamless Transition between Human-like Locomotion in Response to Changing Commands

    H. Huang, W. Cui, T. Zhang, S. Li, J. Han, B. Qin, T. Zhang, L. Zheng, Z. Tang, C. Hu, et al. Think on your feet: Seamless transition between human-like locomotion in response to changing commands. arXiv preprint arXiv:2502.18901, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  38. [38]

    A. Tang, T. Hiraoka, N. Hiraoka, F. Shi, K. Kawaharazuka, K. Kojima, K. Okada, and M. Inaba. Humanmimic: Learning natural locomotion and transitions for humanoid robot via wasserstein adversarial imitation. In 2024 IEEE International Conference on Robotics and Automation (ICRA), 2024

    work page 2024

  39. [39]

    Escontrela, X

    A. Escontrela, X. B. Peng, W. Yu, T. Zhang, A. Iscen, K. Goldberg, and P. Abbeel. Adver- sarial motion priors make good substitutes for complex reward functions. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022. 11

    work page 2022

  40. [40]

    Zargarbashi, J

    F. Zargarbashi, J. Cheng, D. Kang, R. Sumner, and S. Coros. Robotkeyframing: Learning locomotion with high-level objectives via mixture of dense and sparse rewards. arXiv preprint arXiv:2407.11562, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    T. Li, Y . Zhang, C. Zhang, Q. Zhu, J. Sheng, W. Chi, C. Zhou, and L. Han. Learning terrain- adaptive locomotion with agile behaviors by imitating animals. In 2023 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), 2023

    work page 2023

  42. [42]

    Mittal, N

    M. Mittal, N. Rudin, V . Klemm, A. Allshire, and M. Hutter. Symmetry considerations for learning task symmetric robot policies. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 7433–7439, 2024. doi:10.1109/ICRA57147.2024.10611493

    arXiv 2024

  43. [43]

    H. Lee, D. Hwang, D. Kim, H. Kim, J. J. Tai, K. Subramanian, P. R. Wurman, J. Choo, P. Stone, and T. Seno. Simba: Simplicity bias for scaling up parameters in deep reinforce- ment learning. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=jXLiDKsuDo. 12 8 Appendix 8.1 Multi-Objective PPO Proximal Po...

    work page 2025

  44. [2021]

    doi:10.1145/3450626.3459670

    ISSN 1557-7368. doi:10.1145/3450626.3459670. URL http://dx.doi.org/10. 1145/3450626.3459670

    arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.