Pith. sign in

REVIEW 3 major objections 4 minor 62 references

The paper claims that warm-starting PPO with a pretrained inverse-dynamics model lifts locomotion performance by 7.3% and cuts training iterations by 36.9% across nine tasks and three robot bodies.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 09:56 UTC pith:7Q2DJTV5

load-bearing objection A useful warm-starting trick for locomotion RL with a solid empirical sweep, though the headline sample-efficiency number is inflated by a threshold metric that counts initial advantage as convergence. the 3 major comments →

arxiv 2510.12363 v4 pith:7Q2DJTV5 submitted 2025-10-14 cs.RO cs.LG

Pretraining in Actor-Critic Reinforcement Learning for Locomotion

Jiale Fan , Andrei Cramariuc , Tifanny Portela , Marco Hutter This is my paper
classification cs.RO cs.LG
keywords pretraining-finetuningactor-critic reinforcement learninginverse dynamics modelwarm-start initializationreward-free explorationrobot locomotionsample efficiencylegged robots
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Most robot locomotion skills are still learned from scratch, even when they run on the same robot body. This paper argues that what a policy absorbs in its earliest, clumsiest training steps — basic kinematics, contact behavior, how the joints react to commands — is knowledge about the body, not about any particular task, and so it should be captured once and reused. The recipe: let a robot explore with a curiosity-driven policy to gather reward-free transition data; train a Proprioceptive Inverse Dynamics Model (PIDM), a compact MLP that predicts which action moves the joints toward a desired next state; then load that learned backbone into both the actor and critic of PPO (Proximal Policy Optimization) before task-specific training starts. Across nine locomotion tasks on two quadrupeds and one humanoid, the pretrained start improves final performance in eight of nine tasks (tying the ninth) and speeds convergence in all nine, averaging a 7.3% performance gain and a 36.9% reduction in iterations to convergence compared with random initialization of the same architecture. If the recipe holds, the expensive stumbling phase of locomotion RL can be paid for once per robot body instead of once per skill.

Core claim

The central claim: a motor policy separates 'which state do I want next' from 'which action gets me there', and that second mapping — the body's inverse dynamics — is nearly task-independent. The paper labels this Hypothesis 1, supported by a probing experiment: deeper layers of trained policies carry progressively less information about future joint state. The PIDM is built to be that late-stage function, trained by supervised L1 regression on reward-free exploration data, then spliced into the actor and critic as a shared trunk behind randomly initialized task-specific heads and fine-tuned by unmodified PPO. Reported result: +7.3% final performance and 36.9% fewer iterations to converge ve

What carries the argument

The Proprioceptive Inverse Dynamics Model (PIDM) is the key object: a modular MLP mapping a history of proprioception and actions plus a desired next-state change (delta-state) to the action that produces it. An ensemble of five PIDMs drives data collection; the spread of their predictions is an intrinsic reward steering the robot toward uncertain states. The same objective is then trained by supervised regression on that buffer. At fine-tuning, the pretrained backbone becomes the shared trunk of PPO's actor and critic, with a randomly initialized intention-encoder replacing the delta-encoder and a randomly initialized synthesizer replacing the decoder. Modularity is the source of task-agnos

Load-bearing premise

The load-bearing premise is that reward-free exploration data collected on flat and lightly rough ground covers the state distribution every downstream task visits during early training — including the five parkour tasks whose terrains never appear in pretraining — and this overlap is verified for only two flat-terrain tasks.

What would settle it

Repeat the paper's own UMAP overlap analysis (Figure 5a) between the flat-terrain pretraining buffer and the first 100 PPO iterations of a perceptive parkour task (e.g., Climb Up or Jump) started from random initialization. If the overlap is comparable to the flat-terrain case, the coverage rationale survives; if it is small yet warm-starting still helps, the stated mechanism is wrong even though the method works. A sharper mechanistic test: scramble the pretrained backbone's weights while preserving the input–output statistics of the initial policy, and check whether the gains persist; if the

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • New skills on a pretrained robot converged to the same performance level in about a third fewer training iterations, so the pretrained start directly saves compute per downstream task.
  • Because the pretrained weights carry no reward or task information, one PIDM per embodiment can serve tasks with different commands, observations, rewards, curricula, and even terrains not present in pretraining.
  • The method is a drop-in swap: rewards, curricula, observations, and PPO hyperparameters are left untouched, so existing locomotion training pipelines can adopt it without retuning.
  • Better dynamics models translate into better RL: checkpoints with lower PIDM error produced larger downstream gains, suggesting headroom from improved pretraining.
  • A policy trained with this initialization transferred from simulation to a real quadruped, indicating that the warm-started network preserves deployability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper validates the early-training coverage assumption for two flat-terrain tasks only; a direct check of whether flat-terrain exploration data covers the early state distribution of the perceptive parkour tasks would either confirm the stated mechanism or force a revised explanation.
  • The weight-update measurements suggest the pretrained backbone changes how the random heads learn, not just what the initial actions are — a test the paper does not run (freezing the backbone for the first iterations) could separate 'good initial behavior' from 'better optimization geometry' as the causal channel.
  • The success of a single shared embodiment module hints at a wider design principle for legged robots — an embodiment trunk plus task-specific heads — that could compose with skill libraries, multi-task learning, or cross-task distillation beyond the single-task fine-tuning studied here.
  • Because gains scale with dynamics-model accuracy, the recipe is probably not at its ceiling; a larger or temporal backbone could widen the improvements if the RL update stays stable.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This manuscript proposes a task-agnostic pretraining method for actor-critic RL in legged locomotion. A Proprioceptive Inverse Dynamics Model (PIDM) is trained from transitions collected by an exploration policy, and its weights are used to initialize the actor and critic of PPO. Across 9 tasks and 3 embodiments, the authors report a 7.3% final-performance improvement and a 36.9% sample-efficiency improvement relative to random initialization of the same PIDM architecture, with ablations, coverage analysis, and a sim-to-real deployment.

Significance. The contribution is potentially useful: it offers a plug-in initialization that requires no reward-labeled offline data or expert demonstrations, and the main comparison (pretrained vs. random-init PIDM) isolates the effect of pretraining from architecture. Strengths include a clear three-stage pipeline, a reproducible experimental setup (code bundled), ablations for actor/critic and data source, and a sim-to-real demo. The idea is not fully novel—inverse-model pretraining and exploration-based data collection exist—but applying it to warm-start locomotion PPO across multiple embodiments is a reasonable empirical contribution. However, the headline sample-efficiency claim rests on a metric that can count an initial advantage as 'convergence,' and the aggregate statistics are weakened by excluded failed runs and missing error bars.

major comments (3)
  1. [§5.4 (metric definition)] The 'Number of iterations to converge' is defined as iterations to reach 90% of the final performance of PIDM (Random Init), not 90% of the evaluated method's own final performance. If a pretrained run's initial or early performance already exceeds 0.9 × baseline_final, it is counted as converged at iteration 0. Since the pretrained runs often start with an advantage (Figure 6) and have higher final performance than the baseline, the reported '36.9% sample-efficiency improvement' is not a clean measure of convergence speed; it conflates an initial advantage with faster learning. I recommend redefining the threshold relative to each method's own final performance (or a common reference performance), and reporting area-under-curve or full time-to-threshold curves.
  2. [Table 1 / Table 8] The headline numbers are aggregates over five seeds without error bars, and five random-init baseline runs are excluded (Jump: 3/5 failed; Walk and Crouch: 1/5 each). This makes the quantitative conclusions fragile. In particular, the Jump improvement of +5.9% is based on only two successful baseline runs. Please report per-seed results, confidence intervals, and a sensitivity analysis where failed runs are assigned a worst-case performance or convergence time. As it stands, the +7.3% final-performance figure could be partly an artifact of dropping the most unstable baseline runs.
  3. [§4.2 / §5.3] The task-agnostic transfer premise is validated only for two flat-terrain tasks (pedipulation and locomotion). Figure 5a shows coverage of initial exploration for these tasks, but the claim that the initial state distribution is similar across all downstream tasks is used to justify transfer to five perceptive parkour tasks on terrains unseen in pretraining. Either extend the coverage analysis to representative parkour tasks, or moderate the mechanistic claim. The empirical results are encouraging, but the stated mechanism is not fully evidenced.
minor comments (4)
  1. [Abstract] The abstract reports inconsistent numbers: one version states 36.2% sample efficiency and 4.3% task performance, while the body and the other abstract version state 36.9% and 7.3%. Reconcile all occurrences.
  2. [§4.5] The claim that the randomly initialized Action Synthesizer preserves near-unit Gaussian action distribution at initialization is stated without quantitative support. A small experiment or reference to a figure would strengthen this load-bearing design explanation.
  3. [Appendix A.6.2] The modifications to benchmark tasks (removed curricula, unified collision model, changed pedipulation action space) are documented, but there is no quantitative evidence that they leave the relative ranking of methods unchanged. A brief comparison of the vanilla MLP under original versus modified settings would improve comparability.
  4. [Table 1] The derived 'improvement over vanilla MLP' values in the main text (5.0% final, 18.8% sample efficiency) do not exactly match the differences between rows in Table 1 (7.3 - 2.2 = 5.1; -36.9 - (-17.6) = -19.3). Please reconcile or clarify how these were computed.

Circularity Check

0 steps flagged

No significant circularity; the main result is an out-of-sample warm-start comparison against a random-init control.

full rationale

The paper's derivation chain is not circular in the sense that matters here. The pretrained PIDM is fitted by supervised learning on task-agnostic exploration transitions, and the downstream RL evaluation compares a pretrained-initialized actor-critic against the identical architecture with random initialization. No fitted value is recycled into the headline metric: final performance and iteration-to-convergence are measured on separate RL runs whose rewards and terrains are not used to train the PIDM. The data-source ablation (Table 3) and the PIDM-error-vs-RL-performance study (Appendix A.8) provide independent checks that the benefit tracks the pretraining content. The sample-efficiency metric is defined relative to the random-init baseline's final performance, which can partly credit an initial-performance advantage as 'convergence'; this is a legitimate metric-validity concern about the 36.9% claim, not a circularity. The paper's many self-citations (Miki et al., Rudin et al., Hoeller et al., Mittal et al., etc.) are used for standard simulators, baselines, and architecture hyperparameters, not to establish the central claim by citing the authors' own unverified result. No uniqueness theorem, ansatz-by-citation, or definitional identification of input and output is present. The comparison is therefore self-contained and externally falsifiable; score is low.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

No new physical entities are postulated; the PIDM is an algorithm architecture, not a natural-kind entity. The central claim rests on empirical transfer assumptions rather than a formal derivation, so the axiom ledger lists the domain assumptions and hand-tuned components that the empirical results depend on.

free parameters (6)
  • Intrinsic reward scaling factor c_ir = 10
    Table 4; tuned empirically to guide exploration; directly shapes pretraining data distribution.
  • Intrinsic reward clipping threshold r_i_max = 30
    Table 4; tuned empirically.
  • Exploration extrinsic reward weights = 400 (air time), -5.0 (collision), -2e-5 (torques), -5e-2 (velocities), -5e-6 (acceleration), -0.01 (action magnitude), -
    Table 5; hand-designed regularizers; no derivation; affect which transitions are collected.
  • PIDM history length K = 4
    Table 6; chosen by hand to handle partial observability; directly affects input dimension and architecture.
  • PIDM backbone widths = Backend [512,256,128], encoders [128]
    Table 6; architecture choice; paper says model is ~4x vanilla MLP in parameters.
  • Symmetry loss weight = 0.2
    Appendix A.6.2; added to all methods to stabilize PIDM training; not part of original task implementations.
axioms (6)
  • domain assumption Hypothesis 1: a neural-network parameterized policy first formulates the intended target state and afterwards the action necessary to reach that state
    Section 4.1; motivates the PIDM architecture; only correlational evidence in Section 5.2.
  • domain assumption State distributions of randomly initialized policies for different tasks are very similar
    Section 4.2; justifies task-agnostic data collection; verified only for two flat-terrain tasks in Section 5.3.
  • domain assumption Exploration data from flat/basic rough terrain transfers to unseen complex parkour terrains
    Section 4.4 and 5.4; empirically shown but not explained; if false, pretrained weights would not transfer.
  • domain assumption Vanilla-optimal PPO hyperparameters remain appropriate for the PIDM architecture
    Section 5.4; authors state they did not tune learning parameters; possible that tuned hyperparameters change results.
  • ad hoc to paper Randomly initialized Action Synthesizer preserves near-unit Gaussian action distribution at initialization, stabilizing RL
    Section 4.5; empirical design choice, not derived.
  • ad hoc to paper Modifications to benchmark tasks (removed curricula, unified collision model, changed pedipulation action space) do not change the relative ranking of methods
    Appendix A.6.2; necessary for fair comparison but changes the original tasks.

pith-pipeline@v1.3.0-alltime-deepseek · 23532 in / 17585 out tokens · 136852 ms · 2026-08-04T09:56:46.733220+00:00 · methodology

0 comments
read the original abstract

The pretraining-finetuning paradigm has facilitated numerous transformative advancements in artificial intelligence research in recent years. However, in the domain of reinforcement learning (RL) for robot locomotion, individual skills are often learned from scratch despite the high likelihood that some generalizable knowledge is shared across all task-specific policies belonging to the same robot embodiment. This work aims to define a paradigm for pretraining neural network models that encapsulate such knowledge and can subsequently serve as a basis for warm-starting the RL process in classic actor-critic algorithms, such as Proximal Policy Optimization (PPO). We begin with a task-agnostic exploration-based data collection algorithm to gather diverse, dynamic transition data, which is then used to train a Proprioceptive Inverse Dynamics Model (PIDM) through supervised learning. The pretrained weights are then loaded into both the actor and critic networks to warm-start the policy optimization of actual tasks. We systematically validated our proposed method with 9 distinct robot locomotion RL environments comprising 3 different robot embodiments, showing significant benefits of this initialization strategy. Our proposed approach on average improves sample efficiency by 36.2% and task performance by 4.3% compared to random initialization. We further present key ablation studies and empirical analyses that shed light on the mechanisms behind the effectiveness of this method.

Figures

Figures reproduced from arXiv: 2510.12363 by Andrei Cramariuc, Jiale Fan, Marco Hutter, Tifanny Portela.

Figure 1
Figure 1. Figure 1: Method overview: We (i) collect task-agnostic data using an exploration-driven policy, (ii) to train a Proprioceptive Inverse Dynamics Model (PIDM) to capture embodiment-aware dy￾namics, and (iii) initialize the actor-critic networks in PPO to warm-start the RL process. 4 [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Proprioceptive Inverse Dynamics Model (PIDM) ar [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Experiment to probe dynamics knowledge in vanilla policy networks. We analyze the [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: PIDM training and dataset analysis of ANYmal D: For the pretraining dataset we visualize its (a) coverage (green) compared to the initial exploration stages in RL (red and blue) using an UMAP projection and (b) the sample distribution of absolute action magnitudes |at − qt| over different terrains. Finally, in (c) we show the resulting PIDM accuracy across the entire action range as absolute joint errors |… view at source ↗
Figure 7
Figure 7. Figure 7: Network weight update magnitude comparison in the PIDM structured actor network [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Errors of dynamics prediction based on representations from different layers, [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 11
Figure 11. Figure 11: Ablation results for pretraining the actor vs. the critic components. 0 250 500 750 1000 Iteration Number 0 5 10 Mean Curriculum Progress Climb Up 0 250 500 750 1000 Iteration Number 0 5 mean_terrain_level Climb Up 0 250 500 750 1000 Iteration Number 0 5 10 Mean Reward Climb Up PIDM (Exploration data) PIDM (Random Init) PIDM (Pedipulation data) 0 500 1000 1500 2000 2500 Iteration Number 0 2 4 6 8 10 Mean … view at source ↗
Figure 13
Figure 13. Figure 13: PIDM error levels of the 3 checkpoints used to study the relation between PIDM error [PITH_FULL_IMAGE:figures/full_fig_p026_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Study of relation between error Level of PIDM and RL performance. [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Network weight update magnitude comparison. [PITH_FULL_IMAGE:figures/full_fig_p027_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

62 extracted references · 1 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  2. [2]

    Learning markov state abstractions for deep reinforcement learning

    Cameron Allen, Neev Parikh, Omer Gottesman, and George Konidaris. Learning markov state abstractions for deep reinforcement learning. In Marc'Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems...

    2021

  3. [3]

    Pedipulate: Enabling Manipulation Skills using a Quadruped Robot 's Leg , 2024

    Philip Arm, Mayank Mittal, Hendrik Kolvenbach, and Marco Hutter. Pedipulate: Enabling Manipulation Skills using a Quadruped Robot 's Leg , 2024

    2024

  4. [4]

    Scaling mlps: A tale of inductive bias

    Gregor Bachmann, Sotiris Anagnostidis, and Thomas Hofmann. Scaling mlps: A tale of inductive bias. Advances in Neural Information Processing Systems, 36: 0 60821--60840, 2023

    2023

  5. [5]

    A careful examination of large behavior models for multitask dexterous manipulation

    Jose Barreiros, Andrew Beaulieu, Aditya Bhat, Rick Cory, Eric Cousineau, Hongkai Dai, Ching-Hsin Fang, Kunimatsu Hashimoto, Muhammad Zubair Irshad, Masha Itkina, et al. A careful examination of large behavior models for multitask dexterous manipulation. arXiv preprint arXiv:2507.05331, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Dario Bellicoso, Koen Krämer, Markus Stäuble, Dhionis Sako, Fabian Jenelten, Marko Bjelonic, and Marco Hutter

    C. Dario Bellicoso, Koen Krämer, Markus Stäuble, Dhionis Sako, Fabian Jenelten, Marko Bjelonic, and Marco Hutter. Alma - articulated locomotion and manipulation for a torque-controllable robot. pp.\ 8477--8483, 2019. doi:10.1109/ICRA.2019.8794273

    arXiv 2019

  7. [7]

    pi 0: A vision-language-action flow model for general robot control

    Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi 0: A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    RT -2: Vision - Language - Action Models Transfer Web Knowledge to Robotic Control

    Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov...

  9. [9]

    Symmetric reinforcement learning loss for robust learning on diverse tasks and model scales

    Ju-Seung Byun and Andrew Perrault. Symmetric reinforcement learning loss for robust learning on diverse tasks and model scales. arXiv preprint arXiv:2405.17618, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Learning quadrupedal locomotion on deformable terrain

    Suyoung Choi, Gwanghyeon Ji, Jeongsoo Park, Hyeongjun Kim, Juhyeok Mun, \ Jeong Hyun\ Lee, and Jemin Hwangbo. Learning quadrupedal locomotion on deformable terrain. Science Robotics, 8 0 (74), January 2023. ISSN 2470-9476. doi:10.1126/scirobotics.ade2256. Publisher Copyright: 2023 The Authors, some rights reserved

    doi:10.1126/scirobotics.ade2256 2023

  11. [11]

    Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model , 2016

    Paul Christiano, Zain Shah, Igor Mordatch, Jonas Schneider, Trevor Blackwell, Joshua Tobin, Pieter Abbeel, and Wojciech Zaremba. Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model , 2016

    2016

  12. [12]

    Deep reinforcement learning in a handful of trials using probabilistic dynamics models

    Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicol \` o Cesa - Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neu...

    2018

  13. [13]

    Efficient model-based reinforcement learning through optimistic policy search and planning

    Sebastian Curi, Felix Berkenkamp, and Andreas Krause. Efficient model-based reinforcement learning through optimistic policy search and planning. Advances in Neural Information Processing Systems, 33: 0 14156--14170, 2020

    2020

  14. [14]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol...

    doi:10.18653/v1/ 2019

  15. [15]

    Roloma: Robust loco-manipulation for quadruped robots with arms

    Henrique Ferrolho, Vladimir Ivan, Wolfgang Merkt, Ioannis Havoutis, and Sethu Vijayakumar. Roloma: Robust loco-manipulation for quadruped robots with arms. Autonomous Robots, 47 0 (8): 0 1463--1481, 2023

    2023

  16. [16]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.\ 1026--1034, 2015

    2015

  17. [17]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll \'a r, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.\ 16000--16009, 2022

    2022

  18. [18]

    ANYmal Parkour : Learning Agile Navigation for Quadrupedal Robots , 2023

    David Hoeller, Nikita Rudin, Dhionis Sako, and Marco Hutter. ANYmal Parkour : Learning Agile Navigation for Quadrupedal Robots , 2023

    2023

  19. [19]

    Dario Bellicoso, Vassilios Tsounis, Jemin Hwangbo, Karen Bodie, Peter Fankhauser, Michael Bloesch, Remo Diethelm, Samuel Bachmann, Amir Melzer, and Mark Hoepflinger

    Marco Hutter, Christian Gehring, Dominic Jud, Andreas Lauber, C. Dario Bellicoso, Vassilios Tsounis, Jemin Hwangbo, Karen Bodie, Peter Fankhauser, Michael Bloesch, Remo Diethelm, Samuel Bachmann, Amir Melzer, and Mark Hoepflinger. ANYmal - a highly mobile and dynamic quadrupedal robot. In 2016 IEEE / RSJ International Conference on Intelligent Robots and ...

    arXiv 2016

  20. [20]

    Learning agile and dynamic motor skills for legged robots

    Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots. ArXiv preprint, abs/1901.08652, 2019

    Pith/arXiv arXiv 1901

  21. [21]

    Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms, 2021

    Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms, 2021. URL https://arxiv.org/abs/2102.00815

    Pith/arXiv arXiv 2021

  22. [22]

    The role of domain randomization in training diffusion policies for whole-body humanoid control

    Oleg Kaidanov, Firas Al-Hafez, Yusuf Suvari, Boris Belousov, and Jan Peters. The role of domain randomization in training diffusion policies for whole-body humanoid control. arXiv preprint arXiv:2411.01349, 2024

    Pith/arXiv arXiv 2024

  23. [23]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    Pith/arXiv arXiv 2014

  24. [24]

    Actor-critic algorithms

    Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In S. Solla, T. Leen, and K. M\" u ller (eds.), Advances in Neural Information Processing Systems, volume 12. MIT Press, 1999. URL https://proceedings.neurips.cc/paper_files/paper/1999/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf

    1999

  25. [25]

    Learning quadrupedal locomotion over challenging terrain

    Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning quadrupedal locomotion over challenging terrain. Science robotics, 5 0 (47): 0 eabc5986, 2020

    2020

  26. [26]

    Learning to Walk from Three Minutes of Real - World Data with Semi -structured Dynamics Models , 2024

    Jacob Levy, Tyler Westenbroek, and David Fridovich-Keil. Learning to Walk from Three Minutes of Real - World Data with Semi -structured Dynamics Models , 2024

    2024

  27. [27]

    A Survey : Learning Embodied Intelligence from Physical Simulators and World Models , 2025

    Xiaoxiao Long, Qingrui Zhao, Kaiwen Zhang, Zihao Zhang, Dingrui Wang, Yumeng Liu, Zhengjie Shu, Yi Lu, Shouzheng Wang, Xinzhe Wei, Wei Li, Wei Yin, Yao Yao, Jia Pan, Qiu Shen, Ruigang Yang, Xun Cao, and Qionghai Dai. A Survey : Learning Embodied Intelligence from Physical Simulators and World Models , 2025

    2025

  28. [28]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019

    2019

  29. [29]

    Combining physics and deep learning to learn continuous-time dynamics models

    Michael Lutter and Jan Peters. Combining physics and deep learning to learn continuous-time dynamics models. The International Journal of Robotics Research, 42 0 (3): 0 83--107, 2023. doi:10.1177/02783649231169492

    work page doi:10.1177/02783649231169492 2023

  30. [30]

    Umap: Uniform manifold approximation and projection for dimension reduction

    Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018

    Pith/arXiv arXiv 2018

  31. [31]

    Learning robust perceptive locomotion for quadrupedal robots in the wild

    Takahiro Miki, Joonho Lee, Jemin Hwangbo, Lorenz Wellhausen, Vladlen Koltun, and Marco Hutter. Learning robust perceptive locomotion for quadrupedal robots in the wild. ArXiv preprint, abs/2201.08117, 2022

    Pith/arXiv arXiv 2022

  32. [32]

    Orbit: A unified simulation framework for interactive robot learning environments

    Mayank Mittal, Calvin Yu, Qinxi Yu, Jingzhou Liu, Nikita Rudin, David Hoeller, Jia Lin Yuan, Ritvik Singh, Yunrong Guo, Hammad Mazhar, Ajay Mandlekar, Buck Babich, Gavriel State, Marco Hutter, and Animesh Garg. Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters, 8 0 (6): 0 3740--3747, 20...

    arXiv 2023

  33. [33]

    Symmetry considerations for learning task symmetric robot policies

    Mayank Mittal, Nikita Rudin, Victor Klemm, Arthur Allshire, and Marco Hutter. Symmetry considerations for learning task symmetric robot policies. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp.\ 7433--7439. IEEE, 2024

    2024

  34. [34]

    Murphy, Benjamin J

    Michael P. Murphy, Benjamin J. Stephens, Yeuhi Abe, and Alfred A. Rizzi. High degree-of-freedom dynamic manipulation. In Defense, Security, and Sensing, 2012. URL https://api.semanticscholar.org/CorpusID:121683243

    2012

  35. [35]

    Information-directed exploration for deep reinforcement learning

    Nikolay Nikolov, Johannes Kirschner, Felix Berkenkamp, and Andreas Krause. Information-directed exploration for deep reinforcement learning. arXiv preprint arXiv:1812.07544, 2018

    Pith/arXiv arXiv 2018

  36. [36]

    AMP : Adversarial Motion Priors for Stylized Physics - Based Character Control

    Xue Bin Peng, Ze Ma, Pieter Abbeel, Sergey Levine, and Angjoo Kanazawa. AMP : Adversarial Motion Priors for Stylized Physics - Based Character Control . ArXiv preprint, abs/2104.02180, 2021

    Pith/arXiv arXiv 2021

  37. [37]

    ASE : large-scale reusable adversarial skill embeddings for physically simulated characters

    Xue Bin Peng, Yunrong Guo, Lina Halper, Sergey Levine, and Sanja Fidler. ASE : large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions on Graphics, 41 0 (4): 0 1--17, 2022. ISSN 0730-0301, 1557-7368. doi:10.1145/3528223.3530110

    arXiv 2022

  38. [38]

    Whole-body end-effector pose tracking, 2024

    Tifanny Portela, Andrei Cramariuc, Mayank Mittal, and Marco Hutter. Whole-body end-effector pose tracking, 2024

    2024

  39. [39]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  40. [40]

    Learning to walk in minutes using massively parallel deep reinforcement learning

    Nikita Rudin, David Hoeller, Philipp Reist, and Marco Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Conference on robot learning, pp.\ 91--100. PMLR, 2022

    2022

  41. [41]

    Parkour in the Wild : Learning a General and Extensible Agile Locomotion Policy Using Multi -expert Distillation and RL Fine -tuning, 2025

    Nikita Rudin, Junzhe He, Joshua Aurand, and Marco Hutter. Parkour in the Wild : Learning a General and Extensible Agile Locomotion Policy Using Multi -expert Distillation and RL Fine -tuning, 2025

    2025

  42. [42]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  43. [43]

    Rsl-rl: A learning library for robotics research

    Clemens Schwarke, Mayank Mittal, Nikita Rudin, David Hoeller, and Marco Hutter. Rsl-rl: A learning library for robotics research. arXiv preprint arXiv:2509.10771, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Devon Hjelm, Philip Bachman, and Aaron C

    Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R. Devon Hjelm, Philip Bachman, and Aaron C. Courville. Pretraining representations for data-efficient reinforcement learning. In Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, Decembe...

    2021

  45. [45]

    Planning to explore via self-supervised world models

    Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International conference on machine learning, pp.\ 8583--8592. PMLR, 2020

    2020

  46. [46]

    Blind bipedal stair traversal via sim-to-real reinforcement learning, 2021

    Jonah Siekmann, Kevin Green, John Warila, Alan Fern, and Jonathan Hurst. Blind bipedal stair traversal via sim-to-real reinforcement learning, 2021. URL https://arxiv.org/abs/2105.08328

    Pith/arXiv arXiv 2021

  47. [47]

    A unified mpc framework for whole-body dynamic locomotion and manipulation, 2021

    Jean-Pierre Sleiman, Farbod Farshidian, Maria Vittoria Minniti, and Marco Hutter. A unified mpc framework for whole-body dynamic locomotion and manipulation, 2021. URL https://arxiv.org/abs/2103.00946

    Pith/arXiv arXiv 2021

  48. [48]

    Guided Reinforcement Learning for Robust Multi - Contact Loco - Manipulation , 2024

    Jean-Pierre Sleiman, Mayank Mittal, and Marco Hutter. Guided Reinforcement Learning for Robust Multi - Contact Loco - Manipulation , 2024

    2024

  49. [49]

    Perceptive Pedipulation with Local Obstacle Avoidance , 2024

    Jonas Stolle, Philip Arm, Mayank Mittal, and Marco Hutter. Perceptive Pedipulation with Local Obstacle Avoidance , 2024

    2024

  50. [50]

    Gemini robotics: Bringing ai into the physical world

    Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:2503.20020, 2025

    Pith/arXiv arXiv 2025

  51. [51]

    Octo: An open-source generalist robot policy

    Octo Model Team, Dibya Ghosh, Homer Walke, Karl Pertsch, Kevin Black, Oier Mees, Sudeep Dasari, Joey Hejna, Tobias Kreiman, Charles Xu, et al. Octo: An open-source generalist robot policy. arXiv preprint arXiv:2405.12213, 2024

    Pith/arXiv arXiv 2024

  52. [52]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  53. [53]

    Advanced skills through multiple adversarial motion priors in reinforcement learning

    Eric Vollenweider, Marko Bjelonic, Victor Klemm, Nikita Rudin, Joonho Lee, and Marco Hutter. Advanced skills through multiple adversarial motion priors in reinforcement learning. arXiv preprint arXiv:2203.14912, 2022

    Pith/arXiv arXiv 2022

  54. [54]

    Pretraining in Deep Reinforcement Learning : A Survey , 2022

    Zhihui Xie, Zichuan Lin, Junyou Li, Shuai Li, and Deheng Ye. Pretraining in Deep Reinforcement Learning : A Survey , 2022

    2022

  55. [55]

    Neural robot dynamics

    Jie Xu, Eric Heiden, Iretiayo Akinola, Dieter Fox, Miles Macklin, and Yashraj Narang. Neural robot dynamics. arXiv preprint arXiv:2508.15755, 2025

    Pith/arXiv arXiv 2025

  56. [56]

    Neural volumetric memory for visual locomotion control, 2023

    Ruihan Yang, Ge Yang, and Xiaolong Wang. Neural volumetric memory for visual locomotion control, 2023. URL https://arxiv.org/abs/2304.01201

    Pith/arXiv arXiv 2023

  57. [57]

    Distillation-ppo: A novel two-stage reinforcement learning framework for humanoid robot perceptive locomotion, 2025

    Qiang Zhang, Gang Han, Jingkai Sun, Wen Zhao, Chenghao Sun, Jiahang Cao, Jiaxu Wang, Yijie Guo, and Renjing Xu. Distillation-ppo: A novel two-stage reinforcement learning framework for humanoid robot perceptive locomotion, 2025. URL https://arxiv.org/abs/2503.08299

    Pith/arXiv arXiv 2025

  58. [58]

    Intention- Conditioned Flow Occupancy Models , 2025

    Chongyi Zheng, Seohong Park, Sergey Levine, and Benjamin Eysenbach. Intention- Conditioned Flow Occupancy Models , 2025

    2025

  59. [59]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  60. [60]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should not add it explicitly Type <Return> for now, but then later remove the command n...

  61. [61]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@first@sw \@firstoftwo \@ifundefined NAT@b*@#2 \@firstoftwo @num @NAT@ctr \@secondoft...

  62. [62]

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibsetup #1 @NAT@ctr @ @openbib .11em \@plus.33em \@minus.07em 4000 4000 `\.\@m @bibit...