Pith. sign in

REVIEW 3 major objections 6 minor 5 cited by

The paper claims that standard off-policy reinforcement learning can be finetuned on real robots without major algorithmic changes, provided three design choices—retaining simulation or prior data with annealed mixing, warm-starting the rep

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:32 UTC pith:VFE7JYTC

load-bearing objection Solid empirical study of sim-to-online RL on three real robots, but the headline claim that asymmetric updates are critical is undercut by a two-factor confound. the 3 major comments →

arxiv 2602.20220 v2 pith:VFE7JYTC submitted 2026-02-23 cs.RO cs.AI

What Matters for Simulation to Online Reinforcement Learning on Real Robots

Yarden As , Dhruva Tirumala , Ren\'e Zurbr\"ugg , Chenhao Li , Stelian Coros , Andreas Krause , Markus Wulfmeier This is my paper
classification cs.RO cs.AI
keywords sim-to-real transferonline reinforcement learningreal robot learningoff-policy RLsoft actor-criticreplay buffer mixingasymmetric actor-critic updatesdomain randomization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that sim-to-online reinforcement learning on physical robots does not need bespoke algorithms. Across more than 100 real-world training runs on three very different robot types—a manipulation arm, a quadruped, and a race car—the authors show that a standard off-policy method (soft actor-critic, SAC) stays stable when three practices are combined: keep the simulation or prior-experiment data in a separate buffer and anneal the mixing weight from 0.5 to 1; warm-start the online buffer with the pretrained policy if prior data cannot be kept; and update the actor only every 20 critic updates at a reduced learning rate. The payoff, if the claim is right, is that deploying online RL on real hardware takes less engineering effort, and the same recipe works for vision-based, sparse-reward tasks and high-frequency control.

Core claim

The central claim is that the instability seen when finetuning a simulation-trained policy on a real robot is a predictable 'downward spiral,' and that three within-standard-practice choices stop it without modifying the algorithm. The authors document that vanilla SAC overestimates the action-value function in state-action pairs visited after deployment, and that this bias compounds across actor-critic updates. Their recipe combines (1) sampling minibatches as a mixture of the prior buffer and the online buffer, with the mixture weight annealed from 0.5 to 1.0 so the prior regularizes early updates and is dropped later; (2) warming the online buffer with transitions from the pretrained poli

What carries the argument

The load-bearing mechanism is the two-buffer replay mixture, (1-alpha)Unif(D0) + alpha Unif(Donline), combined with an asymmetric actor-critic update schedule (actor learning rate 1e-5, one actor update per 20 critic updates). The paper's theoretical anchor is the approximate policy improvement bound: cumulative improvement is guaranteed only while the Q-value error term stays smaller than the greedy improvement term; retaining low-error prior data and slowing the actor keep that error term in check. The same asymmetry is grounded in two-timescale stochastic approximation, where the critic is allowed to track the value function before the policy changes.

Load-bearing premise

The recipe's exact settings—actor learning rate 1e-5 and one actor update per 20 critic updates—were selected from simulated mismatch sweeps and assumed to transfer to the real robots; if the sim-to-real gap changes which settings are best, the claim that asymmetric updates are critical would be an artifact of tuning.

What would settle it

Run the same three task deployments with data retention and warm starts fixed, but sweep the actor update delay M in {1,5,10,20} and actor learning rate in {1e-5,3e-4} on hardware; if a configuration close to vanilla off-policy RL matches or beats the chosen recipe across all robots, the central claim collapses.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Practitioners can finetune simulation-trained policies on real robots with an unmodified off-policy algorithm by adopting three settings: annealed data mixing, warm-started replay, and delayed low-rate actor updates.
  • Retaining the simulation replay buffer (or data from a few prior real trials) stabilizes online finetuning because critic updates stay concentrated in state-action regions where the critic is already accurate; annealing to alpha=1 ensures the final policy is optimized purely on real-world data.
  • When offline data cannot be retained, warming the online buffer with transitions from the pretrained policy is a fallback that preserves most of the stability benefit, though it is weaker than full data retention.
  • Massively parallel simulation training of off-policy RL works if the update-to-data ratio is raised in proportion to the number of parallel environments; a sufficiently large number of domain-randomized environments (around a thousand) is needed for robust sim-to-real transfer.
  • The recipe transfers across manipulation, locomotion, and navigation, including vision-based sparse-reward manipulation, suggesting the choices are task-agnostic rather than platform-specific.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the mixture-annealing logic is the active ingredient, the same recipe should apply to any off-policy algorithm that uses replay, not only soft actor-critic—the TD3 experiments in the paper already point this way.
  • The fixed five-episode annealing schedule for alpha is a free hyperparameter; a reasonable extension is to tie alpha to measured critic error, annealing only when the online critic becomes trustworthy, which could make the recipe robust to larger sim-to-real gaps.
  • The delayed actor update (M=20) and low actor learning rate likely trade off against each other: a lower learning rate might allow more frequent updates, and higher rates might require longer delays. Mapping this trade-off per task is a cheap, testable next step.
  • The paper's manual-reset episodic setting limits immediate use in fully autonomous deployments; combining this recipe with automatic reset mechanisms is the natural next extension rather than a change to the learning algorithm itself.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies the sim-to-online RL setting: policies pretrained in simulation are fine-tuned online on physical robots. On three platforms — a Franka Panda manipulator, a Unitree Go1 quadruped, and a remote-controlled race car — the authors run a large number of real-world training episodes and ablate design choices including retaining simulation/previous-experiment data via annealed replay mixing (α from 0.5 to 1), warm-starting the online replay buffer, using asymmetric actor-critic updates (actor LR 1e-5, actor updated once every 20 critic updates), and scaling the update-to-data ratio with the number of parallel simulation environments. The paper reports that Soft Actor-Critic with these ingredients fine-tunes successfully and stably, while baselines without them (no data retention, symmetric updates with a shared 3e-4 learning rate) often unlearn the prior policy. The authors also release an open-source training pipeline for the Panda task.

Significance. If the recipe is viewed as a package, this is a valuable and timely contribution: it provides one of the largest real-robot comparisons of sim-to-online RL design choices, covers three very different hardware platforms, reports standard errors, and ships an open-source pipeline for the Franka Panda. The work also extends prior replay-mixing ideas to real robots with annealed mixing, which is a meaningful step. However, the paper's headline attribution — that 'asymmetric updates are critical' — is not established by the experiments as presented, because the key ablation varies the update delay and the actor learning rate simultaneously. The broader conclusion that standard off-policy algorithms can be fine-tuned on real robots with modest changes is credible, but the specific mechanistic claim needs additional isolation experiments or a careful reframing.

major comments (3)
  1. [Section 4 / Figure 4 / Section 5.2 / Figure 10] The claim that 'asymmetric updates are critical' (Section 5.2, Figure 10) is confounded. In Figure 4, the ablations change both the update delay M and the actor learning rate jointly (caption: 'as we increase M and reduce learning rate from 3e-4 to 1e-5'). In Figure 10, the baseline differs from the proposed method in two dimensions: M=1 instead of M=20 and a shared actor/critic LR of 3e-4 instead of an actor LR of 1e-5. The observed instability of the baseline could therefore be due entirely to the 30x smaller actor learning rate, and there is no experiment in either the sim-to-sim sweep or the real-robot study that fixes LR while varying M, or fixes M while varying LR. This is load-bearing because the introduction and conclusion highlight asymmetric updates as a key 'what matters' finding. Please add independent sweeps (e.g., M in {1,5,20} with LR fixed at 1e-5, and LR in {3e-4,1e-5} w
  2. [Section 5.2 / Figures 8-10 / Abstract] The paper's broad quantitative claims — 'significant performance gains' (Section 5.2), 'critical for effective transfer across all robots' (Section 5.2), and 'large-sample empirical study' (Abstract) — rest on three seeds per condition and no statistical tests or raw data release. The plotted standard errors in Figures 8-10 are sizable, so several differences appear qualitative rather than statistically robust. This is not a demand for a full hypothesis-test battery in a real-robot study, but the wording should match the evidence: either add effect sizes, paired comparisons, or confidence intervals where feasible, or hedge the categorical language as observations on three platforms with limited replication.
  3. [Section 4 / Section F / Figure 10] The specific recommended values M=20 and actor LR=1e-5 are selected from simulated mismatch sweeps (Figure 4) and then tested on real robots only at that single operating point against one symmetric baseline. No real-robot sensitivity analysis is provided around M or LR, so it remains possible that the ranking of these settings changes under the sim-to-real gap, and that a different combination (e.g., M=5 or LR=5e-6) would be equally or more stable. The paper should either report a small sensitivity analysis on real hardware or in a more faithful sim-to-sim mismatch setup, or explicitly describe these hyperparameters as task-specific choices rather than universal 'what matters' findings.
minor comments (6)
  1. [Introduction, contribution bullet] 'simply delaying critic updates (akin to Fujimoto et al. [10])' is inaccurate: TD3 delays policy/actor updates, not critic updates. Section 4 uses the correct terminology; please fix the bullet to avoid confusion.
  2. [Section 5.2, 'Recycling data' paragraph] 'each experiment is composed of four trials that only share the same random seed' is confusing. It appears you mean each seed is run across four trials with shared initial randomness; please rephrase for clarity.
  3. [Figure 4 caption] The caption says 'showing significant stability improvements' but no statistical significance test is reported; consider replacing 'significant' with 'clear' or 'consistent' to avoid implying formal tests.
  4. [Figure 8 and text] Figure 8 labels trials P/Q/R/S while the text refers to trials 0/1/2/3; align the labels.
  5. [Appendix D, Figure 17 caption] Typo: 'need seen' should be 'never seen'.
  6. [Data availability] The code release is appreciated, but the paper does not release raw reward trajectories or hyperparameter logs. Given the 'large-sample' claim and the small per-condition seed count, releasing the raw data would substantially strengthen reproducibility and allow readers to evaluate the statistical strength of the claims.

Circularity Check

0 steps flagged

No significant circularity: the real-robot experiments externally validate the recipe, and the self-citation to prior replay-mixing work is not load-bearing.

full rationale

The paper's central claims are empirical and are validated on real hardware against baselines, so they are not constructed from fitted parameters or by definition. The three stabilization techniques (data retention, warm starts, asymmetric updates) are tested on a Franka Panda, Unitree Go1, and race car, with learning curves reported against explicit baselines (Figures 8, 9, 10). The sim-to-sim sweep in Figure 4 is used to select hyperparameters, but the subsequent real-world experiments are external to that sweep; this is model selection/tuning, not a fitted input being renamed as a prediction. The self-citation to Tirumala et al. [20] for replay mixing is a legitimate prior result, but the paper's own Figure 8 and Appendix B independently demonstrate data retention on real robots, so the self-citation is not load-bearing. There is no imported uniqueness theorem, no ansatz smuggled in via self-citation, and no renaming of a known result as a new unification. The skeptical concern that Figure 10 varies both update delay M and actor learning rate simultaneously is a confounding/identifiability issue about which component drives the improvement, not a circularity: the claimed effect is still empirically tested rather than entailed by the inputs. Therefore the circularity score is 0.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

No new entities are introduced. The free parameters are standard RL hyperparameters tuned by the authors rather than derived; the empirical contribution is the recipe, not a theory predicting these values from first principles.

free parameters (6)
  • Actor update delay M = 20
    Chosen from the sim-to-sim sweep in Figure 4; real-world experiments use M=20 (Section F). No derivation.
  • Actor learning rate = 1e-5
    Reduced from 3e-4 to 1e-5 based on stability gains in Figure 4; reported in Section F.
  • Update-to-data ratio eta = 5 (Panda/Race Car), ~1 (Go1)
    Selected as a "conservative UTD" in Section F; not derived and differs by robot.
  • Initial mixing alpha0 and annealing schedule = alpha0=0.5 annealed to 1 over 5 episodes
    Hand-selected in Section 5.2; Section B ablates alpha0 in {0.1, 0.9} but does not derive an optimal schedule.
  • Warm-start buffer size N* = 5000 transitions (Panda/Go1), 1250 (Race Car)
    Set based on prior practice and convenience; not derived from first principles.
  • Number of parallel environments Ne = 512 (Panda), 8192 (Go1)
    Pragmatic choices; Figure 11 shows Ne=128 underperforms Ne=8192 for Go1, but exact values are not derived.
axioms (5)
  • standard math Kakade-Langford performance-difference bound (Eq. 5) is valid for the non-stationary online setting.
    Used to explain the downward spiral; assumes the policy is fixed between updates.
  • domain assumption SAC with BRO/DrQ is a suitable base algorithm for all three tasks.
    Claims about "standard off-policy algorithms" are demonstrated only for this family and one TD3 appendix.
  • domain assumption MuJoCo Playground and domain randomization produce a useful prior with a surmountable sim-to-real gap.
    Section 5.1; if the gap is too large, none of the proposed fixes may work.
  • domain assumption Three platforms (Panda, Go1, race car) are representative of a broader class of robot learning tasks.
    The central generalization "across tasks and hardware" relies on this sampling assumption.
  • domain assumption Episodic human-reset setting and reward functions used on real robots are faithful to the simulation rewards.
    Reward mismatch or unmeasured reset variance would confound the reported learning curves.

pith-pipeline@v1.3.0-alltime-deepseek · 16134 in / 10659 out tokens · 93953 ms · 2026-08-02T21:32:16.757926+00:00 · methodology

0 comments
read the original abstract

We investigate what specific design choices enable successful online reinforcement learning (RL) on physical robots. Across 100 real-world training runs on three distinct robotic platforms, we systematically ablate algorithmic, systems, and experimental decisions that are typically left implicit in prior work. We find that some widely used defaults can be harmful, while a set of robust, readily adopted design choices within standard RL practice yield stable learning across tasks and hardware. These results provide the first large-sample empirical study of such design choices, enabling practitioners to deploy online RL with lower engineering effort.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priors

    cs.LG 2026-06 unverdicted novelty 6.0

    Argues for shifting to diagnosis-driven tension management of offline priors in online RL, supported by a framework on prior roles, experiments showing help-or-hurt reversals, and cross-domain evidence.

  2. When Does Non-Uniform Replay Matter in Reinforcement Learning?

    cs.LG 2026-05 unverdicted novelty 6.0

    Non-uniform replay helps off-policy RL mainly at low replay volumes, high-entropy sampling matters even at similar recency, and Truncated Geometric replay offers a low-overhead practical solution.

  3. Bridging the Gap: Enabling Soft Actor Critic for High Performance Legged Locomotion

    cs.RO 2026-05 unverdicted novelty 5.0

    Targeted changes to policy initialization, critic targets, and return estimation let SAC match PPO performance across legged locomotion tasks in massively parallel simulation.

  4. When Does Non-Uniform Replay Matter in Reinforcement Learning?

    cs.LG 2026-05 unverdicted novelty 5.0

    Non-uniform replay improves RL sample efficiency mainly in low replay-volume regimes, with high-entropy sampling being key even at comparable recency.

  5. When Does Non-Uniform Replay Matter in Reinforcement Learning?

    cs.LG 2026-05 unverdicted novelty 5.0

    Non-uniform replay helps most when replay volume is low; high-entropy sampling remains important, and a truncated geometric distribution delivers better sample efficiency with negligible overhead.

Reference graph

Works this paper leans on

65 extracted references · 18 linked inside Pith · cited by 3 Pith papers

  1. [1]

    Sutton and Andrew G

    Richard S. Sutton and Andrew G. Barto.Reinforcement Learning: An Introduction. 2015. (Cited on page 1)

    2015

  2. [2]

    Chapman & Hall/CRC Artificial Intelligence and Robotics Series

    Marcus Hutter, David Quarel, and Elliot Catt.An Introduction to Universal Artificial Intelligence. Chapman & Hall/CRC Artificial Intelligence and Robotics Series. 2024. (Cited on pages 1 and 11)

    2024

  3. [3]

    Learning agile and dynamic motor skills for legged robots.Science Robotics, 2019.(Cited on pages 1, 4, and 16)

    Jemin Hwangbo, Joonho Lee, Alexey Dosovitskiy, Dario Bellicoso, Vassilios Tsounis, Vladlen Koltun, and Marco Hutter. Learning agile and dynamic motor skills for legged robots.Science Robotics, 2019.(Cited on pages 1, 4, and 16)

    2019

  4. [4]

    Industreal: Transferring contact-rich assembly tasks from simulation to reality.arXiv preprint arXiv:2305.17110, 2023.(Cited on page 1)

    Bingjie Tang, Michael A Lin, Iretiayo Akinola, Ankur Handa, Gaurav S Sukhatme, Fabio Ramos, Dieter Fox, and Yashraj Narang. Industreal: Transferring contact-rich assembly tasks from simulation to reality.arXiv preprint arXiv:2305.17110, 2023.(Cited on page 1)

    Pith/arXiv arXiv 2023

  5. [5]

    Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 2025.(Cited on page 1)

    Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion.The International Journal of Robotics Research, 2025.(Cited on page 1)

    2025

  6. [6]

    Beyondmimic: From motion tracking to versatile humanoid control via guided diffusion.arXiv preprint arXiv:2508.08241, 2025.(Cited on page 1)

    Qiayuan Liao, Takara E Truong, Xiaoyu Huang, Guy Tevet, Koushil Sreenath, and C Karen Liu. Beyondmimic: From motion tracking to versatile humanoid control via guided diffusion.arXiv preprint arXiv:2508.08241, 2025.(Cited on page 1)

    Pith/arXiv arXiv 2025

  7. [7]

    Khurram Javed and Richard S. Sutton. The big world hypothesis and its ramifications for artificial intelligence. InFinding the Frame: An RLC Workshop for Examining Conceptual Frameworks, 2024.(Cited on page 1)

    2024

  8. [8]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, 2018.(Cited on pages 1, 2, 4, 5, and 6)

    2018

  9. [9]

    Mujoco playground.arXiv preprint arXiv:2502.08844, 2025.(Cited on pages 1, 4, 7, 8, 16, 18, and 19)

    Kevin Zakka, Baruch Tabanpour, Qiayuan Liao, Mustafa Haiderbhai, Samuel Holt, Jing Yuan Luo, Arthur Allshire, Erik Frey, Koushil Sreenath, Lueder A Kahrs, et al. Mujoco playground.arXiv preprint arXiv:2502.08844, 2025.(Cited on pages 1, 4, 7, 8, 16, 18, and 19)

    Pith/arXiv arXiv 2025

  10. [10]

    Addressing function approximation error in actor-critic methods

    Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. InInternational conference on machine learning, 2018.(Cited on pages 2, 4, 5, 6, 7, and 17)

    2018

  11. [11]

    Reinforcement learning in robotics: A survey.The International Journal of Robotics Research, 2013.(Cited on page 2)

    Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey.The International Journal of Robotics Research, 2013.(Cited on page 2)

    2013

  12. [12]

    Learning to walk in the real world with minimal human effort.arXiv preprint arXiv:2002.08550, 2020.(Cited on page 2)

    Sehoon Ha, Peng Xu, Zhenyu Tan, Sergey Levine, and Jie Tan. Learning to walk in the real world with minimal human effort.arXiv preprint arXiv:2002.08550, 2020.(Cited on page 2)

    Pith/arXiv arXiv 2002

  13. [13]

    Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar, and Sergey Levine. Cog: Connecting new skills to past experience with offline reinforcement learning.arXiv preprint arXiv:2010.14500, 2020.(Cited on page 2) 11 What Matters for Sim-to-Online Reinforcement Learning on Real Robots

    Pith/arXiv arXiv 2010

  14. [14]

    Awac: Accelerating online reinforcement learning with offline datasets.arXiv preprint arXiv:2006.09359, 2020.(Cited on pages 2 and 5)

    Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets.arXiv preprint arXiv:2006.09359, 2020.(Cited on pages 2 and 5)

    Pith/arXiv arXiv 2006

  15. [15]

    Daydreamer: World models for physical robot learning

    Philipp Wu, Alejandro Escontrela, Danijar Hafner, Pieter Abbeel, and Ken Goldberg. Daydreamer: World models for physical robot learning. InConference on robot learning, 2023.(Cited on page 2)

    2023

  16. [16]

    Finetuning offline world models in the real world

    Yunhai Feng, Nicklas Hansen, Ziyan Xiong, Chandramouli Rajagopalan, and Xiaolong Wang. Finetuning offline world models in the real world. InProceedings of the 7th Conference on Robot Learning, 2023.(Cited on page 2)

    2023

  17. [17]

    Efficient online reinforcement learning fine-tuning need not retain offline data

    Zhiyuan Zhou, Andy Peng, Qiyang Li, Sergey Levine, and Aviral Kumar. Efficient online reinforcement learning fine-tuning need not retain offline data. InThe Thirteenth International Conference on Learning Representations, 2025.(Cited on pages 2, 5, 6, 9, and 17)

    2025

  18. [18]

    Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning.Science Robotics, 2025

    Jianlan Luo, Charles Xu, Jeffrey Wu, and Sergey Levine. Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning.Science Robotics, 2025. (Cited on page 2)

    2025

  19. [19]

    How to train your robot with deep reinforcement learning: lessons we have learned

    Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and Sergey Levine. How to train your robot with deep reinforcement learning: lessons we have learned. The International Journal of Robotics Research, 2021.(Cited on page 2)

    2021

  20. [20]

    Replay across experiments: A natural extension of off-policy RL

    Dhruva Tirumala, Thomas Lampe, Jose Enrique Chen, Tuomas Haarnoja, Sandy Huang, Guy Lever, Ben Moran, Tim Hertweck, Leonard Hasenclever, Martin Riedmiller, Nicolas Heess, and Markus Wulfmeier. Replay across experiments: A natural extension of off-policy RL. InThe Twelfth International Conference on Learning Representations, 2024. (Cited on pages 2, 4, and 6)

    2024

  21. [21]

    Rapidly adapting policies to the real-world via simulation- guided fine-tuning

    Patrick Yin, Tyler Westenbroek, Simran Bagaria, Kevin Huang, Ching-An Cheng, Andrey Kolobov, and Abhishek Gupta. Rapidly adapting policies to the real-world via simulation- guided fine-tuning. InInternational Conference on Learning Representations, 2025.(Cited on page 3)

    2025

  22. [22]

    Legged robots that keep on learning: Fine-tuning locomotion policies in the real world

    Laura Smith, J Chase Kew, Xue Bin Peng, Sehoon Ha, Jie Tan, and Sergey Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. In2022 international conference on robotics and automation, 2022.(Cited on page 3)

    2022

  23. [23]

    A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning.arXiv preprint arXiv:2208.07860, 2022.(Cited on pages 3 and 21)

    Laura Smith, Ilya Kostrikov, and Sergey Levine. A walk in the park: Learning to walk in 20 minutes with model-free reinforcement learning.arXiv preprint arXiv:2208.07860, 2022.(Cited on pages 3 and 21)

    Pith/arXiv arXiv 2022

  24. [24]

    John Wiley & Sons, 2014.(Cited on page 3)

    Martin L Puterman.Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.(Cited on page 3)

    2014

  25. [25]

    Bertsekas.Dynamic Programming and Optimal Control

    Dimitri P. Bertsekas.Dynamic Programming and Optimal Control. Athena Scientific, 2000.(Cited on page 3)

    2000

  26. [26]

    Self-improving reactive agents based on reinforcement learning, planning and teaching.Machine Learning, 1992.(Cited on page 3)

    Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching.Machine Learning, 1992.(Cited on page 3)

    1992

  27. [27]

    Playing atari with deep reinforcement learning

    Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.(Cited on pages 3 and 5)

    Pith/arXiv arXiv 2013

  28. [28]

    Leave no trace: Learning to reset for safe and autonomous reinforcement learning

    Benjamin Eysenbach, Shixiang Gu, Julian Ibarz, and Sergey Levine. Leave no trace: Learning to reset for safe and autonomous reinforcement learning. InInternational Conference on Learning Representations, 2018.(Cited on page 3) 12 What Matters for Sim-to-Online Reinforcement Learning on Real Robots

    2018

  29. [29]

    Autonomous reinforcement learning via subgoal curricula

    Archit Sharma, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Autonomous reinforcement learning via subgoal curricula. InAdvances in Neural Infor- mation Processing Systems, 2021.(Cited on page 3)

    2021

  30. [30]

    Autonomous reinforcement learning: Formalism and bench- marking

    Archit Sharma, Kelvin Xu, Nikhil Sardana, Abhishek Gupta, Karol Hausman, Sergey Levine, and Chelsea Finn. Autonomous reinforcement learning: Formalism and bench- marking. InInternational Conference on Learning Representations, 2022.(Cited on page 3)

    2022

  31. [31]

    A state-distribution matching ap- proach to non-episodic reinforcement learning

    Archit Sharma, Rehaan Ahmad, and Chelsea Finn. A state-distribution matching ap- proach to non-episodic reinforcement learning. InInternational Conference on Machine Learning, 2022.(Cited on page 3)

    2022

  32. [32]

    Conservative q-learning for offline reinforcement learning, 2020.(Cited on page 3)

    Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning, 2020.(Cited on page 3)

    2020

  33. [33]

    Mopo: Model-based offline policy optimization.Interna- tional Conference on Neural Information Processing Systems, 2020.(Cited on page 3)

    Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization.Interna- tional Conference on Neural Information Processing Systems, 2020.(Cited on page 3)

    2020

  34. [34]

    Offline robotic world model: Learning robotic policies without a physics simulator.arXiv preprint arXiv:2504.16680, 2025

    Chenhao Li, Andreas Krause, and Marco Hutter. Offline robotic world model: Learning robotic policies without a physics simulator.arXiv preprint arXiv:2504.16680, 2025. (Cited on page 3)

    arXiv 2025

  35. [35]

    Isaac lab: A gpu-accelerated simulation framework for multi-modal robot learning.arXiv preprint arXiv:2511.04831, 2025.(Cited on page 4)

    Mayank Mittal, Pascal Roth, James Tigue, Antoine Richard, Octi Zhang, Peter Du, Antonio Serrano-Munoz, Xinjie Yao, René Zurbrügg, Nikita Rudin, et al. Isaac lab: A gpu-accelerated simulation framework for multi-modal robot learning.arXiv preprint arXiv:2511.04831, 2025.(Cited on page 4)

    Pith/arXiv arXiv 2025

  36. [36]

    Domain randomization for transferring deep neural networks from simulation to the real world, 2017.(Cited on page 4)

    Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world, 2017.(Cited on page 4)

    2017

  37. [37]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017.(Cited on page 4)

    JohnSchulman,FilipWolski,PrafullaDhariwal,AlecRadford,andOlegKlimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017.(Cited on page 4)

    Pith/arXiv arXiv 2017

  38. [38]

    Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971, 2015.(Cited on pages 4 and 5)

    Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning.arXiv preprint arXiv:1509.02971, 2015.(Cited on pages 4 and 5)

    Pith/arXiv arXiv 2015

  39. [39]

    Maximum a posteriori policy optimisation.arXiv preprint arXiv:1806.06920, 2018.(Cited on page 4)

    Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum a posteriori policy optimisation.arXiv preprint arXiv:1806.06920, 2018.(Cited on page 4)

    Pith/arXiv arXiv 2018

  40. [40]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019.(Cited on page 4)

    Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning.arXiv preprint arXiv:1910.00177, 2019.(Cited on page 4)

    Pith/arXiv arXiv 1910

  41. [41]

    Getting sac to work on a massive parallel simulator: An rl journey with off-policy algorithms.araffin.github.io, Feb 2025

    Antonin Raffin. Getting sac to work on a massive parallel simulator: An rl journey with off-policy algorithms.araffin.github.io, Feb 2025. URLhttps://araffin.github. io/post/sac-massive-sim/.(Cited on pages 4 and 16)

    2025

  42. [42]

    Deterministic policy gradient algorithms

    David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and Martin Riedmiller. Deterministic policy gradient algorithms. InInternational conference on machine learning, 2014.(Cited on page 4)

    2014

  43. [43]

    Equivalence between policy gradients and soft q-learning.arXiv preprint arXiv:1704.06440, 2017.(Cited on page 4)

    John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning.arXiv preprint arXiv:1704.06440, 2017.(Cited on page 4)

    Pith/arXiv arXiv 2017

  44. [44]

    Approximately optimal approximate reinforcement learning

    Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. InProceedings of the nineteenth international conference on machine learning, 2002.(Cited on page 4) 13 What Matters for Sim-to-Online Reinforcement Learning on Real Robots

    2002

  45. [45]

    Efficient online reinforce- ment learning with offline data

    Philip J Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforce- ment learning with offline data. InInternational Conference on Machine Learning, 2023. (Cited on pages 5 and 6)

    2023

  46. [46]

    Hybrid RL: Using both offline and online data can make RL efficient

    Yuda Song, Yifei Zhou, Ayush Sekhari, Drew Bagnell, Akshay Krishnamurthy, and Wen Sun. Hybrid RL: Using both offline and online data can make RL efficient. InThe Eleventh International Conference on Learning Representations, 2023.(Cited on page 5)

    2023

  47. [47]

    Steering your generalists: Improving robotic foundation models via value guidance

    Mitsuhiko Nakamoto, Oier Mees, Aviral Kumar, and Sergey Levine. Steering your generalists: Improving robotic foundation models via value guidance. In8th Annual Conference on Robot Learning, 2024.(Cited on page 5)

    2024

  48. [48]

    Offline reinforce- ment learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020.(Cited on page 5)

    Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforce- ment learning: Tutorial, review, and perspectives on open problems.arXiv preprint arXiv:2005.01643, 2020.(Cited on page 5)

    Pith/arXiv arXiv 2005

  49. [49]

    When to trust your model: Model-based policy optimization.Advances in Neural Information Processing Systems, 2019.(Cited on page 6)

    Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization.Advances in Neural Information Processing Systems, 2019.(Cited on page 6)

    2019

  50. [50]

    Randomized ensembled double q-learning: Learning fast without a model.arXiv preprint arXiv:2101.05982, 2021.(Cited on page 6)

    Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model.arXiv preprint arXiv:2101.05982, 2021.(Cited on page 6)

    Pith/arXiv arXiv 2021

  51. [51]

    Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning

    Michał Nauman, Michał Bortkiewicz, Piotr Miłoś, Tomasz Trzciński, Mateusz Os- taszewski, and Marek Cygan. Overestimation, overfitting, and plasticity in actor-critic: the bitter lesson of reinforcement learning. In41st International Conference on Machine Learning, 2024.(Cited on page 6)

    2024

  52. [52]

    On actor-critic algorithms.SIAM journal on Control and Optimization, 2003.(Cited on page 6)

    Vijay R Konda and John N Tsitsiklis. On actor-critic algorithms.SIAM journal on Control and Optimization, 2003.(Cited on page 6)

    2003

  53. [53]

    Convergence rate of linear two-time-scale stochastic approximation

    Vijay R Konda and John N Tsitsiklis. Convergence rate of linear two-time-scale stochastic approximation. 2004.(Cited on page 6)

    2004

  54. [54]

    Springer.(Cited on page 6)

    Vivek S Borkar.Stochastic approximation: a dynamical systems viewpoint. Springer.(Cited on page 6)

  55. [55]

    Amz driverless: The full autonomous racing system.Journal of Field Robotics, 2020

    Juraj Kabzan, Miguel I Valls, Victor JF Reijgwart, Hubertus FC Hendrikx, Claas Ehmke, Manish Prajapat, Andreas Bühler, Nikhil Gosala, Mehak Gupta, Ramya Sivanesan, et al. Amz driverless: The full autonomous racing system.Journal of Field Robotics, 2020. (Cited on pages 7 and 19)

    2020

  56. [56]

    Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control.Advances in neural information processing systems, 2024.(Cited on page 6)

    Michal Nauman, Mateusz Ostaszewski, Krzysztof Jankowski, Piotr Miłoś, and Marek Cy- gan. Bigger, regularized, optimistic: scaling for compute and sample efficient continuous control.Advances in neural information processing systems, 2024.(Cited on page 6)

    2024

  57. [57]

    Mastering visual continuous control: Improved data-augmented reinforcement learning.arXiv preprint arXiv:2107.09645, 2021.(Cited on page 6)

    Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning.arXiv preprint arXiv:2107.09645, 2021.(Cited on page 6)

    Pith/arXiv arXiv 2021

  58. [58]

    Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem

    C. Daniel Freeman, Erik Frey, Anton Raichuk, Sertan Girgin, Igor Mordatch, and Olivier Bachem. Brax - a differentiable physics engine for large scale rigid body simulation,

  59. [59]

    Shengyi Huang, Rousslan Fernand Julien Dossa, Chang Ye, Jeff Braga, Dipam Chakraborty, Kinal Mehta, and João G.M. Araújo. Cleanrl: High-quality single-file implementations of deep reinforcement learning algorithms.Journal of Machine Learn- ing Research, 2022.(Cited on page 6) 14 What Matters for Sim-to-Online Reinforcement Learning on Real Robots

    2022

  60. [60]

    Parallel𝑞- learning: Scaling off-policy reinforcement learning under massively parallel simulation

    Zechu Li, Tao Chen, Zhang-Wei Hong, Anurag Ajay, and Pulkit Agrawal. Parallel𝑞- learning: Scaling off-policy reinforcement learning under massively parallel simulation. InInternational Conference on Machine Learning, 2023.(Cited on page 16)

    2023

  61. [61]

    Younggyo Seo, Carmelo Sferrazza, Haoran Geng, Michal Nauman, Zhao-Heng Yin, and Pieter Abbeel. Fasttd3: Simple, fast, and capable reinforcement learning for humanoid control.arXiv preprint arXiv:2505.22642, 2025.(Cited on page 16) 15 What Matters for Sim-to-Online Reinforcement Learning on Real Robots Appendix A. Off-Policy Training in Massively-Parallel ...

    Pith/arXiv arXiv 2025

  62. [63]

    Optimizer state not restored on resume.Restoring only model weights but not the optimizer state (momentum, second-moment estimates, learning-rate schedulers, etc.) changes the optimizer dynamics and can substantially alter learning

  63. [64]

    Restoring a critic without its target network.Loading𝑄𝜋𝑛 𝜑 but not its target network ¯𝑄𝜋𝑛 produces inconsistent targets, which effectively lead the critic and thereafter the actor to unlearn

  64. [65]

    These issues are easy to overlook but they change learning dynamics and final performance

    SAC temperature𝛼 (and its optimizer) not restored.Since𝛼 changes during pre- training, failing to restore its value and optimizer state changes the scale of entropy bonus in actor and critic updates, which lead to instability. These issues are easy to overlook but they change learning dynamics and final performance. We explicitly call them out to help pra...

  65. [2021]

    URLhttp://github.com/google/brax.(Cited on pages 6 and 7)