Pith. sign in

REVIEW 4 major objections 5 minor 90 references

The paper argues that a low-cost 3D-printed quadruped, built as a physical counterpart to a popular simulated benchmark, lets reinforcement learning agents learn competent walking directly on hardware in roughly an hour.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

The Open Ant is a physical, open-source quadruped modeled on the Gymnasium Ant that learns to walk from its own experience in about an hour with SARSA(λ) and SAC, and supports sim-to-real transfer.

T0 review reviewed 2026-08-01 challenge →

load-bearing objection A useful open-source quadruped that credibly learns to walk on hardware in about an hour; the main caveats are unvalidated camera tracking and loose statistical reporting, but neither sinks the platform contribution. the 4 major comments →

arxiv 2607.18488 v1 pith:JSMDA3SZ submitted 2026-07-20 cs.RO cs.AIcs.SYeess.SY

The Open Ant: A Robot Platform for Reinforcement Learning Research

Elena Sorina Lupu , Patrick Spieler , Khurram Javed , Kris De Asis , John D. Martin , Martha Steenstrup , Joseph Modayil This is my paper
classification cs.RO cs.AIcs.SYeess.SY
keywords reinforcement learningphysical robotquadruped locomotionsim-to-real transfersoft actor-criticSARSAopen-source hardwareembodied intelligence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that real-robot reinforcement learning can be made routine for researchers who normally work in simulation. It introduces a physical quadruped, the Open Ant, plus a matched simulation, and shows that two very different RL algorithms — a simple linear SARSA(λ) and a deep Soft Actor-Critic — can each learn a competent back-and-forth walking policy from scratch in about an hour of on-robot experience. It also argues that policies trained in the simulation transfer to the physical robot, though the ranking of policies changes across the gap. If true, the platform lowers the cost and time barrier that has kept many RL research groups from including physical experiments in their work.

Core claim

The central claim is that competent walking is reachable within approximately one hour of direct physical interaction for two substantially different algorithms: SARSA(λ) with tile coding and hand-designed motion primitives, and SAC with layer normalization and reward scaling. Learning is non-episodic: a back-and-forth task with a reward-direction switching rule keeps the robot inside a small arena without human resets, and the instantaneous reward is the projection of the robot's planar displacement onto the current goal direction, measured by an overhead camera. The paper also reports that policies learned in a physics-based simulation transfer to reality without any fine-tuning, and that

What carries the argument

The load-bearing object is the Open Ant hardware–simulation pair. The physical robot is about one-third the scale of the simulated benchmark, uses eight commercial servo motors (two per leg), is powered from an AC outlet and connected by USB to an external computer, and is tracked by an overhead camera with fiducial markers. The simulation is system-identified to match the hardware by measuring motor damping, stiffness, velocity gain, foot friction, and contact parameters. The non-episodic back-and-forth task with its reward-direction switching rule is what makes hour-long learning runs feasible without manual resets.

Load-bearing premise

The reward signal relies on overhead camera position estimates that the paper does not validate for accuracy, latency, or failure; if those estimates are unreliable, the learning curves and sim-to-real comparisons may not measure true physical walking progress.

What would settle it

Run the same SAC training protocol while simultaneously measuring the robot's position with a calibrated motion-capture system; if the average-reward curves from the camera-based reward differ materially from the motion-capture ground truth, or if the learned policies fail to walk when evaluated under accurate tracking, the central claim about one-hour learning on hardware would be undermined.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • RL researchers can include physical robot experiments in their evaluations without months of robotics setup.
  • Two algorithm families (linear on-policy and deep off-policy) both succeed on hardware, suggesting the platform is algorithm-agnostic.
  • Sim-to-real transfer works for SAC-trained policies, so researchers can pre-train in simulation and then validate on hardware, even though policy rankings change.
  • The open-source, 3D-printed design with commercial components means repairs can be made in minutes to hours, supporting long-duration learning experiments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the one-hour learning claim holds across more algorithms and tasks, hardware-in-the-loop evaluation could become a standard complement to simulation benchmarks rather than a rare final validation.
  • The reported ranking inversion between simulated and physical policies implies that sim-to-real evaluation with a handful of policies may mislead model selection; a testable implication is that the platform could serve as a low-cost tool for measuring the sim-to-real gap before scale-up.
  • The unvalidated camera tracking and manual resets are the main threats; a natural extension is to replace the overhead camera with a motion-capture ground truth or onboard odometry to disentangle reward-signal quality from the learning algorithm's progress.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces the Open Ant, an open-source quadruped robot platform intended to lower the barrier to physical-robot experiments for RL researchers. The robot mirrors the Gymnasium Ant morphology, is powered and connected via cables, and is accompanied by a MuJoCo simulation. The authors define a non-episodic back-and-forth task with a progress reward computed from overhead AprilTag tracking, and claim that both SARSA(λ) and SAC can learn competent walking from scratch on the physical robot in approximately one hour. They also report that SAC policies trained in simulation transfer to the physical robot, and provide qualitative evidence of the platform's usability through multi-site assembly and a winter-school deployment.

Significance. If the main claims hold, this is a genuinely useful contribution: an affordable ($500–$2200), repairable, AC-powered robot with a matching simulation, released open-source, could make real-hardware RL experiments practical for simulation-centric laboratories. The paper's strengths include open hardware/software, five independent hardware trials per algorithm, supplementary ablations with 30 seeds, detailed system identification, failure-mode and thermal analysis, and a candid discussion of reward drops and simulator exploits. The demonstration that two very different RL algorithms can learn locomotion directly from physical experience is encouraging. However, the central quantitative claims rest on an unvalidated camera-tracking subsystem and a manual-intervention protocol whose effects are not quantified; these issues must be addressed before the reliability and repeatability claims can be fully accepted.

major comments (4)
  1. [§3.2, Eq. (1); §14] The reward r_t and the performance metric in Eq. (3) are computed entirely from the planar position p_t obtained by overhead AprilTag tracking. The paper reports no calibration of this tracking: no static or dynamic accuracy, no latency measurement, no failure-handling statistics. Section 14 itself lists 'camera processing latency' as a topic that 'deserves further analysis.' Because r_t = (p_t − p_{t−1})^T u_t, a lagged or noisy p_t directly biases the learning signal and the cm/s curves in Figures 3–5, and can also trigger the direction flip in Eq. (2) at the wrong time. Moreover, the heading vector used in the observation (Section 3.2) is derived from the same tracking. The authors should supply a validation of the tracking subsystem (e.g., known-translation tests, latency/jitter quantification, dropout rates) and discuss how residual error affects the reported reward curves and the s
  2. [§5, Common Methodology] The learning protocol involves manual interventions: when cables become entangled, the experimenter pauses the experiment, disentangles the robot, and resets it to the start configuration. The paper does not report how often this occurred per trial, how long pauses lasted, or how the reset interacts with the reward computation. If p_t changes discontinuously during a pause/resume, Eq. (1) may credit or penalize the manual displacement unless the reset is explicitly excluded from the reward. More broadly, frequent resetting to a fixed start configuration is an episodic element that qualifies the 'non-episodic / learn forever' framing of Section 3.2. The authors should quantify interventions (count, duration, timing) and demonstrate that learning is not shaped by the resets.
  3. [§5, Figs. 3–4] The repeatability claim ('in every trial, the agent achieved slow yet competent walking behaviors... within approximately one hour') is based on five single-trial curves per condition, with no error bars, no summary statistic at the one-hour mark, and no operational definition of 'competent.' A reader cannot tell whether the reported 2–4 cm/s is meaningfully above a no-movement baseline or whether trial-to-trial variability is small. Please report aggregate statistics (e.g., median and quartiles at fixed time points), define the competence threshold a priori, and give per-trial intervention counts. The same applies to Figure 4, where the claim that 'all trials exhibit stable, visually plausible walking' is hard to verify without quantitative summaries.
  4. [§5, Fig. 5 and §11] The sim-to-real transfer experiment evaluates each policy on a single 10-minute hardware run, yet draws conclusions about transfer success and rank changes. With one run per policy, the differences in Figure 5 may be within hardware or tracking noise; repeated evaluations (at least for a subset of policies) are needed. The text is also internally inconsistent: it says 'ten policies' but then labels policies 1 to 11, and Section 11 says 6,912 policies were trained while Section 5 says 2,304. Please reconcile these numbers and report the exact policy-selection procedure. The caveat that the sim-to-real policies are not directly comparable to the hardware-learning setting is appreciated but weakens the general transfer claim.
minor comments (5)
  1. [§7.1, §10.1] Typos: 'our the robot can continue walking' in §7.1; 'Physical Ant Light' in §10.1 should be 'Physical Ant Lite.'
  2. [§5, Fig. 5] The policy numbering is inconsistent: the text says 'ten policies' but refers to 'policies 1 to 11.' Please clarify.
  3. [§11, Table 5] The number of trained policies is given as 2,304 in Section 5 and 6,912 in Section 11. The product of the listed sweep dimensions does not obviously yield 6,912; please provide the exact count and configuration list.
  4. [§8, Table 4] The XC430-W240 hip motor used in the Physical Ant is omitted from the system-identification table. If its parameters were not measured, state this explicitly rather than deferring to 'a similar analysis can be performed.'
  5. [§4.2, §5] The reward scaling factor was changed to 100 after an initial hardware failure at scale 1.0. Please clarify the timeline: was the final setting selected before the reported five hardware trials, and were the failed runs recorded? Reporting them would improve transparency.

Circularity Check

0 steps flagged

No significant circularity: the paper's claims are empirical hardware demonstrations, not derivations from fitted inputs or self-cited theorems.

full rationale

The paper makes no derivation in the sense required for circularity analysis. Its central claims are empirical demonstrations that SARSA(lambda) and SAC can learn walking policies on a physical robot in about one hour, and that some simulated policies transfer to hardware. The reward in Eq. 1 and the performance metric in Eq. 3 do share the same camera-derived position signal, and the paper explicitly notes camera processing latency and timing as open concerns (Section 14). However, this is a measurement-validity issue, not a circular reduction: the reported result is an observed time series of rewards, not a predicted quantity obtained from a fitted parameter that was itself fit to that quantity. The use of the authors' own simulation to tune SAC components is calibration, not circularity; the sim-to-real transfer is an independent, falsifiable test, and the paper reports that rankings changed and that policies can exploit simulator inaccuracies, which is exactly the kind of outcome that would not occur if the transfer claim were true by construction. The one citation to co-authored prior work (De Asis & Sutton 2024) is used to modify the SAC return definition, but the hardware-learning result does not rest on the validity of that modification, and no uniqueness theorem or ansatz is imported via self-citation. No fitted input is renamed as a prediction, and no known result is merely re-labeled. The paper also discloses limitations such as manual resets for cable entanglement, motor overheating, and simulator exploitation, which further supports treating the empirical results as honest demonstrations rather than constructed equivalences. Overall, no step in the paper reduces, by the paper's own equations or by load-bearing self-citation, to its own inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claims rest on hand-tuned hyperparameters (some on simulator, one on hardware), on the unverified fidelity of the camera-based reward, and on the assumed adequacy of the system-identified MuJoCo model. No new theoretical entities are introduced.

free parameters (4)
  • SAC reward scaling factor = 100.0
    Table 3; scale of 1.0 gave poor hardware performance and a larger value was chosen (validated in sim across 30 seeds) to enable learning.
  • SARSA(λ) hyperparameter set = δt=0.5s, λ=0.964, γ=0.998, ε=0.255, α=0.0001, 4 tiles/dim, 192 tilings, table size 225
    Section 12: selected via Optuna on the Simulated Ant across 100 configurations and 10 seeds; these values are then used on hardware.
  • SAC learning-start and action bounds = learning start 2000 steps; knee 20° range at 50° offset; hip 45° range at 0° offset
    Section 4.2/5: modified from CleanRL defaults; learning start reduced empirically; action bounds set by hand to constrain joint motion.
  • Simulator system-ID parameters (Kv, time constant, solref/solimp) = Kv=2.5 (XL430), 1.5 (XM430); 0.12s motor time constant; adjusted solver parameters
    Section 8: 'tuned to approximately match the response on the Physical Ant'; these are fitted numbers that the sim-to-real transfer claim relies on.
axioms (4)
  • standard math SARSA(λ) and SAC behave as described in the cited literature and converge in this setting (Sutton & Barto 2018; Haarnoja et al. 2018)
    Section 4 relies on standard RL algorithms without proofs of convergence in the non-episodic hardware setting.
  • domain assumption The MuJoCo simulation with identified parameters adequately models the Physical Ant's dynamics for policy transfer
    Section 8; partially validated by transfer results but authors note simulator exploits (Movie 3) and XC430 hip motor is not identified.
  • domain assumption The overhead AprilTag tracking yields accurate position p_t at the control rate for the reward in Equation 1
    Section 3.2; no validation of tracking accuracy or latency reported.
  • domain assumption Manual interventions (pausing, untangling, resetting to start) do not provide a learning signal
    Section 5 Common Methodology; interventions are acknowledged but not quantified for SAC, so their effect on learning is assumed minimal.

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of The Open Ant: A Robot Platform for Reinforcement Learning Research." pith.science (2026). https://pith.science/paper/JSMDA3SZ

@misc{pith2026260718488,
  author       = {Pith},
  title        = {Pith review of: The Open Ant: A Robot Platform for Reinforcement Learning Research},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JSMDA3SZ}},
  note         = {Machine review of arXiv:2607.18488}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Reinforcement learning (RL) research has demonstrated success in both physical and simulated domains; however, the predominant methodology remains rooted in simulations. The predominance of simulations makes translating research to physical reality uncertain for both algorithms and researchers. We propose a physical platform that is designed to simplify the transition. In this paper, we present the Open Ant: a physical variant of the commonly used Gymnasium Ant environment, along with a simulation. We demonstrate that competent walking policies can be learned from scratch in approximately one hour directly from the physical robot's experience for two substantially different RL algorithms: SARSA($\lambda$) and Soft Actor-Critic (SAC). Separately, we show policies that were learned in simulation transfer to reality. We also examine how well the platform supports a nimble experimental ecosystem. Specifically, we observe the speed with which new users from diverse backgrounds achieve their first success with the platform, and how easily the platform can be repaired and updated when hardware issues arise. Both the hardware design and software are available as open-source on GitHub for ease of customization. In summary, we advocate for the use of the Open Ant for RL researchers who frequently use simulated environments, so they can more easily include robot experiments in their evaluations.

Figures

Figures reproduced from arXiv: 2607.18488 by Elena Sorina Lupu, John D. Martin, Joseph Modayil, Khurram Javed, Kris De Asis, Martha Steenstrup, Patrick Spieler.

Figure 1
Figure 1. Figure 1: Physical Ant platform, learning arena, and system overview. (a) An overhead webcam tracks the fiducial markers to compute reward signals and the heading vector of the ant. The robot is connected by cables to AC power and to an external computer where the agent is running. (b) The main components of the Physical Ant. (c) The Gymnasium Ant (Schulman et al., 2015; Towers et al., 2025), which was the inspirati… view at source ↗
Figure 2
Figure 2. Figure 2: Overhead snapshots during run-time learning from physical experience. The cyan circle represents the boundary where the reward direction (the blue arrow) changes direction if the Physical Ant exceeds it. In the first figure, the robot travels towards 11 o’clock. In the second figure, it approaches the boundary. In the third figure, the reward direction flips and in the last two figures, we see the ant trav… view at source ↗
Figure 3
Figure 3. Figure 3: Run-time learning performance of SARSA(λ) executed onboard the Physical Ant across 5 trials. Average reward is computed using Equation 3 on a window of 120 seconds. These 5 trials had a total of one interruption (indicated with circles) caused by the leg failure in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Run-time learning performance using SAC on hardware for 5 trials. Average reward is computed using Equation 3 for both the Physical Ant and the Physical Ant Lite on a window of 120 seconds. Circles indicate interventions, which are manual stops due to cable entanglement with the robot’s legs. The experiment was run on a Macbook M1 with 16 GB of memory. The performance for one of the runs can be seen in Mov… view at source ↗
Figure 5
Figure 5. Figure 5: Evaluating policies learned using SAC on the Simulated Ant with the Physical Ant on the back-and-forth task. We chose ten policies of the Simulated Ant with reward rates between 23 cm/s to 13 cm/s and measured their performance on the Physical Ant. We tested how policies learned on the Simu￾lated Ant performed on the Physical Ant with￾out further learning. We learned 2,304 policies in simulation using SAC … view at source ↗
Figure 6
Figure 6. Figure 6: Different failures observed during the development of the Physical Ant (a) Foot￾tip abrasion from repeated impacts, mitigated with 3D-printed TPU socks. (b) Hip cover fracture due to locomotion stress, redesigned with a stronger interface. (c) Leg shell damage after manual overloading; reinforced with internal ribs. (d) Leg housing cracks from accumulated stress, likely related to (e), interface strengthen… view at source ↗
Figure 7
Figure 7. Figure 7: Free-body diagram of a single leg in stance for the torque calculations. [PITH_FULL_IMAGE:figures/full_fig_p021_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Results for the thermal analysis during stress tests experiments. Motor temperature over time during a one-hour stalled-load experiment for the Physical Ant Lite (XL-series motors) and Physical Ant (XM-series motors). The Physical Ant Lite platform reaches the 72◦ C thermal cutoff and shuts down after approximately 30 minutes, while the Physical Ant platform tolerates higher temperatures (80◦C limit). Inte… view at source ↗
Figure 9
Figure 9. Figure 9: Motor equipped with heat sink. 10.2 Electrical Analysis In this section, we present an analysis of the intermittent communication dropouts observed during run-time learning, as well as other electrical considerations for the Physical Ants [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Electrical investigation of intermittent communication dropouts. (a) Oscilloscope measurement showing the Dynamixel communication bus and supply voltage during a failure event. The oscilloscope was externally triggered by an FTDI USB-to-TTL converter when communication errors occurred. We observed that each error coincides with a brief drop in the supply voltage. (b) The power connection was modified to c… view at source ↗
Figure 11
Figure 11. Figure 11: Simulation results for SARSA(λ). Using the parameters optimized by Optuna, this plot shows the mean performance with ±3 standard errors over 30 random seeds [PITH_FULL_IMAGE:figures/full_fig_p025_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Ablation on three of the components of SAC. Simulation results showing the mean per￾formance with ±3 standard errors over 30 random seeds for evaluating the effect of applying Layer Normalization for the policy and the critic networks, changing when Learning Starts parameter, and the Reward Scaling [PITH_FULL_IMAGE:figures/full_fig_p026_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Reward drop. Performance in simulation during learning of the back-and-forth task using SAC (seed 14). Left: Average reward per second and the action-value function evaluated at the current state-action pair throughout learning. After approximately 40 hours of continual learning, the agent experiences a noticeable drop in performance. Right: A zoomed-in view of the period surrounding this performance degr… view at source ↗
Figure 14
Figure 14. Figure 14: Step timing. This diagram highlights the breakdown of time in an agent-environment step [PITH_FULL_IMAGE:figures/full_fig_p028_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Motion primitives used for the SARSA(λ) algorithm. Each color in the plot and on the robot is associated with a leg part and position (hip or knee, left or right). We chose to design 8 motion primitives in total. 16 List of Movies Mentioned in the Submission • Movie 1. Assembly of the Open Ant. • Movie 2. Learning to walk using SAC. • Movie 3. Simulation results of the robot exploiting the simulator for f… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

90 extracted references · 1 canonical work pages

  1. [1]

    2018 , publisher=

    Reinforcement Learning: An Introduction , author=. 2018 , publisher=

    2018

  2. [2]

    Temporal difference learning and

    Tesauro, Gerald , journal=. Temporal difference learning and

  3. [3]

    Nature , volume=

    Magnetic control of tokamak plasmas through deep reinforcement learning , author=. Nature , volume=. 2022 , publisher=

    2022

  4. [4]

    LeCun, Yan , title =

  5. [5]

    , title =

    Sutton, Richard S. , title =

  6. [6]

    and Naddaf, Yavar and Veness, Joel and Bowling, Michael , journal=

    Bellemare, Marc G. and Naddaf, Yavar and Veness, Joel and Bowling, Michael , journal=. The

  7. [7]

    Mastering the game of

    Silver, David and others , journal=. Mastering the game of. 2016 , publisher=

    2016

  8. [8]

    2025 , eprint=

    Gymnasium: A Standard Interface for Reinforcement Learning Environments , author=. 2025 , eprint=

    2025

  9. [9]

    Nature , volume=

    Autonomous navigation of stratospheric balloons using reinforcement learning , author=. Nature , volume=. 2020 , publisher=

    2020

  10. [10]

    and Bellemare, Marc G

    Machado, Marlos C. and Bellemare, Marc G. and Talvitie, Erik and Veness, Joel and Hausknecht, Matthew and Bowling, Michael , journal=. Revisiting the

  11. [11]

    Brockman, Greg and Cheung, Vicki and Pettersson, Ludwig and Schneider, Jonas and Schulman, John and Tang, Jie and Zaremba, Wojciech , journal=

  12. [12]

    and Sutton, Richard S

    Barto, Andrew G. and Sutton, Richard S. and Anderson, Charles W. , journal=. Neuronlike adaptive elements that can solve difficult learning control problems , year=

  13. [13]

    Journal of Neuroscience , volume=

    A framework for mesencephalic dopamine systems based on predictive Hebbian learning , author=. Journal of Neuroscience , volume=. 1996 , publisher=

    1996

  14. [14]

    Science , volume=

    A neural substrate of prediction and reward , author=. Science , volume=. 1997 , publisher=

    1997

  15. [15]

    and Adams, James L

    Houk, James C. and Adams, James L. and Barto, Andrew G. , title =

  16. [16]

    AAAI , pages=

    Machine learning for fast quadrupedal locomotion , author=. AAAI , pages=

  17. [17]

    Sebastian , booktitle=

    Tedrake, Russ and Zhang, Teresa Weirui and Seung, H. Sebastian , booktitle=. Stochastic policy gradient reinforcement learning on a simple 3. 2004 , volume=

    2004

  18. [18]

    Todorov, Emanuel and Erez, Tom and Tassa, Yuval , booktitle=

  19. [19]

    Conference on Robot Learning , pages=

    Daydreamer: World models for physical robot learning , author=. Conference on Robot Learning , pages=

  20. [20]

    Tassa, Yuval and others , journal=. Deep

  21. [21]

    and others , journal=

    Wurman, Peter R. and others , journal=. Outracing champion. 2022 , publisher=

    2022

  22. [22]

    The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2 , pages=

    Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction , author=. The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2 , pages=

  23. [23]

    , booktitle=

    Deisenroth, Marc and Rasmussen, Carl E. , booktitle=

  24. [24]

    Martin, John and Wang, Jinkun and Englot, Brendan , booktitle=. Sparse

  25. [25]

    The International Journal of Robotics Research , volume=

    Reinforcement learning in robotics: A survey , author=. The International Journal of Robotics Research , volume=. 2013 , publisher=

    2013

  26. [26]

    arXiv preprint arXiv:2110.15245 , year=

    From machine learning to robotics: Challenges and opportunities for embodied intelligence , author=. arXiv preprint arXiv:2110.15245 , year=

    Pith/arXiv arXiv

  27. [27]

    International Conference on Machine Learning , pages=

    Settling the reward hypothesis , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  28. [28]

    Brush-Type

    General Dynamics , howpublished =. Brush-Type. 2020 , url =

    2020

  29. [29]

    An application of

    Abbeel, Pieter and Coates, Adam and Quigley, Morgan and Ng, Andrew , journal=. An application of

  30. [30]

    Proceedings 2001 ICRA

    Autonomous helicopter control using reinforcement learning policy search methods , author=. Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No. 01CH37164) , volume=. 2001 , organization=

    2001

  31. [31]

    , journal=

    Eysenbach, Ben and Levine, Sergey and Salakhutdinov, Russ R. , journal=. Replacing rewards with examples:

  32. [32]

    2019 , journal=

    Striving for simplicity in off-policy deep reinforcement learning , author=. 2019 , journal=

    2019

  33. [33]

    International conference on machine learning , pages=

    Off-policy deep reinforcement learning without exploration , author=. International conference on machine learning , pages=. 2019 , organization=

    2019

  34. [34]

    and Silver, David , year=

    Sutton, Richard S. and Silver, David , year=. Welcome to the

  35. [35]

    2017 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages=

    Domain randomization for transferring deep neural networks from simulation to the real world , author=. 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages=

    2017

  36. [36]

    Sutton and Marlos C

    Richard S. Sutton and Marlos C. Machado and G. Zacharias Holland and David Szepesvari and Finbarr Timbers and Brian Tanner and Adam White , keywords =. Reward-respecting subtasks for. Artificial Intelligence , volume =. 2023 , issn =. doi:https://doi.org/10.1016/j.artint.2023.104001 , url =

    arXiv 2023

  37. [37]

    International Conference on Robotics and Automation , pages=

    Reinforcement learning for robust parameterized locomotion control of bipedal robots , author=. International Conference on Robotics and Automation , pages=

  38. [38]

    Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning , doi =

    Siekmann, Jonah and Green, Kevin and Warila, John and Fern, Alan and Hurst, Jonathan , year =. Blind Bipedal Stair Traversal via Sim-to-Real Reinforcement Learning , doi =

  39. [39]

    Science Robotics , volume =

    David Hoeller and Nikita Rudin and Dhionis Sako and Marco Hutter , title =. Science Robotics , volume =. 2024 , doi =. https://www.science.org/doi/pdf/10.1126/scirobotics.adi7566 , abstract =

    doi:10.1126/scirobotics.adi7566 2024

  40. [40]

    Journal of Machine Learning Research , year =

    Shengyi Huang and others , title =. Journal of Machine Learning Research , year =

  41. [41]

    Achiam, Joshua , title =

  42. [42]

    CoRR , volume =

    Tuomas Haarnoja and Aurick Zhou and Pieter Abbeel and Sergey Levine , title =. CoRR , volume =. 2018 , url =. 1801.01290 , timestamp =

    Pith/arXiv arXiv 2018

  43. [43]

    and Li, Weiping , year =

    Slotine, Jean-Jacques E. and Li, Weiping , year =. Applied nonlinear control , isbn =

  44. [44]

    1996 , publisher=

    Robust adaptive control , author=. 1996 , publisher=

    1996

  45. [45]

    1995 , publisher=

    Nonlinear and adaptive control design , author=. 1995 , publisher=

    1995

  46. [46]

    2012 , publisher=

    Stable adaptive systems , author=. 2012 , publisher=

    2012

  47. [47]

    and Alindogan, Jedidiah and Anderson, Matthew and Chung, Soon-Jo , journal=

    Lupu, Elena Sorina and Xie, Fengze and Preiss, James A. and Alindogan, Jedidiah and Anderson, Matthew and Chung, Soon-Jo , journal=. MAGIC-VFM -. 2025 , volume=

    2025

  48. [48]

    doi:10.1126/scirobotics.abm6597 , url =

    Michael O'Connell and others , title =. doi:10.1126/scirobotics.abm6597 , url =

    doi:10.1126/scirobotics.abm6597

  49. [49]

    2022 , eprint=

    A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learning , author=. 2022 , eprint=

    2022

  50. [50]

    2020 , url =

    Rinu Boney and Jussi Sainio and Mikko Kaivola and Arno Solin and Juho Kannala , title =. 2020 , url =. 2011.03085 , timestamp =

    Pith/arXiv arXiv 2020

  51. [51]

    Nature Machine Intelligence , volume=

    Maximum diffusion reinforcement learning , author=. Nature Machine Intelligence , volume=. 2024 , publisher=

    2024

  52. [52]

    2025 , month =

    De Asis, Kris , title =. 2025 , month =

    2025

  53. [53]

    Policy gradient reinforcement learning for fast quadrupedal locomotion , year=

    Kohl, Nate and Stone, Peter , booktitle=. Policy gradient reinforcement learning for fast quadrupedal locomotion , year=

  54. [54]

    Nature , volume =

    Robots that can adapt like animals , author =. Nature , volume =. 2015 , doi =

    2015

  55. [55]

    Jordan and Pieter Abbeel , journal=

    John Schulman and Philipp Moritz and Sergey Levine and Michael I. Jordan and Pieter Abbeel , journal=. High-Dimensional Continuous Control Using. 2015 , volume=

    2015

  56. [56]

    Gavin Adrian Rummery and Mahesan Niranjan , year=. On-line

  57. [57]

    Sutton and Doina Precup and Satinder Singh , keywords =

    Richard S. Sutton and Doina Precup and Satinder Singh , keywords =. Between. Artificial Intelligence , volume =. 1999 , issn =. doi:https://doi.org/10.1016/S0004-3702(99)00052-1 , url =

    doi:10.1016/s0004-3702(99)00052-1 1999

  58. [58]

    Nicklas Hansen and Hao Su and Xiaolong Wang , booktitle=

  59. [59]

    Temporal Difference Learning for

    Nicklas Hansen and Xiaolong Wang and Hao Su , booktitle=. Temporal Difference Learning for

  60. [60]

    Wu, Philipp and Escontrela, Alejandro and Hafner, Danijar and Goldberg, Ken and Abbeel, Pieter , journal=

  61. [61]

    2024 , howpublished =

    Leslie Kaelbling , title =. 2024 , howpublished =

    2024

  62. [62]

    and Honig, Wolfgang and Sukhatme, Gaurav S

    Preiss, James A. and Honig, Wolfgang and Sukhatme, Gaurav S. and Ayanian, Nora , booktitle=. Crazyswarm: A large nano-quadcopter swarm , year=

  63. [63]

    2025 , eprint=

    Fast Non-Episodic Adaptive Tuning of Robot Controllers with Online Policy Optimization , author=. 2025 , eprint=

    2025

  64. [64]

    2016 , eprint=

    Layer Normalization , author=. 2016 , eprint=

    2016

  65. [65]

    Science Robotics , volume =

    Steven Macenski and Tully Foote and Brian Gerkey and Chris Lalancette and William Woodall , title =. Science Robotics , volume =. 2022 , doi =

    2022

  66. [66]

    The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] , year=

    Deng, Li , journal=. The MNIST Database of Handwritten Digit Images for Machine Learning Research [Best of the Web] , year=

  67. [67]

    Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages =

    Akiba, Takuya and Sano, Shotaro and Yanase, Toshihiko and Ohta, Takeru and Koyama, Masanori , title =. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages =. 2019 , doi =

    2019

  68. [68]

    Olson, Edwin , booktitle=. April. 2011 , volume=

    2011

  69. [69]

    2018 , eprint=

    Setting up a Reinforcement Learning Task with a Real-World Robot , author=. 2018 , eprint=

    2018

  70. [70]

    Rupam Mahmood and Dmytro Korenkevych and Gautham Vasan and William Ma and James Bergstra , title =

    A. Rupam Mahmood and Dmytro Korenkevych and Gautham Vasan and William Ma and James Bergstra , title =. CoRR , volume =. 2018 , url =. 1809.07731 , timestamp =

    Pith/arXiv arXiv 2018

  71. [71]

    Learning From Delayed Rewards , url =

    Watkins, Christopher , year =. Learning From Delayed Rewards , url =

  72. [72]

    2024 , eprint=

    Streaming Deep Reinforcement Learning Finally Works , author=. 2024 , eprint=

    2024

  73. [73]

    Sutton , title =

    Richard S. Sutton , title =. 2025 , howpublished =

    2025

  74. [74]

    2025 , howpublished =

    Leslie Kaelbling , title =. 2025 , howpublished =

    2025

  75. [75]

    Action Space Design in Reinforcement Learning for Robot Motor Skills , booktitle =

    Julian E. Action Space Design in Reinforcement Learning for Robot Motor Skills , booktitle =. 2024 , publisher =

    2024

  76. [76]

    Co-Reyes and Suvansh Sanjeev and Glen Berseth and Abhishek Gupta and Sergey Levine , title =

    John D. Co-Reyes and Suvansh Sanjeev and Glen Berseth and Abhishek Gupta and Sergey Levine , title =. arXiv preprint arXiv:2006.12478 , year =

    Pith/arXiv arXiv 2006

  77. [77]

    Allen and Josh Tenenbaum and Leslie Pack Kaelbling , title =

    Tom Silver and Kelsey R. Allen and Josh Tenenbaum and Leslie Pack Kaelbling , title =. CoRR , volume =. 2018 , url =. 1812.06298 , timestamp =

    Pith/arXiv arXiv 2018

  78. [78]

    Transactions of the ASME Journal of Dynamic Systems , url =

    James Albus , title =. Transactions of the ASME Journal of Dynamic Systems , url =. 1975 , month =

    1975

  79. [79]

    Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding , url =

    Sutton, Richard S , booktitle =. Generalization in Reinforcement Learning: Successful Examples Using Sparse Coarse Coding , url =

  80. [80]

    CoRR , volume =

    Joshua Tobin and Rachel Fong and Alex Ray and Jonas Schneider and Wojciech Zaremba and Pieter Abbeel , title =. CoRR , volume =. 2017 , url =. 1703.06907 , timestamp =

    Pith/arXiv arXiv 2017

Showing first 80 references.

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.