REVIEW 4 major objections 5 minor 30 references
This paper builds a full-workspace benchmark for robotic reach-avoid and shows that DRL agents reach 86-99% success—while policies trained in small tabletop workspaces collapse when tested in the full workspace.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 21:51 UTC pith:UYZ3OXMC
load-bearing objection A useful full-workspace reach-avoid benchmark with a genuinely new cross-evaluation result, but the 'without simplifications' claim is not backed by the simulation fidelity choices. the 4 major comments →
Learning Reach-Avoid Task with Reinforcement Learning: Vectorized Simulation and Benchmark
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The authors claim to present the first comprehensive reach-avoid benchmark that uses each robot's full workspace without the usual simplifications, and to show that PPO with relative joint-position control, a dense distance-based reward, and per-link obstacle-distance observations attains high success rates: 96.1% (UR5e) and 98.8% (Franka) for reaching, and 86.8% and 95.2% with a static spherical obstacle. The central discovery is the extreme sensitivity of learned policies to environment setup: agents trained in a small tabletop workspace reach 100% but drop to 4% in the full workspace, while full-workspace agents keep near-perfect performance on smaller ones; similarly, restricting the ini
What carries the argument
The load-bearing mechanism is a stateless, vectorized environment design that runs thousands of parallel simulation instances and the entire training loop on a single GPU, making full-workspace training tractable. Supporting this are relative joint-position control (the agent outputs small deltas to joint angles), a dense exponential distance reward, and an observation space that includes each robot link's distance to the obstacle—a feature that reduces obstacle collisions by roughly half.
Load-bearing premise
The benchmark's claim to reflect real-world complexity rests on the physics settings (few solver iterations, capsule collision shapes, added joint damping) faithfully reproducing the robots' actual collision and velocity behaviour; if those simplifications miss collisions or distort motion, the reported success rates are not meaningful for real robots.
What would settle it
Run the trained policies on the physical UR5e and Franka arms for the same target and obstacle distributions: if success rates fall sharply or unexpected collisions appear, the simulated dynamics are not faithful. A cheaper check is to re-run evaluation episodes with solver iterations raised from 1 to, say, 32 and compare collision and reach rates; large differences would indicate the tuned settings hide collision behaviour.
If this is right
- Policies trained in a small tabletop workspace reach 100% success but collapse to 4-14% when evaluated in the full workspace, while full-workspace policies transfer perfectly to smaller workspaces.
- Initializing episodes from the full joint range is critical: a policy trained with a limited joint range drops from 98.9% to 60.1% target-reached when evaluated with full-range initialization.
- Including per-link distances to the obstacle in the observation cuts obstacle collisions by roughly half (from about 8% to 4.5%).
- Relative joint-position control is far more stable than absolute position control, staying in the target region for over twice as long and leaving it in only 5% of episodes instead of about 20 times per episode.
- Sparse rewards fail to learn a 1-centimeter target in a full workspace because a random policy reaches it with probability 0.006%, while a dense distance-based reward learns faster and achieves higher precision.
Where Pith is reading between the lines
- The cross-evaluation results imply that 'solving' a tabletop reach-avoid task is not evidence of a general ability; the field should adopt full-workspace, full-range-initialization evaluation as the default for manipulation RL.
- The success of per-link obstacle distances suggests a visual analogue: a learned or estimated distance-to-obstacle channel from depth images could replace engineered state, enabling camera-only reach-avoid policies.
- Because larger obstacles reduce success mostly by removing viable paths rather than by increasing collision rate, the same benchmark design could be extended to dynamic or non-convex obstacles to test planning under increasing geometric constraints.
- The observed sensitivity to initial pose distribution implies that published success rates are incomparable unless the initialization protocol is specified exactly; standardized sampling could become part of the benchmark itself.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces a GPU-accelerated, vectorized reinforcement-learning benchmark for reach and reach-avoid tasks with two robotic arms (UR5e and Franka Emika Robot), built on MuJoCo MJX and the Brax training pipeline. The environment covers the full robot workspace with random initial poses, targets, and static spherical obstacles, and the authors report extensive ablations over reward formulations, action spaces, observation spaces, network sizes, workspace sizes, and obstacle sizes. The headline results are success rates of 96.1% (UR5e) and 98.8% (Franka) for the reach task and 86.8% (UR5e) and 95.2% (Franka) for the reach-avoid task, which the paper labels state-of-the-art. The authors also report large-scale training throughput and claim over 10x speedup over Stable Baselines 3. The code is promised as open source.
Significance. If validated, this benchmark would address a real gap: most existing reach-avoid benchmarks for manipulators use small tabletop workspaces that are too easy, and the paper's finding that performance collapses when moving from PandaGym-style workspaces to full-workspace settings is a useful caution for the community. The paper has notable strengths: the environment is designed for massively parallel GPU training, the evaluation uses 5 seeds and 4096 environments, and the ablations over workspace size, observation space, reward form, and obstacle size are systematic and informative. The cross-workspace generalization results and the obstacle-size overfitting analysis are particularly valuable. However, the central claims of "no simplifications" and "state-of-the-art" are currently unsupported, and several reproducibility-critical details are missing.
major comments (4)
- [Section III-A, Abstract] The abstract and Section III-A claim the benchmark "accurately captures real-world complexities without simplifications," but Section III-A lists three speed-motivated approximations: Newton solver iterations reduced to 1, UR5e end-effector collision geometry changed from cylinder to capsule, and all Franka links replaced by capsule approximations. These changes directly affect collision detection and contact dynamics, which are the central metrics of the benchmark. No fidelity check is provided. I request a quantitative sensitivity study: compare success rates, collision rates, and contact behavior under default/higher solver iterations and under mesh-based collision geometry, or against a real-robot/kinematic validation. If the differences are negligible, report them; if not, the "no simplifications" claim and the headline numbers must be qualified.
- [Section VI, Discussion; Abstract] The paper calls the reported success rates "state-of-the-art," but no prior DRL method is evaluated on this benchmark. The only baselines are random actions, standstill, pseudo-IK, and a reach-task agent without an obstacle. Since the benchmark is new, "state-of-the-art" is undefined without comparing against existing reach-avoid or joint-space DRL methods, e.g., Kumar et al. [23] or a re-implemented prior baseline. Either add quantitative comparisons to prior methods on the same tasks, or replace "state-of-the-art" with a more precise description such as "strong baseline results on a new benchmark."
- [Section I, Introduction] The claim of "over 10x speed-up in training compared to using stable baselines 3" is asserted without any supporting experiment. Section III-B reports environment throughput only (steps/second); it does not compare training wall-clock time, algorithm, hardware, or convergence against Stable Baselines 3. This comparison is load-bearing for the paper's motivation. Either provide the actual benchmark data and protocol, or remove the claim.
- [Section IV-A3, Section IV-B, Eqs. (1)-(3)] The reward functions contain free parameters -- w_reach, w_dist, alpha, w_obst, tau -- but the text never gives their values. These weights directly determine the learned behavior and are essential for reproduction and for interpreting the collision/success trade-offs. The paper states alpha is set to 20, but the other weights are missing. Please report exact values for all reward coefficients (or give a config file path) and specify whether joint velocity/acceleration penalties were used in the final experiments, since Section V-B says they were omitted.
minor comments (5)
- [Abstract] Typo: "the each-avoid task" should be "the reach-avoid task." Also, "Padna" appears in Section V in place of "Panda."
- [Table I, Section IV-C2] The "Pseudo IK" baseline is described only as setting an action to a joint position "known to be able to reach the target." How is this position generated, and why does it succeed only 36.9% of the time? A brief explanation is needed for the baseline to be interpretable.
- [Section II, Related Work] The related-work discussion of previous methods is useful but does not clearly separate methods that use Cartesian-space control from those using joint-space control in terms of the benchmark's evaluation metrics. Also, reference [27] appears twice in the bibliography (same paper as [18]).
- [Section III-A] The damping correction for the UR5e is described qualitatively ("up to 5x real-world specifications"). Please report the actual joint velocity limits and the damping coefficient used, since this changes the robot dynamics and therefore the task difficulty.
- [General] The paper says "The environment and benchmarking code is available as open source at the following link (Blind Review)." In the final version, an actual URL or repository identifier must be provided. Also, Figures 2 and 5-9 would benefit from axis labels and units in several panels.
Circularity Check
No circular derivation: the reported success rates are measured benchmark outcomes, with only a minor non-load-bearing self-citation; fidelity concerns are correctness issues, not circularity.
full rationale
The paper's central contribution is a new benchmark and an empirical evaluation of DRL agents on it. There is no derivation chain whose outputs equal fitted inputs: the headline success rates (96.1%/98.8% reach; 86.8%/95.2% reach-avoid) are measured from trained policies over 4096 environments and 5 seeds, not solved from the reward equations or simulation constants. The reward functions are explicitly credited or conventional: Equations 1 and 2 are standard distance-based rewards, and Equation 3 is 'adapted from [23]' (Kumar et al.), not from the authors' own prior work. The baselines (random, standstill, pseudo-IK) are independent reference points, and the cross-evaluation tables are empirical measurements rather than predictions derived by construction. The only self-citation is in the introduction, '[1], [2]', where [2] is a prior co-authored paper by Eivazi on de-powdering trajectories; this citation is not load-bearing for the benchmark or the reported results, so it does not constitute circularity. The 'state-of-the-art' phrasing is benchmark-relative because the benchmark itself is introduced in this paper, which is a weak self-reference but not an equation-level reduction of a prediction to an input. The Discussion's admission of 'the inherent sim-to-real gap' and the Section III-A approximations (solver iterations reduced to 1, capsule collision geometry, added joint damping) are legitimate threats to the validity of the simulated metrics and to the claim of capturing 'real-world complexities without simplifications', but they are correctness/fidelity concerns, not evidence that a fitted parameter was renamed as a prediction. No circular step meeting the required standard was found.
Axiom & Free-Parameter Ledger
free parameters (5)
- w_reach (semi-sparse reward weight)
- w_dist (distance reward weight)
- alpha (dense reward sharpness) =
20
- w_obst (obstacle penalty weight)
- tau (obstacle penalty distance threshold)
axioms (4)
- domain assumption MuJoCo MJX with reduced solver iterations and capsule collision geometries yields sufficiently accurate dynamics for evaluating reach-avoid.
- domain assumption Uniform sampling from full joint range, with collision rejection, represents the robot's full workspace.
- domain assumption Reward formulations (Eqs. 1-3) are appropriate learning signals for the reach-avoid task.
- standard math PPO and SAC implementations in Brax are correct and converge as intended.
read the original abstract
Deep reinforcement learning (DRL) has a longstanding tradition in addressing the reach-avoid task problem, especially for controlling robotic arms. While this task serves as a baseline environment within the research community, the ability of DRL to effectively learn the each-avoid task in complex and realistic scenarios beyond simplified and restricted tabletop settings remains uncertain. In this paper, we present, for the first time, a comprehensive benchmark for the reachavoid task that accurately captures real-world complexities without simplifications. We demonstrate a diverse range of settings for robotic arm reach-avoid task, which can be used for evaluating DRL research. We achieved this by utilizing the MuJoCo MJX physics engine and parallelizing both the simulation environment and DRL algorithms using the Brax library. We achieved state-of-the-art results with success rates of 96.1% (UR5e) and 98.8% (Franka Emika Robot) for the reach task and 86.8% (UR5e) and 95.2% (Franka) for the static reachavoid task. Our results indicate that while in previous works DRL agents could solve, for example, a reach task in a simplified setting perfectly, their agents performance collapses when evaluated in realistic scenarios. Overall, this work identifies that additional research is still required to claim the successful resolution of the robotic arm reach-avoid task using DRL. The environment and benchmarking code is available as open source at the following link
Figures
Reference graph
Works this paper leans on
-
[1]
Deep reinforcement learning for vision-based robotic grasping: A simulated comparative evaluation of off-policy methods,
D. Quillen, E. Jang, O. Nachum, C. Finn, J. Ibarz, and S. Levine, “Deep reinforcement learning for vision-based robotic grasping: A simulated comparative evaluation of off-policy methods,” in2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018, pp. 6284–6291
2018
-
[2]
Synthesizing depow- dering trajectories for robot arms using deep reinforcement learning,
M. Maurer, S. Seefeldt, J. Seyler, and S. Eivazi, “Synthesizing depow- dering trajectories for robot arms using deep reinforcement learning,” in2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 11 524–11 530
2025
-
[3]
Rapidly-exploring random trees: A new tool for path planning,
S. M. LaValle, “Rapidly-exploring random trees: A new tool for path planning,” 1998
1998
-
[4]
Scalable deep reinforcement learning for vision-based robotic manipulation,
D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V . Vanhouckeet al., “Scalable deep reinforcement learning for vision-based robotic manipulation,” in Conference on robot learning. PMLR, 2018, pp. 651–673
2018
-
[5]
The impact of task underspecification in evaluating deep reinforcement learning,
V . Jayawardana, C. Tang, S. Li, D. Suo, and C. Wu, “The impact of task underspecification in evaluating deep reinforcement learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 23 881–23 893, 2022
2022
-
[6]
Openai gym,
G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016
2016
-
[7]
panda-gym: Open-Source Goal-Conditioned Environments for Robotic Learning,
Q. Gallou ´edec, N. Cazin, E. Dellandr ´ea, and L. Chen, “panda-gym: Open-Source Goal-Conditioned Environments for Robotic Learning,” 4th Robot Learning Workshop: Self-Supervised and Lifelong Learning at NeurIPS, 2021
2021
-
[8]
Mujoco: A physics engine for model-based control,
E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model-based control,” in2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012, pp. 5026–5033
2012
-
[9]
Nvidia isaac simulation,
N. Corporation, “Nvidia isaac simulation,” 2023, accessed: 2023-10-
2023
-
[10]
Available: https://developer.nvidia.com/isaac-sim
[Online]. Available: https://developer.nvidia.com/isaac-sim
-
[11]
Brax–a differentiable physics engine for large scale rigid body simulation,
C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem, “Brax–a differentiable physics engine for large scale rigid body simulation,”arXiv preprint arXiv:2106.13281, 2021
Pith/arXiv arXiv 2021
-
[12]
Proximal policy optimization algorithms,
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017
Pith/arXiv arXiv 2017
-
[13]
Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,
T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” inInternational conference on machine learning. PMLR, 2018, pp. 1861–1870
2018
-
[14]
Stable-baselines3: Reliable reinforcement learning im- plementations,
A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning im- plementations,”Journal of Machine Learning Research, 2021
2021
-
[15]
Solving minimum-cost reach avoid using reinforcement learning,
O. So, C. Ge, and C. Fan, “Solving minimum-cost reach avoid using reinforcement learning,”Advances in Neural Information Processing Systems, vol. 37, pp. 30 951–30 984, 2024
2024
-
[16]
Optlayer-practical constrained optimization for deep reinforcement learning in the real world,
T.-H. Pham, G. De Magistris, and R. Tachibana, “Optlayer-practical constrained optimization for deep reinforcement learning in the real world,” in2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018, pp. 6236–6243
2018
-
[17]
Deep reinforcement learning for collision avoidance of robotic manipulators,
B. Sangiovanni, A. Rendiniello, G. P. Incremona, A. Ferrara, and M. Piastra, “Deep reinforcement learning for collision avoidance of robotic manipulators,” in2018 European Control Conference (ECC). IEEE, 2018, pp. 2063–2068
2018
-
[18]
Accelerating reinforcement learning for reaching using continuous curriculum learning,
S. Luo, H. Kasaei, and L. Schomaker, “Accelerating reinforcement learning for reaching using continuous curriculum learning,” in2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020, pp. 1–8
2020
-
[19]
Revisiting sparse rewards for goal-reaching reinforcement learning,
G. Vasan, Y . Wang, F. Shahriar, J. Bergstra, M. Jagersand, and A. R. Mahmood, “Revisiting sparse rewards for goal-reaching reinforcement learning,”arXiv preprint arXiv:2407.00324, 2024
Pith/arXiv arXiv 2024
-
[20]
Rl-rrt: Kinodynamic motion planning via learning reachability estimators from rl policies,
H.-T. L. Chiang, J. Hsu, M. Fiser, L. Tapia, and A. Faust, “Rl-rrt: Kinodynamic motion planning via learning reachability estimators from rl policies,”IEEE Robotics and Automation Letters, 2019
2019
-
[21]
Reinforcement learning-based algorithm to avoid obstacles by the anthropomorphic robotic arm,
T. Lindner and A. Milecki, “Reinforcement learning-based algorithm to avoid obstacles by the anthropomorphic robotic arm,”Applied Sciences, vol. 12, no. 13, p. 6629, 2022
2022
-
[22]
Deep-reinforcement-learning-based path planning for indus- trial robots using distance sensors as observation,
T. Bhuiyan, L. K ¨astner, Y . Hu, B. Kutschank, and J. Lam- brecht, “Deep-reinforcement-learning-based path planning for indus- trial robots using distance sensors as observation,” in2023 8th Inter- national Conference on Control and Robotics Engineering (ICCRE). IEEE, 2023, pp. 204–210
2023
-
[23]
Y . Wang and H. Kasaei, “Ippo: Obstacle avoidance for robotic ma- nipulators in joint space via improved proximal policy optimization,” arXiv preprint arXiv:2210.00803, 2022
Pith/arXiv arXiv 2022
-
[24]
Joint space control via deep reinforcement learning,
V . Kumar, D. Hoeller, B. Sundaralingam, J. Tremblay, and S. Birch- field, “Joint space control via deep reinforcement learning,” in2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021
2021
-
[25]
Robots,Universal Robots UR5e
U. Robots,Universal Robots UR5e. [Online]. Available: https://www.universal-robots.com/
-
[26]
Robotics,Franka Emika Robot
F. Robotics,Franka Emika Robot. [Online]. Available: https://franka.de/
-
[27]
Gymnasium: A standard interface for reinforcement learning environments,
M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goul ˜ao, A. Kallinteris, M. Krimmel, A. KGet al., “Gymnasium: A standard interface for reinforcement learning environments,”arXiv preprint arXiv:2407.17032, 2024
Pith/arXiv arXiv 2024
-
[28]
Revisiting sparse rewards for goal-reaching reinforcement learning,
G. Vasan, Y . Wang, F. Shahriar, J. Bergstra, M. Jagersand, and A. R. Mahmood, “Revisiting sparse rewards for goal-reaching reinforcement learning,” 2024
2024
-
[29]
Accelerating reinforcement learning for reaching using continuous curriculum learning,
S. Luo, H. Kasaei, and L. Schomaker, “Accelerating reinforcement learning for reaching using continuous curriculum learning,” in2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020
2020
-
[30]
Training larger networks for deep reinforcement learning,
K. Ota, D. K. Jha, and A. Kanezaki, “Training larger networks for deep reinforcement learning,”arXiv preprint arXiv:2102.07920, 2021
Pith/arXiv arXiv 2021
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.