Pith. sign in

REVIEW 4 major objections 5 minor 30 references

This paper builds a full-workspace benchmark for robotic reach-avoid and shows that DRL agents reach 86-99% success—while policies trained in small tabletop workspaces collapse when tested in the full workspace.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 21:51 UTC pith:UYZ3OXMC

load-bearing objection A useful full-workspace reach-avoid benchmark with a genuinely new cross-evaluation result, but the 'without simplifications' claim is not backed by the simulation fidelity choices. the 4 major comments →

arxiv 2607.15935 v1 pith:UYZ3OXMC submitted 2026-07-17 cs.RO cs.LG

Learning Reach-Avoid Task with Reinforcement Learning: Vectorized Simulation and Benchmark

classification cs.RO cs.LG
keywords reinforcement learningreach-avoidrobotic manipulationfull workspaceGPU-accelerated simulationbenchmarkgeneralizationPPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether deep reinforcement learning can solve the robotic-arm reach-avoid task when the workspace is the arm's entire reachable region, not a small tabletop zone. It introduces a massively parallel, GPU-accelerated simulation and training setup, reporting success rates of 96.1% and 98.8% for plain reaching and 86.8% and 95.2% for static reach-avoid on two common arms. Its central demonstration is that environment details dominate: a policy that is perfect in a 30-centimeter tabletop workspace collapses to near-zero success in the full workspace, while full-workspace policies generalize perfectly to smaller ones. The authors argue that workspace size, initial pose distribution, and observation features—especially per-link obstacle distances—are decisive, and that realistic settings plus full documentation are needed to make DRL results meaningful. Why care: the paper provides a reproducible, harder baseline and pinpoints where current DRL still fails in realistic manipulation.

Core claim

The authors claim to present the first comprehensive reach-avoid benchmark that uses each robot's full workspace without the usual simplifications, and to show that PPO with relative joint-position control, a dense distance-based reward, and per-link obstacle-distance observations attains high success rates: 96.1% (UR5e) and 98.8% (Franka) for reaching, and 86.8% and 95.2% with a static spherical obstacle. The central discovery is the extreme sensitivity of learned policies to environment setup: agents trained in a small tabletop workspace reach 100% but drop to 4% in the full workspace, while full-workspace agents keep near-perfect performance on smaller ones; similarly, restricting the ini

What carries the argument

The load-bearing mechanism is a stateless, vectorized environment design that runs thousands of parallel simulation instances and the entire training loop on a single GPU, making full-workspace training tractable. Supporting this are relative joint-position control (the agent outputs small deltas to joint angles), a dense exponential distance reward, and an observation space that includes each robot link's distance to the obstacle—a feature that reduces obstacle collisions by roughly half.

Load-bearing premise

The benchmark's claim to reflect real-world complexity rests on the physics settings (few solver iterations, capsule collision shapes, added joint damping) faithfully reproducing the robots' actual collision and velocity behaviour; if those simplifications miss collisions or distort motion, the reported success rates are not meaningful for real robots.

What would settle it

Run the trained policies on the physical UR5e and Franka arms for the same target and obstacle distributions: if success rates fall sharply or unexpected collisions appear, the simulated dynamics are not faithful. A cheaper check is to re-run evaluation episodes with solver iterations raised from 1 to, say, 32 and compare collision and reach rates; large differences would indicate the tuned settings hide collision behaviour.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Policies trained in a small tabletop workspace reach 100% success but collapse to 4-14% when evaluated in the full workspace, while full-workspace policies transfer perfectly to smaller workspaces.
  • Initializing episodes from the full joint range is critical: a policy trained with a limited joint range drops from 98.9% to 60.1% target-reached when evaluated with full-range initialization.
  • Including per-link distances to the obstacle in the observation cuts obstacle collisions by roughly half (from about 8% to 4.5%).
  • Relative joint-position control is far more stable than absolute position control, staying in the target region for over twice as long and leaving it in only 5% of episodes instead of about 20 times per episode.
  • Sparse rewards fail to learn a 1-centimeter target in a full workspace because a random policy reaches it with probability 0.006%, while a dense distance-based reward learns faster and achieves higher precision.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The cross-evaluation results imply that 'solving' a tabletop reach-avoid task is not evidence of a general ability; the field should adopt full-workspace, full-range-initialization evaluation as the default for manipulation RL.
  • The success of per-link obstacle distances suggests a visual analogue: a learned or estimated distance-to-obstacle channel from depth images could replace engineered state, enabling camera-only reach-avoid policies.
  • Because larger obstacles reduce success mostly by removing viable paths rather than by increasing collision rate, the same benchmark design could be extended to dynamic or non-convex obstacles to test planning under increasing geometric constraints.
  • The observed sensitivity to initial pose distribution implies that published success rates are incomparable unless the initialization protocol is specified exactly; standardized sampling could become part of the benchmark itself.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a GPU-accelerated, vectorized reinforcement-learning benchmark for reach and reach-avoid tasks with two robotic arms (UR5e and Franka Emika Robot), built on MuJoCo MJX and the Brax training pipeline. The environment covers the full robot workspace with random initial poses, targets, and static spherical obstacles, and the authors report extensive ablations over reward formulations, action spaces, observation spaces, network sizes, workspace sizes, and obstacle sizes. The headline results are success rates of 96.1% (UR5e) and 98.8% (Franka) for the reach task and 86.8% (UR5e) and 95.2% (Franka) for the reach-avoid task, which the paper labels state-of-the-art. The authors also report large-scale training throughput and claim over 10x speedup over Stable Baselines 3. The code is promised as open source.

Significance. If validated, this benchmark would address a real gap: most existing reach-avoid benchmarks for manipulators use small tabletop workspaces that are too easy, and the paper's finding that performance collapses when moving from PandaGym-style workspaces to full-workspace settings is a useful caution for the community. The paper has notable strengths: the environment is designed for massively parallel GPU training, the evaluation uses 5 seeds and 4096 environments, and the ablations over workspace size, observation space, reward form, and obstacle size are systematic and informative. The cross-workspace generalization results and the obstacle-size overfitting analysis are particularly valuable. However, the central claims of "no simplifications" and "state-of-the-art" are currently unsupported, and several reproducibility-critical details are missing.

major comments (4)
  1. [Section III-A, Abstract] The abstract and Section III-A claim the benchmark "accurately captures real-world complexities without simplifications," but Section III-A lists three speed-motivated approximations: Newton solver iterations reduced to 1, UR5e end-effector collision geometry changed from cylinder to capsule, and all Franka links replaced by capsule approximations. These changes directly affect collision detection and contact dynamics, which are the central metrics of the benchmark. No fidelity check is provided. I request a quantitative sensitivity study: compare success rates, collision rates, and contact behavior under default/higher solver iterations and under mesh-based collision geometry, or against a real-robot/kinematic validation. If the differences are negligible, report them; if not, the "no simplifications" claim and the headline numbers must be qualified.
  2. [Section VI, Discussion; Abstract] The paper calls the reported success rates "state-of-the-art," but no prior DRL method is evaluated on this benchmark. The only baselines are random actions, standstill, pseudo-IK, and a reach-task agent without an obstacle. Since the benchmark is new, "state-of-the-art" is undefined without comparing against existing reach-avoid or joint-space DRL methods, e.g., Kumar et al. [23] or a re-implemented prior baseline. Either add quantitative comparisons to prior methods on the same tasks, or replace "state-of-the-art" with a more precise description such as "strong baseline results on a new benchmark."
  3. [Section I, Introduction] The claim of "over 10x speed-up in training compared to using stable baselines 3" is asserted without any supporting experiment. Section III-B reports environment throughput only (steps/second); it does not compare training wall-clock time, algorithm, hardware, or convergence against Stable Baselines 3. This comparison is load-bearing for the paper's motivation. Either provide the actual benchmark data and protocol, or remove the claim.
  4. [Section IV-A3, Section IV-B, Eqs. (1)-(3)] The reward functions contain free parameters -- w_reach, w_dist, alpha, w_obst, tau -- but the text never gives their values. These weights directly determine the learned behavior and are essential for reproduction and for interpreting the collision/success trade-offs. The paper states alpha is set to 20, but the other weights are missing. Please report exact values for all reward coefficients (or give a config file path) and specify whether joint velocity/acceleration penalties were used in the final experiments, since Section V-B says they were omitted.
minor comments (5)
  1. [Abstract] Typo: "the each-avoid task" should be "the reach-avoid task." Also, "Padna" appears in Section V in place of "Panda."
  2. [Table I, Section IV-C2] The "Pseudo IK" baseline is described only as setting an action to a joint position "known to be able to reach the target." How is this position generated, and why does it succeed only 36.9% of the time? A brief explanation is needed for the baseline to be interpretable.
  3. [Section II, Related Work] The related-work discussion of previous methods is useful but does not clearly separate methods that use Cartesian-space control from those using joint-space control in terms of the benchmark's evaluation metrics. Also, reference [27] appears twice in the bibliography (same paper as [18]).
  4. [Section III-A] The damping correction for the UR5e is described qualitatively ("up to 5x real-world specifications"). Please report the actual joint velocity limits and the damping coefficient used, since this changes the robot dynamics and therefore the task difficulty.
  5. [General] The paper says "The environment and benchmarking code is available as open source at the following link (Blind Review)." In the final version, an actual URL or repository identifier must be provided. Also, Figures 2 and 5-9 would benefit from axis labels and units in several panels.

Circularity Check

0 steps flagged

No circular derivation: the reported success rates are measured benchmark outcomes, with only a minor non-load-bearing self-citation; fidelity concerns are correctness issues, not circularity.

full rationale

The paper's central contribution is a new benchmark and an empirical evaluation of DRL agents on it. There is no derivation chain whose outputs equal fitted inputs: the headline success rates (96.1%/98.8% reach; 86.8%/95.2% reach-avoid) are measured from trained policies over 4096 environments and 5 seeds, not solved from the reward equations or simulation constants. The reward functions are explicitly credited or conventional: Equations 1 and 2 are standard distance-based rewards, and Equation 3 is 'adapted from [23]' (Kumar et al.), not from the authors' own prior work. The baselines (random, standstill, pseudo-IK) are independent reference points, and the cross-evaluation tables are empirical measurements rather than predictions derived by construction. The only self-citation is in the introduction, '[1], [2]', where [2] is a prior co-authored paper by Eivazi on de-powdering trajectories; this citation is not load-bearing for the benchmark or the reported results, so it does not constitute circularity. The 'state-of-the-art' phrasing is benchmark-relative because the benchmark itself is introduced in this paper, which is a weak self-reference but not an equation-level reduction of a prediction to an input. The Discussion's admission of 'the inherent sim-to-real gap' and the Section III-A approximations (solver iterations reduced to 1, capsule collision geometry, added joint damping) are legitimate threats to the validity of the simulated metrics and to the claim of capturing 'real-world complexities without simplifications', but they are correctness/fidelity concerns, not evidence that a fitted parameter was renamed as a prediction. No circular step meeting the required standard was found.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central performance numbers depend on several undisclosed reward weights and simulation simplifications; these are assumed rather than derived or reported. This limits independent replication and assessment of the SOTA claim.

free parameters (5)
  • w_reach (semi-sparse reward weight)
    Weight for reach reward in semi-sparse formulation (Eq. 1); value not reported in the paper.
  • w_dist (distance reward weight)
    Weight for distance reward in semi-sparse/dense formulation (Eq. 1,2); value not reported.
  • alpha (dense reward sharpness) = 20
    Sharpness of dense reward exponential (Eq. 2), stated as set to 20.
  • w_obst (obstacle penalty weight)
    Weight for obstacle penalty (Eq. 3), value not reported.
  • tau (obstacle penalty distance threshold)
    Distance threshold for obstacle penalty (Eq. 3), value not reported.
axioms (4)
  • domain assumption MuJoCo MJX with reduced solver iterations and capsule collision geometries yields sufficiently accurate dynamics for evaluating reach-avoid.
    Section III-A; if inaccurate, success rates may not reflect real-world performance.
  • domain assumption Uniform sampling from full joint range, with collision rejection, represents the robot's full workspace.
    Section IV-C-1; assumes uniform joint sampling gives representative coverage.
  • domain assumption Reward formulations (Eqs. 1-3) are appropriate learning signals for the reach-avoid task.
    Section IV-A-3; no proof that these rewards align with task objective.
  • standard math PPO and SAC implementations in Brax are correct and converge as intended.
    Relies on standard implementations; no verification in this paper.

pith-pipeline@v1.3.0-alltime-deepseek · 11433 in / 9450 out tokens · 89021 ms · 2026-08-01T21:51:10.573233+00:00 · methodology

0 comments
read the original abstract

Deep reinforcement learning (DRL) has a longstanding tradition in addressing the reach-avoid task problem, especially for controlling robotic arms. While this task serves as a baseline environment within the research community, the ability of DRL to effectively learn the each-avoid task in complex and realistic scenarios beyond simplified and restricted tabletop settings remains uncertain. In this paper, we present, for the first time, a comprehensive benchmark for the reachavoid task that accurately captures real-world complexities without simplifications. We demonstrate a diverse range of settings for robotic arm reach-avoid task, which can be used for evaluating DRL research. We achieved this by utilizing the MuJoCo MJX physics engine and parallelizing both the simulation environment and DRL algorithms using the Brax library. We achieved state-of-the-art results with success rates of 96.1% (UR5e) and 98.8% (Franka Emika Robot) for the reach task and 86.8% (UR5e) and 95.2% (Franka) for the static reachavoid task. Our results indicate that while in previous works DRL agents could solve, for example, a reach task in a simplified setting perfectly, their agents performance collapses when evaluated in realistic scenarios. Overall, this work identifies that additional research is still required to claim the successful resolution of the robotic arm reach-avoid task using DRL. The environment and benchmarking code is available as open source at the following link

Figures

Figures reproduced from arXiv: 2607.15935 by Jonas Weihing, Shahram Eivazi.

Figure 1
Figure 1. Figure 1: Visualization of a randomly sampled initial state for [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of reachable target positions for the UR5e [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison of different network sizes for position [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of different reward formulations. The [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 6
Figure 6. Figure 6: Training curves for the UR5e with PPO across [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 5
Figure 5. Figure 5: Approximate visualization of the four different [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 8
Figure 8. Figure 8: Comparison of different observation spaces for the [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 6 linked inside Pith

  1. [1]

    Deep reinforcement learning for vision-based robotic grasping: A simulated comparative evaluation of off-policy methods,

    D. Quillen, E. Jang, O. Nachum, C. Finn, J. Ibarz, and S. Levine, “Deep reinforcement learning for vision-based robotic grasping: A simulated comparative evaluation of off-policy methods,” in2018 IEEE international conference on robotics and automation (ICRA). IEEE, 2018, pp. 6284–6291

  2. [2]

    Synthesizing depow- dering trajectories for robot arms using deep reinforcement learning,

    M. Maurer, S. Seefeldt, J. Seyler, and S. Eivazi, “Synthesizing depow- dering trajectories for robot arms using deep reinforcement learning,” in2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 11 524–11 530

  3. [3]

    Rapidly-exploring random trees: A new tool for path planning,

    S. M. LaValle, “Rapidly-exploring random trees: A new tool for path planning,” 1998

  4. [4]

    Scalable deep reinforcement learning for vision-based robotic manipulation,

    D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V . Vanhouckeet al., “Scalable deep reinforcement learning for vision-based robotic manipulation,” in Conference on robot learning. PMLR, 2018, pp. 651–673

  5. [5]

    The impact of task underspecification in evaluating deep reinforcement learning,

    V . Jayawardana, C. Tang, S. Li, D. Suo, and C. Wu, “The impact of task underspecification in evaluating deep reinforcement learning,” Advances in Neural Information Processing Systems, vol. 35, pp. 23 881–23 893, 2022

  6. [6]

    Openai gym,

    G. Brockman, V . Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba, “Openai gym,” 2016

  7. [7]

    panda-gym: Open-Source Goal-Conditioned Environments for Robotic Learning,

    Q. Gallou ´edec, N. Cazin, E. Dellandr ´ea, and L. Chen, “panda-gym: Open-Source Goal-Conditioned Environments for Robotic Learning,” 4th Robot Learning Workshop: Self-Supervised and Lifelong Learning at NeurIPS, 2021

  8. [8]

    Mujoco: A physics engine for model-based control,

    E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model-based control,” in2012 IEEE/RSJ international conference on intelligent robots and systems. IEEE, 2012, pp. 5026–5033

  9. [9]

    Nvidia isaac simulation,

    N. Corporation, “Nvidia isaac simulation,” 2023, accessed: 2023-10-

  10. [10]

    Available: https://developer.nvidia.com/isaac-sim

    [Online]. Available: https://developer.nvidia.com/isaac-sim

  11. [11]

    Brax–a differentiable physics engine for large scale rigid body simulation,

    C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem, “Brax–a differentiable physics engine for large scale rigid body simulation,”arXiv preprint arXiv:2106.13281, 2021

  12. [12]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

  13. [13]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” inInternational conference on machine learning. PMLR, 2018, pp. 1861–1870

  14. [14]

    Stable-baselines3: Reliable reinforcement learning im- plementations,

    A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann, “Stable-baselines3: Reliable reinforcement learning im- plementations,”Journal of Machine Learning Research, 2021

  15. [15]

    Solving minimum-cost reach avoid using reinforcement learning,

    O. So, C. Ge, and C. Fan, “Solving minimum-cost reach avoid using reinforcement learning,”Advances in Neural Information Processing Systems, vol. 37, pp. 30 951–30 984, 2024

  16. [16]

    Optlayer-practical constrained optimization for deep reinforcement learning in the real world,

    T.-H. Pham, G. De Magistris, and R. Tachibana, “Optlayer-practical constrained optimization for deep reinforcement learning in the real world,” in2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018, pp. 6236–6243

  17. [17]

    Deep reinforcement learning for collision avoidance of robotic manipulators,

    B. Sangiovanni, A. Rendiniello, G. P. Incremona, A. Ferrara, and M. Piastra, “Deep reinforcement learning for collision avoidance of robotic manipulators,” in2018 European Control Conference (ECC). IEEE, 2018, pp. 2063–2068

  18. [18]

    Accelerating reinforcement learning for reaching using continuous curriculum learning,

    S. Luo, H. Kasaei, and L. Schomaker, “Accelerating reinforcement learning for reaching using continuous curriculum learning,” in2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020, pp. 1–8

  19. [19]

    Revisiting sparse rewards for goal-reaching reinforcement learning,

    G. Vasan, Y . Wang, F. Shahriar, J. Bergstra, M. Jagersand, and A. R. Mahmood, “Revisiting sparse rewards for goal-reaching reinforcement learning,”arXiv preprint arXiv:2407.00324, 2024

  20. [20]

    Rl-rrt: Kinodynamic motion planning via learning reachability estimators from rl policies,

    H.-T. L. Chiang, J. Hsu, M. Fiser, L. Tapia, and A. Faust, “Rl-rrt: Kinodynamic motion planning via learning reachability estimators from rl policies,”IEEE Robotics and Automation Letters, 2019

  21. [21]

    Reinforcement learning-based algorithm to avoid obstacles by the anthropomorphic robotic arm,

    T. Lindner and A. Milecki, “Reinforcement learning-based algorithm to avoid obstacles by the anthropomorphic robotic arm,”Applied Sciences, vol. 12, no. 13, p. 6629, 2022

  22. [22]

    Deep-reinforcement-learning-based path planning for indus- trial robots using distance sensors as observation,

    T. Bhuiyan, L. K ¨astner, Y . Hu, B. Kutschank, and J. Lam- brecht, “Deep-reinforcement-learning-based path planning for indus- trial robots using distance sensors as observation,” in2023 8th Inter- national Conference on Control and Robotics Engineering (ICCRE). IEEE, 2023, pp. 204–210

  23. [23]

    Ippo: Obstacle avoidance for robotic ma- nipulators in joint space via improved proximal policy optimization,

    Y . Wang and H. Kasaei, “Ippo: Obstacle avoidance for robotic ma- nipulators in joint space via improved proximal policy optimization,” arXiv preprint arXiv:2210.00803, 2022

  24. [24]

    Joint space control via deep reinforcement learning,

    V . Kumar, D. Hoeller, B. Sundaralingam, J. Tremblay, and S. Birch- field, “Joint space control via deep reinforcement learning,” in2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021

  25. [25]

    Robots,Universal Robots UR5e

    U. Robots,Universal Robots UR5e. [Online]. Available: https://www.universal-robots.com/

  26. [26]

    Robotics,Franka Emika Robot

    F. Robotics,Franka Emika Robot. [Online]. Available: https://franka.de/

  27. [27]

    Gymnasium: A standard interface for reinforcement learning environments,

    M. Towers, A. Kwiatkowski, J. Terry, J. U. Balis, G. De Cola, T. Deleu, M. Goul ˜ao, A. Kallinteris, M. Krimmel, A. KGet al., “Gymnasium: A standard interface for reinforcement learning environments,”arXiv preprint arXiv:2407.17032, 2024

  28. [28]

    Revisiting sparse rewards for goal-reaching reinforcement learning,

    G. Vasan, Y . Wang, F. Shahriar, J. Bergstra, M. Jagersand, and A. R. Mahmood, “Revisiting sparse rewards for goal-reaching reinforcement learning,” 2024

  29. [29]

    Accelerating reinforcement learning for reaching using continuous curriculum learning,

    S. Luo, H. Kasaei, and L. Schomaker, “Accelerating reinforcement learning for reaching using continuous curriculum learning,” in2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020

  30. [30]

    Training larger networks for deep reinforcement learning,

    K. Ota, D. K. Jha, and A. Kanezaki, “Training larger networks for deep reinforcement learning,”arXiv preprint arXiv:2102.07920, 2021