Pith. sign in

REVIEW 4 major objections 4 minor 31 references

Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a reward-decomposed critic, with separate heads for the sparse task reward and a dense grasp-potential reward, lets real-world robot policies train under 5-25x larger domain randomization than a state-of-the-art…

desk verdict Useful system paper that needs a hardware control and a corrected critic target before its headline gains carry weight. read the letter →

arxiv 2608.09762 v1 pith:FZYQNGC5 submitted 2026-08-10 cs.RO

Changhao Li , Yifang Zhang , Heng Zhang , Davide Torielli , Damiano Gasperini , Arturo Laurenzi , Luca Muratore , Arash Ajoudani
show 1 more author
Nikos Tsagarakis
This is my paper · ORCID
classification cs.RO
keywords real-worldreinforcementlearningrobotmanipulationcentralizedtrainingwithdecentralizedexecutionhybridrewardarchitectureactionspacehuman-in-the-loopRLdomainrandomizationmulti-headcritic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the main obstacle to real-world online reinforcement learning for manipulation is not exploration alone but critic instability: a single Q-function must regress a long-horizon sparse return from noisy RGB observations, and this becomes untenable as workspaces, object poses, and viewpoints vary widely. The proposed HARC framework combines centralized training with decentralized execution and a Hybrid Reward Architecture, splitting the critic into a task-value head and a grasp-value head, with a dense potential-based grasp reward derived from gripper torque or width. Under domain randomization roughly 5-25 times larger than the prior baseline, the framework raises success rates from 60% to 80% on tennis-ball pick-and-place, from 60% to 90% on banana pick-and-place, from 0% to 55% on pot reset, and from 25% to 95% on simulated block relocation, while using fewer expert demonstrations and interventions. If these results hold, real-world RL becomes practical for wider-object-placement manipulation tasks that current online RL systems handle only under tightly constrained setups.

What carries the argument

The central object is the reward-decomposed multi-head critic, which instantiates the identity $Q_\theta(s,a)=\sum_{k\in\{\text{task},\text{grasp}\}} Q^k_\theta(s,a)$. A potential-based grasp signal $\Phi(s)=\operatorname{clip}((x-x_{\min})/(x_{\max}-x_{\min}),0,1)$, with $x$ being gripper torque or width, produces $r_{\mathrm{grasp}}=\gamma\Phi(s')-\Phi(s)+P$. The heads share a feature trunk but have separate TD targets; actor updates use $Q_{\text{task}}+\lambda_g Q_{\text{grasp}}$, and the discrete gripper actor uses a categorical objective over the softmax logits. This decomposition turns one noisy long-horizon value regression into two simpler regressions, which the paper identifies as the reason critic training remains stable under real RGB observations.

What would settle it

Run the pot-reset task with a monolithic critic—same encoders, same actor decomposition, same rewards summed into one Q-function—and compare success rate and critic gradient norms against the multi-head version; if collapse and large gradient norms do not reappear, the paper's attribution of stability to reward decomposition is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that a centralized, reward-decomposed critic lets a continuous arm policy and a discrete gripper policy be trained directly on hardware with far larger domain randomization than previous human-in-the-loop RL, while using less expert data. Concretely, the authors report success-rate improvements from 60% to 80% on tennis ball pick-and-place, from 60% to 90% on banana pick-and-place, from 0% to 55% on pot reset, and from 25% to 95% on simulated block relocation, with total expert data falling from 80, 102, 132, 189 equivalent episodes to 69, 76, 111, 115. The explanation offered is variance reduction: decomposing the critic into task and grasp heads gives each head a simpler temporal-difference target and reduces the destabilizing effect of noisy RGB observations.

Load-bearing premise

The load-bearing premise is that the sparse task reward and the potential-based grasp reward depend on small, largely disjoint subsets of the observations, so splitting the critic reduces target variance; if the grasp potential shares the same informative features as the task reward, the framework's advantage over a single critic disappears.

Editorial extensions

If this is right

  • A training budget of about 160 wall-clock minutes on one GPU can reach 55-95% success on manipulation tasks with wide workspace and pose randomization, rather than only a few centimeters of variation.
  • Expert supervision becomes cheaper: the total equivalent-episode count of demonstrations and interventions falls from 80, 102, 132, 189 to 69, 76, 111, 115 across the four tasks.
  • The gripper policy can be trained from wrist-camera images alone, while the arm policy uses the side view, so heterogeneous per-actor observations are compatible with the shared critic.
  • Because execution remains decentralized, the learned policies require only local observations at deployment, preserving the actor-critic pipeline's practical interface.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to vary the amount of pixel noise in simulation and measure how much the success-rate gap between the multi-head and monolithic critic widens; the paper's variance-reduction story predicts a monotone relationship.
  • The same reward-decomposition recipe should transfer to other auxiliary signals—contact, proximity, or task-progress classifiers—provided each depends on a small, largely disjoint sensor subset, a regime the paper does not explore.
  • The stated disjointness condition implies that the method will lose its advantage if the grasp potential is computed from the same front-view features that drive the task reward; ablating that coupling would locate the boundaries of the framework.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes HARC, a real-world online RL framework that combines centralized training with decentralized execution (CTDE) and a Hybrid Reward Architecture (HRA). It separates a manipulation policy into a continuous Cartesian arm actor and a discrete gripper actor that share a centralized multi-head critic, with one value head for the sparse task reward and one for a potential-based grasp reward. The authors claim that this decomposition, together with a reformulated discrete SAC objective, substantially improves sample efficiency and success rate over the HIL-SERL baseline on tennis-ball pick-and-place, banana pick-and-place, pot reset, and simulated block relocation under larger domain randomization. The paper also contributes a compliant two-finger gripper design used in the real-world experiments.

Significance. If the RL objectives are formally correct and the hardware is matched across methods, the reported results would be a useful step toward real-world online RL under larger workspace randomization, an area where prior systems are often constrained to small perturbations. The paper includes a complete system description, real-robot deployment on two platforms, and a simulation benchmark, which are strengths. However, the central algorithmic equation is internally inconsistent as written, and the hardware used for the baseline is not explicitly controlled, so the headline empirical claims cannot currently be taken at face value. The HRA benefit is also asserted through a conditioning argument rather than demonstrated by ablations or measurements.

major comments (4)
  1. [III-C, Eq. (8)] The TD targets in Eq. (8) are internally inconsistent. V_k(s') is defined as an expectation over the next discrete action, V_k(s') = E_{a'_d ~ pi_psi}[min Q-bar_k(s', a')], but y_task then subtracts alpha_d log pi_psi(a'_d | s') outside that expectation. After the expectation, a'_d is a bound variable and cannot reappear outside; the resulting expression is not the standard discrete SAC target. The correct discrete SAC target would place the entropy term inside the expectation, e.g., y = r + gamma E_{a'_d ~ pi_psi}[min Q-bar(s', a'_d) - alpha_d log pi_psi(a'_d | s')], or, if a single sampled action is used, y = r + gamma (min Q-bar(s', a'_d) - alpha_d log pi_psi(a'_d | s')). Moreover, because the critic is defined over the hybrid action a = (a_c, a_d), V_k(s') must also specify how the next continuous action a'_c enters; as written no expectation over a'_c is shown. Table I additionally reports "Entropy backup False," which makes the presence of the entropy term in y_task confusing. This issue is load-bearing because the paper's central claim relies on the reformulated critic and actor objectives for the categorical gripper policy.
  2. [IV-B and Table II] The hardware used for the HIL-SERL baseline is not reported, which confounds the main comparison. Section IV-B states, "Our compliant palm improves handle grasping reliability beyond the original 1-DoF revolute joint gripper," and Section III-D introduces a new compliant gripper as a contribution. The paper never states whether the HIL-SERL baseline for the pot reset task, or for the custom-manipulator tasks, used the same compliant gripper and the same randomization. If the baseline ran with the original gripper, the Table II result of 0% vs 55% on pot reset, and the abstract's claim of "successfully accomplishing a task where the baseline consistently fails," could be due primarily to the mechanical advantage of the new gripper rather than to HARC. The authors should state explicitly which gripper was mounted for each baseline run and, if the hardware differed, rerun the baseline with the identical gripper.
  3. [III-C] The benefit of the HRA decomposition is asserted through a conditioning argument but is not verified. The paper correctly notes in Section III-C that HRA is most beneficial when "each reward component depends on only a small, largely disjoint subset of the state variables." It then asserts that the task reward is driven mainly by the front view and end-effector pose while the grasp reward is associated with the wrist camera and grasp state, but it provides no measurement, gradient analysis, or ablation supporting this disjointness. If the reward components are not largely disjoint in their dependence on the observations, the variance-reduction argument for the decomposed critic collapses. Since the method's advantage over the monolithic critic is central to the paper, the authors should either provide an ablation with a single shared critic under identical hardware, or report a quantitative proxy for the degree of reward-head disjointness.
  4. [IV-C and Table II] The performance comparison is presented as point estimates without confidence intervals or number of independent runs. Table II reports a single success rate per method and task, and the learning curves in Fig. 4 appear to be single runs. With 20 evaluation episodes, the binomial standard error at 60% success is about 11 percentage points, so the reported gaps are meaningful only if the evaluation protocol is stable and the runs are representative. The paper also does not isolate the contributions of CTDE, HRA, the discrete SAC actor, or the grasp reward weight lambda_g; an ablation study would be needed to attribute the improvement to the proposed components. These points do not invalidate the empirical claims, but they should be reported transparently for a real-world RL paper.
minor comments (4)
  1. [IV-A] The sentence beginning "Additional hyperparameters on All discrete gripper actors..." is duplicated and grammatically incomplete; please rewrite it as a single clear statement about the observations used by the discrete actors.
  2. [Table I] The target entropy is listed as -3 in the shared RLPD parameters and as 0.1 in the HARC parameters. The paper should clarify which target entropy applies to which actor and why the values differ, especially because the discrete actor has three actions.
  3. [III-C, Eq. (6)] The grasp reward is called a potential-based reward shaping (PBRS) term, but r_grasp(s,a) = gamma Phi(s') - Phi(s) + P is not pure PBRS unless P is part of the reward rather than a shaping term. If P is a gripper-switching penalty, write it as a negative term and define its sign explicitly; otherwise the reader cannot tell whether P encourages or discourages switching.
  4. [IV-C, Fig. 5] The conversion from "demo buffer transitions" to "equivalent episodes" is not defined clearly enough in the main text. Please specify the average transitions per episode and whether the comparison controls for episode length and intervention duration.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central derivation is an exact reward decomposition and the performance claims are benchmarked against an external baseline.

full rationale

The paper's core derivation is the HRA decomposition in Eqs. (5)-(8): r is written as r_task + r_grasp, and Q is written as Q_task + Q_grasp by linearity of expectation. This is mathematically exact for additive rewards and is not a fitted quantity renamed as a prediction; the grasp reward is a hand-set potential-based shaping term with fixed normalization bounds and weight lambda_g, and no parameter is fit to the success-rate targets it later reports. The empirical claims are comparisons against HIL-SERL [13], an external baseline, under common RLPD hyperparameters, so the headline improvements are not forced by construction. The remaining concerns are not circularity: the HRA benefit depends on an unstated disjointness assumption about reward components (a correctness risk), and the pot-reset result may be confounded by the new compliant gripper hardware if the baseline was not run with the same gripper (an experimental-control issue). A self-citation to the group's prior gripper [30] supports the hardware description but is not load-bearing for the algorithmic derivation. No step reduces to its own input by definition.

Assumptions & free parameters 3 free parameters · 3 assumptions · 1 invented entities

The method rests on standard RL machinery plus several domain-specific modeling choices. The free parameters are the reward weighting λ_g, the potential-normalization bounds, and the gripper-switching penalty; values are chosen by hand and not varied. The main axioms are the conditional independence of the two actors, the HRA disjoint-support condition, and the validity of potential-based shaping. One hardware entity, the compliant gripper, is introduced and demonstrated on the robot.

free parameters (3)
  • Grasp reward coefficient λ_g = 1.0
    Weight balancing the grasp Q-head in actor losses; reported in Table I without sensitivity analysis.
  • Potential normalization bounds x_min, x_max = Not given
    Clipping bounds for the grasp potential Φ(s), set by hand for each gripper and task.
  • Gripper-switching penalty P = Not given
    Added to the grasp reward; the magnitude is not reported in the paper.
assumptions (3)
  • domain assumption Continuous and discrete actions are conditionally independent given state, enabling the product form in Eq. (3).
    Equation (3) assumes π(a|s)=π_c(a_c|s)π_d(a_d|s); arm motion and gripper command are sampled independently at each control step.
  • domain assumption Each reward component depends on a small, largely disjoint subset of state variables, the HRA condition.
    Section III-C invokes this condition to justify the multi-head critic decomposition; no empirical verification is provided.
  • standard math Potential-based reward shaping preserves the optimal policy when the potential is a function of state and the shaping reward is γΦ(s')-Φ(s).
    Used to define the grasp reward in Eq. (6); this is a standard result, though the original PBRS citation is omitted.
invented entities (1)
  • Compliant two-finger gripper with tendon coupling independent evidence
    purpose: Improve grasping of diverse object shapes (fingertip and enveloping grasps) without adding extra actuators or action dimensions.
    The gripper is physically built and shown in Fig. 3, with videos promised on the project website; it exists independently of the RL framework.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition." pith.science (2026). https://pith.science/paper/FZYQNGC5

@misc{pith2026260809762,
  author       = {Pith},
  title        = {Pith review of: Efficient Real-World Online Reinforcement Learning for Robot Manipulation via Centralized Training and Critic Decomposition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZYQNGC5}},
  note         = {Machine review of arXiv:2608.09762}
}
read the original abstract

Real-world online reinforcement learning (RL) provides a promising approach for training robotic manipulation policies directly in the physical world, avoiding the sim-to-real gap and enabling continuous policy refinement through human-in-the-loop interaction. Recent methods have demonstrated sample-efficient learning through human intervention but remain limited to small randomization ranges and encounter challenges with the non-stationarity induced by concurrently training multiple agents. To address these limitations, we introduce a unified framework that combines centralized training with decentralized execution (CTDE) and a Hybrid Reward Architecture (HRA). This enables multiple actors to share a centralized multi-head critic. The critic is decomposed into task and grasp heads, corresponding to the sparse task reward and a potential-based grasping reward, respectively. We accordingly reformulate the critic and actor objectives to exploit the decomposed Q-values while explicitly accounting for the categorical action distribution of the discrete gripper policy. Experimental results demonstrate that the proposed framework substantially improves both sample efficiency and policy performance. We validate our approach on two robotic arms and a simulated humanoid robot across tennis ball and banana pick-and-place, pot reset, and simulated block relocation tasks under dimension-wise domain randomization, approximately 5-25x larger than those considered in prior work. Compared with a state-of-the-art baseline, our method improves the success rate from 60% to 80% on tennis ball pick-and-place, from 60% to 90% on banana pick-and-place, and from 25% to 95% on simulated block relocation, while also successfully accomplishing a task where the baseline consistently fails. Videos and more details are available at our project website: https://hil-harc.github.io/.

Figures

Figures reproduced from arXiv: 2608.09762 by the authors.

Figure 1
Figure 1. Experimental tasks overview. (a) A pick-and-place tennis ball task with a custom manipulator; (b) A pick-and-place banana task with a custom manipulator and a compliant gripper; (c) A pot reset task with Franka arm; (d) A block relocation task with Unitree G1 and robotiq gripper sample-efficient real-world online RL with human-in-the￾loop (HIL) intervention has attracted growing interest, in￾cluding for embodied-AI … view at source ↗
Figure 2
Figure 2. Overview of training framework. Our online RL framework employs a reward-decomposed multi-head critic with multi-agent CTDE training. During training, a human expert provides demonstrations and corrective interventions via teleoperation, while intervened transitions replace the original policy actions and are stored alongside autonomous experience in a shared replay buffer. Training is asynchronous: a cloud-based le… view at source ↗
Figure 3
Figure 3. Compliant gripper based on the modular 1-DoF revolute joint [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Learning curves for our method and HIL-SERL [13] across all tasks, showing 20-episode running averages of success rate, intervention rate, and [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Comparison of demo buffer transitions across tasks. Light green bars [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

31 extracted references · 11 canonical work pages

  1. [1]

    A review of learning-based dynamics models for robotic manipulation |Science Robotics

    “A review of learning-based dynamics models for robotic manipulation |Science Robotics.” [Online]. Available: https://www.science.org/do i/10.1126/scirobotics.adt1497

    doi:10.1126/scirobotics.adt1497

  2. [2]

    Learning Con- tinuous Control Actions for Robotic Grasping with Reinforcement Learning,

    A. A. Shahid, L. Roveda, D. Piga, and F. Braghin, “Learning Con- tinuous Control Actions for Robotic Grasping with Reinforcement Learning,” in2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Oct. 2020, pp. 4066–4072

    work page 2020

  3. [3]

    General- Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors,

    W. Chen, J. Xu, F. Xiang, X. Yuan, H. Su, and R. Chen, “General- Purpose Sim2Real Protocol for Learning Contact-Rich Manipulation With Marker-Based Visuotactile Sensors,”IEEE Transactions on Robotics, pp. 1–18, 2024, conference Name: IEEE Transactions on Robotics. [Online]. Available: https://ieeexplore.ieee.org/abstract/doc ument/10388459

    work page arXiv 2024

  4. [4]

    Towards bridging the gap: Systematic sim-to-real transfer for diverse legged robots,

    F. Bjelonic, F. Tischhauser, and M. Hutter, “Towards bridging the gap: Systematic sim-to-real transfer for diverse legged robots,”arXiv preprint arXiv:2509.06342, 2025

    arXiv 2025

  5. [5]

    Srl-vic: A variable stiffness-based safe reinforcement learning for contact-rich robotic tasks,

    H. Zhang, G. Solak, G. J. G. Lahr, and A. Ajoudani, “Srl-vic: A variable stiffness-based safe reinforcement learning for contact-rich robotic tasks,”IEEE Robotics and Automation Letters, vol. 9, no. 6, pp. 5631–5638, 2024

    work page 2024

  6. [6]

    Deep reinforcement learning for robotics: A survey of real-world successes,

    C. Tang, B. Abbatematteo, J. Hu, R. Chandra, R. Mart ´ın-Mart´ın, and P. Stone, “Deep reinforcement learning for robotics: A survey of real-world successes,”Annual Review of Control, Robotics, and Autonomous Systems, vol. 8, no. 1, pp. 153–188, 2025

    2025

  7. [7]

    Improving vision-language-action model with online reinforcement learning,

    Y . Guo, J. Zhang, X. Chen, X. Ji, Y .-J. Wang, Y . Hu, and J. Chen, “Improving vision-language-action model with online reinforcement learning,” in2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 15 665–15 672

    work page 2025

  8. [8]

    Hi-wm: Human-in-the-world-model for scalable robot post-training,

    Y . Li, Z. Zhou, Y . Chen, Y . Guo, J. Liu, S. Zhang, J. Chen, and Y . Zhu, “Hi-wm: Human-in-the-world-model for scalable robot post-training,” arXiv preprint arXiv:2604.21741, 2026

    arXiv 2026

Show all 31 references
  1. [9]

    Rl-100: Performant robotic manipulation with real-world reinforcement learning,

    K. Lei, H. Li, D. Yu, Z. Wei, L. Guo, Z. Jiang, Z. Wang, S. Liang, and H. Xu, “Rl-100: Performant robotic manipulation with real-world reinforcement learning,”arXiv preprint arXiv:2510.14830, 2025

    2025

  2. [10]

    Rlinf-user: A unified and extensible system for real-world online policy learning in embodied ai,

    H. Zang, S. Yu, H. Lin, T. Zhou, Z. Huang, Z. Guo, X. Xu, J. Zhou, Y . Sheng, S. Zhanget al., “Rlinf-user: A unified and extensible system for real-world online policy learning in embodied ai,”arXiv preprint arXiv:2602.07837, 2026

    2026

  3. [11]

    Towards Autonomous Reinforcement Learning for Real-World Robotic Manipulation With Large Language Models,

    N. Turcato, M. Iovino, A. Synodinos, A. Dalla Libera, R. Carli, and P. Falco, “Towards Autonomous Reinforcement Learning for Real-World Robotic Manipulation With Large Language Models,” IEEE Robotics and Automation Letters, vol. 10, no. 9, pp. 8850–8857, Sep. 2025. [Online]. A...

    2025

  4. [12]

    Serl: A software suite for sample- efficient robotic reinforcement learning,

    J. Luo, Z. Hu, C. Xu, Y . L. Tan, J. Berg, A. Sharma, S. Schaal, C. Finn, A. Gupta, and S. Levine, “Serl: A software suite for sample- efficient robotic reinforcement learning,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 16 961–16 969

    2024

  5. [13]

    Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning,

    J. Luo, C. Xu, J. Wu, and S. Levine, “Precise and dexterous robotic manipulation via human-in-the-loop reinforcement learning,”Science Robotics, vol. 10, no. 105, p. eads5033, 2025

    2025

  6. [14]

    Vla-rl: Towards masterful and general robotic ma- nipulation with scalable reinforcement learning,

    G. Lu, W. Guo, C. Zhang, Y . Zhou, H. Jiang, Z. Gao, Y . Tang, and Z. Wang, “Vla-rl: Towards masterful and general robotic ma- nipulation with scalable reinforcement learning,”arXiv preprint arXiv:2505.18719, 2025

    2025 arXiv

  7. [15]

    Reinflow: Fine-tuning flow matching policy with online reinforcement learning,

    T. Zhang, C. Yu, S. Su, and Y . Wang, “Reinflow: Fine-tuning flow matching policy with online reinforcement learning,”Advances in Neural Information Processing Systems, vol. 38, pp. 106 282–106 319, 2026

    2026

  8. [16]

    Hybrid reward architecture for reinforcement learning,

    H. Van Seijen, M. Fatemi, J. Romoff, R. Laroche, T. Barnes, and J. Tsang, “Hybrid reward architecture for reinforcement learning,” Advances in neural information processing systems, vol. 30, 2017

    2017

  9. [17]

    Efficient online reinforcement learning with offline data,

    P. J. Ball, L. Smith, I. Kostrikov, and S. Levine, “Efficient online reinforcement learning with offline data,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 1577–1594

    2023

  10. [18]

    Deep reinforcement learning in param- eterized action space,

    M. Hausknecht and P. Stone, “Deep reinforcement learning in param- eterized action space,”arXiv preprint arXiv:1511.04143, 2015

    2015 arXiv

  11. [19]

    Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space,

    J. Xiong, Q. Wang, Z. Yang, P. Sun, L. Han, Y . Zheng, H. Fu, T. Zhang, J. Liu, and H. Liu, “Parametrized deep q-networks learning: Reinforcement learning with discrete-continuous hybrid action space,” arXiv preprint arXiv:1810.06394, 2018

    arXiv

  12. [20]

    Multi-pass q-networks for deep reinforcement learning with parameterised action spaces,

    C. J. Bester, S. D. James, and G. D. Konidaris, “Multi-pass q-networks for deep reinforcement learning with parameterised action spaces,” arXiv preprint arXiv:1905.04388, 2019

    1905 arXiv

  13. [21]

    Multi-agent actor-critic for mixed cooperative-competitive environments,

    R. Lowe, Y . I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mor- datch, “Multi-agent actor-critic for mixed cooperative-competitive environments,”Advances in neural information processing systems, vol. 30, 2017

    2017

  14. [22]

    The surprising effectiveness of ppo in cooperative multi-agent games,

    C. Yu, A. Velu, E. Vinitsky, J. Gao, Y . Wang, A. Bayen, and Y . Wu, “The surprising effectiveness of ppo in cooperative multi-agent games,”Advances in neural information processing systems, vol. 35, pp. 24 611–24 624, 2022

    2022

  15. [23]

    Asynchronous actor-critic for multi-agent reinforcement learning,

    Y . Xiao, W. Tan, and C. Amato, “Asynchronous actor-critic for multi-agent reinforcement learning,”Advances in Neural Information Processing Systems, vol. 35, pp. 4385–4400, 2022

    2022

  16. [24]

    Action decoupled sac reinforcement learning with discrete-continuous hybrid action spaces,

    Y . Xu, Y . Wei, K. Jiang, L. Chen, D. Wang, and H. Deng, “Action decoupled sac reinforcement learning with discrete-continuous hybrid action spaces,”Neurocomputing, vol. 537, pp. 141–151, 2023

    2023

  17. [25]

    Effective multi-agent deep reinforcement learning control with relative entropy regularization,

    C. Miao, Y . Cui, H. Li, and X. Wu, “Effective multi-agent deep reinforcement learning control with relative entropy regularization,” IEEE Transactions on Automation Science and Engineering, vol. 22, pp. 3704–3718, 2024

    2024

  18. [26]

    Soft actor-critic algorithms and applications. arxiv 2018,

    T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V . Ku- mar, H. Zhu, A. Gupta, P. Abbeelet al., “Soft actor-critic algorithms and applications. arxiv 2018,”arXiv preprint arXiv:1812.05905, 1812

    2018 arXiv

  19. [27]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778

    2016

  20. [28]

    An introduction to centralized training for decentralized execution in cooperative multi-agent reinforcement learning,

    C. Amato, “An introduction to centralized training for decentralized execution in cooperative multi-agent reinforcement learning,”arXiv preprint arXiv:2409.03052, 2024

    2024 arXiv

  21. [29]

    Soft actor-critic for discrete action settings,

    P. Christodoulou, “Soft actor-critic for discrete action settings,”arXiv preprint arXiv:1910.07207, 2019

    1910 arXiv

  22. [30]

    A high-force gripper with embedded multimodal sensing for powerful and perception driven grasping,

    E. Del Bianco, D. Torielli, F. Rollo, D. Gasperini, A. Laurenzi, L. Baccelliere, L. Muratore, M. Roveri, and N. G. Tsagarakis, “A high-force gripper with embedded multimodal sensing for powerful and perception driven grasping,” in2024 IEEE-RAS 23rd International Conference on ...

    2024

  23. [31]

    Root mean square layer normalization,

    B. Zhang and R. Sennrich, “Root mean square layer normalization,” Advances in neural information processing systems, vol. 32, 2019

    2019

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.