Pith. sign in

REVIEW 3 major objections 5 minor 44 references

Enhancing Exploration with Diffusion Policies in Hybrid Off-Policy RL: Application to Non-Prehensile Manipulation

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A hybrid diffusion-policy actor-critic with a maximum-entropy objective improves exploration and raises real-robot success in non-prehensile manipulation to 72%.

desk verdict Solid diffusion-policy RL method with a real reporting problem: the 53→72 headline aggregates planar and 6D runs; the 6D-only gain is not statistically significant. read the letter →

arxiv 2411.14913 v2 pith:QF3DGSQF submitted 2024-11-22 cs.RO

Huy Le , Tai Hoang , Miroslav Gabriel , Gerhard Neumann , Ngo Anh Vien This is my paper · ORCID
classification cs.RO
keywords reinforcementlearningdiffusionpoliciesnon-prehensilemanipulationhybridactionspacemaximumentropyRLsim2realtransferconsistencymodelssoftactor-critic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that a robot learning non-prehensile manipulation—pushing, flipping, aligning 6D poses—explores more effectively when the continuous part of its hybrid action policy is a diffusion model and the training objective includes a maximum-entropy regularizer. The proposed algorithm, HyDo, selects discrete contact points by Q-value maximization and generates continuous motion parameters by sampling a diffusion policy, with an entropy term over the full denoising chain derived as a lower bound via structured variational inference. In simulation, HyDo beats the HACMan baseline on unseen object categories and instances in 6D pose alignment. In zero-shot transfer to a real Franka Panda robot, the average success rate over planar and 6D goals rises from 53% for HACMan to 68% for HyDo and 72% for HyDo+CM, while behavior-diversity metrics also increase. If this holds, the payoff is a general recipe for diverse exploration in hybrid-action robotic manipulation that transfers to unseen objects and real hardware.

What carries the argument

The object that carries the argument is the HyDo actor-critic objective, in which the continuous motion-parameter policy $\pi_m$ is a diffusion model (a policy that generates actions by denoising random noise over $K$ steps) and the discrete contact-point policy $\pi_{\text{loc}}$ is a softmax over per-point Q-values. The load-bearing identity is the lower-bound maximum-entropy objective $J_\pi(\theta) = \sum_t \mathbb{E}_{s_t,a^{0:K}_t\sim\pi_\theta}[ r(s_t,a^0_t) - \alpha \sum_{k=0}^K \log \pi_\theta(a^{k-1}_t | a^k_t, k, s_t) ]$, obtained from a variational lower bound on $\log p(O_{1:T})$ with a factorized variational distribution $q$ over the denoising chain. This one expression supplies the actor loss (via reparameterized gradients) and the entropy term in the soft Bellman target, and the same entropy logic extends to the location policy through $\log \pi_{\text{loc}}$. The consistency-model variant replaces the DDPM sampler with a consistency-model inference step, reusing the same objective and update rules.

What would settle it

Run HyDo with the entropy term replaced by a direct sample-based estimate of the entropy of the executed action distribution $\pi(a^0|s)$ (e.g., histogram or k-NN entropy over many sampled actions at fixed states); if success rates do not drop or the ranking of policies does not match the path-entropy surrogate, the mechanism attributed to the entropy term is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that the combination of a diffusion-model motion policy and a maximum-entropy objective—both applied inside a hybrid actor-critic that already chooses discrete contact points—produces exploration that is measurably more diverse and more successful for 6D non-prehensile manipulation. The new objective is $J_\pi(\theta) = \sum_t \mathbb{E}_{s_t,a^{0:K}_t\sim\pi_\theta}[ r(s_t,a^0_t) - \alpha \sum_{k=0}^K \log \pi_\theta(a^{k-1}_t | a^k_t, k, s_t) ]$, which the authors derive as a lower bound on the maximum-reward likelihood using structured variational inference with a variational distribution $q$ that factorizes over the denoising chain. This objective feeds both the reparameterized actor gradient and the soft Bellman target, while the location policy keeps its softmax-over-Q form. The authors show that HyDo and HyDo+CM (a consistency-model variant) outperform HACMan and its diffusion/consistency ablations on unseen categories and instances in simulation, achieve higher behavior entropy on fixed tasks, and raise the real-robot success average from 53% to 72% in zero-shot transfer.

Load-bearing premise

The claim rests on assuming that the summed log-probabilities of the diffusion denoising steps measure the true diversity of the action the robot executes, an equality the paper derives as a lower bound but never proves.

Editorial extensions

If this is right

  • On simulated 6D pose alignment, HyDo achieves higher interquartile-mean success than HACMan on unseen categories (0.843 vs 0.760) and unseen instances (0.884 vs 0.818).
  • Adding entropy regularization to diffusion or consistency-model policies improves success by roughly 10 to 15 percentage points over the unregularized diffusion/consistency baselines HACMan+Diff and HACMan+CM.
  • In zero-shot real-robot transfer, the diffusion-based entropy-regularized variants outperform HACMan on both planar and 6D goals, with HyDo+CM reaching 72% average success.
  • The consistency-model variant HyDo+CM matches or exceeds the diffusion variant's success while needing fewer denoising steps; at $K=5$ it reaches 0.787 success with 7.51 ms inference time versus 0.684 for HyDo.
  • All HyDo variants show higher behavior entropy than HACMan and HyDo (w/o Diff), indicating more diverse action sequences under identical start and goal poses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's headline 53% to 72% averages planar and 6D real-robot trials; reading Table II directly, the 6D-only gain is from 54% (HACMan) to 62% (HyDo) and 64% (HyDo+CM).
  • If the variational lower bound is loose, the diversity gains may reflect diffusion-policy expressiveness or implicit regularization rather than true maximum-entropy exploration; a direct test is to replace the path-entropy term with a sample-based estimate of the true action entropy.
  • The same actor-critic pattern—Q-maximized discrete selection plus entropy-regularized diffusion sampling for continuous parameters—should transfer to other hybrid-action robot tasks such as grasping, tool use, or any contact-rich manipulation where motion parameters are continuous.
  • The consistency-model result suggests that diffusion-policy exploration in online RL can be made cheap enough for real-time control, so pairing entropy-regularized diffusion with faster samplers is a promising direction for closed-loop manipulation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes HyDo, a hybrid off-policy maximum-entropy reinforcement learning algorithm for non-prehensile manipulation. The continuous motion-parameter policy is modeled as a diffusion or consistency model, while discrete contact-point selection is handled by a Q-value-based softmax policy. The authors derive a surrogate maximum-entropy objective via structured variational inference and evaluate HyDo in simulation and zero-shot sim2real tasks, reporting improved success rates and behavior diversity compared with HACMan baselines.

Significance. If the central claims hold, the work is a useful step for non-prehensile manipulation: it combines diffusion/consistency policies with maximum-entropy exploration in a hybrid action space, and the simulation study uses IQM with confidence intervals, which is a strength. The qualitative diversity analysis and ablations over denoising steps also provide useful evidence. However, the headline real-world result is misreported, and the theoretical identification of the path-entropy term with action entropy is not established; both issues affect the paper's main claims.

major comments (3)
  1. [Abstract, Section I, Section V-B.2, Table II] The claim in the abstract and introduction that HyDo improves success from 53% to 72% 'on a real-world 6D pose alignment task' is not supported by Table II. The 53% figure is HACMan's combined planar+6D total (26/50 + 27/50 = 53/100) and the 72% figure is HyDo+CM's combined total (40/50 + 32/50 = 72/100). For the 6D task alone, HACMan achieves 27/50 (54%) and HyDo+CM achieves 32/50 (64%); a two-sided Fisher exact test on these counts gives p ≈ 0.40, so the word 'significantly' is not justified for the 6D-specific comparison. Please correct the abstract and introduction, report planar and 6D results separately with exact counts and statistical tests, and do not describe the combined average as a 6D-task result.
  2. [Section IV-B, Eq. (4)] The regularizer Σ_{k=0}^K log πθ(a_{k-1} | a_k, k, s) is a sum of reverse-chain log-densities, not the entropy H(πθ(a0|s)) of the action distribution πθ(a0|s). The structured variational inference derivation lower-bounds log p(O1:T) by an expectation of reward minus a path log-density term; it does not show that this path term equals or bounds -log πθ(a0|s) or H(πθ(a0|s)). Thus the claim that the method optimizes a principled maximum-entropy objective for the diffusion policy is not established. Please either provide a proof that the path entropy is a valid surrogate for the action entropy, or recast the objective as a heuristic regularizer and temper the theoretical contribution accordingly.
  3. [Section V-B.2, Table II] The real-robot evaluation uses only 10 trials per object per condition and 50 trials per condition overall. Even for the combined planar+6D comparison, no confidence intervals or significance tests are reported, and for the 6D-only comparison the observed gap (27/50 vs 32/50) is within sampling error. The manuscript should report exact binomial confidence intervals and appropriate tests for each task configuration, and should avoid the phrase 'significantly improved' unless a test supports it.
minor comments (5)
  1. [Section IV-B, Eq. (4)] The summation in Eq. (4) runs from k=0 to K, but the term for k=0 refers to log πθ(a_{-1} | a_0, 0, s), which is undefined; the summation should start at k=1, consistent with Eq. (6) and the variational derivation.
  2. [Section V-B.2] The sentence reporting average success rates of 53%, 64%, 68%, and 72% should state explicitly that these numbers are averaged over planar and 6D goals combined; the table shows separate planar and 6D columns, so the current text invites the misinterpretation that the 6D task alone improved from 53% to 72%.
  3. [Section V-C] The behavior-entropy formula uses log_{|B|} π(β), which is a normalized entropy, but the notation is not defined; please clarify that this is normalized by log|B| so that the reported values are comparable across tasks with different |B|.
  4. [Global] There are several typos and stylistic issues, including 'varity' in Section I and 'intracble' in Section IV-B; the paper would benefit from a careful proofreading pass.
  5. [Section V and Algorithm 1] The manuscript does not provide hyperparameter values for α1, α2, β, and the number of denoising steps K for the main experiments, nor does it include code or a reproducibility statement; adding these would strengthen the empirical contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the maximum-entropy objective is a standard ELBO bound and the central empirical claims are measured against external baselines.

full rationale

The paper's derivation chain is self-contained and does not reduce to its own inputs. The maximum-entropy diffusion-policy objective in Eq. 4 is a conventional soft actor-critic-style loss in which the intractable action density of a diffusion policy is replaced by the sum of per-step denoising log-likelihoods. The structured variational inference argument in Section IV-B is a standard ELBO derivation: it applies Jensen's inequality and introduces a variational distribution q that is parameterized by the policy πθ, producing a lower bound on the reward likelihood that contains the same path-entropy regularizer. This is a mathematical identity-style bound, not a fitted parameter renamed as a prediction. The hybrid framework builds on HACMan [7] (external) and Feldman et al. [6]; although [6] shares an author, the central empirical claim is evaluated against HACMan and HACMan+Diff/CM baselines, not against the authors' prior results. The diversity metric follows Jia et al. [10] with overlapping authors, but the diversity values are computed from the authors' own rollouts and compared across methods. The abstract's '53% to 72%' phrasing aggregates planar and 6D real-robot trials in Table II, and the 6D-only improvement is not reported with a significance test; this is a reporting/correctness concern, not circularity. The main unverified modeling choice—using path entropy Σ_k log pθ(a_{k-1}|a_k,s) as a surrogate for action entropy H(π(a0|s))—is an assumption with no proof of equality or lower bound, but it is not a reduction by construction. No step of the derivation is equivalent, by definition, to its own input.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central theoretical derivation rests on several assumptions that are stated but not proven. First, the control-as-inference factorization p(O_t|s_t,a_t)=exp(r(s_t,a_t)/α) is standard but domain-specific. Second, the variational distribution is taken to be the diffusion policy path, and the path entropy is used in place of the action entropy; this is an approximation the paper does not justify in detail. Third, the discrete location policy is a Gibbs softmax over Q-values taken from HACMan. Fourth, the reparameterization trick is assumed to yield a valid gradient estimator through the denoising chain. No new physical entities are introduced, and the only free parameters are the entropy coefficients and softmax temperature, whose values are not reported.

free parameters (4)
  • Entropy temperature α1 for location policy = not reported
    Balances exploration vs exploitation for discrete contact-point selection; the paper says 'Adjust temperature α' in Algorithm 1 but does not report the value or tuning procedure.
  • Entropy coefficient α2 for motion policy = not reported
    Balances the diffusion-path entropy term in the actor and critic objectives; value not stated.
  • Softmax temperature β for location policy = not reported
    Controls the sharpness of the discrete location policy (Eq. 3), taken from HACMan; value not reported.
  • Number of denoising steps K = evaluated at 5, 10, 20, 50
    The main results do not state which K is used; the ablation in Tab IV shows performance varies with K, so K is a free parameter affecting the central claim.
assumptions (4)
  • domain assumption The optimality variable factorization p(O_t|s_t,a_t) = exp(r(s_t,a_t)/α) with temperature α
    Standard control-as-inference assumption from Levine 2018, used in the lower-bound derivation in Section IV-B.
  • ad hoc to paper The variational distribution q factorizes as the diffusion policy path, and the entropy of the diffusion path approximates the entropy of the action a0
    The derivation replaces H(π(a0|s)) with Σ_k log q(a_{k-1}|a_k,s), which is not generally equal to the action entropy; this is a modeling approximation stated in Section IV-B.
  • domain assumption The location policy is a Gibbs softmax over Q-values with temperature β (Eq. 3)
    Taken from HACMan; the exploration over discrete points is controlled by this softmax.
  • standard math The reparameterization trick applies to the diffusion denoising steps, enabling gradient backpropagation through the sampling chain
    Used in Eq. 6 to derive the policy gradient.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enhancing Exploration with Diffusion Policies in Hybrid Off-Policy RL: Application to Non-Prehensile Manipulation." pith.science (2026). https://pith.science/paper/QF3DGSQF

@misc{pith2026241114913,
  author       = {Pith},
  title        = {Pith review of: Enhancing Exploration with Diffusion Policies in Hybrid Off-Policy RL: Application to Non-Prehensile Manipulation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QF3DGSQF}},
  note         = {Machine review of arXiv:2411.14913}
}
read the original abstract

Learning diverse policies for non-prehensile manipulation is essential for improving skill transfer and generalization to out-of-distribution scenarios. In this work, we enhance exploration through a two-fold approach within a hybrid framework that tackles both discrete and continuous action spaces. First, we model the continuous motion parameter policy as a diffusion model, and second, we incorporate this into a maximum entropy reinforcement learning framework that unifies both the discrete and continuous components. The discrete action space, such as contact point selection, is optimized through Q-value function maximization, while the continuous part is guided by a diffusion-based policy. This hybrid approach leads to a principled objective, where the maximum entropy term is derived as a lower bound using structured variational inference. We propose the Hybrid Diffusion Policy algorithm (HyDo) and evaluate its performance on both simulation and zero-shot sim2real tasks. Our results show that HyDo encourages more diverse behavior policies, leading to significantly improved success rates across tasks - for example, increasing from 53% to 72% on a real-world 6D pose alignment task. Project page: https://leh2rng.github.io/hydo

Figures

Figures reproduced from arXiv: 2411.14913 by the authors.

Figure 1
Figure 1. Overview of HyDo: The network takes point clouds, goal flow, and segmentation (indicating object and background points) as input. These are passed [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Set of five objects used for real robot evaluations. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. A real robot task showcases the multi-modalities of action sequences, [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Left: A pushing task with 24 behavioral modes. Right: Behavior entropy of different methodologies [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Left: A push and align task with 2 behavioral modes. Right: Pareto plot between Entropy and Success Rates. where B is a set of task-specific behavior descriptors. We first evaluate our method on a task similar to [10], where the robot pushes a cube across a red line (s…
Figure 6
Figure 6. Figure 6: Cluttered environment. The manipulated object is shown in color, the [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

44 extracted references · 24 canonical work pages

  1. [1]

    More than a million ways to be pushed. a high-fidelity experimental dataset of planar pushing,

    K.-T. Yu, M. Bauza, N. Fazeli, and A. Rodriguez, “More than a million ways to be pushed. a high-fidelity experimental dataset of planar pushing,” in 2016 IEEE/RSJ international conference on intelligent robots and systems (IROS) . IEEE, 2016, pp. 30–37

    work page 2016

  2. [2]

    Universal manipulation policy network for articulated objects,

    Z. Xu, Z. He, and S. Song, “Universal manipulation policy network for articulated objects,” IEEE robotics and automation letters , vol. 7, no. 2, pp. 2447–2454, 2022

    2022

  3. [3]

    Contact mode guided motion planning for quasidynamic dexterous manipulation in 3d,

    X. Cheng, E. Huang, Y . Hou, and M. T. Mason, “Contact mode guided motion planning for quasidynamic dexterous manipulation in 3d,” in 2022 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2022, pp. 2730–2736

    work page 2022

  4. [4]

    Robust execution of contact-rich motion plans by hybrid force-velocity control,

    Y . Hou and M. T. Mason, “Robust execution of contact-rich motion plans by hybrid force-velocity control,” in 2019 International Conference on Robotics and Automation (ICRA) . IEEE, 2019

    work page 2019

  5. [5]

    Where2act: From pixels to actions for articulated 3d objects,

    K. Mo, L. J. Guibas, M. Mukadam, A. Gupta, and S. Tulsiani, “Where2act: From pixels to actions for articulated 3d objects,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 6813–6823

    2021

  6. [6]

    A hybrid ap- proach for learning to shift and grasp with elaborate motion primitives,

    Z. Feldman, H. Ziesche, N. A. Vien, and D. D. Castro, “A hybrid ap- proach for learning to shift and grasp with elaborate motion primitives,” in 2022 International Conference on Robotics and Automation, ICRA . IEEE, 2022, pp. 6365–6371

    work page 2022

  7. [7]

    HACMan: Learning hybrid actor-critic maps for 6d non-prehensile manipulation,

    W. Zhou, B. Jiang, F. Yang, C. Paxton, and D. Held, “HACMan: Learning hybrid actor-critic maps for 6d non-prehensile manipulation,” in Conference on Robot Learning (CoRL) , vol. 229. PMLR, 2023

    work page 2023

  8. [8]

    Neural probabilistic motor primitives for humanoid control,

    J. Merel, L. Hasenclever, A. Galashov, A. Ahuja, V . Pham, G. Wayne, Y . W. Teh, and N. Heess, “Neural probabilistic motor primitives for humanoid control,” in International Conference on Learning Represen- tations, 2018

    work page 2018

Show all 44 references
  1. [9]

    One solution is not all you need: Few-shot extrapolation via structured maxent rl,

    S. Kumar, A. Kumar, S. Levine, and C. Finn, “One solution is not all you need: Few-shot extrapolation via structured maxent rl,” Advances in Neural Information Processing Systems , 2020

    2020

  2. [10]

    Towards diverse behaviors: A benchmark for imitation learning with human demonstrations,

    X. Jia, D. Blessing, X. Jiang, M. Reuss, A. Donat, R. Lioutikov, and G. Neumann, “Towards diverse behaviors: A benchmark for imitation learning with human demonstrations,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  3. [11]

    Generative modeling by estimating gradients of the data distribution,

    Y . Song and S. Ermon, “Generative modeling by estimating gradients of the data distribution,” Advances in neural information processing systems, vol. 32, 2019

    2019

  4. [12]

    Denoising diffusion probabilistic models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” Advances in neural information processing systems , 2020

    2020

  5. [13]

    Consistency models as a rich and efficient policy class for reinforcement learning,

    Z. Ding and C. Jin, “Consistency models as a rich and efficient policy class for reinforcement learning,” in The Twelfth International Conference on Learning Representations , 2023

    2023

  6. [14]

    Diffusion policy: Visuomotor policy learning via action diffusion,

    C. Chi, S. Feng, Y . Du, Z. Xu, E. Cousineau, B. Burchfiel, and S. Song, “Diffusion policy: Visuomotor policy learning via action diffusion,” arXiv preprint arXiv:2303.04137 , 2023

    2023 arXiv

  7. [15]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” in International conference on machine learning . PMLR, 2018, pp. 1861–1870

    2018

  8. [16]

    Diffusion policies as an expres- sive policy class for offline reinforcement learning,

    Z. Wang, J. J. Hunt, and M. Zhou, “Diffusion policies as an expres- sive policy class for offline reinforcement learning,” in The Eleventh International Conference on Learning Representations , 2023

    2023

  9. [17]

    Idql: Implicit q-learning as an actor-critic method with diffusion policies,

    P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine, “Idql: Implicit q-learning as an actor-critic method with diffusion policies,” arXiv preprint arXiv:2304.10573 , 2023

    2023 arXiv

  10. [18]

    Contrastive energy prediction for exact energy-guided diffusion sampling in offline rein- forcement learning,

    C. Lu, H. Chen, J. Chen, H. Su, C. Li, and J. Zhu, “Contrastive energy prediction for exact energy-guided diffusion sampling in offline rein- forcement learning,” in International Conference on Machine Learning . PMLR, 2023, pp. 22 825–22 855

    2023

  11. [19]

    Reasoning with latent diffusion in offline reinforcement learning,

    S. Venkatraman, S. Khaitan, R. T. Akella, J. Dolan, J. Schneider, and G. Berseth, “Reasoning with latent diffusion in offline reinforcement learning,” in The Twelfth International Conference on Learning Repre- sentations, 2024

    2024

  12. [20]

    Learning multimodal behaviors from scratch with diffusion policy gra- dient,

    Z. Li, R. Krohn, T. Chen, A. Ajay, P. Agrawal, and G. Chalvatzaki, “Learning multimodal behaviors from scratch with diffusion policy gra- dient,” in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  13. [21]

    Goal conditioned imitation learning using score-based diffusion policies,

    M. Reuss, M. Li, X. Jia, and R. Lioutikov, “Goal conditioned imitation learning using score-based diffusion policies,” in Robotics: Science and Systems, 2023

    2023

  14. [22]

    Imitating human behaviour with diffusion models,

    T. Pearce, T. Rashid, A. Kanervisto, D. Bignell, M. Sun, R. Georgescu, S. V . Macua, S. Z. Tan, I. Momennejad, K. Hofmann et al. , “Imitating human behaviour with diffusion models,” arXiv preprint arXiv:2301.10677, 2023

    2023 arXiv

  15. [23]

    Offline reinforcement learning via high-fidelity generative behavior modeling,

    H. Chen, C. Lu, C. Ying, H. Su, and J. Zhu, “Offline reinforcement learning via high-fidelity generative behavior modeling,” inThe Eleventh International Conference on Learning Representations , 2022

    2022

  16. [24]

    Diffcps: Diffusion model based constrained policy search for offline reinforcement learning,

    L. He, L. Zhang, J. Tan, and X. Wang, “Diffcps: Diffusion model based constrained policy search for offline reinforcement learning,” arXiv preprint arXiv:2310.05333, 2023

    2023 arXiv

  17. [25]

    Reinforcement learning by reward-weighted regression for operational space control,

    J. Peters and S. Schaal, “Reinforcement learning by reward-weighted regression for operational space control,” in Proceedings of the 24th international conference on Machine learning , 2007, pp. 745–750

    2007

  18. [26]

    Aligning text-to-image models using human feedback,

    K. Lee, H. Liu, M. Ryu, O. Watkins, Y . Du, C. Boutilier, P. Abbeel, M. Ghavamzadeh, and S. S. Gu, “Aligning text-to-image models using human feedback,” arXiv preprint arXiv:2302.12192 , 2023

    2023 arXiv

  19. [27]

    Training diffu- sion models with reinforcement learning,

    K. Black, M. Janner, Y . Du, I. Kostrikov, and S. Levine, “Training diffu- sion models with reinforcement learning,” in The Twelfth International Conference on Learning Representations , 2024

    2024

  20. [28]

    Feedback efficient online fine-tuning of diffusion models,

    M. Uehara, Y . Zhao, K. Black, E. Hajiramezanali, G. Scalia, N. L. Diamant, A. M. Tseng, S. Levine, and T. Biancalani, “Feedback efficient online fine-tuning of diffusion models,” in International Conference on Machine Learning (ICML) , 2024

    2024

  21. [29]

    Fine-tuning of continuous-time diffusion models as entropy-regularized control,

    M. Uehara, Y . Zhao, K. Black, E. Hajiramezanali, G. Scalia, N. L. Diamant, A. M. Tseng, T. Biancalani, and S. Levine, “Fine-tuning of continuous-time diffusion models as entropy-regularized control,”CoRR, vol. abs/2402.15194, 2024

    2024 arXiv

  22. [30]

    Learning a diffu- sion model policy from rewards via q-score matching,

    M. Psenka, A. Escontrela, P. Abbeel, and Y . Ma, “Learning a diffu- sion model policy from rewards via q-score matching,” arXiv preprint arXiv:2312.11752, 2023

    2023 arXiv

  23. [31]

    Learning to grasp the ungraspable with emergent extrinsic dexterity,

    W. Zhou and D. Held, “Learning to grasp the ungraspable with emergent extrinsic dexterity,” in Conference on Robot Learning . PMLR, 2023, pp. 150–160

    2023

  24. [32]

    HACMan++: Spatially-Grounded Motion Primitives for Manipulation,

    B. Jiang, Y . Wu, W. Zhou, C. Paxton, and D. Held, “HACMan++: Spatially-Grounded Motion Primitives for Manipulation,” inProceedings of Robotics: Science and Systems , Delft, Netherlands, July 2024

    2024

  25. [33]

    Movement primitive diffusion: Learning gentle robotic manipulation of deformable objects,

    P. M. Scheikl, N. Schreiber, C. Haas, N. Freymuth, G. Neumann, R. Lioutikov, and F. Mathis-Ullrich, “Movement primitive diffusion: Learning gentle robotic manipulation of deformable objects,” CoRR, vol. abs/2312.10008, 2023

    2023 arXiv

  26. [34]

    Prodmp: A unified perspective on dynamic and probabilistic movement primitives,

    G. Li, Z. Jin, M. V olpp, F. Otto, R. Lioutikov, and G. Neumann, “Prodmp: A unified perspective on dynamic and probabilistic movement primitives,” IEEE Robotics and Automation Letters , 2023

    2023

  27. [35]

    R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press, 2018

    2018

  28. [36]

    Addressing function approxi- mation error in actor-critic methods,

    S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approxi- mation error in actor-critic methods,” in International conference on machine learning. PMLR, 2018, pp. 1587–1596

    2018

  29. [37]

    Consistency models,

    Y . Song, P. Dhariwal, M. Chen, and I. Sutskever, “Consistency models,” in International Conference on Machine Learning . PMLR, 2023, pp. 32 211–32 252

    2023

  30. [38]

    Efficient diffusion policies for offline reinforcement learning,

    B. Kang, X. Ma, C. Du, T. Pang, and S. Yan, “Efficient diffusion policies for offline reinforcement learning,” Advances in Neural Information Processing Systems, vol. 36, 2024

    2024

  31. [39]

    A minimalist approach to offline reinforce- ment learning,

    S. Fujimoto and S. S. Gu, “A minimalist approach to offline reinforce- ment learning,” Advances in neural information processing systems , vol. 34, pp. 20 132–20 145, 2021

    2021

  32. [40]

    Reinforcement learning and control as probabilistic infer- ence: Tutorial and review,

    S. Levine, “Reinforcement learning and control as probabilistic infer- ence: Tutorial and review,” arXiv preprint arXiv:1805.00909 , 2018

    2018 arXiv

  33. [41]

    robosuite: A modular simulation framework and benchmark for robot learning,

    Y . Zhu, J. Wong, A. Mandlekar, R. Mart´ın-Mart´ın, A. Joshi, S. Nasiriany, and Y . Zhu, “robosuite: A modular simulation framework and benchmark for robot learning,” arXiv preprint arXiv:2009.12293 , 2020

    2009 arXiv

  34. [42]

    Mujoco: A physics engine for model- based control,

    E. Todorov, T. Erez, and Y . Tassa, “Mujoco: A physics engine for model- based control,” in 2012 IEEE/RSJ international conference on intelligent robots and systems . IEEE, 2012, pp. 5026–5033

    2012

  35. [43]

    Deep reinforcement learning at the edge of the statistical precipice,

    R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Belle- mare, “Deep reinforcement learning at the edge of the statistical precipice,” Advances in Neural Information Processing Systems, vol. 34, 2021

    2021

  36. [44]

    CORN: Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Ob- jects,

    Y . Cho, J. Han, Y . Cho, and B. Kim, “CORN: Contact-based Object Representation for Nonprehensile Manipulation of General Unseen Ob- jects,” in International Conference on Learning Representations (ICLR), 2024

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.