Pith. sign in

REVIEW 2 major objections 5 minor 47 references

A single diffusion policy trained from scratch with online RL solves multi-shape block pushing and transfers zero-shot to real robots under tested variations.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 08:29 UTC pith:WL576MLT

load-bearing objection Strong sim multi-task diffusion-from-scratch result with a clean reweighted ELBO; zero-shot hardware claim is real but thinly documented. the 2 major comments →

arxiv 2607.10892 v1 pith:WL576MLT submitted 2026-07-12 cs.RO

A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer

Haitong Ma , Haldun Balim , Yang Hu , Bo Dai , Na Li This is my paper
classification cs.RO
keywords diffusion policyonline reinforcement learningblock pushingsim-to-real transfermulti-task manipulationcurriculum learningobjective-centric representationenergy-guided score matching
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that one diffusion policy can learn multi-task, contact-rich block pushing from sparse rewards without expert demonstrations. It trains the policy from scratch with online RL by reweighting the usual diffusion training loss with the exponential of the action-value function, then pairs that objective with reverse curriculum generation and objective-centric state and action representations. In simulation the resulting controller reaches 100 percent success on T-, C-, and L-shaped blocks and still succeeds on an unseen I-shape, while Gaussian-policy baselines largely fail. The same policy, using only pose estimates, is then run on a real robot and succeeds under changes in goal pose, block shape, block weight, and surface friction. A sympathetic reader cares because the result suggests that the multi-modality of diffusion policies can carry both multi-task competence and zero-shot sim-to-real transfer for contact-rich manipulation when the right curriculum and observation design remove the exploration barrier.

Core claim

A single diffusion policy, trained from scratch by online reinforcement learning with an energy-guided score-matching loss, reverse curriculum generation, and objective-centric state representations, solves multi-shape sparse-reward block-pushing tasks at 100 percent success in simulation and transfers zero-shot to real-world block pushing under the tested variations in goal pose, shape, weight, and surface friction.

What carries the argument

Energy-guided score matching (ESM): the ordinary diffusion evidence-lower-bound loss reweighted by the exponential of the current action-value function, which is exactly the closed-form policy-mirror-descent update and can be used as a drop-in policy module inside any actor-critic RL algorithm.

Load-bearing premise

The main sim-to-real gap (surface friction and related contact forces) does not push the closed-loop system outside the region where the learned multi-modal policy remains stabilizing, so domain randomization or residual adaptation can be skipped.

What would settle it

Run the identical zero-shot policy on a surface whose friction (or block mass) lies outside the range already tested and record whether success rate collapses while pose estimation remains accurate; if it does, the zero-shot claim fails for that gap size.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • One multi-modal policy class can replace separate single-task controllers for families of contact-rich pushing tasks when curriculum and relative coordinates are used.
  • Gaussian-policy RL methods remain inadequate for the same multi-task sparse-reward setting even after identical curriculum and representation design.
  • Accurate pose estimation plus relative coordinates is sufficient for zero-shot transfer of diffusion policies when contact dynamics stay inside the stabilizing region of the learned policy.
  • The same ESM loss can serve as a fine-tuning module for existing behavior-cloned diffusion policies without redesigning the network.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the multi-modality of the diffusion policy is what preserves a stabilizing skill under friction mismatch, the same pipeline may transfer to other planar contact tasks (e.g., sliding assembly) without domain randomization.
  • The simple shape encoding already yields partial zero-shot generalization to an unseen I-block; richer image- or geometry-based encodings could extend the same controller to open-set object sets.
  • Because the method only needs pose estimates, any improvement in marker-free registration immediately upgrades the real-world success envelope without retraining the policy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes training a single diffusion policy from scratch via online RL for multi-task sparse-reward block pushing across shapes (T/C/L). It introduces an energy-guided score matching (ESM) loss that reweights the standard DDPM ELBO by exp(Q/λ) so that the diffusion model targets the closed-form policy-mirror-descent update (Eq. 2, Prop. 1, Appendix A). Exploration is aided by reverse curriculum generation (expanding start–goal distance) and objective-centric observations/actions expressed in the block frame plus a simple 4-D shape code. In simulation the method reaches 100 % success with short episodes (Tables I–II), substantially outperforming SAC, PPO, TD3 and several recent diffusion-RL baselines; ablations confirm the value of curriculum and frame design (Fig. 7). The same policy is claimed to transfer zero-shot to a UR5e under variations of goal pose, shape, weight and surface friction (Abstract, §I, §V-B, Table III).

Significance. If the claims hold, the work supplies a practical, demonstration-free route to multi-modal policies for contact-rich multi-task manipulation and shows that diffusion policies can be trained end-to-end with online RL while retaining zero-shot sim-to-real robustness under the tested conditions. The derivation that ESM recovers the mirror-descent target (Appendix A) is clean and avoids the baseline-function or truncation issues of related weighted-score methods. Strong simulation tables, clear ablations, and an explicit multi-shape generalization test (including an unseen I-block) constitute solid empirical evidence for the algorithmic contribution. The real-world transfer result, if fully substantiated, would be a useful data point for the community on when multi-modal policies can absorb unmodeled friction without domain randomization.

major comments (2)
  1. [§V-B, Table III] The central zero-shot claim (Abstract; §I-c; §V-B) rests on 36 real-world tasks whose per-condition success counts, distances, steps, failure modes, pose-estimation error statistics, and exact friction/weight values are never tabulated. Table III reports only a 9-task subset (3 shapes × 3 trials) for the SAC comparison; the remaining 27 tasks and all quantitative metrics appear only as qualitative statements and an external website. Because the paper itself argues (§IV-A, §VII) that transfer succeeds only when contact dynamics remain inside the stabilizing region of the multi-modal policy, an incompletely reported hardware sample leaves the decisive empirical premise unverifiable from the archival record.
  2. [§IV-A, §VII] No quantitative characterization of the residual sim-to-real gap is supplied (measured friction coefficients, contact stiffness, pose-estimation RMSE under occlusion, etc.). The argument in §IV-A that “the sim-to-real gap will not pose a significant challenge au as long as it does not affect system stability” is therefore an untested modeling assumption rather than a measured fact; without those numbers the zero-shot success cannot be attributed to policy multi-modality versus simply a small gap.
minor comments (5)
  1. [§I] The citation for catastrophic forgetting is left as “[?]” in the introduction.
  2. [§III-B, Fig. 2] Shape-encoding vector is described as living in [0.0,2.0]^4 yet never specified numerically for the four training shapes; a short table or equation would aid reproducibility.
  3. [§III-C, Appendix B] Curriculum parameters M=10, N=50 and the exact spatial expansion schedule appear only in Appendix B; they should be stated in the main text or a hyper-parameter table.
  4. [Fig. 7] Figure 7 ablation curves lack error bars or seed-wise variance, making it hard to judge whether the “acceptable success rates” after ablation are statistically reliable.
  5. [§IV-B] The point-cloud registration pipeline (§IV-B) is only sketched; a short quantitative accuracy number (e.g., mean rotational error under partial occlusion) would strengthen the claim that state-based inputs enable easy transfer.

Circularity Check

0 steps flagged

No significant circularity: ESM loss is a standard reweighting of the DDPM ELBO to match the closed-form policy-mirror-descent target; empirical success rates are measured, not forced by construction.

full rationale

The paper’s central technical claim is a practical policy-loss (ESM, Eq. 4) obtained by reweighting the ordinary DDPM evidence lower bound (Eq. 3) with the importance ratio exp(Q/λ) that appears in the closed-form solution of KL-regularized policy mirror descent (Eq. 2). Appendix A proves equivalence by direct substitution of the target density into the reweighted score-matching objective; the derivation is self-contained, uses only standard Gaussian-kernel identities, and does not define the loss in terms of the later-reported success rates. Curriculum parameters (M, N), temperature λ, and the objective-centric coordinate transform are design choices whose values are stated explicitly; they are not fitted to any headline metric. Simulation tables (I–II) and real-world counts (Table III) are ordinary empirical evaluations against external baselines (SAC, PPO, QVPO, etc.). No uniqueness theorem, self-citation chain, or fitted constant is invoked to force the multi-task or zero-shot claims. Consequently the derivation chain contains no circular steps.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

The central claims rest on standard MDP/RL and DDPM machinery, a closed-form KL-regularized policy update, and several engineering design choices (curriculum schedule, local-frame observations, shape codes, pose estimation). Free parameters are algorithmic knobs, not physics constants fitted to force success. No new physical entity is postulated; ESM is a loss reweighting, not a new particle or force.

free parameters (5)
  • KL temperature λ in exp(Q/λ) reweighting
    Controls how sharply the diffusion policy is pulled toward high-Q actions; value not reported as fixed from theory and must be chosen for training stability.
  • Curriculum stages M and success threshold N
    Set to M=10, N=50 (Appendix B); directly gate exploration difficulty and thus whether sparse-reward learning succeeds.
  • Curriculum spatial expansion schedule
    Initial (x,y) box grows as 30 + m/M·100 around canvas center 256; hand-designed, not derived.
  • Diffusion noise schedule {α_t, σ_t} and horizon T
    Standard DDPM hyperparameters that affect policy expressiveness and training; not uniquely determined by the theory section.
  • Shape encoding vector in [0,2]^4
    Ad hoc discrete/continuous codes for T/C/L; proof-of-concept design that conditions multi-task behavior.
axioms (5)
  • standard math Policy mirror descent with KL penalty has closed-form Boltzmann update π ∝ π_old exp(Q/λ) (Eq. 1–2).
    Standard result used to justify the ESM target policy; invoked in §II-B and Appendix A.
  • standard math DDPM noise-prediction ELBO is equivalent to matching the noise-perturbed score (Prop. 2 / Vincent connection).
    Background diffusion theory used to equate reweighted score matching with fitting the target policy.
  • domain assumption Block-pushing can be modeled as a sparse-reward MDP with state-based observations and velocity actions in local frames.
    Defines the training problem (§II-A, §III-B); success metrics depend on this task formalization.
  • domain assumption Accurate block pose (ArUco or registration + Kalman / sim backup) is available at control rate for state-based policy input.
    Required for zero-shot transfer of a state-based policy (§IV-B); not learned end-to-end from pixels.
  • ad hoc to paper Reverse curriculum by expanding start–goal distance is a valid exploration scaffold for this sparse-reward task.
    Adopted from Florensa et al. style curricula and tuned for this environment (§III-C, Appendix B); ablation shows dependence.
invented entities (2)
  • Energy-guided score matching (ESM) loss no independent evidence
    purpose: Train a diffusion policy online to match the KL-regularized optimal policy without sampling from the intractable normalized Boltzmann target.
    Defined in Eq. (4) as exp(Q)-reweighted DSM; related to prior weighted score-matching RL methods but presented as a simplified module without baseline or Q truncation.
  • Objective-centric multi-task observation (block-frame states + shape code) no independent evidence
    purpose: Share one policy across goals and shapes by baking relative geometry and a shape token into the state.
    Design choice in §III-B / Fig. 2; enables multi-goal generalization without explicit absolute goal conditioning.

pith-pipeline@v1.1.0-grok45 · 17587 in / 3782 out tokens · 43042 ms · 2026-07-14T08:29:09.051110+00:00 · methodology

0 comments
read the original abstract

Diffusion policies have shown promising empirical performance in representing and learning complex maneuvers for robots using behavior cloning (BC). In this paper, we explore training diffusion policies from scratch using reinforcement learning (RL) for multi-task robotic manipulation. Specifically, we aim to train a single diffusion policy for block-pushing tasks with multiple shapes. The proposed framework features a simple policy loss function, which is a reweighted evidence lower bound used in BC-based diffusion policy training and can seamlessly serve as the policy learning module in RL algorithms. To address the exploration challenges arising from the absence of demonstrations, we incorporate reverse curriculum generation and objective-centric representations. Combined with the expressiveness of diffusion policies, our design supports learning of multi-task block-pushing policies in our sparse-reward simulation setting. We further evaluate whether the trained diffusion policy transfers in zero-shot to real-world tasks under varying environmental conditions including goal positions, block shapes, block weights and surface friction, providing evidence that this pipeline can transfer to our real-world block-pushing setup under the tested variations.

Figures

Figures reproduced from arXiv: 2607.10892 by Bo Dai, Haitong Ma, Haldun Balim, Na Li, Yang Hu.

Figure 1
Figure 1. Figure 1: (Upper) The pipeline we propose to train the diffusion policies with RL to solve the multi-task block-pushing task. (Lower) The learned diffusion policy can achieve zero-shot transfer to real-world block-pushing tasks with different block shapes, block weights, frictions, and goal poses. More figures and videos are available at https://diffusionpolicyrl.site/. the real-world setting under various weights, … view at source ↗
Figure 2
Figure 2. Figure 2: Observation space design: objective-centric representation [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: Note that the naive shape encoding used here is just [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Figure 4: An example point cloud registration results when large additional [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Screenshots of zero-shot generalization of the proposed ESM policy [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Ablation studies of ESM (proposed) and SAC (best baseline). The [PITH_FULL_IMAGE:figures/full_fig_p005_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Real-world configuration of the block-pushing tasks. Setup. We evaluate the real-world performance of the learned diffusion pol￾icy using the setup shown in [PITH_FULL_IMAGE:figures/full_fig_p005_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: All evaluation tasks in the real-world setting. Figures show the final [PITH_FULL_IMAGE:figures/full_fig_p006_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 19 linked inside Pith

  1. [1]

    Denoising Diffusion Probabilistic Models,

    J. Ho, A. Jain, and P. Abbeel, “Denoising Diffusion Probabilistic Models,” inAdvances in Neural Information Processing Systems, vol. 33. Curran Associates, Inc., 2020, pp. 6840–6851

    2020

  2. [2]

    Deep Unsupervised Learning using Nonequilibrium Thermodynam- ics,

    J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep Unsupervised Learning using Nonequilibrium Thermodynam- ics,” inProceedings of the 32nd International Conference on Machine Learning. PMLR, June 2015, pp. 2256–2265

    2015

  3. [3]

    Score-Based Generative Modeling through Stochastic Differential Equations,

    Y . Song, J. Sohl-Dickstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, “Score-Based Generative Modeling through Stochastic Differential Equations,” Feb. 2021

    2021

  4. [4]

    Diffusion models beat gans on image synthesis,

    P. Dhariwal and A. Nichol, “Diffusion models beat gans on image synthesis,”Advances in neural information processing systems, vol. 34, pp. 8780–8794, 2021

    2021

  5. [5]

    Plan- ning with diffusion for flexible behavior synthesis,

    M. Janner, Y . Du, J. B. Tenenbaum, and S. Levine, “Plan- ning with diffusion for flexible behavior synthesis,”arXiv preprint arXiv:2205.09991, 2022

    Pith/arXiv arXiv 2022

  6. [6]

    Diffusion policy: Visuomotor policy learning via ac- tion diffusion,

    C. Chi, Z. Xu, S. Feng, E. Cousineau, Y . Du, B. Burchfiel, R. Tedrake, and S. Song, “Diffusion policy: Visuomotor policy learning via ac- tion diffusion,”The International Journal of Robotics Research, p. 02783649241273668, 2023

    2023

  7. [7]

    Learning universal policies via text-guided video generation,

    Y . Du, S. Yang, B. Dai, H. Dai, O. Nachum, J. Tenenbaum, D. Schu- urmans, and P. Abbeel, “Learning universal policies via text-guided video generation,”Advances in Neural Information Processing Sys- tems, vol. 36, 2024

    2024

  8. [8]

    Diffusion policies as an expres- sive policy class for offline reinforcement learning,

    Z. Wang, J. J. Hunt, and M. Zhou, “Diffusion policies as an expres- sive policy class for offline reinforcement learning,”arXiv preprint arXiv:2208.06193, 2022

    Pith/arXiv arXiv 2022

  9. [9]

    Offline reinforce- ment learning via high-fidelity generative behavior modeling,

    H. Chen, C. Lu, C. Ying, H. Su, and J. Zhu, “Offline reinforce- ment learning via high-fidelity generative behavior modeling,”arXiv preprint arXiv:2209.14548, 2022

    Pith/arXiv arXiv 2022

  10. [10]

    Idql: Implicit q-learning as an actor-critic method with diffusion policies,

    P. Hansen-Estruch, I. Kostrikov, M. Janner, J. G. Kuba, and S. Levine, “Idql: Implicit q-learning as an actor-critic method with diffusion policies,”arXiv preprint arXiv:2304.10573, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Energy-weighted flow matching for offline reinforcement learning,

    S. Zhang, W. Zhang, and Q. Gu, “Energy-weighted flow matching for offline reinforcement learning,”arXiv preprint arXiv:2503.04975, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning,

    H. He, C. Bai, K. Xu, Z. Yang, W. Zhang, D. Wang, B. Zhao, and X. Li, “Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning,”Advances in neural information processing systems, vol. 36, pp. 64 896–64 917, 2023

    2023

  13. [13]

    Metadiffuser: Diffusion model as conditional planner for offline meta- rl,

    F. Ni, J. Hao, Y . Mu, Y . Yuan, Y . Zheng, B. Wang, and Z. Liang, “Metadiffuser: Diffusion model as conditional planner for offline meta- rl,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 26 087–26 105

    2023

  14. [14]

    Policy representation via diffusion probability model for reinforcement learning,

    L. Yang, Z. Huang, F. Lei, Y . Zhong, Y . Yang, C. Fang, S. Wen, B. Zhou, and Z. Lin, “Policy representation via diffusion probability model for reinforcement learning,”arXiv preprint arXiv:2305.13122, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Diffusion-based reinforcement learning via q-weighted variational policy optimization,

    S. Ding, K. Hu, Z. Zhang, K. Ren, W. Zhang, J. Yu, J. Wang, and Y . Shi, “Diffusion-based reinforcement learning via q-weighted variational policy optimization,”arXiv preprint arXiv:2405.16173, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Diffusion policy policy optimization,

    A. Z. Ren, J. Lidard, L. L. Ankile, A. Simeonov, P. Agrawal, A. Majumdar, B. Burchfiel, H. Dai, and M. Simchowitz, “Diffusion policy policy optimization,”arXiv preprint arXiv:2409.00588, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Proximal policy optimization algorithms,

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization algorithms,”arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  18. [18]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor,” inInternational conference on machine learning. PMLR, 2018, pp. 1861–1870

    2018

  19. [19]

    Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner architectures,

    L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V . Mnih, T. Ward, Y . Doron, V . Firoiu, T. Harley, I. Dunning,et al., “Impala: Scalable dis- tributed deep-rl with importance weighted actor-learner architectures,” inInternational conference on machine learning. PMLR, 2018, pp. 1407–1416

    2018

  20. [20]

    Walk these ways: Tuning robot control for generalization with multiplicity of behavior,

    G. B. Margolis and P. Agrawal, “Walk these ways: Tuning robot control for generalization with multiplicity of behavior,” inConference on Robot Learning. PMLR, 2023, pp. 22–31

    2023

  21. [21]

    Hierarchical diffu- sion policy for kinematics-aware multi-task robotic manipulation,

    X. Ma, S. Patidar, I. Haughton, and S. James, “Hierarchical diffu- sion policy for kinematics-aware multi-task robotic manipulation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 18 081–18 090

    2024

  22. [22]

    Domain randomization for transferring deep neural networks from simulation to the real world,

    J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel, “Domain randomization for transferring deep neural networks from simulation to the real world,” in2017 IEEE/RSJ international con- ference on intelligent robots and systems (IROS). IEEE, 2017, pp. 23–30

    2017

  23. [23]

    Streaming diffusion policy: Fast policy synthesis with variable noise diffusion models,

    S. H. Høeg, Y . Du, and O. Egeland, “Streaming diffusion policy: Fast policy synthesis with variable noise diffusion models,”arXiv preprint arXiv:2406.04806, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Crossway diffu- sion: Improving diffusion-based visuomotor policy via self-supervised learning,

    X. Li, V . Belagali, J. Shang, and M. S. Ryoo, “Crossway diffu- sion: Improving diffusion-based visuomotor policy via self-supervised learning,” in2024 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2024, pp. 16 841–16 849

    2024

  25. [25]

    Mirror descent policy optimization,

    M. Tomar, L. Shani, Y . Efroni, and M. Ghavamzadeh, “Mirror descent policy optimization,”arXiv preprint arXiv:2005.09814, 2020

    Pith/arXiv arXiv 2005

  26. [26]

    Trust region policy optimization,

    J. Schulman, “Trust region policy optimization,”arXiv preprint arXiv:1502.05477, 2015

    Pith/arXiv arXiv 2015

  27. [27]

    Advantage-weighted regression: Simple and scalable off-policy reinforcement learning,

    X. B. Peng, A. Kumar, G. Zhang, and S. Levine, “Advantage-weighted regression: Simple and scalable off-policy reinforcement learning,” arXiv preprint arXiv:1910.00177, 2019

    Pith/arXiv arXiv 1910

  28. [28]

    Awac: Accelerating online reinforcement learning with offline datasets,

    A. Nair, A. Gupta, M. Dalal, and S. Levine, “Awac: Accelerating online reinforcement learning with offline datasets,”arXiv preprint arXiv:2006.09359, 2020

    Pith/arXiv arXiv 2006

  29. [29]

    Goal-conditioned reinforcement learning: Problems and solutions,

    M. Liu, M. Zhu, and W. Zhang, “Goal-conditioned reinforcement learning: Problems and solutions,”arXiv preprint arXiv:2201.08299, 2022

    Pith/arXiv arXiv 2022

  30. [30]

    Re- verse curriculum generation for reinforcement learning,

    C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel, “Re- verse curriculum generation for reinforcement learning,” inConference on robot learning. PMLR, 2017, pp. 482–495

    2017

  31. [31]

    Implicit behavioral cloning,

    P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and J. Tompson, “Implicit behavioral cloning,” inConference on robot learning. PMLR, 2022, pp. 158– 168

    2022

  32. [32]

    Addressing function approxi- mation error in actor-critic methods,

    S. Fujimoto, H. Hoof, and D. Meger, “Addressing function approxi- mation error in actor-critic methods,” inInternational conference on machine learning. PMLR, 2018, pp. 1587–1596

    2018

  33. [33]

    Learning a diffusion model policy from rewards via q-score matching,

    M. Psenka, A. Escontrela, P. Abbeel, and Y . Ma, “Learning a diffusion model policy from rewards via q-score matching,”arXiv preprint arXiv:2312.11752, 2023

    Pith/arXiv arXiv 2023

  34. [34]

    Diffusion Actor-Critic with Entropy Regulator,

    Y . Wang, L. Wang, Y . Jiang, W. Zou, T. Liu, X. Song, W. Wang, L. Xiao, J. Wu, J. Duan, and S. E. Li, “Diffusion Actor-Critic with Entropy Regulator,” Dec. 2024

    2024

  35. [35]

    Hind- sight experience replay,

    M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welin- der, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba, “Hind- sight experience replay,”Advances in neural information processing systems, vol. 30, 2017

    2017

  36. [36]

    Con- trastive learning as goal-conditioned reinforcement learning,

    B. Eysenbach, T. Zhang, S. Levine, and R. R. Salakhutdinov, “Con- trastive learning as goal-conditioned reinforcement learning,”Ad- vances in Neural Information Processing Systems, vol. 35, pp. 35 603– 35 620, 2022

    2022

  37. [37]

    Goal-conditioned reinforce- ment learning with imagined subgoals,

    E. Chane-Sane, C. Schmid, and I. Laptev, “Goal-conditioned reinforce- ment learning with imagined subgoals,” inInternational conference on machine learning. PMLR, 2021, pp. 1430–1440

    2021

  38. [38]

    Curriculum learning,

    Y . Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” inProceedings of the 26th annual international conference on machine learning, 2009, pp. 41–48

    2009

  39. [39]

    Learning curriculum policies for reinforce- ment learning,

    S. Narvekar and P. Stone, “Learning curriculum policies for reinforce- ment learning,”arXiv preprint arXiv:1812.00285, 2018

    Pith/arXiv arXiv 2018

  40. [40]

    Curriculum learning for reinforcement learning domains: A framework and survey,

    S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor, and P. Stone, “Curriculum learning for reinforcement learning domains: A framework and survey,”Journal of Machine Learning Research, vol. 21, no. 181, pp. 1–50, 2020

    2020

  41. [41]

    Learning to fly in seconds,

    J. Eschmann, D. Albani, and G. Loianno, “Learning to fly in seconds,” IEEE Robotics and Automation Letters, 2024

    2024

  42. [42]

    Data-efficient control policy search using residual dynamics learning,

    M. Saveriano, Y . Yin, P. Falco, and D. Lee, “Data-efficient control policy search using residual dynamics learning,” in2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2017, pp. 4709–4715

    2017

  43. [43]

    Asap: Aligning simulation and real-world physics for learning agile humanoid whole-body skills,

    T. He, J. Gao, W. Xiao, Y . Zhang, Z. Wang, J. Wang, Z. Luo, G. He, N. Sobanbab, C. Pan,et al., “Asap: Aligning simulation and real-world physics for learning agile humanoid whole-body skills,”arXiv preprint arXiv:2502.01143, 2025

    Pith/arXiv arXiv 2025

  44. [44]

    Diffusion spectral representation for reinforcement learning,

    D. Shribak, C.-X. Gao, Y . Li, C. Xiao, and B. Dai, “Diffusion spectral representation for reinforcement learning,”arXiv preprint arXiv:2406.16121, 2024

    Pith/arXiv arXiv 2024

  45. [45]

    A Connection Between Score Matching and Denoising Autoencoders,

    P. Vincent, “A Connection Between Score Matching and Denoising Autoencoders,”Neural Computation, vol. 23, no. 7, pp. 1661–1674, July 2011. APPENDIX A. Derivations of Energy-Guided Score Matching We first revisit the closed-form optimal solution of the DSM loss (3), and then show that our ESM loss (4) induces the same optimal solution to conclude loss equivalence

    2011

  46. [46]

    thenoise-perturbed score func- tion[44]

    Noise prediction to learn noise-perturbed score func- tion:We first revisit an existing result showing that the optimal solution of the noise prediction modelϵ θ(at, s, t) is−σ t∇at log ˜πt(at|s), i.e. thenoise-perturbed score func- tion[44]. Recall that the termscore functionrefers to ∇x logp(x), the gradient of the log-density function. Before stating t...

  47. [47]

    equivalent

    ESM loss as an equivalent reweighted loss.:Consider an equivalent loss with reweighting functiong, i.e., Lg(θ, s, t) = Z g(at, s, t)∥ϵ θ(at;s, t) +σ t∇at log ˜πt(at|st)∥2 dat, (5) whereg(a t, s, t)is a strictly positive function everywhere in the action spaceAfor any fixedsandt. The loss is called “equivalent” since we can indeed show that any reweighted ...