Pith. sign in

REVIEW 3 major objections 5 minor 40 references

Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that the intractable intermediate energy in energy-guided diffusion policies for offline RL reduces to a closed-form analytic target, and that training an energy network against this target improves guided policy…

desk verdict Useful empirical recipe for diffusion-policy guidance, but the 'analytic' energy target is not actually controlled: Appendix I.2's error bound is wrong. read the letter →

arxiv 2505.01822 v1 pith:YOQCE3VU submitted 2025-05-03 cs.LG cs.AI

Jifeng Hu , Sili Huang , Zhejian Yang , Shengchao Hu , Li Shen , Hechang Chen , Lichao Sun , Yi Chang
show 1 more author
Dacheng Tao
This is my paper · ORCID
classification cs.LGcs.AI
keywords offlinereinforcementlearningdiffusionpolicyenergy-guidedsamplingintermediateenergylog-expectationapproximationposteriorGaussiananalyticguidanceD4RLbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes Analytic Energy-guided Policy Optimization (AEPO), a way to turn the intractable log-expectation energy that steers diffusion policies in offline reinforcement learning into a closed-form analytic target. In energy-guided diffusion sampling, guiding a learned behavior policy toward high returns requires the intermediate energy $E_t(s,a_t)=\log \mathbb{E}_{a_0\sim\mu_{0|t}(a_0|a_t,s)}[e^{\beta Q(s,a_0)}]$, which is hard to estimate because of the logarithm outside the expectation. The paper derives an approximation of this energy from a Taylor expansion of the Q function and a Gaussian approximation of the posterior over clean actions, then trains a neural energy model against that target to supply the guidance during generation. Across D4RL offline RL tasks, the resulting guided diffusion policy matches or surpasses a wide set of diffusion-based and non-diffusion baselines. The significance is that guidance becomes computable and trainable without hand-designed conditioning or contrastive estimation of the full expectation.

What carries the argument

The machinery is Equation (25), the analytic simplification of the log-expectation intermediate energy. It converts the intractable quantity into three computable terms: a Q-value at a pivot action, a directional term involving the Q-gradient and the posterior mean, and a variance-scaled gradient norm. The posterior mean $\tilde\mu_{0|t}$ is reparameterized from the trained noise predictor, the isotropic variance $\tilde\sigma_{0|t}^2$ is estimated from the expected noise prediction norm, and the energy network $E_\Theta(s,a_t,t)$ is fitted by regression against this target. Differentiating this trained network with respect to $a_t$ supplies the intermediate guidance used in the reverse diffusion update.

What would settle it

On a task whose behavior policy has two well-separated action modes, compute the paper's analytic target from Equation (25) and compare it with a high-sample Monte Carlo estimate of $\log\mathbb{E}_{a_0\sim\mu_{0|t}}[e^{\beta Q(s,a_0)}]$ at matched denoising steps; if the energy error is large or does not track the gap between AEPO and an empirical-posterior variant, the Gaussian posterior approximation is the failing component. A small energy error on such multimodal tasks would instead support the approximation's practical reach.

Watch

Extended reading notes

Core claim

The central discovery is that, under Gaussian diffusion transitions, the exact intermediate energy $E_t(s,a_t)=\log\mathbb{E}_{a_0\sim\mu_{0|t}}[e^{\beta Q(s,a_0)}]$ can be approximated in closed form. Starting from the observation that previous classifier-guided and DPS-style energies are inexact because they do not preserve $p_t(x_t)\propto q_t(x_t)e^{-E_t(x_t)}$, the paper expands the Q function around a pivot action $\bar a$ and uses the moment-generating function of a Gaussian posterior to obtain the analytic target $\beta Q(s,\bar a)+\beta Q'(s,\bar a)^\top(\tilde\mu_{0|t}-\bar a)+\frac12\beta^2\tilde\sigma_{0|t}^2\|Q'(s,\bar a)\|^2$. The posterior mean comes from the diffusion noise predictor, and two closed-form isotropic variance estimates are proposed. Training an energy network against this target and differentiating it during reverse diffusion yields the guided policy the paper evaluates.

Load-bearing premise

The load-bearing premise is that the posterior distribution of the clean action given a noisy action is an isotropic Gaussian whose mean and variance come from the noise predictor, so the moment-generating and Taylor steps can replace the true log-expectation; for real offline datasets this posterior is often neither Gaussian nor isotropic, and the resulting bias is not measured.

Editorial extensions

If this is right

  • AEPO gives a training signal for the intermediate energy that is analytic rather than contrastive or hand-specified, so the guidance term is reproducible from the Q function and noise predictor alone.
  • The method separates behavior cloning (the diffusion model) from policy improvement (the energy guidance), allowing either component to be swapped without redesigning the guided sampling loop.
  • Guidance rescaling makes performance robust across guidance strengths, so deployment does not require per-task tuning of the guidance weight.
  • On the D4RL tasks evaluated, including Gym-MuJoCo, maze2d, antmaze, and Adroit, the aggregate scores are at or above those of prior diffusion-based offline RL methods, which is the empirical claim that would need to hold for the method to be preferred.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that Equation (25) could be evaluated online during inference without training a separate energy network, since all ingredients are functions of the current noisy action, the noise predictor, and the Q-gradient; the network only adds a fitted smoothing.
  • A testable extension is to relax the isotropic Gaussian posterior to a learned diagonal covariance or mixture, then measure whether the energy error on multimodal offline action distributions shrinks; the paper's ablations compare only two closed-form covariance estimates.
  • Because the approximation is first-order in Q, a second-order correction containing $\nabla^2_a Q$ and the posterior covariance is a natural next test for whether curvature of the Q function, rather than posterior mismatch, is the dominant source of energy error.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Analytic Energy-guided Policy Optimization (AEPO), a diffusion-model-based policy for offline reinforcement learning. The method defines an intermediate energy E_t(s,a_t) = log E_{a0 ~ mu_{0|t}}[exp(beta Q(s,a0))] and approximates this intractable log-expectation by a Taylor expansion of Q around a reference action bar a, followed by an application of the Gaussian moment-generating function. The resulting target, Equation (25), is used to train an energy network E_Theta via regression, and the learned energy gradient guides the diffusion sampling process. The paper reports strong empirical results on D4RL benchmarks, including Gym-MuJoCo, maze2d, antmaze, and Adroit, with comparisons against more than 30 baselines.

Significance. If the analytical target in Equation (25) were valid, the paper would make a useful theoretical and practical contribution: it would replace a costly Monte Carlo estimation of the intermediate log-expectation energy with a closed-form expression and connect it to exact energy-guided diffusion in offline RL. The empirical evaluation is broad, with 30+ tasks and ablations on posterior approximations, Q-function training, and guidance rescaling, which is a clear strength. However, the central theoretical claim is not currently established: the error analysis supporting Equation (25) is mathematically invalid, and the Gaussian-posterior assumption is not justified for realistic offline action distributions. As a result, the analytic target remains an uncontrolled approximation, and the significance of the theoretical contribution is conditional on fixing these issues.

major comments (3)
  1. [Appendix I.2 and Eq. (25)] The error bound supporting Equation (25) is invalid. The exact second-order Taylor remainder of Q(s,a0) around bar a is R2(a0) = (1/2)(a0 - bar a)^T H_Q(c)(a0 - bar a), where H_Q is the Hessian of Q with respect to a0. Appendix I.2 instead writes R2(a) = (1/2)[(∇Q)^T(a - bar a)]^2 and bounds this by (1/2)nu^2 using ||∇Q|| ||a - bar a|| ≤ nu. This squared-gradient term is a first-order quantity, not a bound on the Hessian quadratic form: it can vanish even when the Hessian is large, and the omitted ||H_Q|| factor can make the true remainder arbitrarily large. Moreover, the Taylor error enters inside the exponential before the moment-generating step, so the bias in log E[e^{βQ}] is on the order of β times the pointwise Taylor error, not its square. No regime is identified in which the curvature of the neural-network Q function is small enough for the first-order expansion to be accurate, and no valid bound on the approximation error of Equation (25) is provided.
  2. [Sec. 3.2, Eqs. (18)-(22)] The derivation assumes that the posterior mu_{0|t}(a0|at,s) is an isotropic Gaussian whose mean is given by Eq. (18) and whose scalar variance is given by Eq. (22). For an arbitrary offline action distribution, the exact posterior under Gaussian perturbation is not an isotropic Gaussian; in general it is a mixture or has no closed Gaussian form, so the moment-generating step leading to Eq. (17) is exact only under the stated Gaussian assumption. The paper does not quantify the posterior mismatch, nor does it validate the scalar variance estimate in Eq. (22), which discards all off-diagonal covariance structure. Because L_IE in Eq. (26) regresses directly onto this approximate target, the learned energy network inherits an unquantified bias from the posterior approximation in addition to the Taylor error.
  3. [Sec. 4 and Table 4] The empirical evaluation would be much more convincing if it isolated the effect of the proposed analytic target. A natural control is to replace Equation (25) with a Monte Carlo estimator of log E[e^{βQ}] obtained by sampling a0 from the learned posterior mu_{0|t} during inference, while keeping the rest of the pipeline unchanged. Without such an ablation, it is difficult to attribute the strong D4RL results to the analytic approximation rather than to the guidance-rescaling heuristic in Eq. (45) or to the expectile-based Q-function training. Additionally, Table 4 reports ranges for beta, tau, and omega and Fig. 6 shows sensitivity analyses on D4RL tasks; the paper should state clearly whether these hyperparameters are selected per task on the same test sets, since that would affect the significance of the benchmark comparisons.
minor comments (5)
  1. [Sec. 3.1, after Eq. (15)] The sentence 'Reminding that the posterior distribution mu_{0|t}(a0|at,s) is also a Gaussian distribution' is overly strong; in general the posterior under the Gaussian perturbation is not Gaussian. Please rephrase as an assumption and justify it, or add a discussion of when it is a reasonable approximation.
  2. [Sec. 3.4, Eq. (45)] The rescaling rule divides by ||∇at log pi_t(at|s)||; the case where this norm is zero is not discussed. Please state that the normalization is applied with a small numerical constant or otherwise handle the degenerate case.
  3. [Appendix I.2, Eq. (46)] The notation in the definition bar a = a - nu * Q'_psi(s,a0)/||Q'_psi(s,a0)|| mixes a0 and a; please clarify which action is used in the gradient and in the Taylor expansion, since the expansion is in a0 around bar a and the target Q'(s,bar a) is later replaced by Q'(s,a).
  4. [Sec. 3.2, Eq. (22)] The variance estimate in Eq. (22) uses a single scalar 1/d E||epsilon_theta||^2 for all dimensions. It would be helpful to report the empirical spread of ||epsilon_theta|| across timesteps, since the approximation quality likely varies strongly with t.
  5. [Appendix G.3] The third posterior approximation in Eq. (44) is derived but never used in experiments. Please either remove it or explain why it is omitted from the main results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the analytic energy target is a supervised regression target computed from learned Q and diffusion-model components; approximation-quality issues are correctness risks, not circular reasoning.

full rationale

The paper's derivation chain is self-contained in the sense required by the circularity review. The exact intermediate energy E_t(s,a_t) = log E[e^{beta Q(s,a0)}] is defined in Equation (12) and proved via Bayes' rule in Appendix E; this identity is not assumed from the quantity being predicted. The analytic approximation in Equation (25) follows from a first-order Taylor expansion of Q and the moment-generating function of an assumed Gaussian posterior, with posterior mean and variance estimated from the trained diffusion model (Equations 18, 22, 24). The energy network E_Theta is then trained by supervised regression onto this computed target (Equation 26), which is a standard function-approximation setup rather than a prediction that reduces to its own input. The Q function itself is learned by expectile regression (Equation 27), independently of the energy network, and final claims are evaluated against D4RL benchmarks, which are external to the fitted energy target. The main weaknesses identified by the skeptical reader--the non-Gaussianity of the true posterior and the uncontrolled Taylor remainder, including the incorrect remainder bound in Appendix I.2--are validity and correctness concerns, not circularity: they do not make the derivation equivalent to its inputs by construction. Self-citations (e.g., Hu et al. 2023) appear only as a comparison baseline, not as load-bearing justification for the central derivation, so they do not raise the circularity score. Hyperparameter tuning on the same D4RL tasks is a statistical overfitting risk, which is outside the definition of circularity used here.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central approximation relies on four free hyperparameters and several assumptions that are either standard or ad hoc. The most consequential assumptions are the Gaussian isotropic posterior and the accuracy of the first-order Taylor expansion; these are not established and the appended error analysis is incorrect. No new physical or mathematical entities are introduced.

free parameters (4)
  • inverse temperature beta = tuned from {1,2,3,4,5}, exact per-task values not reported
    Controls the strength of the energy guidance. Appendix B.3 and Table 4 indicate beta is searched over this range and selected to maximize D4RL performance.
  • expectile weight tau = tuned from {0.5,0.6,0.7,0.8}, exact per-task values not reported
    Controls the Q-function training via expectile regression. Figure 6 and Table 4 show performance varies with tau and a value like 0.7 is preferred.
  • step size nu for reference action bar a = 0.001
    Used in Equation (46) to set bar a near a. The value is chosen to keep the Taylor expansion local; the error analysis in Appendix I.2 is flawed.
  • guidance degree omega = ablated over 0.0 to 3.0
    Used at inference to weight the energy gradient. The rescaling method is designed to be robust to omega, but performance still depends on the chosen omega.
assumptions (5)
  • standard math The constrained RL problem has optimal policy pi*(a|s) proportional to mu(a|s) exp(beta Q(s,a)).
    Derived in Appendix C from a KL-constrained policy optimization; standard duality argument for KL-regularized RL.
  • standard math The exact intermediate energy satisfies p_t(x_t) proportional to q_t(x_t) exp(-E_t(x_t)) with E_t = -log E[e^{-beta E(x0)}].
    Proven in Appendix D and E using the law of total probability and Bayes rule; relies on the forward transition being shared between guided and unguided processes.
  • ad hoc to paper The posterior distribution mu0|t(a0|at,s) is Gaussian and isotropic.
    Assumed in Section 3.2 (Posterior 1). For a Gaussian forward process and a non-Gaussian data distribution, the posterior is generally not Gaussian; the paper provides no justification for multimodality.
  • ad hoc to paper The first-order Taylor expansion of Q(s,a0) around bar a is sufficiently accurate.
    Invoked in Section 3.1 to linearize Q inside the exponential. The error bound attempted in Appendix I.2 is mathematically invalid, so this remains an unverified assumption.
  • domain assumption The expectile-trained Q function approximates the true action value well enough for guidance.
    Standard in offline RL (IQL style). The paper uses Q trained with expectile regression and argues it avoids OOD overestimation, but this is an inductive assumption about the learned value function.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning." pith.science (2026). https://pith.science/paper/YOQCE3VU

@misc{pith2026250501822,
  author       = {Pith},
  title        = {Pith review of: Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YOQCE3VU}},
  note         = {Machine review of arXiv:2505.01822}
}
read the original abstract

Conditional decision generation with diffusion models has shown powerful competitiveness in reinforcement learning (RL). Recent studies reveal the relation between energy-function-guidance diffusion models and constrained RL problems. The main challenge lies in estimating the intermediate energy, which is intractable due to the log-expectation formulation during the generation process. To address this issue, we propose the Analytic Energy-guided Policy Optimization (AEPO). Specifically, we first provide a theoretical analysis and the closed-form solution of the intermediate guidance when the diffusion model obeys the conditional Gaussian transformation. Then, we analyze the posterior Gaussian distribution in the log-expectation formulation and obtain the target estimation of the log-expectation under mild assumptions. Finally, we train an intermediate energy neural network to approach the target estimation of log-expectation formulation. We apply our method in 30+ offline RL tasks to demonstrate the effectiveness of our method. Extensive experiments illustrate that our method surpasses numerous representative baselines in D4RL offline reinforcement learning benchmarks.

Figures

Figures reproduced from arXiv: 2505.01822 by the authors.

Figure 2
Figure 2. Guidance rescale ablation of AEPO on Gym-MuJoCo walker2d-medium-expert task. The x-axis denotes training steps. an action is OOD, thus enhancing the robustness on OOD actions. Recently, generative models (He et al., 2023; Li et al., 2023; Wang et al., 2024), such as diffusion models, are proposed to augment the offline datasets with synthetic experiences or train a world model for planning. 5.2. Generative Policy Op… view at source ↗
Figure 3
Figure 3. Guidance rescale ablation of AEPO on D4RL Gym-MuJoCo halfcheetah-medium-expert task. The y-axis and x-axis denote the normalized score and training steps, respectively. B.2. Additional Rescaling Ablation Apart from the guidance rescaling ablation on walker2d-medium-expert task, we also conduct the experiments on halfcheetach-medium-expert, hopper-medium-expert, and walker2d-medium tasks, where the results are shown … view at source ↗
Figure 4
Figure 4. Guidance rescale ablation of AEPO on D4RL Gym-MuJoCo hopper-medium-expert task. The y-axis and x-axis denote the normalized score and training steps, respectively [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Guidance rescale ablation of AEPO on D4RL Gym-MuJoCo walker2d-medium task. The y-axis and x-axis denote the normalized score and training steps, respectively. B.3. Parameter Sensitivity We investigate the influence of hyperparameters of β and τ that are used in loss LI…
Figure 6
Figure 6. Figure 6: Parameter sensitivity of D4RL Gym-MuJoCo tasks [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 3 canonical work pages

  1. [1]

    Is conditional generative model- ing all you need for decision-making? arXiv preprint arXiv:2211.15657,

    Ajay, A., Du, Y ., Gupta, A., Tenenbaum, J., Jaakkola, T., and Agrawal, P. Is conditional generative model- ing all you need for decision-making? arXiv preprint arXiv:2211.15657,

    arXiv

  2. [6]

    Simple hierarchical planning with diffusion

    Chen, C., Deng, F., Kawaguchi, K., Gulcehre, C., and Ahn, S. Simple hierarchical planning with diffusion. arXiv preprint arXiv:2401.02644,

    arXiv

  3. [7]

    Offline rein- forcement learning via high-fidelity generative behavior modeling

    Chen, H., Lu, C., Ying, C., Su, H., and Zhu, J. Offline rein- forcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209.14548,

    arXiv

  4. [8]

    Score reg- ularized policy optimization through diffusion behavior

    Chen, H., Lu, C., Wang, Z., Su, H., and Zhu, J. Score reg- ularized policy optimization through diffusion behavior. arXiv preprint arXiv:2310.07297,

    arXiv

  5. [10]

    K., and Levine, S

    Dasari, S., Mees, O., Zhao, S., Srirama, M. K., and Levine, S. The ingredients for robotic diffusion transformers. arXiv preprint arXiv:2410.10088,

    arXiv

  6. [11]

    and Jin, C

    Ding, Z. and Jin, C. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309.16984,

    arXiv

  7. [12]

    Diffuserlite: Towards real-time diffusion plan- ning

    Dong, Z., Hao, J., Yuan, Y ., Ni, F., Wang, Y ., Li, P., and Zheng, Y . Diffuserlite: Towards real-time diffusion plan- ning. arXiv preprint arXiv:2401.15443,

    arXiv

  8. [13]

    D4rl: Datasets for deep data-driven reinforcement learning

    Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine, S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219,

    arXiv 2004

Show all 40 references
  1. [14]

    Off-policy deep reinforcement learning without exploration

    Fujimoto, S., Meger, D., and Precup, D. Off-policy deep reinforcement learning without exploration. In Interna- tional conference on machine learning, pp. 2052–2062. PMLR,

  2. [16]

    and Salimans, T

    Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598,

    arXiv

  3. [17]

    Instructed diffuser with temporal condition guidance for offline reinforcement learning

    Hu, J., Sun, Y ., Huang, S., Guo, S., Chen, H., Shen, L., Sun, L., Chang, Y ., and Tao, D. Instructed diffuser with temporal condition guidance for offline reinforcement learning. arXiv preprint arXiv:2306.04875,

    arXiv

  4. [18]

    B., and Levine, S

    Janner, M., Du, Y ., Tenenbaum, J. B., and Levine, S. Plan- ning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.09991,

    arXiv

  5. [19]

    Efficient planning in a com- pact latent action space.arXiv preprint arXiv:2208.10291,

    Jiang, Z., Zhang, T., Janner, M., Li, Y ., Rockt ¨aschel, T., Grefenstette, E., and Tian, Y . Efficient planning in a com- pact latent action space.arXiv preprint arXiv:2208.10291,

    arXiv

  6. [20]

    Offline reinforce- ment learning with implicit q-learning

    Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning. arXiv preprint arXiv:2110.06169,

    arXiv

  7. [21]

    Offline rein- forcement learning: Tutorial, review, and perspectives on open problems

    Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643,

    2005 arXiv

  8. [22]

    Learning multimodal behaviors from scratch with diffusion policy gradient

    Li, Z., Krohn, R., Chen, T., Ajay, A., Agrawal, P., and Chalvatzaki, G. Learning multimodal behaviors from scratch with diffusion policy gradient. arXiv preprint arXiv:2406.00681,

    arXiv

  9. [23]

    Adaptdiffuser: Diffusion models as adaptive self-evolving planners

    Liang, Z., Mu, Y ., Ding, M., Ni, F., Tomizuka, M., and Luo, P. Adaptdiffuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.01877,

    arXiv

  10. [24]

    Dataset distillation for offline reinforcement learning

    Light, J., Liu, Y ., and Hu, Z. Dataset distillation for offline reinforcement learning. arXiv preprint arXiv:2407.20299,

  11. [25]

    Selfbc: Self behavior cloning for offline reinforcement learning

    Liu, S., Bai, C., Guo, Z., Zhang, H., Sharma, G., and Liu, Y . Selfbc: Self behavior cloning for offline reinforcement learning. arXiv preprint arXiv:2408.02165,

    arXiv

  12. [26]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps

    Lu, C., Zhou, Y ., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775–5787, 2022a. Lu, C., Zhou, Y ., Bao, F., Chen, J., Li, C., and Zhu,...

    arXiv

  13. [27]

    Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning

    Mao, L., Xu, H., Zhan, X., Zhang, W., and Zhang, A. Diffusion-dice: In-sample diffusion guidance for offline reinforcement learning. arXiv preprint arXiv:2407.20109, 2024a. Mao, Y ., Zhang, H., Chen, C., Xu, Y ., and Ji, X. Supported value regularization for offline reinforcem...

    arXiv

  14. [29]

    V ., Tan, S

    Pearce, T., Rashid, T., Kanervisto, A., Bignell, D., Sun, M., Georgescu, R., Macua, S. V ., Tan, S. Z., Momenne- jad, I., Hofmann, K., et al. Imitating human behaviour with diffusion models. arXiv preprint arXiv:2301.10677,

    arXiv

  15. [30]

    B., Kumar, A., Zhang, G., and Levine, S

    Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177,

    1910 arXiv

  16. [32]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  17. [33]

    P., Kumar, A., Er- mon, S., and Poole, B

    Song, Y ., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Er- mon, S., and Poole, B. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456,

    2011 arXiv

  18. [34]

    Wang, K., Zhao, H., Luo, X., Ren, K., Zhang, W., and Li, D

    URL https://spaces.ac.cn/ archives/9246. Wang, K., Zhao, H., Luo, X., Ren, K., Zhang, W., and Li, D. Bootstrapped transformer for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:34748–34761, 2022a. Wang, R., Frans, K., Abbeel, P., Levine, ...

    arXiv

  19. [35]

    J., and Zhou, M

    Wang, Z., Hunt, J. J., and Zhou, M. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06193, 2022b. Wen, X., Yu, X., Yang, R., Chen, H., Bai, C., and Wang, Z. Towards robust offline-to-online reinforcement learning v...

    arXiv

  20. [36]

    Behavior regu- larized offline reinforcement learning

    Wu, Y ., Tucker, G., and Nachum, O. Behavior regu- larized offline reinforcement learning. arXiv preprint arXiv:1911.11361,

    1911 arXiv

  21. [37]

    Improving diffusion-based image synthesis with context prediction

    Yang, L., Liu, J., Hong, S., Zhang, Z., Huang, Z., Cai, Z., Zhang, W., and Cui, B. Improving diffusion-based image synthesis with context prediction. Advances in Neural Information Processing Systems, 36, 2024b. Yang, S., Wang, Z., Zheng, H., Feng, Y ., and Zhou, M. A behavior...

    arXiv

  22. [38]

    Revisiting generative policies: A simpler reinforcement learning algorithmic perspective

    Zhang, J., Xue, R., Niu, Y ., Chen, Y ., Yang, J., Li, H., and Liu, Y . Revisiting generative policies: A simpler reinforcement learning algorithmic perspective. arXiv preprint arXiv:2412.01245, 2024a. Zhang, Z., Liu, R., Hanocka, R., and Aberman, K. Tedi: Temporally-entangled...

    2024 arXiv

  23. [39]

    In lines 3-13, we sample data from the dataset and train the noise prediction modelϵθ that is used to obtain∇at logµt(at|s) =−ϵθ(s,at,t )/σt, the intermediate energyEΘ(s,at,t ) that can be used to approximate the intermediate guidance∇atEt(s,at), and the Q functionQψ that is u...

    2017

  24. [40]

    We use ˜µ0|t = Eµ0|t(a0|at,s)[a0] to derivate the third equation

    it as ˜Σ0|t(at) = Eµ0|t(a0|at,s) (a0− ˜µ0|t)(a0− ˜µ0|t)⊤ = Eµ0|t(a0|at,s) h (a0−u0)− (˜µ0|t−u0) (a0−u0)− (˜µ0|t−u0) ⊤i = Eµ0|t(a0|at,s) (a0−u0)(a0−u0)⊤ − (˜µ0|t−u0)(µ0|t−u0)⊤, (40) where we add a constant vector u0 that has the same dimension with ˜µ0|t in the second equation....

    2021

  25. [2013]

    Awac: Accel- erating online reinforcement learning with offline datasets

    Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accel- erating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359,

    2006 arXiv

  26. [2018]

    G., and Levine, S

    9 Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning Hansen-Estruch, P., Kostrikov, I., Janner, M., Kuba, J. G., and Levine, S. Idql: Implicit q-learning as an actor- critic method with diffusion policies. arXiv preprint arXiv:2304.10573,

    arXiv

  27. [2019]

    Learning complex dexterous manipulation with deep reinforcement learning and demonstrations

    Rajeswaran, A., Kumar, V ., Gupta, A., Vezzani, G., Schul- man, J., Todorov, E., and Levine, S. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. arXiv preprint arXiv:1709.10087,

    arXiv

  28. [2020]

    Pessimistic bootstrapping for uncertainty- driven offline reinforcement learning

    Bai, C., Wang, L., Yang, Z., Deng, Z., Garg, A., Liu, P., and Wang, Z. Pessimistic bootstrapping for uncertainty- driven offline reinforcement learning. arXiv preprint arXiv:2202.11566,

    arXiv

  29. [2021]

    and Dulac-Arnold, G

    Argenson, A. and Dulac-Arnold, G. Model-based offline planning. arXiv preprint arXiv:2008.05556,

    2008 arXiv

  30. [2022]

    Estimating the optimal covariance with imperfect mean in diffusion probabilistic models

    Bao, F., Li, C., Sun, J., Zhu, J., and Zhang, B. Estimating the optimal covariance with imperfect mean in diffusion probabilistic models. arXiv preprint arXiv:2206.07309, 2022a. Bao, F., Li, C., Zhu, J., and Zhang, B. Analytic-dpm: an ana- lytic estimate of the optimal reverse...

    arXiv

  31. [2023]

    T., Klasky, M

    Chung, H., Kim, J., Mccann, M. T., Klasky, M. L., and Ye, J. C. Diffusion posterior sampling for general noisy in- verse problems. arXiv preprint arXiv:2209.14687,

    arXiv

  32. [2024]

    Cao, P., Zhou, F., Song, Q., and Yang, L

    URL https://openai.com/research/ video-generation-models-as-world-simulators . Cao, P., Zhou, F., Song, Q., and Yang, L. Controllable generation with text-to-image diffusion models: A survey. arXiv preprint arXiv:2403.04279,

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.