{"id":"2a9a7ff0-1c78-42c0-a80b-0a40816e0f7a","arxiv_id":"2505.01822","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AEPO approximates the log-expectation in energy-guided diffusion policy sampling using Taylor expansion and the Gaussian moment-generating function, and reports state-of-the-art average scores on D4RL offline RL benchmarks.","lead":"The authors propose AEPO, a method that guides diffusion-based policies in offline reinforcement learning by approximating an intractable intermediate energy term with a closed-form formula. It reports strong D4RL benchmark results, often surpassing closely related energy-guided diffusion policies, though the theoretical analysis relies on approximations and the method is not fully reproducible from the paper.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (25) rests on a first-order Taylor expansion whose remainder is uncontrolled; Appendix I.2's error bound squares the gradient instead of the Hessian quadratic form, so the 'analytic' energy target can be arbitrarily biased.","rationale":"The reader's verdict is CONDITIONAL with medium correctness risk, and the identified weakest assumption already flags the Gaussian-posterior assumption and the invalid Taylor remainder analysis in Appendix I.2. My stress-test pass reaches the same conclusion, focusing on the most load-bearing sub-step: even for an exactly Gaussian posterior, the first-order Taylor expansion used to derive Eq. (25) has an uncontrolled Hessian remainder, and the paper's attempted error bound is mathematically wrong (squaring the gradient rather than the Hessian quadratic form). This undermines the central theoretical claim of a closed-form, analytically justified intermediate energy. The empirical D4RL results are still plausible evidence that the training procedure can produce useful guidance in practice, so the concern does not demand rejection; it does demand either correcting the error analysis, restricting the claim to settings where curvature is provably small, or reframing Eq. (25) as a heuristic approximation with empirical validation. Since the reader already recommended CONDITIONAL and this concern strengthens that recommendation rather than moving it, the verdict is unchanged.","tokens_in":27867,"tokens_out":4283,"duration_ms":50286,"concrete_test":"Construct a 1D or 2D synthetic problem with a known two-mode Gaussian mixture behavior policy, a quadratic Q(s,a) = c ||a||^2 with large c, and the paper's a_bar update from Eq. (46). Compute the exact intermediate energy E_t(s,a_t) = log integral mu0|t(a0|a_t,s) e^{beta Q(s,a0)} da0 by high-resolution numerical quadrature at several diffusion times t. Compare against Eq. (25) using the true posterior mean and variance. If the absolute error exceeds a small threshold (e.g., 0.2 in log-energy units) or the gradient of the approximate energy with respect to a_t differs in sign from the exact gradient, the analytic target is unreliable. Additionally, recompute the Taylor remainder with the true Hessian quadratic form; if it exceeds (1/2)nu^2, Appendix I.2's bound is demonstrably invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central derivation replaces log E_{a0~mu0|t}[e^{beta Q(s,a0)}] with the first-order Taylor expression in Eq. (25). The exact second-order remainder of that expansion about a_bar is R2(a0) = (1/2)(a0-a_bar)^T H_Q(c)(a0-a_bar), a Hessian quadratic form. Appendix I.2 instead writes R2(a) = (1/2)[(nabla Q)^T(a-a_bar)]^2 and bounds this by (1/2)nu^2 using ||a-a_bar|| = nu. This is not a bound on the second-order remainder: the gradient term is first-order and can vanish while the Hessian term is large, and the proposed bound omits ||H_Q|| entirely. The error also enters inside the exponential before the moment-generating step, so the bias in log E[e^{beta Q}] is on the order of the pointwise Taylor error, not its square. For neural-network Q functions in D4RL (typical action dimensions 6-60, trained by expectile regression), no regime is identified in which curvature is small enough for the first-order expansion to be valid. Even granting the Gaussian posterior assumption, Eq. (25) is therefore an uncontrolled approximation rather than a closed-form solution, and the guidance in Eq. (13) has no guarantee of pointing toward high-Q actions. The non-Gaussianity of the true posterior for multimodal offline action distributions compounds the issue, but the Taylor step alone is sufficient to invalidate the claimed analytic target.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Analytic Energy-guided Policy Optimization (AEPO), a diffusion-model-based policy for offline reinforcement learning. The method defines an intermediate energy E_t(s,a_t) = log E_{a0 ~ mu_{0|t}}[exp(beta Q(s,a0))] and approximates this intractable log-expectation by a Taylor expansion of Q around a reference action bar a, followed by an application of the Gaussian moment-generating function. The resulting target, Equation (25), is used to train an energy network E_Theta via regression, and the learned energy gradient guides the diffusion sampling process. The paper reports strong empirical results on D4RL benchmarks, including Gym-MuJoCo, maze2d, antmaze, and Adroit, with comparisons against more than 30 baselines.","tokens_in":28195,"tokens_out":5570,"duration_ms":57182,"significance":"If the analytical target in Equation (25) were valid, the paper would make a useful theoretical and practical contribution: it would replace a costly Monte Carlo estimation of the intermediate log-expectation energy with a closed-form expression and connect it to exact energy-guided diffusion in offline RL. The empirical evaluation is broad, with 30+ tasks and ablations on posterior approximations, Q-function training, and guidance rescaling, which is a clear strength. However, the central theoretical claim is not currently established: the error analysis supporting Equation (25) is mathematically invalid, and the Gaussian-posterior assumption is not justified for realistic offline action distributions. As a result, the analytic target remains an uncontrolled approximation, and the significance of the theoretical contribution is conditional on fixing these issues.","major_comments":[{"comment":"The error bound supporting Equation (25) is invalid. The exact second-order Taylor remainder of Q(s,a0) around bar a is R2(a0) = (1/2)(a0 - bar a)^T H_Q(c)(a0 - bar a), where H_Q is the Hessian of Q with respect to a0. Appendix I.2 instead writes R2(a) = (1/2)[(∇Q)^T(a - bar a)]^2 and bounds this by (1/2)nu^2 using ||∇Q|| ||a - bar a|| ≤ nu. This squared-gradient term is a first-order quantity, not a bound on the Hessian quadratic form: it can vanish even when the Hessian is large, and the omitted ||H_Q|| factor can make the true remainder arbitrarily large. Moreover, the Taylor error enters inside the exponential before the moment-generating step, so the bias in log E[e^{βQ}] is on the order of β times the pointwise Taylor error, not its square. No regime is identified in which the curvature of the neural-network Q function is small enough for the first-order expansion to be accurate, and no valid bound on the approximation error of Equation (25) is provided.","section":"Appendix I.2 and Eq. (25)"},{"comment":"The derivation assumes that the posterior mu_{0|t}(a0|at,s) is an isotropic Gaussian whose mean is given by Eq. (18) and whose scalar variance is given by Eq. (22). For an arbitrary offline action distribution, the exact posterior under Gaussian perturbation is not an isotropic Gaussian; in general it is a mixture or has no closed Gaussian form, so the moment-generating step leading to Eq. (17) is exact only under the stated Gaussian assumption. The paper does not quantify the posterior mismatch, nor does it validate the scalar variance estimate in Eq. (22), which discards all off-diagonal covariance structure. Because L_IE in Eq. (26) regresses directly onto this approximate target, the learned energy network inherits an unquantified bias from the posterior approximation in addition to the Taylor error.","section":"Sec. 3.2, Eqs. (18)-(22)"},{"comment":"The empirical evaluation would be much more convincing if it isolated the effect of the proposed analytic target. A natural control is to replace Equation (25) with a Monte Carlo estimator of log E[e^{βQ}] obtained by sampling a0 from the learned posterior mu_{0|t} during inference, while keeping the rest of the pipeline unchanged. Without such an ablation, it is difficult to attribute the strong D4RL results to the analytic approximation rather than to the guidance-rescaling heuristic in Eq. (45) or to the expectile-based Q-function training. Additionally, Table 4 reports ranges for beta, tau, and omega and Fig. 6 shows sensitivity analyses on D4RL tasks; the paper should state clearly whether these hyperparameters are selected per task on the same test sets, since that would affect the significance of the benchmark comparisons.","section":"Sec. 4 and Table 4"}],"minor_comments":[{"comment":"The sentence 'Reminding that the posterior distribution mu_{0|t}(a0|at,s) is also a Gaussian distribution' is overly strong; in general the posterior under the Gaussian perturbation is not Gaussian. Please rephrase as an assumption and justify it, or add a discussion of when it is a reasonable approximation.","section":"Sec. 3.1, after Eq. (15)"},{"comment":"The rescaling rule divides by ||∇at log pi_t(at|s)||; the case where this norm is zero is not discussed. Please state that the normalization is applied with a small numerical constant or otherwise handle the degenerate case.","section":"Sec. 3.4, Eq. (45)"},{"comment":"The notation in the definition bar a = a - nu * Q'_psi(s,a0)/||Q'_psi(s,a0)|| mixes a0 and a; please clarify which action is used in the gradient and in the Taylor expansion, since the expansion is in a0 around bar a and the target Q'(s,bar a) is later replaced by Q'(s,a).","section":"Appendix I.2, Eq. (46)"},{"comment":"The variance estimate in Eq. (22) uses a single scalar 1/d E||epsilon_theta||^2 for all dimensions. It would be helpful to report the empirical spread of ||epsilon_theta|| across timesteps, since the approximation quality likely varies strongly with t.","section":"Sec. 3.2, Eq. (22)"},{"comment":"The third posterior approximation in Eq. (44) is derived but never used in experiments. Please either remove it or explain why it is omitted from the main results.","section":"Appendix G.3"}],"recommendation":"major_revision","confidential_remarks":"I agree with the stress-test assessment: the Appendix I.2 error bound is indeed invalid, and the Gaussian-posterior assumption is a further correctness risk. The experimental results are broad and the paper has useful ingredients, but the central theoretical claim needs either a corrected error analysis or a clear reframing as a heuristic approximation. I recommend major revision rather than rejection, since the empirical contribution may be salvageable independently of the overclaimed analytic derivation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about AEPO. First, it introduces a genuinely new practical trick: replacing the intractable log-expectation intermediate energy in guided diffusion policies with the closed-form expression in Eq. (25), built from a first-order Taylor expansion of Q and the Gaussian moment-generating function. That is a reasonable engineering idea, and the empirical work is broad: 30+ D4RL tasks, competitive numbers against many baselines, and useful ablations on posterior choices, Q-training methods, and guidance rescaling. Second, the theoretical story is not as clean as the title suggests. The Taylor remainder analysis in Appendix I.2 is mathematically invalid, and the Gaussian posterior assumption is unverified. I agree with the stress-test note: the 'analytic' target is an uncontrolled approximation.\n\nLet me be specific. The exact intermediate energy E_t = log E[e^{\\beta Q}] was already known (Lu et al. 2023). The new content is the approximation. But the error bound in I.2 writes R2(a) = (1/2)[(\\nabla Q)^T(a-a_bar)]^2 and bounds it by (1/2)\\nu^2. That is not a second-order remainder; the actual remainder is (1/2)(a-a_bar)^T H_Q(c)(a-a_bar). The gradient term can vanish while the Hessian term is large, and the bound omits ||H_Q|| entirely. Worse, the Taylor error enters inside the exponential, so the bias in log E[e^{\\beta Q}] scales with the pointwise error, not its square. For neural-network Q functions in D4RL (action dimensions 6-60), no regime is identified where curvature is small. And the posterior is assumed isotropic Gaussian; real offline action distributions are often multimodal, and the paper never checks this. So Eq. (25) is a heuristic target, not a closed-form solution.\n\nThat said, the empirical method may still work, and the experiments support that possibility. The paper is honest about comparing many baselines, and the guidance-rescaling trick is simple and useful. The citation pattern looks fair: the key prior work (Lu et al.) is cited, and the novelty claim is appropriately scoped. The flaws are fixable: provide code, correct or remove the error bound, validate the posterior on a multimodal task, and report per-task hyperparameters and standard deviations (most tables lack error bars). None of these issues clearly breaks the method, but they prevent a full accept as-is.\n\nWho is this for? Researchers working on diffusion policies and guided sampling in offline RL. It deserves serious peer review, but with major revision. If I were the editor, I'd send it to reviewers and ask specifically for the theory to be fixed or the claims to be toned down.","headline":"Useful empirical recipe for diffusion-policy guidance, but the 'analytic' energy target is not actually controlled: Appendix I.2's error bound is wrong.","tokens_in":28737,"tokens_out":3218,"would_cite":true,"duration_ms":31590,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the intractable intermediate energy in energy-guided diffusion policies for offline RL reduces to a closed-form analytic target, and that training an energy network against this target improves guided policy…","keywords":["offline reinforcement learning","diffusion policy","energy-guided sampling","intermediate energy","log-expectation approximation","posterior Gaussian approximation","analytic guidance","D4RL benchmark"],"falsifier":"On a task whose behavior policy has two well-separated action modes, compute the paper's analytic target from Equation (25) and compare it with a high-sample Monte Carlo estimate of $\\log\\mathbb{E}_{a_0\\sim\\mu_{0|t}}[e^{\\beta Q(s,a_0)}]$ at matched denoising steps; if the energy error is large or does not track the gap between AEPO and an empirical-posterior variant, the Gaussian posterior approximation is the failing component. A small energy error on such multimodal tasks would instead support the approximation's practical reach.","tokens_in":27646,"feed_emoji":"🎯","tokens_out":14139,"duration_ms":126663,"temperature":0.7,"pith_summary":"This paper proposes Analytic Energy-guided Policy Optimization (AEPO), a way to turn the intractable log-expectation energy that steers diffusion policies in offline reinforcement learning into a closed-form analytic target. In energy-guided diffusion sampling, guiding a learned behavior policy toward high returns requires the intermediate energy $E_t(s,a_t)=\\log \\mathbb{E}_{a_0\\sim\\mu_{0|t}(a_0|a_t,s)}[e^{\\beta Q(s,a_0)}]$, which is hard to estimate because of the logarithm outside the expectation. The paper derives an approximation of this energy from a Taylor expansion of the Q function and a Gaussian approximation of the posterior over clean actions, then trains a neural energy model against that target to supply the guidance during generation. Across D4RL offline RL tasks, the resulting guided diffusion policy matches or surpasses a wide set of diffusion-based and non-diffusion baselines. The significance is that guidance becomes computable and trainable without hand-designed conditioning or contrastive estimation of the full expectation.","feed_headline":"Closed-form energy guides diffusion policies to top offline RL scores","feed_subtitle":"The method replaces the intractable log-expectation with an analytic estimate and beats strong baselines on D4RL.","key_machinery":"The machinery is Equation (25), the analytic simplification of the log-expectation intermediate energy. It converts the intractable quantity into three computable terms: a Q-value at a pivot action, a directional term involving the Q-gradient and the posterior mean, and a variance-scaled gradient norm. The posterior mean $\\tilde\\mu_{0|t}$ is reparameterized from the trained noise predictor, the isotropic variance $\\tilde\\sigma_{0|t}^2$ is estimated from the expected noise prediction norm, and the energy network $E_\\Theta(s,a_t,t)$ is fitted by regression against this target. Differentiating this trained network with respect to $a_t$ supplies the intermediate guidance used in the reverse diffusion update.","core_discovery":"The central discovery is that, under Gaussian diffusion transitions, the exact intermediate energy $E_t(s,a_t)=\\log\\mathbb{E}_{a_0\\sim\\mu_{0|t}}[e^{\\beta Q(s,a_0)}]$ can be approximated in closed form. Starting from the observation that previous classifier-guided and DPS-style energies are inexact because they do not preserve $p_t(x_t)\\propto q_t(x_t)e^{-E_t(x_t)}$, the paper expands the Q function around a pivot action $\\bar a$ and uses the moment-generating function of a Gaussian posterior to obtain the analytic target $\\beta Q(s,\\bar a)+\\beta Q'(s,\\bar a)^\\top(\\tilde\\mu_{0|t}-\\bar a)+\\frac12\\beta^2\\tilde\\sigma_{0|t}^2\\|Q'(s,\\bar a)\\|^2$. The posterior mean comes from the diffusion noise predictor, and two closed-form isotropic variance estimates are proposed. Training an energy network against this target and differentiating it during reverse diffusion yields the guided policy the paper evaluates.","pith_inferences":["The paper leaves implicit that Equation (25) could be evaluated online during inference without training a separate energy network, since all ingredients are functions of the current noisy action, the noise predictor, and the Q-gradient; the network only adds a fitted smoothing.","A testable extension is to relax the isotropic Gaussian posterior to a learned diagonal covariance or mixture, then measure whether the energy error on multimodal offline action distributions shrinks; the paper's ablations compare only two closed-form covariance estimates.","Because the approximation is first-order in Q, a second-order correction containing $\\nabla^2_a Q$ and the posterior covariance is a natural next test for whether curvature of the Q function, rather than posterior mismatch, is the dominant source of energy error."],"forward_implications":["AEPO gives a training signal for the intermediate energy that is analytic rather than contrastive or hand-specified, so the guidance term is reproducible from the Q function and noise predictor alone.","The method separates behavior cloning (the diffusion model) from policy improvement (the energy guidance), allowing either component to be swapped without redesigning the guided sampling loop.","Guidance rescaling makes performance robust across guidance strengths, so deployment does not require per-task tuning of the guidance weight.","On the D4RL tasks evaluated, including Gym-MuJoCo, maze2d, antmaze, and Adroit, the aggregate scores are at or above those of prior diffusion-based offline RL methods, which is the empirical claim that would need to hold for the method to be preferred."],"supporting_citations":[{"why":"Defines the exact log-expectation intermediate energy and the contrastive training method that AEPO replaces with an analytic target.","marker":"Lu et al., 2023"},{"why":"Introduces classifier-guided diffusion planning with Q values and defines the MSE intermediate energy the paper argues is inexact.","marker":"Janner et al., 2022"},{"why":"Introduces diffusion posterior sampling, the DPS intermediate energy the paper contrasts with the exact log-expectation energy.","marker":"Chung et al., 2022"},{"why":"Provides the reparameterized posterior mean and optimal reverse covariance estimate used for Posterior 1.","marker":"Bao et al., 2022b"},{"why":"Supplies the covariance reformulation used for Posterior 2.","marker":"Su, 2022"},{"why":"Supplies the expectile-regression Q and V training used to learn the critic in AEPO.","marker":"Kostrikov et al., 2021"},{"why":"Provides the D4RL datasets and normalized score metric used in the evaluations.","marker":"Fu et al., 2020"},{"why":"Defines the denoising diffusion objective used to train the noise predictor whose prediction norm enters the posterior variance.","marker":"Ho et al., 2020"}],"fun_headline_variants":["Closed-form energy unlocks diffusion RL gains","Analytic guidance fixes diffusion policy energy","Exact energy target boosts offline RL diffusion","AEPO: analytic energy for top D4RL scores","Closed-form log-expectation drives RL wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the posterior distribution of the clean action given a noisy action is an isotropic Gaussian whose mean and variance come from the noise predictor, so the moment-generating and Taylor steps can replace the true log-expectation; for real offline datasets this posterior is often neither Gaussian nor isotropic, and the resulting bias is not measured.","fun_headline_variants_meta":{"raw":{"variants":["Closed-form energy unlocks diffusion RL gains","Analytic guidance fixes diffusion policy energy","Exact energy target boosts offline RL diffusion","AEPO: analytic energy for top D4RL scores","Closed-form log-expectation drives RL wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1387,"prompt_tokens":931,"completion_tokens":456,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":387}},"tokens_in":547,"tokens_out":456,"duration_ms":4191,"temperature":1.0,"reasoning_tokens":387,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:09:15.302541+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a task whose behavior policy has two well-separated action modes, compute the paper's analytic target from Equation (25) and compare it with a high-sample Monte Carlo estimate of $\\log\\mathbb{E}_{a_0\\sim\\mu_{0|t}}[e^{\\beta Q(s,a_0)}]$ at matched denoising steps; if the energy error is large or does not track the gap between AEPO and an empirical-posterior variant, the Gaussian posterior approximation is the failing component. A small energy error on such multimodal tasks would instead support the approximation's practical reach.","supporting_citations":[],"review_version":1}