{"id":"7926a87c-13c3-450e-871b-cd5783d6e471","arxiv_id":"2502.00361","paper_version":4,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Diffusion policies can be trained online with reweighted score matching using only Q-functions, and the resulting DPMD and SDAC algorithms beat SAC and prior diffusion-policy RL on most MuJoCo tasks.","lead":"This paper introduces a reweighted score matching loss that lets diffusion policies be trained online from value functions, without sampling from the optimal policy. The resulting algorithms, DPMD and SDAC, beat soft actor-critic and prior diffusion-policy RL methods on most MuJoCo benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SDAC's soft critic relies on an unvalidated Gaussian surrogate for log pi(a|s) and a self-normalized loss whose at-dependent reweighting is not the Eq (14) expectation; this is the weakest load-bearing link for the max-entropy claim.","rationale":"The reader's weakest assumption identifies the same gap: the implemented SDAC uses an approximation for log pi(a|s) and a normalization that changes the objective relative to Eq (14). My read of the derivations confirms that the RSM identities in Eqs (13)-(14) are mathematically sound; the DPMD loss is a valid weighted denoising score matching objective, and the SDAC expectation with unnormalized exp(Q/lambda) weights is also valid. The vulnerability is in the practical algorithm: the critic target in Appendix C.1.2 substitutes the Gaussian noise log-density for the true policy log-density, and the logsumexp step in Algorithm 2 is an at-dependent reweighting, not merely an s-dependent one. These are not covered by the RSM theorem, so the max-entropy interpretation of SDAC is conditional on unvalidated approximations. The empirical overstatement in Section 5.2.2 (e.g., SDAC on Ant, DPMD on Swimmer in Table 1) is a separate reporting issue and does not change the methodological verdict. The reader's CONDITIONAL verdict remains appropriate; no adjustment is needed.","tokens_in":20636,"tokens_out":17110,"duration_ms":186825,"concrete_test":"Run SDAC on Walker2d-v4 with the published hyperparameters, but replace the Gaussian log-likelihood in Eq (29) with a Monte Carlo estimate of log pi(a|s) for the actual batch-argmax-plus-Gaussian policy (sample many action batches per state-action and estimate the resulting density), keeping the policy-update loss unchanged. If the final return or the soft-Bellman residual of the learned Q changes materially (e.g., more than 5-10%), the Gaussian surrogate is load-bearing and the current evidence does not establish that SDAC optimizes the max-entropy objective.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The RSM identity and the DPMD/SDAC loss derivations in Eqs (13)-(14) are internally consistent as expectations; the mathematical core appears sound. The load-bearing gap is in the translation to Algorithm 2 and Appendix C.1.2. The SDAC critic target (29) replaces log pi(a|s) with the log density of the additive Gaussian exploration noise, but the actual policy is a batch-argmax selection followed by additive noise, followed by reverse diffusion sampling at the next state. The density of this composed policy is not the Gaussian log-density, so the learned Q is not the soft Q of the executed or learned policy. If the critic is biased, then exp(Q/lambda) is not the max-entropy energy function that the RSM loss is designed to match, and the SDAC max-entropy claim does not follow from the theorem. Separately, the logsumexp normalization in Algorithm 2 is described as 'another reweighting on the s space,' but because the sampled a0^(i) depend on a_t through phi0|t(·|a_t), the normalization is an a_t-dependent, self-normalized importance weighting. The resulting finite-K objective is not the expectation in Eq (14); it may preserve the pointwise score optimum in the infinite-capacity limit, but the paper provides no analysis of the finite-K bias or its effect on the learned policy. These are practical-algorithm premises, not flaws in the RSM identity itself, but they are exactly the premises needed for the SDAC portion of the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Reweighted Score Matching (RSM), a family of losses for training diffusion policies in online RL. The core idea is that the denoising score matching objective can be reweighted by any strictly positive function without changing the pointwise optimal score, and that two particular reweightings yield tractable losses for policy mirror descent (DPMD, Eq. 13) and max-entropy RL (SDAC, Eq. 14), avoiding both sampling from the target policy and backpropagation through the reverse diffusion process. The authors provide a toy Boltzmann-sampling experiment, a MuJoCo benchmark study against SAC, PPO, TD3, and several recent diffusion-policy RL methods, and a comparison of GPU memory and training time.","tokens_in":20981,"tokens_out":17034,"duration_ms":172605,"significance":"The RSM identity is clean and useful: if it holds, diffusion policies can be trained online with cost comparable to denoising score matching while preserving the expressiveness of energy-based policies. The derivations of Eqs. (13) and (14) are internally consistent under the stated support assumptions, and the mathematical core is not circular: the reweighted losses follow from the definitions of the target policies and standard score-matching identities. The release of code and the memory/time comparisons are also valuable. The main risk is the gap between the exact RSM losses and the implemented SDAC algorithm, specifically the Gaussian surrogate for log pi(a|s) in the critic, plus an overstatement in the empirical summary. These issues are load-bearing for the SDAC max-entropy claim and for the claimed all-task superiority, respectively.","major_comments":[{"comment":"The soft policy evaluation for SDAC replaces log pi(a|s) in Eq. (29) with the log density of the additive Gaussian exploration noise. The policy actually executed is the batch-argmax selector (16) composed with additive Gaussian noise and, at the next state, reverse diffusion sampling; the log-density of this composed policy is not the Gaussian log-density. Consequently the learned Q is not the soft Q of the executed or learned policy, and exp(Q/lambda) is not the energy function of the max-entropy policy (5) that Eq. (14) is designed to match. The paper provides no analysis or empirical validation of this surrogate; it should either estimate the true log pi(a|s) of the composed policy, prove that the surrogate introduces negligible bias in the Bellman target, or explicitly reposition SDAC's max-entropy claim as heuristic.","section":"Appendix C.1.2, Eq. (29); Section 4.2 (Log likelihood computation)"},{"comment":"The logsumexp normalization replaces the weight exp(Q(s, a0)/lambda) in Eq. (14) with a self-normalized weight. For finite K this estimator is biased for the expectation in Eq. (14), and because the sampled a0^(i) are drawn from phi0|t(·|at), the normalizer depends on at; calling it 'another reweighting on the s space' is inaccurate. In the infinite-K limit the normalized objective is a different but still valid RSM loss (it reweights by h_t rather than h_t Z(s) p_t), so the pointwise optimum is preserved in the infinite-capacity limit, but the paper should state this reasoning and either analyze the finite-K bias or acknowledge it as an unmodeled approximation.","section":"Algorithm 2, line 11; Section 4.2 (Numerical stability)"},{"comment":"The claim that the proposed algorithms 'outperform all the baselines in all OpenAI Gym MuJoCo environments' is contradicted by Table 1. On Pusher, DPMD (-30.43 ± 0.37) and SDAC (-32.53 ± 5.27) are worse than TD3 (-25.07 ± 1.01) and PPO (-25.52 ± 2.60); on Swimmer, DPMD (79.3 ± 52.5) is worse than DPPO (106.1 ± 6.5) and DACER (103.0 ± 45.8). The abstract's more guarded statement, 'outperform recent diffusion-policy online RLs on most tasks,' is consistent with the table, but the main-text overclaim should be corrected.","section":"Section 5.2.2, Table 1"},{"comment":"The EMA normalization of Q changes the exponent in Eq. (13) from Q/lambda to (Q - mu)/(sigma lambda). This is not a pure constant shift: it changes the effective KL coefficient to sigma lambda and introduces a batch-dependent scaling of the loss. The paper should either state that this is a heuristic stabilization separate from the exact derivation of Eq. (13), or show that the normalized objective preserves the optimum of Eq. (13).","section":"Algorithm 1, line 10; Section 4.2 (Numerical stability)"}],"minor_comments":[{"comment":"The displayed loss uses a single exp term without an explicit average over i; as written it is ambiguous whether the K samples are averaged, and the parentheses around Q_e(s, a0^(i))/lambda_e are misplaced. It should read something like (1/K) sum_i exp(Q_e(s, a0^(i))/lambda_e - log sum_j exp(Q_e(s, a0^(j))/lambda_e)) ||s_theta(at;s,t) - nabla_{at} log phi_t(a0^(i)|at)||^2.","section":"Algorithm 2, line 11"},{"comment":"The text says 'sample a batch of cations and choose the one with the highest Q-value'; 'cations' should be 'actions'.","section":"Section 4.2"},{"comment":"The summation runs from t=0 to T, but the forward process and noise schedule are defined for t=1,...,T; the sum should presumably start at t=1.","section":"Eq. (7)"},{"comment":"The integral J(s) uses p0(a|s) where the action variable should be a0, and the proportionality factor between phi0|t and qt|0 is not written out; this makes the reverse sampling trick harder to follow than necessary.","section":"Remark 4.1"},{"comment":"The GitHub URL contains spaces ('diffusion policy online rl') and will not resolve as a URL; it should be a single hyphenated or percent-encoded string.","section":"Section 5.2.1"},{"comment":"The environment names are inconsistent: the table uses 'Inverted2Pendulum' while the text uses 'InvertedDoublePendulum'; the naming should be unified.","section":"Table 1 and main text"},{"comment":"In the main-text display of Eq. (11), the notation 'at ~ q0|t(·|at)' uses at on both sides of the conditioning bar; this should be a0 ~ q0|t(·|at).","section":"Proposition 3.1"}],"recommendation":"major_revision","confidential_remarks":"The mathematical core of the RSM identity is sound, and the DPMD loss derivation is solid modulo the EMA normalization. The SDAC portion needs substantial revision because the implemented critic and loss do not implement the claimed max-entropy objective. The empirical overclaim in Section 5.2.2 should be fixed regardless. With those changes, the paper is likely suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The RSM trick is real. The derivation of Eqs. (13)–(14) is standard reweighted score matching plus a change of measure, and it does what the paper says: train a diffusion policy against Q without sampling from the target policy or backpropagating through the reverse diffusion. The two losses are genuinely new as RL algorithms; DPMD especially is clean and likely useful. I was less worried than I expected about novelty. Vincent, Karras, and iDEM all carry the reweighting idea in some form, but the mirror-descent and max-entropy instantiations for online RL are not in the prior work cited.\n\nThe math holds up under scrutiny. The Tweedie and score-matching derivations are correct under the stated support assumptions, and the reverse-sampling trick is a neat way to swap p0 for a known sampler. I would trust Proposition 3.1 and the loss derivations.\n\nThe empirical section is where the paper starts to strain. The abstract and Section 5.2.2 claim a 41.7% improvement on Walker2d over SAC, but Table 1 gives DPMD at 35% and SDAC at 23.6%. That number is not in the table. The all-environment superiority claim also does not survive a close read: SDAC loses to DPMD on Ant and underperforms DIPO on Swimmer. The computation table is nice, but the training details are thin. The code URL in the text has a typo, and key hyperparameters like SDAC's K and the exploration noise scale after batch action sampling are not reported. Reproduction would involve guesswork.\n\nThe weakest link is SDAC's practical critic. Eq. (14) is fine as an expectation, but Algorithm 2 replaces log pi(a|s) with the log density of the additive Gaussian noise during soft policy evaluation. The executed policy is batch-argmax plus Gaussian noise and then reverse diffusion at the next state, so the Gaussian log-density is not the density of that composed policy. The learned Q is not necessarily the soft Q of the executed or learned policy. The logsumexp normalization is also an a_t-dependent self-normalized weighting, not a reweighting on the s-space, and the paper gives no finite-K analysis. These issues do not damage the RSM identity itself, but they mean the max-entropy claim for SDAC is not actually established by the theorem. The authors should either analyze the batch-argmax policy density or soften the claim to what the experiments support.\n\nWho is this for? Anyone building diffusion policies for online control. I would send it to a serious referee with instructions to push hard on the empirical claims and ask for SDAC ablations. The central idea is worth engaging.","headline":"The RSM identity is real and the two losses are genuine contributions; the paper's main soft spots are empirical overclaiming and an unanalyzed SDAC critic approximation, not the core math.","tokens_in":21512,"tokens_out":4643,"would_cite":true,"duration_ms":51249,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reweighted score matching lets diffusion policies train online without sampling from the optimal policy, and yields algorithms that outperform SAC on MuJoCo benchmarks.","keywords":["diffusion policy","online reinforcement learning","reweighted score matching","energy-based policy","policy mirror descent","max-entropy reinforcement learning","denoising score matching","soft actor-critic"],"falsifier":"On a small MDP with a known soft-optimal policy, run SDAC with the Gaussian-noise surrogate for $\\log \\pi(a|s)$ and with an unbiased estimate of the diffusion policy's true log-likelihood, and compare both learned action distributions to Boltzmann($Q/\\lambda$); disagreement beyond sampling error, or a mismatch between Algorithm 2's empirical loss and the expectation in Eq (14), would show the implemented objective is not the exact reweighted loss.","tokens_in":20421,"feed_emoji":"🤖","tokens_out":10460,"duration_ms":89198,"temperature":0.7,"pith_summary":"The paper's central claim is that denoising score matching can be reweighted by any strictly positive function without changing its optimum, and that this freedom removes the main obstacle to training diffusion policies in online reinforcement learning, namely the need to sample from the optimal policy. With a Q-function-derived reweighting, the loss becomes a tractable expectation over the current policy or any full-support sampling distribution, yielding two algorithms: DPMD for policy mirror descent and SDAC for max-entropy policies. This matters because diffusion policies can represent the multimodal energy-based policies that arise as closed-form solutions in these settings, while Gaussian actors such as SAC cannot. The paper reports that the resulting algorithms outperform recent diffusion-policy RL methods on most MuJoCo tasks, with more than 120% improvement over SAC on Humanoid and Ant.","feed_headline":"Reweighted loss drops optimal-policy sampling for diffusion RL","feed_subtitle":"A reweighted loss keeps score matching's optimum but drops its sampling need; DPMD beats SAC by >120% on two MuJoCo tasks.","key_machinery":"The central object is the reweighted score-matching loss $L_g(\\theta;s,t) = \\int g(a_t;s) \\|s_\\theta(a_t;s,t) - \\nabla_{a_t} \\log p_t(a_t|s)\\|^2 \\, da_t$, whose minimizer is the noise-perturbed score for any strictly positive $g$. The argument rides on Tweedie's identity, which expresses the unknown score $\\nabla_{a_t} \\log p_t(a_t|s)$ as an expectation of the Gaussian corruption score over the posterior $q_{0|t}$, and on the reverse sampling trick, which notes that $\\phi_{0|t}(\\tilde{a}_0|a_t) = \\mathcal{N}(\\tilde{a}_0; a_t/\\sqrt{\\bar\\alpha_t}, ((1-\\bar\\alpha_t)/\\bar\\alpha_t)I)$ is proportional to the forward kernel $q_{t|0}(a_t|a_0)$ and shares its score with respect to $a_t$. These identities let importance weights $\\exp(Q(s,a_0)/\\lambda)$ be evaluated on samples that are not from the optimal policy, converting intractable score matching against the optimal policy into tractable expectations over the current policy or a chosen density $h_t$.","core_discovery":"The paper establishes that, for any strictly positive weighting $g(a_t;s)$, minimizing $L_g(\\theta;s,t) = \\int g(a_t;s) \\|s_\\theta(a_t;s,t) - \\nabla_{a_t} \\log p_t(a_t|s)\\|^2 \\, da_t$ has the same optimum as standard denoising score matching, namely $s_{\\theta^*}(a_t;s,t) = \\nabla_{a_t} \\log p_t(a_t|s)$. With $g = Z_{MD}(s) p_t(a_t|s)$, the loss equals, up to constants, the DPMD objective $\\mathbb{E}_{a_0 \\sim \\pi_{\\mathrm{old}}, a_t \\sim q_{t|0}}[\\exp(Q(s,a_0)/\\lambda) \\|s_\\theta(a_t;s,t) - \\nabla_{a_t} \\log q_{t|0}(a_t|a_0)\\|^2]$, which is estimable from samples of the current policy. With $g = h_t(a_t|s) Z(s) p_t(a_t|s)$ and the reverse sampling trick, where $\\phi_{0|t}(\\tilde{a}_0|a_t) \\propto q_{t|0}(a_t|\\tilde{a}_0)$ with matching scores, the SDAC objective becomes an expectation over any full-support $h_t$ and the reverse Gaussian. The paper claims this preserves the optimal solution and the low cost of denoising score matching while eliminating the need to sample from the target policy, and validates the two algorithms on MuJoCo.","pith_inferences":["The same reweighting construction should transfer to any setting where a reliable energy or Q-function is available, such as offline RL with learned critics, removing the need for expert action samples for diffusion-policy fitting.","Since the SDAC loss's sampling distribution $h_t$ is arbitrary subject to full support, it could be used as an explicit exploration knob: shaping $h_t$ changes which actions are weighted in training while the optimum stays fixed, an avenue the paper does not fully explore.","A natural stress test of the identity would be to apply RSM to other energy-based policy objectives, such as KL-regularized objectives beyond the two studied here, and check numerically that the learned score still matches the target Boltzmann distribution."],"forward_implications":["Diffusion policies can be trained for online RL with roughly the memory and compute cost of denoising score matching, avoiding backpropagation through the reverse diffusion process.","DPMD trains a diffusion policy to represent the mirror-descent update $\\pi_{MD}$ in (2), and SDAC trains one to represent the max-entropy policy in (5), without projecting these energy-based policies onto Gaussians.","Because any strictly positive $g$ works, RSM also applies to general known-energy distributions; the paper demonstrates this on Gaussian-mixture and Two-Moon Boltzmann sampling tasks.","Combined with batch action sampling and tuned exploration noise, the reweighted losses keep exploitation while preserving the expressiveness of diffusion policies.","On MuJoCo benchmarks the best DPMD/SDAC variants outperform all compared baselines, exceeding SAC by more than 120% on Humanoid and Ant."],"supporting_citations":[{"why":"Defines the denoising score matching loss (7) that RSM generalizes by reweighting.","marker":"(Ho et al., 2020)"},{"why":"Provides the noise-perturbed EBM and score-based view that justifies diffusion policies as solvers for energy-based policies.","marker":"(Song & Ermon, 2019)"},{"why":"Supplies Tweedie's identity, used to equate the reweighted loss with a tractable denoising objective.","marker":"(Efron, 2011)"},{"why":"Establishes the equivalence between score matching and denoising score matching that underlies Proposition 3.1.","marker":"(Vincent, 2011)"},{"why":"SAC is the max-entropy baseline whose Gaussian projection limits expressiveness and which DPMD/SDAC are compared against.","marker":"(Haarnoja et al., 2018)"},{"why":"Provides the batch action sampling trick (16) adopted in both algorithms.","marker":"(Ding et al., 2024a)"},{"why":"Supplies the automatic exploration-noise scheme and the reverse-process policy parametrization that RSM avoids.","marker":"(Wang et al., 2024)"},{"why":"QSM is a diffusion-policy online RL baseline whose score-from-learned-Q approach RSM improves on.","marker":"(Psenka et al., 2023)"},{"why":"Motivates the Boltzmann sampling applications of RSM beyond RL.","marker":"(Akhound-Sadegh et al., 2024)"}],"fun_headline_variants":["Reweighted score matching lets diffusion RL skip target sampling","Diffusion RL without optimal-policy sampling: RSM wins big","DPMD: reweighted loss, no target sampling, +120% vs SAC","Soft diffusion actor-critic: reweighted loss for online RL","No sampling needed: diffusion policies rival SAC in MuJoCo"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Q-function used as the energy oracle is reliable, and that SDAC's practical approximations, namely the Gaussian-noise stand-in for $\\log \\pi(a|s)$ and the per-sample logsumexp normalization treated as a state-only reweighting, do not change the learned optimum.","fun_headline_variants_meta":{"raw":{"variants":["Reweighted score matching lets diffusion RL skip target sampling","Diffusion RL without optimal-policy sampling: RSM wins big","DPMD: reweighted loss, no target sampling, +120% vs SAC","Soft diffusion actor-critic: reweighted loss for online RL","No sampling needed: diffusion policies rival SAC in MuJoCo"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000265,"raw_usage":{"total_tokens":1692,"prompt_tokens":1115,"completion_tokens":577,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":731,"completion_tokens_details":{"reasoning_tokens":484}},"tokens_in":731,"tokens_out":577,"duration_ms":5319,"temperature":1.0,"reasoning_tokens":484,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T19:18:38.065452+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a small MDP with a known soft-optimal policy, run SDAC with the Gaussian-noise surrogate for $\\log \\pi(a|s)$ and with an unbiased estimate of the diffusion policy's true log-likelihood, and compare both learned action distributions to Boltzmann($Q/\\lambda$); disagreement beyond sampling error, or a mismatch between Algorithm 2's empirical loss and the expectation in Eq (14), would show the implemented objective is not the exact reweighted loss.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the automatic exploration-noise scheme and the reverse-process policy parametrization that RSM avoids."}],"review_version":1}