{"id":"ebac44f0-aff1-4934-98f5-9ac7567ac733","arxiv_id":"2507.01381","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"DSAC-D couples a diffusion-based value distribution network with a diffusion policy, but its headline state-of-the-art claim is contradicted by its own benchmark table.","lead":"This paper combines two diffusion models into one reinforcement learning algorithm: one diffusion model represents the distribution of future rewards, and another represents the action policy. The authors report top results on nine simulated control tasks and multiple driving styles on a small vehicle, but their own benchmark table does not support the top-result claim.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (11) does not define a policy update because the maximand is independent of pi, and the diffusion policy has no tractable log-density; the claimed MDPI convergence is therefore unsupported.","rationale":"The reader correctly flags the GMM entropy estimate as an unvalidated assumption, but the more fundamental defect is upstream: the policy-improvement step in Eq. (11) as written does not define an update, because pi appears nowhere in the maximand when the expectation is over the current policy or replay data. Even after a typo correction, the diffusion policy has no tractable log-density, so the log pi(a|s) term in the objective cannot be computed and no policy-gradient objective J_pi(omega) is specified in Algorithm 1. The GMM entropy H_hat is at best a proxy for the entropy term and is never related to the log-density needed for policy improvement; N and K are not stated, so the bias cannot be assessed. These are internal gaps, not disagreements with community consensus. The empirical claim is also internally contradicted: Table I shows DSAC-D is not state-of-the-art on HalfCheetah-v3 (16,409 vs DAC 17,177 and DSAC-T 17,025) and is tied on Reacher-v2 and InvertedDoublePendulum-v3, contrary to the abstract's 'SOTA in all 9 control tasks'. Because both the theoretical foundation and the headline empirical result are unsupported, the existing REJECT verdict should stand. I found no independent support such as a code release, machine-checked proofs, or parameter-free derivations that would offset these gaps.","tokens_in":1011,"tokens_out":1079,"duration_ms":109197,"concrete_test":"Independently re-derive Eq. (11): maximize E[ E_{Z^pi_old}[Z^pi_old(s,a)] - alpha log pi(a|s) ] over pi and verify that the optimizing pi satisfies pi_new(a|s) proportional to exp(Q^pi_old(s,a)/alpha), as in soft policy improvement; if the maximand is invariant to pi, the update is undefined. Then run DSAC-D on HalfCheetah-v3 with the diffusion policy replaced by a Gaussian-mixture policy with known density, and compare the GMM entropy estimate of Eqs. (16)-(18) with the true entropy over the same state batch; if the difference is nonzero and changes alpha in Eq. (19), the entropy assumption is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim, that Q^pi_k strictly monotonically increases under Multimodal Distributional Policy Iteration, rests on Eq. (11) being a genuine soft policy-improvement step. As written, pi_new = arg max_pi E[ E_{Z^pi_old}[Z^pi_old(s,a)] - alpha log pi_old(a|s) ]; the maximand contains only pi_old when the expectation is taken over the current policy or replay data, so the argmax over pi is undefined. If the intended expression replaces log pi_old by log pi, the update still cannot be evaluated for a diffusion policy, because a diffusion policy has no closed-form density: the paper never gives log pi(a|s) or a tractable surrogate. The only density estimate provided is the GMM entropy H_hat in Eqs. (16)-(18), which estimates H_s, not log pi(a|s), and whose bias depends on unstated K and N and on EM initialization. Algorithm 1 likewise invokes a policy-gradient objective J_pi(omega) that is never defined. Consequently, the asserted proof of monotonic improvement has a missing premise; the convergence claim is unsupported. The empirical SOTA claim is separately contradicted by Table I: on HalfCheetah-v3, DSAC-D's 16409 ± 477 is below DAC's 17177 ± 176 and DSAC-T's 17025 ± 157.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes DSAC-D, an actor-critic algorithm that uses a diffusion model for the value distribution and a diffusion policy, with a claimed Multimodal Distributional Policy Iteration (MDPI) convergence guarantee, state-of-the-art performance on nine MuJoCo tasks, reduced value-estimation bias, and multimodal trajectory behavior in real-vehicle tests. Section III-A introduces the distributional soft Bellman relation and the policy improvement step, Section III-B describes a diffusion value network, Section III-C uses a GMM entropy estimate for adaptive entropy tuning, and Section IV reports MuJoCo and vehicle experiments.","tokens_in":11178,"tokens_out":4774,"duration_ms":50850,"significance":"The paper targets a real gap: standard unimodal value distributions can be biased, and diffusion policies can represent multimodal action distributions. The combination is timely, and the real-vehicle demonstration of multimodal trajectories is a useful qualitative result. However, the manuscript's two headline claims—the MDPI monotone-convergence guarantee and SOTA on all nine tasks—are not supported. The policy improvement step in Eq. (11) is mathematically undefined as written, the convergence proof is only announced, the algorithm pseudo-code references objectives that are never defined, and Table I contradicts the SOTA claim. These are load-bearing issues, so the paper cannot be accepted in its present form.","major_comments":[{"comment":"The proposed multimodal distributional policy improvement step is not a valid argmax: the expression E[E_{Z^π_old}[Z^π_old(s,a)] − α log π_old(a|s)] is independent of the optimization variable π, so π_new is undefined. If the intended expression replaces log π_old by log π, the update still cannot be computed because the diffusion policy is not given a closed-form log-density anywhere in the paper. The statement immediately after Eq. (11) that 'we can prove' Q^{π_k} strictly monotonically increases is therefore unsupported; no theorem statement or proof appears.","section":"Section III-A, Eq. (11)"},{"comment":"The pseudo-code calls for updates θ ← θ − β_z ∇_θ J_z(θ) and ω ← ω + β_π ∇_ω J_π(ω), but neither objective is defined in the text. Eq. (15) gives a simplified diffusion loss for the value network, but no J_z; no policy-gradient objective J_π is given at all. Without these definitions, the algorithm as presented is not reproducible and the claimed connection to the MDPI framework cannot be checked.","section":"Algorithm 1 / Sections III-B and III-C"},{"comment":"The text claims DSAC-D 'outperforms all baselines in all tasks,' but Table I shows DSAC-D's HalfCheetah-v3 return (16409 ± 477) is below DAC (17177 ± 176) and DSAC-T (17025 ± 157), and Reacher-v2 is a tie across several algorithms. Table II also fails to show consistent bias suppression: on HalfCheetah-v3 DSAC-D's relative bias is 21.45, worse than SAC's −4.82, and on Reacher-v2 DDPG's −0.28 beats DSAC-D's −3.85. The SOTA and bias-suppression claims are therefore contradicted by the paper's own data.","section":"Section IV-B, Tables I and II"},{"comment":"The GMM entropy estimate Ĥ depends on unstated hyperparameters K and N and on EM initialization, and no validation is provided against the true (unknown) policy entropy. Since α is adapted from Ĥ and α enters Eq. (10) and the policy update, the correctness of the policy-improvement step is contingent on an unvalidated estimator. The paper should state K, N, and any safeguards, and provide evidence that Ĥ is unbiased or conservatively bounded.","section":"Section III-C, Eqs. (16)-(19)"}],"minor_comments":[{"comment":"The text says 'we run 20 parallel environment interactions per iteration' and later 'conduct 9 independent tests for each algorithm,' while Table I reports 'standard deviation over five runs.' These numbers should be reconciled.","section":"Section IV-A"},{"comment":"The notation in Eqs. (8)-(9) mixes r_i and r, and the definition of Z^π(s,a) at state-action pair (s,a) is inconsistent with the inner expectation notation; please clean up the indices.","section":"Section II"},{"comment":"Figure 4 measures PPO and TRPO by number of network updates while other algorithms are measured by environment interactions; this makes the comparison non-uniform and should be justified.","section":"Section IV-B"},{"comment":"The added exploration noise term a = a + λ_α * N(0,I) introduces a hyperparameter λ_α that is never defined or reported.","section":"Algorithm 1"},{"comment":"The claimed 'total average return improvement of over 10%' is not derived or tabulated anywhere; please report the aggregate calculation or remove the claim.","section":"Abstract and Section V"}],"recommendation":"reject","confidential_remarks":"Given that the central theoretical claim is unsupported and the empirical headline is contradicted by the paper's own tables, I do not think revision within the normal scope can fix this; rejection is appropriate. If the authors can supply a valid policy-improvement derivation, define the missing objectives, and correct the empirical claims, a new submission might be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Nothing wrong with the kernel here. The idea of putting a diffusion model on both the value distribution and the policy, with a GMM entropy estimate to keep the SAC-style objective computable, is a legitimate new combination. The real-vehicle experiments showing multi-modal trajectories are a useful addition; that sort of hardware demonstration is rare in RL papers and hard to fake. I also appreciate that the authors are explicit that the diffusion policy has no closed-form density, which is why they fit a GMM.\n\nThat said, the central theoretical claim is not supported. Equation (11) is the policy improvement step, but as written the maximand contains only pi_old—the expectation is over Z^pi_old and the log term is log pi_old. The argmax over pi is undefined. If they intended log pi inside, the paper never provides a tractable log-density for the diffusion policy; the GMM entropy H_s estimates the per-state entropy, not log pi(a|s). So the monotone convergence claim for MDPI has a missing premise. Algorithm 1 references a policy-gradient objective J_pi(omega) that is never defined. These aren't cosmetic issues; the guarantee the paper advertises is not derived.\n\nThe empirical side is also oversold. Table I shows HalfCheetah-v3 where DSAC-D (16409) is below both DAC (17177) and DSAC-T (17025), and Reacher is a three-way tie. That directly contradicts the abstract's 'SOTA in all 9 control tasks' and the text's 'outperforms all baselines in all tasks.' The bias-suppression claim in Table II is also mixed: on HalfCheetah and Swimmer, DSAC-D's bias is worse than SAC's. The paper's own numbers undo the headline.\n\nReproducibility is a problem too: the GMM component count K, the action sample count N, and the diffusion hyperparameters are not stated, so the entropy estimate that drives alpha and the policy update cannot be replicated.\n\nWho benefits? Someone working on diffusion RL might take the idea and fix the theory. But as it stands the paper is not reliable enough to cite for its claims. I'd send it out for review only if I expected the referees to enforce a major rewrite; otherwise a desk reject with an invitation to resubmit a corrected version would be appropriate. My verdict is reject in current form.","headline":"Interesting dual-diffusion combination undermined by an undefined policy update and an overclaimed SOTA result.","tokens_in":11722,"tokens_out":3233,"would_cite":false,"duration_ms":39887,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dual-diffusion actor-critic algorithm claims to learn multimodal value distributions and policies while converging to an optimal policy.","keywords":["Distributional reinforcement learning","Diffusion policy","Diffusion value network","Soft actor-critic","Multimodal value distribution","Entropy estimation","Value estimation bias","Continuous control"],"falsifier":"Compare the GMM-entropy estimate $\\hat H$ against a large-sample Monte Carlo entropy of the diffusion policy on the same states; if the two diverge substantially, Eq. (11) is not an exact policy improvement and the MDPI convergence theorem does not apply. Alternatively, check the published Table I: DSAC-D does not achieve the maximum return in every task, since DAC's HalfCheetah-v3 score (17177) exceeds DSAC-D's (16409), so the 'SOTA in all 9 tasks' claim is contradicted by the paper's own table.","tokens_in":10633,"feed_emoji":"🤖","tokens_out":5937,"duration_ms":58527,"temperature":0.7,"pith_summary":"This paper tries to establish that reinforcement learning agents can represent both value functions and policies as multimodal distributions and still converge to an optimal policy. The authors propose DSAC-D, which models the value distribution with a diffusion value network and the policy with a diffusion policy network, alternating between distributional policy evaluation and improvement steps they call Multimodal Distributional Policy Iteration. They claim that the multimodal Q-value strictly increases with each iteration, and that the resulting algorithm suppresses value-estimation bias and achieves state-of-the-art returns on all nine MuJoCo tasks tested, with real-vehicle experiments showing multimodal driving trajectories. The practical stakes are that distributional RL need not be limited to unimodal value representations, and that diffusion models can supply the needed flexibility in both value and policy networks.","feed_headline":"Dual-diffusion RL algorithm targets value bias and SOTA control scores","feed_subtitle":"DSAC-D's distributional policy iteration claims monotone Q-improvement and multimodal driving trajectories.","key_machinery":"The central object is the distributional Bellman equation with entropy regularisation, Eq. (9), which the paper calls the multimodal distributional policy evaluation step. Because diffusion policies lack an analytical density, the entropy $\\hat H$ driving the adaptive temperature $\\alpha$ is estimated by fitting a Gaussian mixture model to $N$ sampled actions per state via EM; the policy improvement step then uses this estimated entropy. The diffusion value network generates return samples through reverse denoising, and the combination forms the MDPI loop the proof of monotone Q-improvement relies on.","core_discovery":"The central claim is that a distributional Bellman self-consistency condition with an entropy term, $Z^\\pi(s,a) \\overset{D}{=} r + \\gamma[Z(s',a') - \\alpha\\log\\pi(a'|s')]$, together with a greedy policy improvement step, yields a Multimodal Distributional Policy Iteration (MDPI) in which the multimodal Q-value $Q^{\\pi_k}(s,a)$ strictly monotonically increases with iteration $k$. The paper further claims that implementing the value distribution with a diffusion network (generating return samples by reverse denoising) and the policy with a diffusion policy network yields DSAC-D, an algorithm that learns multimodal policies, suppresses value overestimation bias, and outperforms baselines on all nine MuJoCo control tasks, with real-vehicle tests producing multimodal obstacle-avoidance trajectories.","pith_inferences":["A testable extension would be to replace the GMM entropy estimate with a learned density or a Monte Carlo estimator; if convergence behavior changes materially, the entropy estimate is the active ingredient.","The same MDPI framework could be applied to risk-sensitive objectives (e.g., CVaR) by changing the distributional criterion, since the framework is distributional rather than expectation-only.","The diffusion value network could be reused as an uncertainty estimator for out-of-distribution detection, since it returns a full return distribution rather than a scalar."],"forward_implications":["If MDPI converges as claimed, distributional RL can represent multimodal value functions without losing soft policy improvement guarantees.","DSAC-D's dual-diffusion design gives a concrete recipe for applying diffusion models inside actor-critic algorithms without an analytical policy density.","The claimed >10% average return improvement and bias suppression would make DSAC-D a strong candidate for high-dimensional continuous control and autonomous driving.","Real-vehicle demonstrations imply the same machinery can produce multimodal driving trajectories reflecting distinct driving styles.","The GMM entropy estimator (Eq. 18) becomes a reusable component for entropy-regularised diffusion policies."],"supporting_citations":[{"why":"Establishes distributional RL and the distributional Bellman equation, which the value network is built on.","marker":"[4]"},{"why":"Defines denoising diffusion probabilistic models, the generator behind the diffusion value and policy networks.","marker":"[7]"},{"why":"Introduces diffusion actor-critic with entropy regulator, the direct precursor for the diffusion policy and entropy scheme.","marker":"[11]"},{"why":"Supplies the maximum-entropy soft actor-critic objective and adaptive entropy coefficient.","marker":"[16]"},{"why":"Provides the distributional soft actor-critic baseline (DSAC-T) that this paper extends.","marker":"[17]"},{"why":"Serves as an off-policy baseline in the MuJoCo comparisons.","marker":"[13]"},{"why":"Serves as an off-policy baseline emphasising clipped Q-learning.","marker":"[14]"},{"why":"Serves as an on-policy baseline in the MuJoCo comparisons.","marker":"[15]"},{"why":"Serves as an on-policy baseline in the MuJoCo comparisons.","marker":"[18]"},{"why":"Provides the GOPS solver and vehicle environment used in the real-vehicle tests.","marker":"[9]"}],"fun_headline_variants":["Dual-diffusion RL targets value bias, hits SOTA on MuJoCo","DSAC-D: diffusion value and policy nets improve returns by 10%","Multimodal distributional policy iteration with diffusion policy","Diffusion-based distributional RL suppresses bias, tops all tasks","Distributional SAC with diffusion policy achieves SOTA across 9 tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The algorithm's convergence rests on the assumption that the entropy of the diffusion policy is accurately captured by a Gaussian mixture fit to a small number of sampled actions; if that estimate is biased, the policy update is not a true soft policy improvement and the claimed monotone convergence can fail.","fun_headline_variants_meta":{"raw":{"variants":["Dual-diffusion RL targets value bias, hits SOTA on MuJoCo","DSAC-D: diffusion value and policy nets improve returns by 10%","Multimodal distributional policy iteration with diffusion policy","Diffusion-based distributional RL suppresses bias, tops all tasks","Distributional SAC with diffusion policy achieves SOTA across 9 tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000357,"raw_usage":{"total_tokens":1939,"prompt_tokens":954,"completion_tokens":985,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":892}},"tokens_in":570,"tokens_out":985,"duration_ms":10299,"temperature":1.0,"reasoning_tokens":892,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:53:15.168713+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the GMM-entropy estimate $\\hat H$ against a large-sample Monte Carlo entropy of the diffusion policy on the same states; if the two diverge substantially, Eq. (11) is not an exact policy improvement and the MDPI convergence theorem does not apply. Alternatively, check the published Table I: DSAC-D does not achieve the maximum return in every task, since DAC's HalfCheetah-v3 score (17177) exceeds DSAC-D's (16409), so the 'SOTA in all 9 tasks' claim is contradicted by the paper's own table.","supporting_citations":[{"cited_title":"& Munos, R","cited_arxiv_id":null,"evidence_quote":"Establishes distributional RL and the distributional Bellman equation, which the value network is built on."},{"cited_title":"& Abbeel, P","cited_arxiv_id":null,"evidence_quote":"Defines denoising diffusion probabilistic models, the generator behind the diffusion value and policy networks."},{"cited_title":"& Others Diffusion actor-critic with entropy regulator.Advances In Neural Information Processing Systems.37pp","cited_arxiv_id":null,"evidence_quote":"Introduces diffusion actor-critic with entropy regulator, the direct precursor for the diffusion policy and entropy scheme."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the distributional soft actor-critic baseline (DSAC-T) that this paper extends."},{"cited_title":"& Meger, D","cited_arxiv_id":null,"evidence_quote":"Serves as an off-policy baseline emphasising clipped Q-learning."},{"cited_title":"& Moritz, P","cited_arxiv_id":null,"evidence_quote":"Serves as an on-policy baseline in the MuJoCo comparisons."},{"cited_title":"& Others GOPS: A general optimal control problem solver for autonomous driving and industrial control appli- cations.Communications In Transportation Research.3pp","cited_arxiv_id":null,"evidence_quote":"Provides the GOPS solver and vehicle environment used in the real-vehicle tests."}],"review_version":1}