Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

This paper claims that the intractable entropy of diffusion policies can be replaced by a factorized evidence lower bound, making diffusion policies trainable from scratch in online cooperative multi-agent reinforcement learning with 2.5x t

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 21:57 UTC pith:XZREHQPH

load-bearing objection OMAD is a plausible new combination—online off-policy MARL with factorized diffusion policies and a joint distributional critic—but the entropy-surrogate mechanism at its core is never actually measured, and a couple of the empirical claims are slightly over-stated. the 4 major comments →

arxiv 2602.18291 v2 pith:XZREHQPH submitted 2026-02-20 cs.AI

Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies

Zhuoran Li , Hai Zhong , Xun Wang , Qingxin Xia , Lihua Zhang , Longbo Huang This is my paper
classification cs.AI
keywords diffusion policiesmulti-agent reinforcement learningmaximum entropy RLcentralized training with decentralized executiondistributional Q-learningsample efficiencyonline off-policy learningcooperative control
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish that diffusion policies—highly expressive generative action models normally reserved for offline settings—can be trained from scratch in online multi-agent reinforcement learning. Its central move is to replace the intractable entropy of the joint diffusion policy with a tractable evidence lower bound (ELBO), which lets maximum-entropy exploration and soft value targets be computed. Around that surrogate the paper builds a CTDE algorithm: a centralized distributional critic guides synchronized updates of decentralized per-agent diffusion policies, with an auto-tuned temperature that keeps the ELBO above a target. If the claims hold, online MARL can use multimodal, expressive policies and reach strong coordination with substantially fewer environment interactions. The key caveat the paper itself acknowledges is that the ELBO only lower-bounds the true entropy, and the gap is unquantified during training.

Core claim

OMAD's core claim is that the obstacle to online diffusion MARL—intractable likelihoods blocking entropy-based exploration—can be bypassed by optimizing a relaxed objective. Theorem 1 shows the joint policy entropy is lower-bounded by the sum of per-agent ELBOs along denoising trajectories; the algorithm maximizes this bound, incorporates it into the distributional Bellman target, and uses it to auto-tune the temperature. A centralized distributional critic, trained with batch normalization and stop-gradients, supplies a shared signal for simultaneously updating all decentralized diffusion policies. The paper reports that on MPE and MAMuJoCo this yields state-of-the-art returns across 10 tas

What carries the argument

The load-bearing object is the factorized entropy ELBO: for factorized decentralized policies, Theorem 1 states that the joint policy entropy is lower-bounded by the sum of per-agent evidence lower bounds along each agent's denoising chain. This bound converts the intractable max-entropy objective into a computable sum, enabling three mechanisms: an exploration bonus in the policy objective, an entropy-augmented target for the centralized distributional critic, and an auto-tuned temperature that keeps the aggregate ELBO at a target level. The critic's joint distributional value is what synchronizes the per-agent updates and gives the denoising trajectories a shared coordination signal.

Load-bearing premise

The whole exploration and temperature-tuning mechanism rests on treating the ELBO—a lower-bound estimate of policy entropy—as if it were the true entropy; the gap between them is a divergence the paper never measures and only assumes shrinks to zero at convergence.

What would settle it

Track the per-agent gap between the ELBO and a Monte-Carlo estimate of the true marginal policy entropy during early training in a small MPE task. If the gap is large at the stage when exploration matters, or if the auto-tuned temperature rises while the true entropy falls, then the exploration and coordination benefits attributed to the entropy surrogate are not coming from the mechanism the paper claims.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Diffusion policies become usable in online MARL without offline demonstrations, broadening generative policy classes to non-stationary cooperative settings.
  • A tractable entropy surrogate plus a centralized distributional critic provides a template for other implicit or generative policies in multi-agent reinforcement learning.
  • The reported 2.5x to 5x sample-efficiency gains imply that coordinated exploration in high-dimensional joint action spaces can be achieved with far fewer environment interactions.
  • Auto-tuning the temperature against the ELBO removes manual exploration tuning, a practical obstacle to adopting diffusion policies.
  • The synchronized joint update structure suggests a way to mitigate non-stationarity without relying on fragmented per-agent losses in CTDE.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural test the paper does not run is to replace the distributional critic with an expectation-based joint critic while keeping the entropy surrogate; that would separate the contribution of the value distribution from the contribution of the ELBO to the reported gains.
  • If the ELBO gap stays small in other tasks, the same relaxation could extend to other likelihood-free policies, such as normalizing flows or energy-based policies, in online MARL; the paper only establishes the mechanism for diffusion.
  • The state-coverage experiment shows broader exploration, but the link from coverage to final returns is correlational; a direct comparison against a uniform-noise exploration baseline would test whether expressiveness or entropy is the driving cause.
  • The diffusion baselines were implemented with centralized execution, so a stronger comparison would test CTDE variants of those baselines to rule out the possibility that some gains come from execution-mode mismatch rather than the proposed method itself.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes OMAD, an online off-policy multi-agent RL framework in which each agent's policy is a denoising diffusion process. Because the exact entropy of a diffusion policy is intractable, the authors prove (Theorem 1) a variational lower bound on the joint entropy of factorized decentralized policies and use this bound in three places: the policy objective (Eq. 11), the distributional critic target (Eq. 12), and the automatic temperature update (Eq. 15). Training is centralized with a joint distributional critic using CrossQ, while execution is decentralized. The paper reports large sample-efficiency gains over HATD3, HASAC, and two diffusion baselines on MPE and MAMuJoCo, with state-coverage and ablation studies.

Significance. If the proposed surrogate were a faithful entropy estimate, this would be a meaningful step: it gives a tractable way to use expressive diffusion policies in online MARL, with a clean factorization and a single synchronized policy loss. The main inequality in Theorem 1 is correct, and the empirical study is broad (10 tasks, 5 seeds, ablations, state-coverage visualization). However, the central identification of the ELBO with entropy is not established; in fact l = H - KL, so the objective being optimized is not maximum entropy as claimed. This is a major correctness-risk concern, not a disagreement with consensus, because it can be tested empirically and theoretically. The empirical results are strong but do not yet isolate the proposed mechanism.

major comments (4)
  1. [§4 Theorem 1, Appendix B, §5.2 Eqs. (11)-(15)] The surrogate l is not an entropy estimator. From the definitions in Appendix B, l_{πθ_i} = H(πθ_i(a_i0|s)) - KL(πθ_i(a_i1:H|a_i0,s) || π(a_i1:H|a_i0,s)). Thus Eq. (11) maximizes entropy minus a KL penalty, Eq. (12) uses this biased quantity as a reward, and Eq. (15) tunes α against it. The Appendix D.3 claim that the gap vanishes at convergence is not proved and is questionable: for a perfectly fitted diffusion model, the reverse conditional given a0 is the forward posterior, not the forward noising conditional, so this KL is generally positive. Please quantify the gap (e.g., with sample-based entropy estimates) or reframe the objective as an entropy-consistency regularizer.
  2. [§5.2 Eq. (15)] Because l = H - KL, the auto-tuned constraint l ≥ H_target corresponds to H ≥ H_target + KL. If KL is large or varies during training, α no longer controls the true exploration entropy; it can inject excessive stochasticity when KL is large or be too conservative when KL is small. This undermines the maximum-entropy interpretation and the claim that the method maintains a principled exploration budget.
  3. [§6.3, Fig. 6] No ablation isolates the proposed entropy surrogate. Figure 6 varies α for the full OMAD objective but never sets α=0 or substitutes a different exploration bonus, so the observed gains could come from the CrossQ-style distributional critic or the diffusion policy class alone. The central mechanism should be tested by (i) removing the l bonus, (ii) replacing it with a matched Gaussian-entropy bonus, and (iii) reporting the empirical gap H-l during training.
  4. [Table 3, §6.2] The headline SOTA claim is overstated. Table 3 reports the maximum over training of the mean return; with 5 seeds this is an optimistic statistic and not a standard final-performance measure. Also, in Physical Deception (N=2) OMAD (45.1±4.1) is not better than HASAC (45.4±3.1), so 'new state-of-the-art across 10 diverse scenarios' is inaccurate; the D.3 text correctly says 'nearly all'. Please report final or median performance and correct the abstract.
minor comments (5)
  1. [§1 Introduction] The paragraph beginning 'In order to address the limitations of policy expressiveness...' is duplicated verbatim.
  2. [Table 1 and Algorithm 1] The policy soft-update parameter is listed as ρ=0.0, but the text calls the update 'soft' and Algorithm 1 writes θ'_i ← ρθ'_i + (1-ρ)θ_i. With ρ=0 this is a hard copy; please clarify the intended update.
  3. [Eq. (14)] The term log πθ_i(a_iH|s) is independent of θ because a_iH is drawn from a fixed Gaussian prior; it can be dropped or the notation should indicate that it is a constant.
  4. [Fig. 3] The text describes 'blue and orange areas' as the coverage, but the figure caption lists red, green, blue, and orange regions. The legend/description should be harmonized.
  5. [§6.1] The diffusion baselines MADPMD and MASDAC are not described in the main text; please state their key differences from OMAD (e.g., centralized vs decentralized execution, objective) in the experiment section.

Circularity Check

0 steps flagged

No significant circularity: OMAD's ELBO surrogate is a valid, explicitly acknowledged lower bound, and the SOTA claims rest on external benchmarks rather than on a fitted parameter or a self-citation chain.

full rationale

The derivation chain is self-contained rather than circular. Theorem 1 is proved from the standard factorized-policy assumption (π(a|s)=∏_i π_i(a_i|s), cited to HARL [74]) and the standard variational ELBO for a diffusion trajectory; the single-agent version is cited to DIME [8], but Appendix B re-derives the bound directly, so its validity does not rest on that citation. The surrogate l_{πθ_i} used in Eqs. (11)-(15) is explicitly presented as a tractable lower bound on the intractable entropy, not as the entropy itself, and the paper openly states it is substituting the ELBO for exact entropy in the max-entropy objective, the distributional critic target, and the temperature dual. The gap H − Σ_i l_{πθ_i} is a KL divergence; it is not a fitted parameter, a renamed target, or a quantity defined to equal the output. Appendix D.3 explicitly admits the approximation gap is unquantified and only 'theoretically vanishes' at convergence, which is a correctness/verification limitation rather than a circularity. The centralized critic is trained against external environment rewards, and the claimed improvements are evaluated on external benchmarks (MPE, MAMuJoCo) against multiple baselines. The only self-citation ([35], offline MARL diffusion) appears in related work and is not load-bearing for the online coordination mechanism. No uniqueness theorem or ansatz is imported from the authors' prior work to force the design choice. Therefore no step in the paper reduces by construction to its own inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 3 axioms · 0 invented entities

The central theoretical claim rests on the factorization assumption and on the unproven adequacy of the ELBO surrogate. The empirical performance claims depend on task-specific hyperparameters (V_max, learning rates) and hand-chosen thresholds (H_target, ξ). No new physical or mathematical entities are postulated.

free parameters (5)
  • target_entropy_H_target = 4*dim(A)
    Hand-chosen exploration budget used in temperature auto-tuning (Eq. 15, Table 1).
  • initial_temperature_alpha = 1.0
    Initial value for the temperature before auto-tuning (Table 1).
  • distributional_entropy_coefficient_xi = 0.005
    Ad-hoc entropy regularization on the critic's return distribution (Eq. 13, Table 1).
  • V_max_per_task = e.g., 1200 (Ant 2x4), 50000 (Walker2d), 200 (MPE)
    Support bound for the distributional critic, hand-tuned per task; Fig. 4 shows sensitivity to this value.
  • learning_rate_per_task = varies 3.5e-5 to 1e-3
    Task-specific learning rates in Table 2; standard hyperparameter tuning affects the empirical results.
axioms (3)
  • domain assumption Joint policy factorizes exactly: πθ(a|s) = Π_i πθ_i(a_i|s)
    Used to decompose joint entropy into a sum of per-agent entropies (Theorem 1, Eq. 16). It excludes correlated action dependencies; standard in CTDE but a restriction.
  • ad hoc to paper The ELBO l_πθi is a valid surrogate for the true policy entropy in the maximum-entropy objective
    Eq. (11) and (15) replace H with l. The gap (a KL divergence) is not quantified; Appendix D.3 states it 'theoretically vanishes' at convergence but provides no proof.
  • ad hoc to paper The entropy bonus on the distributional critic (ξ H(Z_ϕ)) improves stability
    Added in Eq. (13) without derivation; no ablation isolates this term.

pith-pipeline@v1.3.0-alltime-deepseek · 21276 in / 14412 out tokens · 126789 ms · 2026-08-02T21:57:03.455861+00:00 · methodology

0 comments
read the original abstract

Online Multi-Agent Reinforcement Learning (MARL) is a prominent framework for efficient agent coordination. Crucially, enhancing policy expressiveness is pivotal for achieving superior performance. Diffusion-based generative models are well-positioned to meet this demand, having demonstrated remarkable expressiveness and multimodal representation in image generation and offline settings. Yet, their potential in online MARL remains largely under-explored. A major obstacle is that the intractable likelihoods of diffusion models impede entropy-based exploration and coordination. To tackle this challenge, we propose among the first \underline{O}nline off-policy \underline{MA}RL framework using \underline{D}iffusion policies (\textbf{OMAD}) to orchestrate coordination. Our key innovation is a relaxed policy objective that maximizes scaled joint entropy, facilitating effective exploration without relying on tractable likelihood. Complementing this, within the centralized training with decentralized execution (CTDE) paradigm, we employ a joint distributional value function to optimize decentralized diffusion policies. It leverages tractable entropy-augmented targets to guide the simultaneous updates of diffusion policies, thereby ensuring stable coordination. Extensive evaluations on MPE and MAMuJoCo establish our method as the new state-of-the-art across $10$ diverse tasks, demonstrating a remarkable $2.5\times$ to $5\times$ improvement in sample efficiency.

Figures

Figures reproduced from arXiv: 2602.18291 by Hai Zhong, Lihua Zhang, Longbo Huang, Qingxin Xia, Xun Wang, Zhuoran Li.

Figure 1
Figure 1. Figure 1: The CTDE framework of OMAD. The left panel illustrates Decentralized Execution, [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Learning curves comparing OMAD with state-of-the-art online MARL baselines (HATD3 [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: State coverage comparison on representative dimensions (1 and 21) at 250k steps. We [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Ablation study on Distributional Q-function hyperparameters. [PITH_FULL_IMAGE:figures/full_fig_p013_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Ablation study on the number of denoising steps. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Efficacy of Auto-Tuning vs. Fixed Entropy Coefficients. [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Multi-agent particle environments (MPE) and Multi-agent HalfCheetah task in MuJoCo [PITH_FULL_IMAGE:figures/full_fig_p027_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of learned diffusion policies across four distinct MAMuJoCo tasks. We [PITH_FULL_IMAGE:figures/full_fig_p032_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Coordinated Diffusion: Generating Multi-Agent Behavior Without Multi-Agent Demonstrations

    cs.RO 2026-05 unverdicted novelty 7.0

    CoDi decomposes the multi-agent diffusion score into pre-trained single-agent policies plus a gradient-free cost guidance term to generate coordinated behavior from single-agent data alone.

Reference graph

Works this paper leans on

78 extracted references · 3 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Reducing over- estimation bias in multi-agent domains using double centralized critics.Advances in Neural Information Processing Systems: Deep Reinforcement Learning Workshop, 2019

    Johannes Ackermann, Volker Gabler, Takayuki Osa, and Masashi Sugiyama. Reducing over- estimation bias in multi-agent domains using double centralized critics.Advances in Neural Information Processing Systems: Deep Reinforcement Learning Workshop, 2019

    2019

  2. [2]

    Is conditional generative modeling all you need for decision making? InThe Eleventh International Conference on Learning Representations, 2023

    Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? InThe Eleventh International Conference on Learning Representations, 2023

    2023

  3. [3]

    Heterogeneous agent q-weighted policy optimization

    Anonymous. Heterogeneous agent q-weighted policy optimization. InSubmitted to The Four- teenth International Conference on Learning Representations, 2025. under review

    2025

  4. [4]

    A distributional perspective on reinforce- ment learning

    Marc G Bellemare, Will Dabney, and R´ emi Munos. A distributional perspective on reinforce- ment learning. InInternational conference on machine learning, pages 449–458. PMLR, 2017

    2017

  5. [5]

    The complexity of decentralized control of markov decision processes.Mathematics of operations research, 27(4):819–840, 2002

    Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes.Mathematics of operations research, 27(4):819–840, 2002. xv

    2002

  6. [6]

    Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity

    Aditya Bhatt, Daniel Palenicek, Boris Belousov, Max Argus, Artemij Amiranashvili, Thomas Brox, and Jan Peters. Crossq: Batch normalization in deep reinforcement learning for greater sample efficiency and simplicity. InThe Twelfth International Conference on Learning Repre- sentations, 2024

    2024

  7. [7]

    Video generation models as world simulators.OpenAI Blog, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators.OpenAI Blog, 2024

    2024

  8. [8]

    Dime: Diffusion-based maximum entropy reinforcement learn- ing

    Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palenicek, Jan Peters, Georgia Chal- vatzaki, and Gerhard Neumann. Dime: Diffusion-based maximum entropy reinforcement learn- ing. InForty-second International Conference on Machine Learning, 2025

    2025

  9. [9]

    Offline reinforcement learn- ing via high-fidelity generative behavior modeling

    Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learn- ing via high-fidelity generative behavior modeling. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  10. [10]

    Multi-agent systems for robotic autonomy with llms

    Junhong Chen, Ziqi Yang, Haoyuan G Xu, Dandan Zhang, and George Mylonas. Multi-agent systems for robotic autonomy with llms. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 4194–4204, 2025

    2025

  11. [11]

    Novelty-guided data reuse for efficient and diversified multi-agent reinforcement learning

    Yangkun Chen, Kai Yang, Jian Tao, and Jiafei Lyu. Novelty-guided data reuse for efficient and diversified multi-agent reinforcement learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 15930–15938, 2025

    2025

  12. [12]

    Continuous q-score matching: Diffusion guided reinforcement learning for continuous-time control

    HUA Chengxiu, Jiawen Gu, and Yushun Tang. Continuous q-score matching: Diffusion guided reinforcement learning for continuous-time control. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  13. [13]

    Diffusion policy: Visuomotor policy learning via action diffusion

    Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin Burchfiel, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. InProceedings of Robotics: Science and Systems (RSS), 2023

    2023

  14. [14]

    Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533, 2020

    Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip HS Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge?arXiv preprint arXiv:2011.09533, 2020

    Pith/arXiv arXiv 2011

  15. [15]

    Diffusion-based reinforcement learning via q-weighted variational policy optimization

    Shutong Ding, Ke Hu, Zhenhao Zhang, Kan Ren, Weinan Zhang, Jingyi Yu, Jingya Wang, and Ye Shi. Diffusion-based reinforcement learning via q-weighted variational policy optimization. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  16. [16]

    Maximum entropy reinforcement learning with diffusion policy

    Xiaoyi Dong, Jian Cheng, and Xi Sheryl Zhang. Maximum entropy reinforcement learning with diffusion policy. InForty-second International Conference on Machine Learning, 2025. xvi

    2025

  17. [17]

    Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning

    Linjiajie Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, and Bingyi Jing. Diffusion actor-critic: Formulating constrained policy iteration as diffusion noise regression for offline reinforcement learning. InThe Thirteenth International Conference on Learning Representations, 2024

    2024

  18. [18]

    Counterfactual multi-agent policy gradients

    Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon White- son. Counterfactual multi-agent policy gradients. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  19. [19]

    INS: Interaction-aware synthesis to enhance offline multi-agent reinforcement learning

    Yuqian Fu, Yuanheng Zhu, Jian Zhao, Jiajun Chai, and Dongbin Zhao. INS: Interaction-aware synthesis to enhance offline multi-agent reinforcement learning. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  20. [20]

    Reinforcement learning with deep energy-based policies

    Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. InInternational conference on machine learning, pages 1352–

  21. [21]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. InInternational conference on machine learning, pages 1861–1870. Pmlr, 2018

    2018

  22. [22]

    Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023

    Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Denoising diffusion probabilistic models.Advances in Neural Information Processing Systems, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in Neural Information Processing Systems, 33:6840–6851, 2020

    2020

  24. [24]

    Value diffusion reinforcement learning

    Xiaoliang Hu, Fuyun Wang, Tong Zhang, and Zhen Cui. Value diffusion reinforcement learning. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  25. [25]

    Diffuseloco: Real-time legged locomotion control with diffusion from offline datasets

    Xiaoyu Huang, Yufeng Chi, Ruofeng Wang, Zhongyu Li, Xue Bin Peng, Sophia Shao, Borivoje Nikolic, and Koushil Sreenath. Diffuseloco: Real-time legged locomotion control with diffusion from offline datasets. In8th Annual Conference on Robot Learning, 2024

    2024

  26. [26]

    Diffuseloco: Real-time legged locomotion control with diffusion from offline datasets

    Xiaoyu Huang, Yufeng Chi, Ruofeng Wang, Zhongyu Li, Xue Bin Peng, Sophia Shao, Borivoje Nikolic, and Koushil Sreenath. Diffuseloco: Real-time legged locomotion control with diffusion from offline datasets. InConference on Robot Learning, pages 1567–1589. PMLR, 2025

    2025

  27. [27]

    Planning with diffusion for flexible behavior synthesis

    Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. InInternational Conference on Machine Learning, 2022

    2022

  28. [28]

    Agent- centric actor-critic for asynchronous multi-agent reinforcement learning

    Whiyoung Jung, Sunghoon Hong, Deunsol Yoon, Kanghoon Lee, and Woohyung Lim. Agent- centric actor-critic for asynchronous multi-agent reinforcement learning. InForty-second In- ternational Conference on Machine Learning, 2025. xvii

    2025

  29. [29]

    Efficient diffusion poli- cies for offline reinforcement learning.Advances in Neural Information Processing Systems, 36:67195–67212, 2023

    Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion poli- cies for offline reinforcement learning.Advances in Neural Information Processing Systems, 36:67195–67212, 2023

    2023

  30. [30]

    Enhancing cooperative multi-agent reinforcement learning with state modelling and adversarial exploration

    Andreas Kontogiannis, Konstantinos Papathanasiou, Yi Shen, Giorgos Stamou, Michael M Zavlanos, and George Vouros. Enhancing cooperative multi-agent reinforcement learning with state modelling and adversarial exploration. InForty-second International Conference on Ma- chine Learning, 2025

    2025

  31. [31]

    Gta: Generative trajectory aug- mentation with guidance for offline reinforcement learning.Advances in Neural Information Processing Systems, 37:56766–56801, 2024

    Jaewoo Lee, Sujin Yun, Taeyoung Yun, and Jinkyoo Park. Gta: Generative trajectory aug- mentation with guidance for offline reinforcement learning.Advances in Neural Information Processing Systems, 37:56766–56801, 2024

    2024

  32. [32]

    Dof: A diffusion factorization framework for offline multi-agent decision making

    Chao Li, Ziwei Deng, Chenxing Lin, Wenqi Chen, Yongquan Fu, Weiquan Liu, Chenglu Wen, Cheng Wang, and Siqi Shen. Dof: A diffusion factorization framework for offline multi-agent decision making. InInternational Conference on Learning Representations, 2025

    2025

  33. [33]

    Race: improve multi-agent reinforcement learning with representation asymmetry and collaborative evolution

    Pengyi Li, Jianye Hao, Hongyao Tang, Yan Zheng, and Xian Fu. Race: improve multi-agent reinforcement learning with representation asymmetry and collaborative evolution. InInter- national Conference on Machine Learning, pages 19490–19503. PMLR, 2023

    2023

  34. [34]

    Revisiting cooperative off-policy multi-agent reinforcement learning

    Yueheng Li, Guangming Xie, and Zongqing Lu. Revisiting cooperative off-policy multi-agent reinforcement learning. InForty-second International Conference on Machine Learning, 2025

    2025

  35. [35]

    Beyond conservatism: Diffusion policies in offline multi-agent reinforcement learning, 2023

    Zhuoran Li, Ling Pan, and Longbo Huang. Beyond conservatism: Diffusion policies in offline multi-agent reinforcement learning, 2023

    2023

  36. [36]

    Maximum entropy heterogeneous-agent reinforcement learning

    Jiarong Liu, Yifan Zhong, Siyi Hu, Haobo Fu, QIANG FU, Xiaojun Chang, and Yaodong Yang. Maximum entropy heterogeneous-agent reinforcement learning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  37. [37]

    Multi-agent actor-critic for mixed cooperative-competitive environments.Advances in neural information processing systems, 30, 2017

    Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments.Advances in neural information processing systems, 30, 2017

    2017

  38. [38]

    Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning

    Cheng Lu, Huayu Chen, Jianfei Chen, Hang Su, Chongxuan Li, and Jun Zhu. Contrastive energy prediction for exact energy-guided diffusion sampling in offline reinforcement learning. InInternational Conference on Machine Learning, pages 22825–22855. PMLR, 2023

    2023

  39. [39]

    Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.Advances in neural information processing systems, 35:5775–5787, 2022

    Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps.Advances in neural information processing systems, 35:5775–5787, 2022

    2022

  40. [40]

    Synthetic experience replay

    Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. Advances in Neural Information Processing Systems, 36:46323–46344, 2023. xviii

    2023

  41. [41]

    Efficient online reinforcement learning for diffusion policy

    Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy. InForty-second International Conference on Machine Learning, 2025

    2025

  42. [42]

    Coordinated multi-robot exploration under communication constraints using decentralized markov decision processes

    La¨ etitia Matignon, Laurent Jeanpierre, and Abdel-Illah Mouaddib. Coordinated multi-robot exploration under communication constraints using decentralized markov decision processes. InTwenty-sixth AAAI conference on artificial intelligence, 2012

    2012

  43. [43]

    Springer, 2016

    Frans A Oliehoek and Christopher Amato.A concise introduction to decentralized POMDPs. Springer, 2016

    2016

  44. [44]

    Optimal and approximate q-value functions for decentralized pomdps.Journal of Artificial Intelligence Research, 32:289–353, 2008

    Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps.Journal of Artificial Intelligence Research, 32:289–353, 2008

    2008

  45. [45]

    Regularized softmax deep multi-agent q-learning.Advances in Neural Information Processing Systems, 34:1365–1377, 2021

    Ling Pan, Tabish Rashid, Bei Peng, Longbo Huang, and Shimon Whiteson. Regularized softmax deep multi-agent q-learning.Advances in Neural Information Processing Systems, 34:1365–1377, 2021

    2021

  46. [46]

    Imitating human behaviour with diffusion models

    Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, et al. Imitating human behaviour with diffusion models. InThe Eleventh International Conference on Learning Representations, 2024

    2024

  47. [47]

    Facmac: Factored multi-agent centralised policy gradients.Advances in Neural Information Processing Systems, 34:12208–12221, 2021

    Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin B¨ ohmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients.Advances in Neural Information Processing Systems, 34:12208–12221, 2021

    2021

  48. [48]

    Learning a diffusion model policy from rewards via q-score matching

    Michael Psenka, Alejandro Escontrela, Pieter Abbeel, and Yi Ma. Learning a diffusion model policy from rewards via q-score matching. InInternational Conference on Machine Learning, pages 41163–41182. PMLR, 2024

    2024

  49. [49]

    Offline multi- agent reinforcement learning via score decomposition, 2025

    Dan Qiao, Wenhao Li, Shanchao Yang, Hongyuan Zha, and Baoxiang Wang. Offline multi- agent reinforcement learning via score decomposition, 2025

    2025

  50. [50]

    Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. InInternational conference on machine learning, pages 4295–4304. PMLR, 2018

    2018

  51. [51]

    Diffusion policy policy op- timization

    Allen Z Ren, Justin Lidard, Lars Lien Ankile, Anthony Simeonov, Pulkit Agrawal, Anirudha Majumdar, Benjamin Burchfiel, Hongkai Dai, and Max Simchowitz. Diffusion policy policy op- timization. InCoRL 2024 Workshop on Mastering Robot Manipulation in a World of Abundant Data, 2024

    2024

  52. [52]

    Cambridge University Press, 2019

    Simo S¨ arkk¨ a and Arno Solin.Applied stochastic differential equations, volume 10. Cambridge University Press, 2019. xix

    2019

  53. [53]

    Deep unsuper- vised learning using nonequilibrium thermodynamics

    Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. InInternational Conference on Machine Learning, pages 2256–2265. PMLR, 2015

    2015

  54. [54]

    Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning

    Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. InInternational conference on machine learning, pages 5887–5896. PMLR, 2019

    2019

  55. [55]

    Generative modeling by estimating gradients of the data distribution.Advances in Neural Information Processing Systems, 32, 2019

    Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution.Advances in Neural Information Processing Systems, 32, 2019

    2019

  56. [56]

    Score-based generative modeling through stochastic differential equations

    Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021

    2021

  57. [57]

    Value- decomposition networks for cooperative multi-agent learning based on team reward

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zam- baldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning based on team reward. InPro- ceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems...

    2085

  58. [58]

    Multiagent cooperation and competition with deep reinforcement learning.PloS one, 12(4):e0172395, 2017

    Ardi Tampuu, Tambet Matiisen, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning.PloS one, 12(4):e0172395, 2017

    2017

  59. [59]

    Multi-agent reinforcement learning: Independent vs

    Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. InPro- ceedings of the tenth international conference on machine learning, pages 330–337, 1993

    1993

  60. [60]

    Qplex: Duplex dueling multi-agent q-learning

    Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. Qplex: Duplex dueling multi-agent q-learning. InInternational Conference on Learning Representations, 2021

    2021

  61. [61]

    Diffusion actor-critic with entropy regulator

    Yinuo Wang, Likun Wang, Yuxuan Jiang, Wenjun Zou, Tong Liu, Xujie Song, Wenxuan Wang, Liming Xiao, Jiang Wu, Jingliang Duan, et al. Diffusion actor-critic with entropy regulator. InAdvances in Neural Information Processing Systems, 2024

    2024

  62. [62]

    Enhanced dacer algorithm with high diffusion efficiency.arXiv preprint arXiv:2505.23426, 2025

    Yinuo Wang, Likun Wang, Mining Tan, Wenjun Zou, Xujie Song, Wenxuan Wang, Tong Liu, Guojian Zhan, Tianze Zhu, Shiqi Liu, et al. Enhanced dacer algorithm with high diffusion efficiency.arXiv preprint arXiv:2505.23426, 2025

    arXiv 2025

  63. [63]

    Diffusion policies as an expressive policy class for offline reinforcement learning

    Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. InThe Eleventh International Conference on Learning Representations, 2023

    2023

  64. [64]

    Learning intractable multimodal policies with repa- rameterization and diversity regularization

    Ziqi Wang, Jiashun Liu, and Ling Pan. Learning intractable multimodal policies with repa- rameterization and diversity regularization. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025. xx

    2025

  65. [65]

    Latent diffusion planning for imitation learning

    Amber Xie, Oleh Rybkin, Dorsa Sadigh, and Chelsea Finn. Latent diffusion planning for imitation learning. InForty-second International Conference on Machine Learning, 2025

    2025

  66. [66]

    Multi-agent reinforcement learning with communication-constrained priors

    Guang Yang, Jingwen Qiao, Tianpei Yang, Yanqing Wu, Jing Huo, Xingguo Chen, and Yang Gao. Multi-agent reinforcement learning with communication-constrained priors. InThe Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  67. [67]

    Policy representation via diffusion probability model for reinforcement learning.arXiv preprint arXiv:2305.13122, 2023

    Long Yang, Zhixiong Huang, Fenghao Lei, Yucun Zhong, Yiming Yang, Cong Fang, Shiting Wen, Binbin Zhou, and Zhouchen Lin. Policy representation via diffusion probability model for reinforcement learning.arXiv preprint arXiv:2305.13122, 2023

    Pith/arXiv arXiv 2023

  68. [68]

    Fine-tuning diffusion policies with backpropagation through diffusion timesteps, 2025

    Ningyuan Yang, Jiaxuan Gao, Feng Gao, Yi Wu, and Chao Yu. Fine-tuning diffusion policies with backpropagation through diffusion timesteps, 2025

    2025

  69. [69]

    The surprising effectiveness of ppo in cooperative multi-agent games.Advances in neural in- formation processing systems, 35:24611–24624, 2022

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games.Advances in neural in- formation processing systems, 35:24611–24624, 2022

    2022

  70. [70]

    Turbodiffusion: Accelerating video diffusion models by 100-200 times

    Jintao Zhang, Kaiwen Zheng, Kai Jiang, Haoxu Wang, Ion Stoica, Joseph E Gonzalez, Jianfei Chen, and Jun Zhu. Turbodiffusion: Accelerating video diffusion models by 100-200 times. arXiv preprint arXiv:2512.16093, 2025

    arXiv 2025

  71. [71]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. InProceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023

    2023

  72. [72]

    Scaling in-the-wild training for diffusion- based illumination harmonization and editing by imposing consistent light transport

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Scaling in-the-wild training for diffusion- based illumination harmonization and editing by imposing consistent light transport. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  73. [73]

    Revisiting multi-agent world modeling from a diffusion-inspired perspective

    Yang Zhang, Xinran Li, Jianing Ye, Shuang Qiu, Delin Qu, Xiu Li, Chongjie Zhang, and Chenjia Bai. Revisiting multi-agent world modeling from a diffusion-inspired perspective. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

    2025

  74. [74]

    Heterogeneous-agent reinforcement learning.Journal of Machine Learning Research, 25(32):1– 67, 2024

    Yifan Zhong, Jakub Grudzien Kuba, Xidong Feng, Siyi Hu, Jiaming Ji, and Yaodong Yang. Heterogeneous-agent reinforcement learning.Journal of Machine Learning Research, 25(32):1– 67, 2024

    2024

  75. [75]

    Smarts: An open-source scalable multi- agent rl training school for autonomous driving

    Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, et al. Smarts: An open-source scalable multi- agent rl training school for autonomous driving. InConference on robot learning, pages 264–

  76. [76]

    Madiff: Offline multi-agent learning with diffusion models.Advances in Neural Information Processing Systems, 37:4177–4206, 2024

    Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Er- mon, and Weinan Zhang. Madiff: Offline multi-agent learning with diffusion models.Advances in Neural Information Processing Systems, 37:4177–4206, 2024. xxi

    2024

  77. [77]

    Maximum entropy inverse reinforcement learning

    Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. InAaai, volume 8, pages 1433–1438. Chicago, IL, USA, 2008. A Detailed Related Works Here we propose our detailed related works about online MARL, diffusion policies in offline RL and single-agent online RL scenarios. We also discuss online...

    2008

  78. [78]

    In spite of policy-centric ap- proaches, DIMA [73] uses diffusion models as environment dynamics to boost data efficiency

    and NCDPO [68]), score approximation (MaxEntDP [16]) and Q-weighted diffusion objectives with uniform exploration (QVPO [15], HAQO [3] and RSM [41]). In spite of policy-centric ap- proaches, DIMA [73] uses diffusion models as environment dynamics to boost data efficiency. However, extending these methods to online MARL faces three hurdles: computational i...

    arXiv 2048