REVIEW 2 major objections 7 minor 63 references
Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition
T0 review · 2 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read OMSD claims that offline cooperative multi-agent reinforcement learning can avoid out-of-distribution joint actions by decomposing the joint behavior policy into a chain of per-agent conditional scores, estimated by diffusion models…
desk verdict Useful empirical paper with an overstated theoretical wrapper; the sequential score-decomposition idea is worth taking seriously, but Eq. (8) does not match Eq. (6). read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the sequential decomposition of the joint behavior policy into per-agent conditional distributions, together with the score functions of those conditionals. For each agent $i$, a diffusion model is trained to estimate the conditional score $\nabla_{a_i}\log\mu_i(a_i|s,a_{1:i-1})$ through its noise prediction, conditioning on the state and on the actions of earlier agents; during policy update, agent $i$ conditions on the freshly updated policies of its prefix agents, which is what converts a single global distributional constraint into coordinated, decentralized, in-sample updates. A shared joint-action critic trained with expectile regression supplies the top-down reward signal, and the $\beta$-weighted conditional score supplies the bottom-up in-distribution constraint, and the paper claims this combination avoids both uncoordinated Q-gradients and out-of-distribution joint actions.
What would settle it
On a two- or three-agent bandit with two optimal coordinated modes and strongly correlated agents, compare the OMSD update against a full chain-rule update that includes the neglected suffix terms $\sum_{j>i}\nabla_{a_i}\log\mu_j(a_j|s,a_{1:j-1})$; if the truncated update selects a different mode, or lands on out-of-distribution joint actions with materially higher probability, the paper's 'unbiased' claim fails. A direct quantitative check is to compute the norm of the dropped terms on the actual MPE and MaMuJoCo datasets and show whether ignoring them changes the learned policy's support.
Extended reading notes
Core claim
The central claim is that policy-based offline MARL fails structurally because the joint behavior policy in offline data is multimodal and cannot be written as a product of independent marginals, yet standard methods regularize each agent against exactly such a product. OMSD replaces the product with a chain of conditionals, $\mu(a|s) = \prod_{i=1}^n \mu_i(a_i|s, a_{1:i-1})$, and derives each agent's policy gradient as a joint-action Q-gradient plus the conditional behavior score: $\nabla_{\theta_i} L = E[\nabla_{a_i} Q_\phi(s,a) + \frac{1}{\beta}\nabla_{a_i}\log\mu_i(a_i|s,a_{i-})]\nabla_{\theta_i}\pi_{\theta_i}(s)$, where the score is read out of a diffusion model conditioned on the prefix agents' actions. The paper proves that the independent factorization of a two-mode optimal policy yields $2^n$ spurious modes with total variation distance approaching 1 as the number of agents grows, and the empirical sections show OMSD outperforming the compared baselines on 11 of 13 MPE and MaMuJoCo tasks, with the largest gains on the most multimodal datasets.
Load-bearing premise
The paper treats each agent's conditional behavior score as the complete regularization signal, ignoring the extra terms that the exact chain-rule expansion of the joint policy would add from later agents' conditional distributions; if those neglected terms are large, the gradient no longer minimizes the stated objective and the promised safety against out-of-distribution joint actions is not guaranteed.
Editorial extensions
If this is right
- Offline cooperative MARL can be built on a joint-action critic plus per-agent conditional behavior scores, bypassing value decomposition and the coverage biases that come with it.
- Multimodal datasets, from multiple Nash equilibria, anonymized agent identities, or mixed-quality demonstrations, become usable instead of imposing conflicting regularization on independent policies.
- Because the diffusion model is used only to read out score estimates and never to sample actions, training is coordinated while execution stays lightweight and fully decentralized.
- The method's largest reported margins come on medium and random datasets, where multimodality is most pronounced, suggesting the gain comes specifically from preserving modes rather than from better single-mode imitation.
- Sequential conditioning acts as a safeguard: when early agents produce slightly out-of-distribution actions, later agents still receive in-distribution conditional context and pull the joint policy back toward the data support.
Reading between the lines
- The agent ordering is a free parameter the paper does not tune: since the analysis shows the dropped suffix terms in the exact chain-rule expansion carry the approximation error, permuting the order should measurably change performance, and an order chosen to minimize that error could be learned from the dataset.
- The same chain-of-conditionals idea transfers to partially observable settings by conditioning scores on observation-action histories rather than joint states, and to heterogeneous teams by conditioning on groups of agents rather than single agents.
- Proposition 4.1 implies a sharp, testable contrast: distribution shift under the factorized baseline grows exponentially in the number of agents, while the chain decomposition should degrade only with the number of conditioning steps, and measuring that gap on a controlled task would isolate the mechanism the paper claims.
- A natural correction to the truncation concern would be an update that keeps the omitted suffix terms by back-propagating through later agents' conditional score models during training; the paper's own BRPO-IGO versus OMSD ablation already brackets how much the decomposition choice matters.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes OMSD, an offline cooperative MARL algorithm. It models the joint behavior policy with a chain-rule factorization and learns each per-agent conditional score with a diffusion model; these scores are used as behavior regularizers in per-agent policy updates guided by a centralized IQL critic. The authors argue that this sequential score decomposition avoids the multimodality and coordination failures of IGO-style product factorization, and they report large improvements over prior offline MARL methods on MPE and MaMuJoCo benchmarks, with ablations and t-SNE visualizations supporting the in-distribution claim. The central theoretical assertion is that the resulting per-agent update is an 'unbiased' sequential decomposition of the joint-policy KL constraint.
Significance. If the theoretical claim were correct, the paper would make a meaningful contribution: it identifies a concrete failure mode of product-factorized behavior regularization under multimodal offline data and provides a diffusion-based estimator that acts on per-agent scores while preserving decentralized execution. Strengths of the empirical work are the externally evaluated rollouts on standard benchmarks, five-seed reporting, ablations over the regularization temperature and score-decomposition variants, and honest acknowledgment of the two tasks where OMSD underperforms. However, the central 'unbiased' derivation is not correct as written, so the paper's main methodological guarantee is not established; the empirical results are nevertheless substantial and the gap appears fixable by revising the objective or the claims.
major comments (2)
- [§4.2, Eqs. (6)–(8)] The derivation of the per-agent update is not the gradient of the stated KL objective. Under the chain-rule factorization μ(a|s)=∏_{j=1}^n μ_j(a_j|s,a_{<j}), the score of the joint behavior policy with respect to a_i is ∇_{a_i} log μ(a|s) = ∇_{a_i} log μ_i(a_i|s,a_{<i}) + Σ_{j>i} ∇_{a_i} log μ_j(a_j|s,a_{<j}). Eq. (8) keeps only the first term. In the n=2 case, the dropped term ∇_{a_1} log μ_2(a_2|s,a_1) measures exactly how strongly agent 2's behavior depends on agent 1's action, which is the coordination structure the method claims to preserve. Eq. (6) also writes the regularizer as DKL(π_{θ_i}π_{θ_{-i}} || μ_i μ_{-i}) without defining μ_{-i}; if μ_{-i}=∏_{j≠i} μ_j, then the gradient of that KL contains the suffix terms, and if μ_{-i} is meant to absorb them, the equality to the joint KL is lost. Thus Eqs. (7)–(8) do not descend the stated objective or the surrogate in Eq. (21); the labels 'unbiased' (Sec. 4.2) and 'strongly guarantees' (Appendix G.3) are unsupported. Please provide a corrected derivation, include the suffix terms via a joint score estimator, or explicitly reformulate the objective and remove the guarantee language.
- [§4.3 and Appendix G.3, Eq. (20)] The conditional diffusion models are trained on prefixes a_{i-} drawn from the dataset, but at policy-update time Eq. (8) evaluates ∇_{a_i} log μ_i(a_i|s,a_{i-}) at prefixes sampled from the updated policies π_{θ_{i-}}(s). No correction is provided for this covariate shift, and the claim in Sec. 4.2 that the conditional structure 'ensures that the current agent is updated with respect to a meaningful, in-distribution context' is not established by the equations. This is a second, distinct distribution-shift source that is central to the paper's OOD-safety claim; it should be analyzed, or the claim should be restricted to the empirical t-SNE evidence.
minor comments (7)
- [Algorithm 1, line 12] The algorithm refers to Eq. (7) for the policy update, but the practical update implemented in the paper is Eq. (8).
- [§3.1] The phrase 'episodic infinite horizon problem' is contradictory; the paper should specify either an episodic finite-horizon or an infinite-horizon discounted setting.
- [Appendix G.3] Reference [16] is cited for classifier-free guidance, but [16] is IDQL, not a classifier-free-guidance diffusion paper; the citation should be corrected.
- [Abstract and Conclusion] The claim to be 'the first work to explicitly address the distributional gap between offline and online MARL' is too strong given earlier works on OOD joint actions, such as AlberDICE, COMADICE, and SIPO; the novelty claim should be softened.
- [Appendix H] The running-time statements are inconsistent: the text says OMSD took 10 hours for HalfCheetah and 1 hour for MPE, then lists separate 10-hour pretraining and 12-hour policy-training times; the total should be clarified.
- [Table 2] The caption states that shaded columns represent the proposed method, but no shading is visible in the manuscript; use boldface, an asterisk, or a printed shading to mark OMSD.
- [General] Minor typographical issues should be fixed: 'infactorization' in the §4.1 heading, 'classsifier' in Appendix G.3, 'comparation' in §5.3, and inconsistent spelling of 'Dilac/DiLac'.
Circularity Check
No circularity: OMSD's behavior models and critics are fitted to offline data, while claims are evaluated by external environment rollouts; the only self-citation is not load-bearing.
full rationale
The paper's derivation chain is self-contained rather than circular. The sequential score functions and the joint-action critic are pretrained on the offline dataset (Appendix C.4–C.5, Eq. 20) and then used only as regularizers in the policy extraction step (Eq. 8, Algorithm 1). Reported performance is measured by rollout returns in external benchmarks (MPE and MaMuJoCo), not by any quantity fitted into the algorithm, so the 'state-of-the-art' claim is not forced by construction. The KL-to-score derivation in Eq. 7–8 may contain a correctness gap—the suffix coupling terms Σ_{j>i} ∇_{a_i} log μ_j are omitted—but that is a mathematical derivation error, not circular reasoning: Eq. 8 is not definitionally identical to the objective in Eq. 6. The only author-overlap citation is [46], used for the generic observation that multiple Nash equilibria cause multimodal datasets (Appendix B), and it is paired with an independent citation [13]; it does not carry the paper's central argument. No fitted input is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via self-citation. Hence the paper does not exhibit circularity.
Assumptions & free parameters
free parameters (2)
- β (behavior regularization temperature) =
Per task and dataset, from {0.001, 0.005, 0.01, 0.02, 0.05, 0.1, 0.2, 0.5}; see Tables 6-9
- IQL critic temperature and expectile τ =
Per task and dataset; temperature in {1,3,5,7,10}, τ in {0.3,0.5,0.7,0.9}; see Tables 4-5
assumptions (5)
- standard math Chain rule of probability: µ(a|s)=Π_i µ_i(a_i|s,a_{1:i-1})
- domain assumption Diffusion score approximation: a well-trained denoising model approximates ∇_{a_i} log µ_i(a_i|s,a_{i-}) at low noise levels
- domain assumption Deterministic Dilac policies can stand in for stochastic policies in score evaluation and prefix sampling
- domain assumption IQL expectile regression provides a pessimistic enough Q-function for joint actions
- ad hoc to paper Proposition 4.1 assumes the factorized approximation is learned by maximum likelihood (forward KL)
Cite this review
Pith. "Pith review of Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition." pith.science (2026). https://pith.science/paper/BWSHQW4X
@misc{pith2026250505968,
author = {Pith},
title = {Pith review of: Offline Multi-agent Reinforcement Learning via Sequential Score Decomposition},
year = {2026},
howpublished = {\url{https://pith.science/paper/BWSHQW4X}},
note = {Machine review of arXiv:2505.05968}
}
read the original abstract
Offline cooperative multi-agent reinforcement learning (MARL) faces unique challenges due to distributional shifts, particularly stemming from the high dimensionality of joint action spaces and the presence of out-of-distribution joint action selections. In this work, we highlight that a fundamental challenge in offline MARL arises from the multi-equilibrium nature of cooperative tasks, which induces a highly multimodal joint behavior policy space coupled with heterogeneous-quality behavior data. This makes it difficult for individual policy regularization to align with a consistent coordination pattern, leading to the policy distribution shift problems. To tackle this challenge, we design a sequential score function decomposition method that distills per-agent regularization signals from the joint behavior policy, which induces coordinated modality selection under decentralized execution constraints. Then we leverage a flexible diffusion-based generative model to learn these score functions from multimodal offline data, and integrate them into joint-action critics to guide policy updates toward high-reward, in-distribution regions under a shared team reward. Our approach achieves state-of-the-art performance across multiple particle environments and Multi-agent MuJoCo benchmarks consistently. To the best of our knowledge, this is the first work to explicitly address the distributional gap between offline and online MARL, paving the way for more generalizable offline policy-based MARL methods.
Figures
Figures from the paper (9 more)
Reference graph
Works this paper leans on
-
[1]
Principal component analysis
Hervé Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433–459, 2010
2010
-
[2]
A Model-Based Solution to the Offline Multi-Agent Reinforcement Learning Coordination Problem
Paul Barde, Jakob Foerster, Derek Nowrouzezahrai, and Amy Zhang. A model-based solu- tion to the offline multi-agent reinforcement learning coordination problem. arXiv preprint arXiv:2305.17198, 2023
work page Pith review arXiv 2023
-
[3]
Dota 2 with large scale deep reinforcement learning
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Dkebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019
arXiv 1912
-
[4]
The complexity of decentralized control of markov decision processes
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4):819–840, 2002
work page 2002
-
[5]
The Viet Bui, Thanh Hong Nguyen, and Tien Mai. Comadice: Offline cooperative multi- agent reinforcement learning with stationary distribution shift regularization. arXiv preprint arXiv:2410.01954, 2024
work page Pith review arXiv 2024
-
[6]
Powernet: Multi-agent deep reinforcement learning for scalable powergrid control
Dong Chen, Kaian Chen, Zhaojian Li, Tianshu Chu, Rui Yao, Feng Qiu, and Kaixiang Lin. Powernet: Multi-agent deep reinforcement learning for scalable powergrid control. IEEE Transactions on Power Systems, 37(2):1007–1017, 2021
work page 2021
-
[7]
Score regularized policy optimization through diffusion behavior
Huayu Chen, Cheng Lu, Zhengyi Wang, Hang Su, and Jun Zhu. Score regularized policy optimization through diffusion behavior. In The Twelfth International Conference on Learning Representations, 2024
work page 2024
-
[8]
Offline reinforcement learning via high-fidelity generative behavior modeling
Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209.14548, 2022
arXiv 2022
Show all 63 references
-
[9]
Consistency models as a rich and efficient policy class for reinforcement learning
Zihan Ding and Chi Jin. Consistency models as a rich and efficient policy class for reinforcement learning. arXiv preprint arXiv:2309.16984, 2023
2023 arXiv
-
[10]
Putting data at the centre of offline multi-agent reinforcement learning
Claude Formanek, Louise Beyers, Callum Rhys Tilbury, Jonathan P Shock, and Arnu Preto- rius. Putting data at the centre of offline multi-agent reinforcement learning. arXiv preprint arXiv:2409.12001, 2024
2024 arXiv
-
[11]
Off-the-grid marl: Datasets and baselines for offline multi-agent reinforcement learning
Claude Formanek, Asad Jeewa, Jonathan Shock, and Arnu Pretorius. Off-the-grid marl: Datasets and baselines for offline multi-agent reinforcement learning. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 2442–2444, 2023
2023
-
[12]
Dis- pelling the mirage of progress in offline marl through standardised baselines and evaluation
Juan Formanek, Callum R Tilbury, Louise Beyers, Jonathan Shock, and Arnu Pretorius. Dis- pelling the mirage of progress in offline marl through standardised baselines and evaluation. Advances in Neural Information Processing Systems, 37:139650–139672, 2024
2024
-
[13]
Henriques
Tim Franzmeyer, Edith Elkind, Philip Torr, Jakob Nicolaus Foerster, and Joao F. Henriques. Select to perfect: Imitating desired behavior from large multi-agent data. In The Twelfth International Conference on Learning Representations, 2024
2024
-
[14]
Rein- forcement learning-based consensus reaching in large-scale social networks
Shijun Guo, Haoran Xu, Guangqiang Xie, Di Wen, Yangru Huang, and Peixi Peng. Rein- forcement learning-based consensus reaching in large-scale social networks. In International Conference on Neural Information Processing, pages 169–183. Springer, 2023
2023
-
[15]
Dynamic programming for partially observable stochastic games
Eric A Hansen, Daniel S Bernstein, and Shlomo Zilberstein. Dynamic programming for partially observable stochastic games. In AAAI, 2004
2004
-
[16]
Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023
Philippe Hansen-Estruch, Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as an actor-critic method with diffusion policies.arXiv preprint arXiv:2304.10573, 2023. 10
2023 arXiv
-
[17]
Denoising diffusion probabilistic models
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In NeurIPS, 2020
2020
-
[18]
Actor-attention-critic for multi-agent reinforcement learning
Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. In International conference on machine learning, pages 2961–2970. PMLR, 2019
2019
-
[19]
Offline decentralized multi-agent reinforcement learning
Jiechuan Jiang and Zongqing Lu. Offline decentralized multi-agent reinforcement learning. arXiv preprint arXiv:2108.01832, 2021
2021 arXiv
-
[20]
Efficient diffusion policies for offline reinforcement learning
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024
2024
-
[21]
Offline reinforcement learning with implicit q-learning
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169, 2021
2021 arXiv
-
[22]
Trust region policy optimisation in multi-agent reinforcement learning
JG Kuba, R Chen, M Wen, Y Wen, F Sun, J Wang, and Y Yang. Trust region policy optimisation in multi-agent reinforcement learning. In ICLR 2022-10th International Conference on Learning Representations, page 1046. The International Conference on Learning Representations (ICLR), 2022
2022
-
[23]
Offline reinforcement learning: Tutorial, review, and perspectives on open problems
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020
2005 arXiv
-
[24]
Dof: A diffusion factorization framework for offline multi-agent reinforcement learning
Chao Li, Ziwei Deng, Chenxing Lin, Wenqi Chen, Yongquan Fu, Weiquan Liu, Chenglu Wen, Cheng Wang, and Siqi Shen. Dof: A diffusion factorization framework for offline multi-agent reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025
2025
-
[25]
Beyond conservatism: Diffusion policies in offline multi-agent reinforcement learning
Zhuoran Li, Ling Pan, and Longbo Huang. Beyond conservatism: Diffusion policies in offline multi-agent reinforcement learning. arXiv preprint arXiv:2307.01472, 2023
2023 arXiv
-
[26]
A kernelized stein discrepancy for goodness-of-fit tests
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized stein discrepancy for goodness-of-fit tests. In ICML, 2016
2016
-
[27]
Offline multi-agent reinforcement learning via in-sample sequential policy optimization
Zongkai Liu, Qian Lin, Chao Yu, Xiawei Wu, Yile Liang, Donghui Li, and Xuetao Ding. Offline multi-agent reinforcement learning via in-sample sequential policy optimization. InProceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 19068–19076, 2025
2025
-
[28]
Multi-agent actor-critic for mixed cooperative-competitive environments
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017
2017
-
[29]
Efficient and scalable reinforcement learning for large-scale network control
Chengdong Ma, Aming Li, Yali Du, Hao Dong, and Yaodong Yang. Efficient and scalable reinforcement learning for large-scale network control. Nature Machine Intelligence, pages 1–15, 2024
2024
-
[30]
Learning to coordinate from offline datasets with uncoordinated behavior policies
Jinming Ma and Feng Wu. Learning to coordinate from offline datasets with uncoordinated behavior policies. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, pages 1258–1266, 2023
2023
-
[31]
Dynamic economic emissions dispatch optimisation using multi-agent reinforcement learning
Patrick Mannion, Karl Mason, Sam Devlin, Jim Duggan, and Enda Howley. Dynamic economic emissions dispatch optimisation using multi-agent reinforcement learning. In Proceedings of the Adaptive and Learning Agents workshop (at AAMAS 2016), 2016
2016
-
[32]
Diffusion-dice: In- sample diffusion guidance for offline reinforcement learning
Liyuan Mao, Haoran Xu, Xianyuan Zhan, Weinan Zhang, and Amy Zhang. Diffusion-dice: In- sample diffusion guidance for offline reinforcement learning. arXiv preprint arXiv:2407.20109, 2024
2024 arXiv
-
[33]
Alberdice: Addressing out-of-distribution joint actions in offline multi-agent rl via alternating stationary distribution correction estimation
Daiki E Matsunaga, Jongmin Lee, Jaeseok Yoon, Stefanos Leonardos, Pieter Abbeel, and Kee-Eung Kim. Alberdice: Addressing out-of-distribution joint actions in offline multi-agent rl via alternating stationary distribution correction estimation. arXiv preprint arXiv:2311.02194, 2023. 11
2023 arXiv
-
[34]
Offline pre-trained multi-agent decision transformer: One big sequence model tackles all smac tasks
Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu. Offline pre-trained multi-agent decision transformer: One big sequence model tackles all smac tasks. arXiv preprint arXiv:2112.02845, 2021
2021 arXiv
-
[35]
Awac: Accelerating online reinforcement learning with offline datasets
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020
2006 arXiv
-
[36]
Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification
Ling Pan, Longbo Huang, Tengyu Ma, and Huazhe Xu. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification. In International Conference on Machine Learning, pages 17221–17237. PMLR, 2022
2022
-
[37]
Facmac: Factored multi-agent centralised policy gradients
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Alexandre Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. In Advances in Neural Information Processing Systems, volume 34, pages 12208–12221, 2021
2021
-
[38]
A survey on offline reinforcement learning: Taxonomy, review, and open problems
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023
2023
-
[39]
Monotonic value function factorisation for deep multi-agent reinforcement learning
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder De Witt, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent reinforcement learning. Journal of Machine Learning Research, 21(178):1–51, 2020
2020
-
[40]
Counterfac- tual conservative q learning for offline multi-agent reinforcement learning
Jianzhun Shao, Yun Qu, Chen Chen, Hongchang Zhang, and Xiangyang Ji. Counterfac- tual conservative q learning for offline multi-agent reinforcement learning. arXiv preprint arXiv:2309.12696, 2023
2023 arXiv
-
[41]
Deep unsuper- vised learning using nonequilibrium thermodynamics
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In ICML, 2015
2015
-
[42]
Generative modeling by estimating gradients of the data distribution.yang-song.net, May 2021
Yang Song. Generative modeling by estimating gradients of the data distribution.yang-song.net, May 2021
2021
-
[43]
Score-based generative modeling through stochastic differential equations
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020
2011 arXiv
-
[44]
Arena: A general evaluation platform and building toolkit for multi-agent intelligence
Yuhang Song, Jianyi Wang, Thomas Lukasiewicz, Zhenghua Xu, Mai Xu, Zihan Ding, and Lianlong Wu. Arena: A general evaluation platform and building toolkit for multi-agent intelligence. In AAAI, 2020
2020
-
[45]
Corl: Research-oriented deep offline reinforcement learning library
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. Advances in Neural Information Processing Systems, 36:30997–31020, 2023
2023
-
[46]
Learning from good trajectories in offline multi-agent reinforcement learning
Qi Tian, Kun Kuang, Furui Liu, and Baoxiang Wang. Learning from good trajectories in offline multi-agent reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 11672–11680, 2023
2023
-
[47]
Coordination failure in cooperative offline marl
Callum Rhys Tilbury, Claude Formanek, Louise Beyers, Jonathan P Shock, and Arnu Pretorius. Coordination failure in cooperative offline marl. arXiv preprint arXiv:2407.01343, 2024
2024 arXiv
-
[48]
Offline multi- agent reinforcement learning with knowledge distillation
Wei-Cheng Tseng, Tsun-Hsuan Johnson Wang, Yen-Chen Lin, and Phillip Isola. Offline multi- agent reinforcement learning with knowledge distillation. Advances in Neural Information Processing Systems, 35:226–237, 2022
2022
-
[49]
Visualizing data using t-sne
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
2008
-
[50]
Offline multi-agent reinforce- ment learning with implicit global-to-local value regularization
Xiangsen Wang, Haoran Xu, Yinan Zheng, and Xianyuan Zhan. Offline multi-agent reinforce- ment learning with implicit global-to-local value regularization. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 12
2023
-
[51]
Order matters: Agent-by-agent policy optimization
Xihuai Wang, Zheng Tian, Ziyu Wan, Ying Wen, Jun Wang, and Weinan Zhang. Order matters: Agent-by-agent policy optimization. arXiv preprint arXiv:2302.06205, 2023
2023 arXiv
-
[52]
Diffusion policies as an expressive policy class for offline reinforcement learning
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023
2023
-
[53]
Macca: Offline multi-agent reinforcement learning with causal credit assignment
Ziyan Wang, Yali Du, Yudi Zhang, Meng Fang, and Biwei Huang. Macca: Offline multi-agent reinforcement learning with causal credit assignment. arXiv preprint arXiv:2312.03644, 2023
2023 arXiv
-
[54]
Multi-agent reinforcement learning is a sequence modeling problem
Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 35:16509–16521, 2022
2022
-
[55]
Behavior regularized offline reinforcement learning
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361, 2019
1911 arXiv
-
[56]
An overview of multi-agent reinforcement learning from game theoretical perspective
Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:2011.00583, 2020
2011 arXiv
-
[57]
Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning
Yiqin Yang, Xiaoteng Ma, Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34:10299–10312, 2021
2021
-
[58]
Multi-agent reinforcement learning: A selective overview of theories and algorithms
Kaiqing Zhang, Zhuoran Yang, and Tamer Ba¸ sar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of reinforcement learning and control, pages 321–384, 2021
2021
-
[59]
Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning
Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. Fop: Factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In International conference on machine learning, pages 12491–12500. PMLR, 2021
2021
-
[60]
The ai economist: Improving equality and productivity with ai-driven tax policies
Stephan Zheng, Alexander Trott, Sunil Srinivasa, Nikhil Naik, Melvin Gruesbeck, David C Parkes, and Richard Socher. The ai economist: Improving equality and productivity with ai-driven tax policies. arXiv preprint arXiv:2004.13332, 2020
2004 arXiv
-
[61]
guide-then-select
Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. Madiff: Offline multi-agent learning with diffusion models. Advances in Neural Information Processing Systems, 37:4177–4206, 2024. 13 A Related Works Several related work...
2024
-
[63]
and MADiff [61]. Table 5: IQL Training Hyperparameters in MaMuJoCo Environment Task Hyper Parameter Name Value Global Training Steps/Epoch 10000 Epochs 200 2-HalfCheetah 200 Expert temperature 3.0 Expert τ 0.7 Medium temperature 3.0 Medium τ 0.7 Medium-Replay temperature 3.0 M...
-
[512]
The key hyperparameters for OMSD are summarized in Table 3
The training process consists of 1.0 million gradient steps for MaMuJoCo tasks and 0.1 million gradient steps for MPE tasks. The key hyperparameters for OMSD are summarized in Table 3. Table 3: Hyper-Parameters for OMSD Algorithm Hyper-Parameter Name Value All Batch Size 512 A...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.