Pith. sign in

REVIEW 2 major objections 7 minor 51 references

Action Dependency Graphs for Globally Optimal Coordinated Reinforcement Learning

T0 review · 2 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In cooperative multi-agent reinforcement learning, a sparse action dependency graph derived from the coordination graph can turn locally optimal policies into globally optimal ones.

desk verdict A clean and genuinely new sufficient condition for sparse action-dependent policies to be globally optimal, but the empirical sections overstate what the theorem can certify. read the letter →

arxiv 2506.00797 v1 pith:NIFYGARO submitted 2025-06-01 cs.LG cs.AIcs.SYeess.SYmath.OC

Jianglin Ding , Jingcheng Tang , Gangshan Jing This is my paper
classification cs.LGcs.AIcs.SYeess.SYmath.OC
keywords multi-agentreinforcementlearningcoordinationgraphsactiondependencyglobaloptimalitypolicyiterationaction-dependentpoliciescooperativeMARLvaluedecomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether cooperative multi-agent reinforcement learning can reach the globally optimal joint policy without making every agent's policy depend on every other agent's action. Its answer is yes whenever the problem has a coordination graph: a policy in which each agent conditions on a sparse, acyclic set of earlier agents' actions can have the property that every locally optimal response is globally optimal. The load-bearing condition is $N_d(i) = N_c(i[+])$ for every agent $i$, linking the action dependency graph to the coordination graph. If true, the result says scalability and optimality are not in tension for structured cooperative tasks, and sparse action-dependent policies can replace fully dense auto-regressive policies. The paper also gives a tabular policy-iteration algorithm that provably converges to the global optimum in finite steps under this condition.

What carries the argument

The central objects are the coordination graph (an undirected graph whose edges carry pairwise terms in a decomposition of the state-action value function) and the action dependency graph (a directed acyclic graph whose edges say which earlier agents' actions each agent's policy may condition on). The mechanism is the suffix-neighborhood identity $N_d(i) = N_c(i[+])$: agent $i$ must depend on exactly the agents outside the suffix $i[+]$ that are neighbors (in the coordination graph) of some agent in $i[+]$. Under this identity, Lemmas B.1--B.4 split $Q$ into a term that depends on the suffix actions plus a term that depends only on earlier actions, so a greedy backward induction over agents is equivalent to maximizing jointly over all actions. Algorithm 1 is the tabular action-dependent multi-agent policy iteration that realizes this induction; Algorithm 2 greedily chooses the agent ordering to keep the resulting ADG sparse.

What would settle it

Exhaustively search all small finite Markov games with a fixed sparse coordination graph, small action sets, and an ADG satisfying $N_d(i) = N_c(i[+])$: if any $G_d$-locally optimal deterministic policy has value below $V^*$, the theorem is false; a brute-force enumeration over deterministic policies and exact value iteration settles it.

Watch

Extended reading notes

Core claim

On its own terms, the paper proves Theorem 4.3: let $G_d$ be an action dependency graph for a policy $\pi$, let $G_c$ be a coordination graph for the state-action value $Q^\pi$, and suppose $N_d(i) = N_c(i[+])$ for all agents $i$, where $i[+] = \{i, i+1, \ldots, n\}$ is the suffix of the topological ordering. Then any $G_d$-locally optimal policy is globally optimal, so $V^\pi = V^*$. The proof works by induction from the last agent: under the condition, the value function splits so that each agent's local maximization step can be chained into a maximization over the whole suffix, turning a sequence of coordinated best responses into a global argmax. The same structural condition makes the proposed action-dependent multi-agent policy iteration converge in finite terms to a globally optimal policy, and the paper demonstrates empirically that sparse ADGs integrated into common policy-based and value-based deep MARL algorithms match or exceed fully dense ADGs on coordination polymatrix games, adaptive traffic signal control, and StarCraft II.

Load-bearing premise

The proof requires the state-action value function to decompose exactly as a sum of pairwise terms over the coordination graph for the policies involved (or at least for the converged policy); in the deep learning experiments this decomposition is approximate or hand-constructed, so the global-optimality guarantee is not strictly inherited.

Editorial extensions

If this is right

  • For any finite Markov game whose value function decomposes exactly over a coordination graph, a sparse ADG constructed from condition (14) gives the same global-optimality guarantee as a fully dense auto-regressive graph, with fewer dependencies and lower per-step cost.
  • The result subsumes existing special cases: when both graphs are empty, any Nash equilibrium is globally optimal (the VDN-style case), and when the CG is complete, the fully dense ADG recovers the known auto-regressive optimality result.
  • Algorithm 1 converges in finitely many policy iterations to a globally optimal policy, providing a tabular baseline that is provably optimal rather than merely Nash.
  • The ADG construction can be layered onto policy-based and value-based deep MARL algorithms, where sparse ADGs empirically match dense ADGs on the tested benchmarks and avoid the max-plus inference step used by coordination-graph value methods.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the sharpest sparse ADG depends on the agent ordering, the sparsity problem is effectively an elimination-order problem: the analogy to variable elimination suggests that finding the optimally sparse ADG for a given CG is computationally hard in general, so greedy index heuristics are the practical answer.
  • A testable extension is to learn the coordination graph along with the ADG from data: if the learned graph is used, condition (14) becomes approximate, and the empirical question is how much the global-optimality guarantee degrades as the learned CG departs from the true decomposition.
  • The same suffix-condition idea may carry over to hypergraph coordination structures, where local value terms involve more than two agents; the paper's proof structure suggests a generalized condition on hyperedge neighborhoods, though the paper does not establish this.
  • In continuous-action settings, Algorithm 1's sequential argmax would become sequential gradient ascent, and the theorem would need a convexity or unimodality assumption; one could test whether the ADG ordering still improves coordination relative to independent policies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. This paper proposes action dependency graphs (ADGs), directed acyclic graphs that specify which agents condition on the actions of others, and studies when such action-dependent policies can achieve global optimality in cooperative MARL. The main theoretical result (Theorem 4.3) states that if the state-action value function Q^π decomposes exactly over a coordination graph Gc (Eq. 6), and the policy's ADG satisfies Nd(i) = Nc(i[+]) for every agent i, then any Gd-locally optimal policy is globally optimal. The paper also contributes a tabular action-dependent multi-agent policy iteration algorithm (Algorithm 1) with a convergence proof (Theorem 5.2), a greedy construction of sparse ADGs (Algorithm 2), and an integration of the ADG idea into MAPPO and QMIX, with experiments on coordination polymatrix games, adaptive traffic signal control, and StarCraft II.

Significance. The theoretical contribution is a clean, machine-checkable-looking proof that sparse action dependency structures, derived from a coordination graph, can certify global optimality of locally optimal policies. This generalizes earlier dense auto-regressive policy results and connects action-dependent policies to variable-elimination-style structures. The tabular algorithm's convergence proof is a valid extension of policy iteration. However, the practical scope is narrow: the exact decomposition of Q^π over the CG is a strong premise that is not verified in the deep RL experiments, so the empirical results should be viewed as heuristic rather than as instances of the theoretical guarantee. If the authors address this gap, the theoretical part is a solid contribution.

major comments (2)
  1. [Sections 7.2-7.3, Theorem 4.3] The global optimality guarantee in Theorem 4.3 is conditional on the exact coordinate-graph decomposition of Q^π stated in Eq. (6). In the ATSC and SMAC experiments, the CG is constructed from adjacency (Tables 16-17) or is artificial (Tables 18-19), and the trained QMIX/MAPPO value functions are not constrained to decompose as a sum of pairwise edge functions. The paper provides no verification that the learned Q^π satisfies Eq. (6) for the converged policy. Consequently, Theorem 4.3 does not apply to these experiments, and the empirical results cannot be cited as evidence for the global-optimality claim. Please either add a quantitative check of the decomposition residual, or explicitly state that the deep RL results are heuristic extensions and adjust the abstract and conclusion accordingly.
  2. [Proposition D.1, Remark 5.3, and Section 8] The class of Markov games for which an exact CG exists is very restrictive: Proposition D.1 requires both the reward and the transition model to decompose additively over the same edge set, with P(s'|s,a)=Σ_{(i,j)∈Ec} P_ij(s'|s,a_i,a_j). This is not the usual approximation notion of a coordination graph, and the paper gives no natural multi-step examples outside one-step polymatrix games. The conclusion's statement that 'the empirical results affirm the robustness and applicability of our approach in more general scenarios' overstates what the theory supports. A discussion of how far the exact condition is from the experimental setting, and whether an approximate-decomposition error bound exists, is needed to prevent readers from attributing to the theorem what is actually a heuristic.
minor comments (7)
  1. [Definition 4.1, Eq. (13)] The left-hand side E_{π_{-Nd(i)}}[Q^π(s,a)] is written without indicating that the actions of agents in Nd(i) are fixed; the notation should explicitly be E_{π_{-Nd(i)}}[Q^π(s, a_{Nd(i)}, a_{-Nd(i)})] to be unambiguous.
  2. [Appendix B, proof of Lemma B.4] In Eq. (29) the term Q2(s, a'_{Nd(i)}, a_{i-\Nd(i)}) is written with three argument groups, whereas Q2 is defined in Eq. (26) as depending on a_{i-}; this is likely a typo for Q2(s, a'_{i-}).
  3. [Section 7, Figures 5, 7, and 9] The empirical curves show only mean values over ten seeds, with no error bars, confidence regions, or statistical significance tests; claims such as 'faster convergence' would be better supported by reporting variance or a bootstrap interval.
  4. [Section 5, Algorithm 1 and Theorem 5.2] The convergence proof uses Assumption 5.1 (singleton argmax), but the algorithm description mentions a practical tie-breaking relaxation only in a paragraph. It would be helpful to restate the algorithm with the explicit relaxation included and to clarify whether the finite-time optimality proof carries over under that tie-breaking rule.
  5. [Section 4.3, Algorithm 2] The greedy criterion 'assign index n-i to a vertex minimizing the size of Nc((n-i)[+])' is stated without intuition or a small worked example; adding a sentence explaining that this reduces the maximum in-degree of the resulting ADG would improve readability.
  6. [Introduction, Section 1] The claim that prior studies [4,6,9] 'focus exclusively on fully dense ADGs' is accurate for those references but might be misread as ignoring [7], which studies general Bayesian-network policies; the related-work section does acknowledge [7], so a small rephrasing of the contribution statement could avoid this ambiguity.
  7. [Throughout, technical notation] There are several instances where π_{(k+1)[+]} appears in the proof of Theorem 4.3 where π_{k+} (or π_{(k+1)[+]}) is intended; the notation is confusing, and a consistent use of the appendix's i+ and i[+] conventions would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the global-optimality theorem is derived from explicit decomposition and local-optimality assumptions, not from fitted parameters or self-citations.

full rationale

The paper's central result (Theorem 4.3) is a conditional theorem: if a policy is Gd-locally optimal (Definition 4.1) and Gc is a coordination graph of Qπ (Eq. 6), then the graph condition Nd(i)=Nc(i[+]) implies Vπ=V*. The proof in Appendix B supplies lemmas B.1-B.4 and an induction that derives the Bellman-optimal fixed point from these assumptions; the conclusion is not assumed in the definition of Gd-local optimality, which is a strictly weaker local best-response criterion (Remark 4.2). The CG-decomposition premise is stated as an assumption, with Proposition D.1 giving sufficient conditions under which it holds exactly; this is a hypothesis of the theorem, not a fitted input renamed as a prediction. Algorithm 1's convergence proof (Theorem C.1) shows the limit is Gd-locally optimal and then applies Theorem 4.3, again a proof step rather than a circular invocation. The sparse ADG (Algorithm 2) is constructed from the CG purely by graph minimization; no parameter is fit to the observed returns and then reported as a prediction. Self-citations (e.g., Ref. [5]) appear only in motivational context about Nash equilibria and are not load-bearing for the proof. The deep-RL experiments do assume or approximate a CG without verifying exact Qπ decomposition, which is an external-validity/correctness caveat, not circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the structural assumption that the value function decomposes over the coordination graph, plus standard RL assumptions and a technical uniqueness condition for convergence. No free parameters are fitted to data in the theory.

assumptions (4)
  • domain assumption The state-action value function Qπ admits a coordination graph decomposition (Eq. 6) for all policies π or at least for the converged policy.
    This is the premise of Theorem 4.3 and Theorem 5.2. It requires the reward and transition model to decompose over the CG (Proposition D.1), which does not hold generally.
  • standard math The agent indices follow a topological order of the ADG (j < i for all j in Nd(i)).
    Used throughout Section 4 and the proofs; every DAG has a topological order, so this is without loss of generality.
  • ad hoc to paper Assumption 5.1: the greedy argmax in Algorithm 1 is a singleton for all iterations and agents.
    Required for the finite-time convergence proof of Theorem 5.2; the paper says it can be relaxed in practice with tie-checking.
  • standard math The Markov game is finite with discount factor gamma in [0,1), and the Bellman optimality operator has a unique fixed point.
    Standard RL assumptions, invoked implicitly throughout the optimality and convergence proofs.
invented entities (1)
  • Action Dependency Graph (ADG) independent evidence
    purpose: A directed acyclic graph specifying which other agents' actions each agent's policy conditions on.
    This is a formal modeling construct introduced in Definition 3.3, not a physical entity. Its utility is evidenced by the theorem predicting global optimality under the graph condition, which is falsifiable in new environments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Action Dependency Graphs for Globally Optimal Coordinated Reinforcement Learning." pith.science (2026). https://pith.science/paper/NIFYGARO

@misc{pith2026250600797,
  author       = {Pith},
  title        = {Pith review of: Action Dependency Graphs for Globally Optimal Coordinated Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NIFYGARO}},
  note         = {Machine review of arXiv:2506.00797}
}
read the original abstract

Action-dependent individual policies, which incorporate both environmental states and the actions of other agents in decision-making, have emerged as a promising paradigm for achieving global optimality in multi-agent reinforcement learning (MARL). However, the existing literature often adopts auto-regressive action-dependent policies, where each agent's policy depends on the actions of all preceding agents. This formulation incurs substantial computational complexity as the number of agents increases, thereby limiting scalability. In this work, we consider a more generalized class of action-dependent policies, which do not necessarily follow the auto-regressive form. We propose to use the `action dependency graph (ADG)' to model the inter-agent action dependencies. Within the context of MARL problems structured by coordination graphs, we prove that an action-dependent policy with a sparse ADG can achieve global optimality, provided the ADG satisfies specific conditions specified by the coordination graph. Building on this theoretical foundation, we develop a tabular policy iteration algorithm with guaranteed global optimality. Furthermore, we integrate our framework into several SOTA algorithms and conduct experiments in complex environments. The empirical results affirm the robustness and applicability of our approach in more general scenarios, underscoring its potential for broader MARL challenges.

Figures

Figures reproduced from arXiv: 2506.00797 by the authors.

Figure 1
Figure 1. A coordination graph (a) and an action dependency graph (b). In many practical scenarios such as sensor networks [31], wind farms [32], mobile networks [33], etc., the value function Q can be approximated as the sum of local value functions, each depending on the states and actions of a subset of agents. A widely used approach to representing this decomposition is the use of the coordination graph (CG) [8], which ca… view at source ↗
Figure 2
Figure 2. A polymatrix game on a line CG. When using deterministic independent policies, the joint policies π = (1, 1, 1) and π = (0, 0, 0) are both agent-by-agent optimal. However, only π = (0, 0, 0) is globally optimal. Although π = (1, 1, 1) is suboptimal, agents lack suf￾ficient motivation to unilaterally change their actions, highlighting the limitations of indepen￾dent policies in ensuring optimal cooperation. Now consi… view at source ↗
Figure 3
Figure 3. Different index orders of agents result in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: ADGs generated by Algorithm 2 for CG topologies: line, ring, and star. Although a fully dense ADG Gd ensures that any Gd-locally optimal policy achieves global optimality, training such policies can be com￾putationally costly, limiting scalability. Thus, for sparse CGs…
Figure 5
Figure 5. Figure 5: Results of coordination polymatrix game. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: 3x3 road network. ATSC is a benchmark with a clear cooperation structure. We perform ATSC experiments using the Simulation of Urban Mobility (SUMO) platform [41], with the objective of optimizing traffic signal opera￾tions to enhance vehicular flow. Operating under par…
Figure 7
Figure 7. Figure 7: Results of ATSC. We augment the QMIX and MAPPO algorithms with ADG and compare their performance with the Deep Coor￾dination Graph (DCG) algorithm [18] across ten independent experiments with distinct random seeds, as re￾ported in [PITH_FULL_IMAGE:figures/full_fig_p00…
Figure 8
Figure 8. Figure 8: SMAC MMM2 We adopt the SMAC [43] benchmark to evaluate the performance of our method in complex environments. Recognizing the difficulty of inferring fixed CGs in StarCraft II, we introduce an artificially constructed CG and derive the corresponding ADG, as illustrated…
Figure 9
Figure 9. Figure 9: Results of SMAC. The results reveal that QMIX with a sparse ADG outperforms the baseline QMIX algorithm, whereas QMIX with a dense ADG exhibits inferior performance. This suggests that dense ADG policies may require a greater volume of samples to achieve comparable per…
Figure 10
Figure 10. Figure 10: The CG and sparse ADG of polymatrix coordination game. [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 40 canonical work pages

  1. [1]

    Multi-agent reinforcement learning: A selective overview of theories and algorithms

    Kaiqing Zhang, Zhuoran Yang, and Tamer Ba¸ sar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pages 321–384, 2021. 9

    work page 2021

  2. [2]

    A review of cooperative multi-agent deep reinforce- ment learning

    Afshin Oroojlooy and Davood Hajinezhad. A review of cooperative multi-agent deep reinforce- ment learning. Applied Intelligence, 53(11):13677–13722, 2023

    2023

  3. [3]

    Revisiting some common practices in cooperative multi-agent reinforcement learning

    Wei Fu, Chao Yu, Zelai Xu, Jiaqi Yang, and Yi Wu. Revisiting some common practices in cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 6863–6877. PMLR, 2022

    work page 2022

  4. [4]

    Towards global optimality in cooperative marl with sequential transformation

    Jianing Ye, Chenghao Li, Jianhao Wang, Qianchuan Zhao, and Chongjie Zhang. Towards global optimality in cooperative marl with sequential transformation. 2022

    work page 2022

  5. [5]

    Gangshan Jing, He Bai, Jemin George, Aranya Chakrabortty, and Piyush K. Sharma. Dis- tributed multiagent reinforcement learning based on graph-induced local value functions. IEEE Transactions on Automatic Control, 69(10):6636–6651, 2024

    work page 2024

  6. [6]

    Multiagent reinforcement learning: Rollout and policy iteration

    Dimitri Bertsekas. Multiagent reinforcement learning: Rollout and policy iteration. IEEE/CAA Journal of Automatica Sinica, 8(2):249–272, 2021

    work page 2021

  7. [7]

    Context-aware bayesian network actor-critic methods for cooperative multi-agent reinforcement learning

    Dingyang Chen and Qi Zhang. Context-aware bayesian network actor-critic methods for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5327–5350. PMLR, 2023

    work page 2023

  8. [8]

    Coordinated reinforcement learning

    Carlos Guestrin, Michail G Lagoudakis, and Ronald Parr. Coordinated reinforcement learning. In Proceedings of the 19th International Conference on Machine Learning , pages 227–234, 2002

    work page 2002

Show all 51 references
  1. [9]

    More centralized training, still decentralized execution: Multi-agent conditional policy factorization

    Jiangxing Wang, Deheng Ye, and Zongqing Lu. More centralized training, still decentralized execution: Multi-agent conditional policy factorization. arXiv preprint arXiv:2209.12681, 2022

    2022 arXiv

  2. [10]

    Multi-agent reinforcement learning: Independent vs

    Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceed- ings of the 10th International Conference on Machine Learning, pages 330–337, 1993

    1993

  3. [11]

    Value- decomposition networks for cooperative multi-agent learning based on team reward

    Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value- decomposition networks for cooperative multi-agent learning based on team reward. In Pro- ceedings o...

    2018

  4. [12]

    Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning

    Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4295–4304. PMLR, 2018

    2018

  5. [13]

    Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning

    Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pages 5887–5896. PMLR, 2019

    2019

  6. [14]

    Multi-agent actor-critic for mixed cooperative-competitive environments

    Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems, 30, 2017

    2017

  7. [15]

    Counterfactual multi-agent policy gradients

    Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. InProceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  8. [16]

    Actor-attention-critic for multi-agent reinforcement learning

    Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 2961–2970. PMLR, 2019

    2019

  9. [17]

    The surprising effectiveness of ppo in cooperative multi-agent games

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:24611–24624, 2022

    2022

  10. [18]

    Deep coordination graphs

    Wendelin Böhmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In Interna- tional Conference on Machine Learning, pages 980–991. PMLR, 2020. 10

    2020

  11. [19]

    Deep implicit coordination graphs for multi-agent reinforcement learning

    Sheng Li, Jayesh K Gupta, Peter Morales, Ross Allen, and Mykel J Kochenderfer. Deep implicit coordination graphs for multi-agent reinforcement learning. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 764–772, 2021

    2021

  12. [20]

    Context-aware sparse deep coordination graphs

    Tonghan Wang, Liang Zeng, Weijun Dong, Qianlan Yang, Yang Yu, and Chongjie Zhang. Context-aware sparse deep coordination graphs. In International Conference on Learning Representations, 2022

    2022

  13. [21]

    Analysing factoriza- tions of action-value networks for cooperative multi-agent reinforcement learning

    Jacopo Castellini, Frans A Oliehoek, Rahul Savani, and Shimon Whiteson. Analysing factoriza- tions of action-value networks for cooperative multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems, 35(2):25, 2021

    2021

  14. [22]

    Biasing coevolutionary search for optimal multiagent behaviors

    Liviu Panait, Sean Luke, and R Paul Wiegand. Biasing coevolutionary search for optimal multiagent behaviors. IEEE Transactions on Evolutionary Computation, 10(6):629–645, 2006

    2006

  15. [23]

    Bounded approximate decentralised coordination via the max-sum algorithm

    Alex Rogers, Alessandro Farinelli, Ruben Stranders, and Nicholas R Jennings. Bounded approximate decentralised coordination via the max-sum algorithm. Artificial Intelligence, 175 (2):730–759, 2011

    2011

  16. [24]

    Nonserial dynamic programming

    Umberto Bertele and Francesco Brioschi. Nonserial dynamic programming. Academic Press, Inc., 1972

    1972

  17. [25]

    Gcs: Graph-based coordination strategy for multi-agent rein- forcement learning

    Jingqing Ruan, Yali Du, Xuantang Xiong, Dengpeng Xing, Xiyun Li, Linghui Meng, Haifeng Zhang, Jun Wang, and Bo Xu. Gcs: Graph-based coordination strategy for multi-agent rein- forcement learning. In Proceedings of the 21st International Conference on Autonomous Agents and Mult...

    2022

  18. [26]

    Ace: Cooperative multi-agent q-learning with bidirectional action-dependency

    Chuming Li, Jie Liu, Yinmin Zhang, Yuhong Wei, Yazhe Niu, Yaodong Yang, Yu Liu, and Wanli Ouyang. Ace: Cooperative multi-agent q-learning with bidirectional action-dependency. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 8536–8544, 2023

    2023

  19. [27]

    Backpropagation through agents

    Zhiyuan Li, Wenshuai Zhao, Lijun Wu, and Joni Pajarinen. Backpropagation through agents. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 13718–13726, 2024

    2024

  20. [28]

    Group-aware coordination graph for multi-agent reinforce- ment learning

    Wei Duan, Jie Lu, and Junyu Xuan. Group-aware coordination graph for multi-agent reinforce- ment learning. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 3926–3934, 2024

    2024

  21. [29]

    Is centralized training with decentralized execution framework centralized enough for marl? arXiv preprint arXiv:2305.17352, 2023

    Yihe Zhou, Shunyu Liu, Yunpeng Qing, Kaixuan Chen, Tongya Zheng, Yanhao Huang, Jie Song, and Mingli Song. Is centralized training with decentralized execution framework centralized enough for marl? arXiv preprint arXiv:2305.17352, 2023

    2023 arXiv

  22. [30]

    Multi-agent reinforcement learning is a sequence modeling problem

    Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. Advances in Neural Information Processing Systems, 35:16509–16521, 2022

    2022

  23. [31]

    Coordinated multi-agent reinforcement learning in net- worked distributed POMDPs

    Chongjie Zhang and Victor Lesser. Coordinated multi-agent reinforcement learning in net- worked distributed POMDPs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 764–770, 2011

    2011

  24. [32]

    Learning to coordinate with coordination graphs in repeated single-stage multi-agent decision problems

    Eugenio Bargiacchi, Timothy Verstraeten, Diederik Roijers, Ann Nowé, and Hado Hasselt. Learning to coordinate with coordination graphs in repeated single-stage multi-agent decision problems. In International Conference on Machine Learning, pages 482–490. PMLR, 2018

    2018

  25. [33]

    Coordinated reinforcement learning for optimizing mobile networks

    Maxime Bouton, Hasan Farooq, Julien Forgeat, Shruti Bothe, Meral Shirazipour, and Per Karlsson. Coordinated reinforcement learning for optimizing mobile networks. arXiv preprint arXiv:2109.15175, 2021

    2021 arXiv

  26. [34]

    Understanding value decomposition algo- rithms in deep cooperative multi-agent reinforcement learning.arXiv preprint arXiv:2202.04868, 2022

    Zehao Dou, Jakub Grudzien Kuba, and Yaodong Yang. Understanding value decomposition algo- rithms in deep cooperative multi-agent reinforcement learning.arXiv preprint arXiv:2202.04868, 2022. 11

    2022 arXiv

  27. [35]

    On the global convergence rates of decentralized softmax gradient play in markov potential games

    Runyu Zhang, Jincheng Mei, Bo Dai, Dale Schuurmans, and Na Li. On the global convergence rates of decentralized softmax gradient play in markov potential games. Advances in Neural Information Processing Systems, 35:1923–1935, 2022

    1923

  28. [36]

    Trust region policy optimisation in multi-agent reinforcement learning

    JG Kuba, R Chen, M Wen, Y Wen, F Sun, J Wang, and Y Yang. Trust region policy optimisation in multi-agent reinforcement learning. InInternational Conference on Learning Representations, page 1046, 2022

    2022

  29. [37]

    On minmax theorems for multiplayer games

    Yang Cai and Constantinos Daskalakis. On minmax theorems for multiplayer games. In Proceedings of the 20nd Annual ACM-SIAM Symposium on Discrete Algorithms, pages 217–

  30. [38]

    Team theory and person-by-person optimization with binary decisions

    Dario Bauso and Raffaele Pesenti. Team theory and person-by-person optimization with binary decisions. SIAM Journal on Control and Optimization, 50(5):3011–3028, 2012

    2012

  31. [39]

    Collaborative multiagent reinforcement learning by payoff propagation

    Jelle R Kok and Nikos Vlassis. Collaborative multiagent reinforcement learning by payoff propagation. Journal of Machine Learning Research, 7, 2006

    2006

  32. [40]

    Reinforcement learning: An introduction

    Richard S Sutton. Reinforcement learning: An introduction. A Bradford Book, 2018

    2018

  33. [41]

    Microscopic traffic simulation using sumo

    Pablo Alvarez Lopez, Michael Behrisch, Laura Bieker-Walz, Jakob Erdmann, Yun-Pang Flöt- teröd, Robert Hilbrich, Leonhard Lücken, Johannes Rummel, Peter Wagner, and Evamarie Wießner. Microscopic traffic simulation using sumo. InThe 21st IEEE International Conference on Intellig...

    2018

  34. [42]

    Lucas N. Alegre. SUMO-RL. https://github.com/LucasAlegre/sumo-rl, 2019

    2019

  35. [43]

    Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philiph H. S. Torr, Jakob Foerster, and Shimon Whiteson. The StarCraft Multi-Agent Challenge. CoRR, abs/1902.04043, 2019

    1902 arXiv

  36. [44]

    Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pages 1861–1870. PMLR, 2018

    2018

  37. [45]

    Multi-agent reinforcement learning for networked system control

    Tianshu Chu, Sandeep Chinchali, and Sachin Katti. Multi-agent reinforcement learning for networked system control. arXiv preprint arXiv:2004.01339, 2020

    2004 arXiv

  38. [46]

    Proximal policy optimization algorithms

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    2017 arXiv

  39. [47]

    Abstract dynamic programming

    Dimitri Bertsekas. Abstract dynamic programming. Athena Scientific, 2022

    2022

  40. [48]

    Dynamical model of traffic congestion and numerical simulation

    Masako Bando, Katsuya Hasebe, Akihiro Nakayama, Akihiro Shibata, and Yuki Sugiyama. Dynamical model of traffic congestion and numerical simulation. Physical Review E, 51(2): 1035, 1995

    1995

  41. [49]

    High- dimensional continuous control using generalized advantage estimation

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438, 2015

    2015 arXiv

  42. [50]

    Heterogeneous-agent mirror learning: A continuum of solutions to cooperative marl

    Jakub Grudzien Kuba, Xidong Feng, Shiyao Ding, Hao Dong, Jun Wang, and Yaodong Yang. Heterogeneous-agent mirror learning: A continuum of solutions to cooperative marl. arXiv preprint arXiv:2208.01682, 2022

    2022 arXiv

  43. [51]

    Albrecht

    Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V . Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2...

    2021

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.