Pith. sign in

REVIEW 4 major objections 4 minor 80 references

A Survey of Multi Agent Reinforcement Learning: Federated Learning and Cooperative and Noncooperative Decentralized Regimes

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Multi-agent reinforcement learning falls into three regimes — federated, decentralized-cooperative, and noncooperative — unified by one formalism and distinguished only by communication topology and incentives.

desk verdict A sensible three-regime taxonomy and clear restatements of key algorithms, but the broken reference list and unfulfilled promises of numerical performance undercut the survey's central claim to be a reliable map of the field. read the letter →

arxiv 2507.06278 v1 pith:BWVAQ3R3 submitted 2025-07-08 cs.MA cs.AIcs.LG

Kemboi Cheruiyot , Nickson Kiprotich , Vyacheslav Kungurtsev , Kennedy Mugo , Vivian Mwirigi , Marvin Ngesa This is my paper
classification cs.MAcs.AIcs.LG MSC 68T0591A0691A1590C40
keywords multi-agentreinforcementlearningfederateddecentralizednoncooperativeMARLNashequilibriummean-fieldgamesactor-criticalgorithmsgossipnetworks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This survey claims that the whole of multi-agent reinforcement learning can be organized into three interaction regimes: federated RL, where distributed agents train local models and exchange only parameters through a central server to preserve privacy; cooperative decentralized RL, where agents cooperate over peer-to-peer gossip networks with no central authority; and noncooperative RL, where self-interested agents pursue private rewards and the organizing concept is the Nash equilibrium. The paper's thesis is that all three share a common formal foundation — Markov decision processes, policy gradients, actor-critic methods, and two-time-scale stochastic approximation — and that their differences are structural, not fundamental. If the taxonomy is right, a practitioner can use it to choose the correct formalism, algorithm family, and theoretical guarantees for a given multi-agent deployment. The survey also collects the known results, including heterogeneity-dependent suboptimality for federated algorithms, almost-sure convergence for consensus-based decentralized actor-critic, and the hardness of exact equilibrium computation.

What carries the argument

The paper's organizing device is the three-way distinction in interaction topology: star-shaped central aggregation (FRL), peer-to-peer gossip graphs (CDRL), and competitive incentive structures resolved by game-theoretic equilibria (NMARL). The load-bearing machinery inside each regime is specific. For FRL it is the aggregation operator — FedAvg-style weighted averaging of local Q-tables or policies, written as $\pi_{\text{global}} = F(\{\pi_i\}, \{w_i\})$ — which produces the QAvg, PAvg, DQNAvg, and DDPGAvg algorithms whose suboptimality is bounded by environment heterogeneity. For CDRL it is the Networked Multi-Agent MDP with factored joint policy $\pi_\theta(s,a) = \prod_i \pi^i_{\theta_i}(s, a_i)$, the decentralized policy-gradient theorem derived from the log-derivative trick, and consensus-based critic updates that are contractive in expectation under a spectral-norm condition on the consensus matrices, combined with two-time-scale actor-critic updates. For NMARL it is the Nash equilibrium concept in Markov games, extended through generalized and mean-field variants to handle shared constraints and large populations. Across all three regimes, Borkar's two-time-scale stochastic approximation theorem is the convergence-analysis engine: the faster timescale settles first, so the slower updates see a quasi-static environment.

What would settle it

Look up arXiv:2108.11887: the reference list assigns the same identifier to two different works, Lyu et al. (2021), 'Privacy-preserving federated reinforcement learning for mobile crowdsensing,' and Qi et al. (2021), 'Federated reinforcement learning: Techniques, applications, and open challenges.' Since the definitions of FRL, HFRL, and VFRL in Section 3 are attributed to Qi et al. (2021), checking which paper actually carries that identifier, and whether the quoted definitions appear in it, settles whether the survey's foundational attributions are trustworthy; the same verification applies to the venue-less 'Gavi, K. (2025)' entry cited in the RL background section.

Watch

Extended reading notes

Core claim

On the paper's own terms, the contribution is a map with three territories. Federated RL is defined as the integration of federated learning and RL under privacy protection: agents hold local MDPs and exchange only parameters through an aggregator, in horizontal and vertical variants depending on whether agents share state-action spaces or observe different features of the environment. Cooperative decentralized RL is formalized as a Networked Multi-Agent MDP over a time-varying, jointly connected gossip graph, where a factored joint policy and a decentralized policy-gradient theorem support consensus-based actor-critic algorithms with almost-sure convergence. Noncooperative MARL is formalized as Markov games in which each agent maximizes its own reward, with Nash equilibrium and its refinements — generalized Nash equilibrium, $\varepsilon$-Nash, local Nash, Markov perfect, and mean-field Nash equilibrium — serving as the solution concepts. The survey asserts that the three regimes share the same analytical machinery, and it records the known theoretical guarantees for each: Jin et al.'s results that the QAvg and PAvg federated algorithms converge to suboptimal solutions whose gap grows with environment heterogeneity, Zhang et al.'s convergence theorems for decentralized actor-critic, and the PPAD-completeness of equilibrium computation that motivates mean-field approximations.

Load-bearing premise

The survey's value as a map depends on its reference list being real and accurately summarized; the list itself cites two different works under the same arXiv identifier (arXiv:2108.11887) and one entry with no publication venue, so part of the foundation for the federated and background sections cannot be checked by a reader.

Editorial extensions

If this is right

  • A privacy-critical distributed-learning deployment using FRL must expect the aggregated policy to be suboptimal in heterogeneous environments, with the gap growing with divergence in transition dynamics; personalized aggregation via environment embeddings is the survey's proposed remedy for this gap.
  • In cooperative settings with a time-varying, jointly connected gossip network, consensus-based actor-critic converges almost surely and matches centralized cumulative return, so intermittent communication does not cost asymptotic performance.
  • In noncooperative settings, exact Nash equilibrium computation is PPAD-complete, so the tractable route to scale is mean-field approximation: agents interact with the aggregate population distribution and converge to a self-consistent mean-field Nash equilibrium.
  • Because all three regimes rest on the same MDP, policy-gradient, and two-time-scale machinery, techniques developed in one — variance reduction, personalization, event-triggered or asynchronous communication — transfer to the others.
  • The taxonomy implies that the first design question in any multi-agent system is not which RL algorithm to use but which regime the deployment lives in, since communication topology and incentive alignment, not the underlying algorithm, pick the formalism.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My inference: the taxonomy doubles as a decision rule the survey leaves implicit — choose FRL when data privacy is the binding constraint, CDRL when no trusted coordinator exists, and NMARL when rewards conflict; the survey can be read as a selection guide for these three cases.
  • My inference: the heterogeneity-suboptimality theorems imply that naive averaging is a poor default for real fleets (robots, vehicles, sensors) whose environments differ systematically, making personalized or clustered aggregation the practical default rather than a refinement.
  • My inference: a genuine open evaluation the survey does not perform would benchmark one task suite under all three regimes and measure the explicit privacy, performance, and communication trade-offs; the survey's comparisons are structural and theoretical, not empirical.
  • My inference: the shared two-time-scale convergence skeleton suggests a transfer program — convergence proofs for consensus critics in CDRL could be adapted to personalized FRL aggregation and to NMARL learning dynamics, since all three are analyzed with the same ODE machinery.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This manuscript is a survey of multi-agent reinforcement learning organized around three interaction regimes: Federated RL (FRL), Cooperative Decentralized RL (CDRL), and Noncooperative MARL. For each regime, it provides MDP-style formulations, representative algorithms (e.g., QAvg, PAvg, networked actor-critic, mean-field RL), theoretical guarantees drawn from the literature, and lists of open problems. The survey is heavily built on a small number of primary sources, especially Qi et al. (2021) for FRL, Jin et al. (2022) for federated algorithms, and Zhang et al. (2018) for decentralized actor-critic methods, with a broad but shallow recitation of standard references for noncooperative MARL. The abstract also promises to include highlights and limitations of numerical performance, but no numerical results or benchmark tables appear anywhere in the manuscript.

Significance. If the coverage and citations were reliable, the paper would be a useful organizational map of three related MARL areas, and its three-regime taxonomy is a sensible pedagogical frame. It also collects several important theoretical results, such as the suboptimality of QAvg/PAvg under environment heterogeneity, the decentralized policy gradient theorem, and mean-field Nash equilibrium, and it presents pseudocode for representative algorithms. The paper contains no original derivations, no code, and no experiments, so its value is purely expository. That value is currently undermined by a reference list with duplicated and unverifiable entries and by an abstract promise of numerical performance that the manuscript does not deliver.

major comments (4)
  1. [References; Sections 2, 3.1, 3.4, 3.5] The reference list cannot support the survey's central claim of being a reliable map of the literature. Lyu et al. (2021) and Qi et al. (2021) are both assigned arXiv:2108.11887; Jin et al. (2022a) and Jin et al. (2022b) are the same paper cited as distinct works; Zeng et al. (2024a)/(2024b), Foerster et al. (2018b)/(2018c), and Hernandez-Leal et al. (2019a)/(2019b) are duplicate entries; and Gavi (2025) is listed with venue "unknown" but is used in Section 2 to support claims about recent advances. Because the FRL definitions in Section 3.1 and the algorithm descriptions in Section 3.5 rest on Qi et al. and Jin et al., the reader cannot verify which source supports which claim. This must be corrected by deduplicating and fully identifying every reference, or the affected claims must be re-sourced.
  2. [Abstract; Sections 3-5] The abstract promises "highlights and limitations of numerical performance," but the manuscript contains no numerical results, benchmark tables, empirical comparisons, or performance figures in Sections 3-5. Section 4.2.6 states that experiments validate the algorithms but reports no data. The authors must either revise the abstract to remove this promise or add a consolidated summary of the numerical results reported in the surveyed papers.
  3. [Section 3.5.2, Algorithm 2] Algorithm 2 (PAvg) line 4 uses the gradient term ∂g_{d0,k}(πk_t)/∂π(a|s), but the quantity g_{d0,k} is never defined. The only defined objective is g_{d0}(π) in Eq. (22), which averages over all environments and has no per-agent subscript. Without a definition of g_{d0,k}, the pseudocode is not executable and the reader cannot verify that the algorithm faithfully represents Jin et al. (2022b).
  4. [Section 5.8.2] The text describes "Meta-MARL Lanctot (2017)" as a method that "leverages experience from past interactions to quickly adapt to new opponents." The cited work (Lanctot et al., 2017) is a game-theoretic framework for multi-agent reinforcement learning, not a meta-learning method. This misattribution, together with the duplicate references noted above, indicates that the survey's summaries of individual papers cannot be taken at face value until the bibliography is verified.
minor comments (4)
  1. [Abstract; Section 1] The abstract uses "Federal Reinforcement Learning" and "Federal RL," while the rest of the paper uses "Federated Reinforcement Learning"; please make the terminology consistent.
  2. [Section 2.7 and Section 3.6] The manuscript numbers two different results as "Theorem 1": Borkar's two-time-scale theorem in Section 2.7 and the dependence of the optimal policy on the initial state distribution in Section 3.6. The theorems should be renumbered consecutively.
  3. [Section 3.5.3, Algorithm 3] Line 6 of Algorithm 3 writes the DQN update as θk ← θk − α∇θk (r + γ max_a' Q_{θk}(s',a') − Q_{θk}(s,a))^2, which is not the standard DQN loss because the target is differentiated with respect to the same parameters; the target should be treated as a constant, for example by using a target network.
  4. [References] The reference for Huh and Mohapatra (2024) lacks venue information; please complete the bibliographic entry.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: the survey reports external results without fitting parameters, self-defined predictions, or author-uniqueness claims.

full rationale

This is a survey paper, not a paper that derives new results from fitted inputs. Its theoretical statements are explicitly attributed to external works: the FRL convergence theorems are presented as results of Jin et al. (2022b), the decentralized actor-critic convergence analysis is attributed to Zhang et al. (2018), and Borkar's two-time-scale theorem is quoted from Borkar (2008). No parameter is fitted and then renamed as a prediction; no quantity is defined in terms of its own target; and no uniqueness theorem from the authors' own prior work is invoked to force a choice. The reference list contains no apparent self-citations by the present authors, and the survey's organizing taxonomy of Federated, Decentralized, and Noncooperative MARL is a descriptive classification rather than a result derived from itself. Bibliographic anomalies such as duplicated arXiv identifiers (Lyu et al. 2021 and Qi et al. 2021 both citing arXiv:2108.11887) and the 'unknown' venue for Gavi (2025) are serious correctness and integrity concerns for a survey, but they do not constitute circular reasoning: the paper is not defining its target in terms of a suspect citation, nor is it using a self-citation chain as its only support. The manuscript also promises 'highlights and limitations of numerical performance' yet contains no numerical experiments, which is a completeness or accuracy gap rather than a circularity defect. Therefore the appropriate circularity score is 0, with the caveat that bibliography quality and the abstract's performance promise should be examined under correctness and integrity review.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The survey introduces no free parameters or invented entities. It relies on standard mathematical results (e.g., Borkar's two-time-scale theorem) and on the assumptions stated in the surveyed papers. The integrity of the central claim depends on the correctness of the cited literature, which is the main risk.

assumptions (3)
  • standard math Borkar's two-time-scale stochastic approximation convergence theorem
    Used in Section 2.7 and Section 4.2.5 as the backbone of actor-critic convergence analysis.
  • domain assumption NM-MDP assumptions: jointly connected time-varying graph, ergodic Markov chain, bounded rewards, column-stochastic consensus matrices in expectation, full-rank feature matrices
    Stated in Section 4.2.2 as conditions for Zhang et al. (2018)'s convergence results; the survey relies on these conditions when presenting the theorems.
  • domain assumption FedRL environment heterogeneity assumptions: shared state and action spaces, possibly different transition kernels Pi
    Implicit in Section 3.5 when reproducing QAvg and PAvg guarantees from Jin et al. (2022b).

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Survey of Multi Agent Reinforcement Learning: Federated Learning and Cooperative and Noncooperative Decentralized Regimes." pith.science (2026). https://pith.science/paper/BWVAQ3R3

@misc{pith2026250706278,
  author       = {Pith},
  title        = {Pith review of: A Survey of Multi Agent Reinforcement Learning: Federated Learning and Cooperative and Noncooperative Decentralized Regimes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BWVAQ3R3}},
  note         = {Machine review of arXiv:2507.06278}
}
read the original abstract

The increasing interest in research and innovation towards the development of autonomous agents presents a number of complex yet important scenarios of multiple AI Agents interacting with each other in an environment. The particular setting can be understood as exhibiting three possibly topologies of interaction - centrally coordinated cooperation, ad-hoc interaction and cooperation, and settings with noncooperative incentive structures. This article presents a comprehensive survey of all three domains, defined under the formalism of Federal Reinforcement Learning (RL), Decentralized RL, and Noncooperative RL, respectively. Highlighting the structural similarities and distinctions, we review the state of the art in these subjects, primarily explored and developed only recently in the literature. We include the formulations as well as known theoretical guarantees and highlights and limitations of numerical performance.

Figures

Figures reproduced from arXiv: 2507.06278 by the authors.

Figure 1
Figure 1. Horizontal Federated Reinforcement Learning [PITH_FULL_IMAGE:figures/full_fig_p016_1.png] view at source ↗
Figure 2
Figure 2. An example of horizontal federated reinforcement learning (HFRL) [PITH_FULL_IMAGE:figures/full_fig_p016_2.png] view at source ↗
Figure 3
Figure 3. Vertical Federated Reinforcement Learning [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: An example of vertical federated reinforcement learning (VFRL) ar [PITH_FULL_IMAGE:figures/full_fig_p020_4.png]
Figure 5
Figure 5. Figure 5: A dynamic communication topology illustrating time-varying connec [PITH_FULL_IMAGE:figures/full_fig_p035_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

80 extracted references · 75 canonical work pages

  1. [1]

    Balduzzi, D., Racanière, S., Martens, J., and Foerster, J. N. (2018). The mechanics of n-player differentiable games. International Conference on Machine Learning (ICML)

    work page 2018

  2. [2]

    and Olsder, G

    Başar, T. and Olsder, G. J. (1999). Dynamic noncooperative game theory. SIAM Series in Applied Mathematics

    work page 1999

  3. [3]

    Borkar, V. (2008). Stochastic approximation. A dynamical systems viewpoint

    work page 2008

  4. [4]

    Borkar, V. (2024). Stochastic approximation with two time scales: The general case

    work page 2024

  5. [5]

    and Veloso, M

    Bowling, M. and Veloso, M. (2002). Multiagent learning using a variable learning rate. Artificial Intelligence , 136(2):215--250

    work page 2002

  6. [6]

    and Delarue, F

    Carmona, R. and Delarue, F. (2018). Probabilistic theory of mean field games. Springer

    work page 2018

  7. [7]

    Chen, Y., Gu, B., Chen, K., Huang, Y., Su, Q., and Huang, Z. (2021). Federated reinforcement learning with localized value functions. In International Conference on Machine Learning , pages 1666--1676. PMLR

    work page 2021

  8. [8]

    Massively Multiagent Minigames for Training Generalist Agents

    Choe, K. W., Sullivan, R., and Su \'a rez, J. (2024). Massively multiagent minigames for training generalist agents. arXiv preprint , arXiv:2406.05071

    work page Pith review arXiv 2024

Show all 80 references
  1. [9]

    W., and Papadimitriou, C

    Daskalakis, C., Goldberg, P. W., and Papadimitriou, C. H. (2009). The complexity of computing a nash equilibrium. SIAM Journal on Computing , 39(1):195--259

    2009

  2. [10]

    S., Alami, R., and Simonin, O

    Dibangoye, J. S., Alami, R., and Simonin, O. (2018). Learning to act in decentralized partially observable MDP s. Proceedings of the AAAI Conference on Artificial Intelligence , 32(1)

    2018

  3. [11]

    and Johansson, K

    Dimos, N. and Johansson, K. H. (2014). Decentralized event-triggered control and optimization. IFAC Proceedings Volumes , 47(3):981--986

    2014

  4. [12]

    Doan, T. T. (2021). Nonlinear two-time-scale stochastic approximation: Convergence and finite-time performance

    2021

  5. [13]

    Doan, T. T. and Romberg, J. (2019). Finite-time performance of distributed two-time-scale stochastic approximation

    2019

  6. [14]

    and Pang, J.-S

    Facchinei, F. and Pang, J.-S. (2003a). Finite-dimensional variational inequalities and complementarity problems. Springer , 1:1--60

    2003

  7. [15]

    and Pang, J.-S

    Facchinei, F. and Pang, J.-S. (2003b). Finite-Dimensional Variational Inequalities and Complementarity Problems . Springer

    2003

  8. [16]

    and Pang, J.-S

    Facchinei, F. and Pang, J.-S. (2010). Generalized nash equilibrium problems. Annals of Operations Research , 186:173–210

    2010

  9. [17]

    Finn, C., Abbeel, P., and Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning (ICML)

    2017

  10. [18]

    N., Chen, R

    Foerster, J. N., Chen, R. Y., Al-Shedivat, M., Whiteson, S., Abbeel, P., and Mordatch, I. (2018a). Learning with opponent-learning awareness. International Conference on Autonomous Agents and Multiagent Systems (AAMAS)

    2018

  11. [19]

    N., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S

    Foerster, J. N., Farquhar, G., Afouras, T., Nardelli, N., and Whiteson, S. (2018b). Counterfactual multi-agent policy gradients. Proceedings of the AAAI Conference on Artificial Intelligence , 32

    2018

  12. [20]

    N., Farquhar, G., Afouras, T., Torr, P

    Foerster, J. N., Farquhar, G., Afouras, T., Torr, P. H. S., and Whiteson, S. (2018c). Counterfactual multi-agent policy gradients. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (AAAI-18) , pages 2974--2982

    2018

  13. [21]

    Gao, J., Zhang, Y., Xu, Z., and Lin, Y. (2022). Meta-learning for equilibrium selection in multi-agent reinforcement learning. Artificial Intelligence Journal , 306:1--25

    2022

  14. [22]

    Gavi, K. (2025). Integrating reinforcement learning and neural networks for autonomous decision-making in embedded systems. unknown

    2025

  15. [23]

    L., Rabbat, R., and Savvides, M

    Gupta, P. L., Rabbat, R., and Savvides, M. (2021). Federated reinforcement learning with privacy-preserving event-triggered communication. IEEE Transactions on Neural Networks and Learning Systems , 33(11):6913--6927

    2021

  16. [24]

    Hernandez-Leal, P., Kartal, B., and Taylor, M. E. (2019a). A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems , 33:750–797

    2019

  17. [25]

    Hernandez-Leal, P., Kartal, B., and Taylor, M. E. (2019b). A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems , 33(3):750--797

    2019

  18. [26]

    and Wellman, M

    Hu, J. and Wellman, M. P. (2003). Nash q-learning for general-sum stochastic games. Journal of Machine Learning Research , 4:1039--1069

    2003

  19. [27]

    P., and Caines, P

    Huang, M., Malhame, R. P., and Caines, P. E. (2006). Large-population cost-coupled lqg problems with nonuniform agents: Individual-mass behavior and decentralized \( \)-nash equilibria. IEEE Transactions on Automatic Control , 52(9):1560--1571

    2006

  20. [28]

    and Mohapatra, P

    Huh, D. and Mohapatra, P. (2024). Multi-agent reinforcement learning: A comprehensive survey

    2024

  21. [29]

    and Sha, F

    Iqbal, S. and Sha, F. (2021). Randomized entity-wise factorization for multi-agent reinforcement learning. International Conference on Machine Learning (ICML)

    2021

  22. [30]

    Jiang, J., Dun, C., Lu, W., Li, X., and Yu, Y. (2020). Graph-based multi-agent reinforcement learning: A survey. arXiv preprint arXiv:2006.07281

    2020 arXiv

  23. [31]

    Jin, H., Peng, Y., Yang, W., Wang, S., and Zhang, Z. (2022a). Federated reinforcement learning with environment heterogeneity

    2022

  24. [32]

    Jin, H., Peng, Y., Yang, W., Wang, S., and Zhang, Z. (2022b). Federated reinforcement learning with environment heterogeneity. In International Conference on Artificial Intelligence and Statistics , pages 18--37. PMLR

    2022

  25. [33]

    D., Narasimhan, K., Saeedi, A., and Tenenbaum, J

    Kulkarni, T. D., Narasimhan, K., Saeedi, A., and Tenenbaum, J. B. (2016). Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. Advances in Neural Information Processing Systems (NeurIPS) , 29

    2016

  26. [34]

    Lanctot, M. e. a. (2017). A unified game-theoretic approach to multiagent reinforcement learning. NeurIPS

    2017

  27. [35]

    Lasry, J. M. and Lions, P. L. (2007). Mean-field games. Japanese Journal of Mathematics , 2:229--260

    2007

  28. [36]

    Li, R., Wang, J., and Gao, Z. (2023). Adaptive event-triggered communication for multi-agent reinforcement learning in resource-constrained networks. IEEE Internet of Things Journal , 10(12):10908--10920

    2023

  29. [37]

    Li, Y., Sun, J., Gong, M., and Grosse, R. (2021). Directed acyclic graph neural networks. In International Conference on Learning Representations

    2021

  30. [38]

    Lian, X., Zhang, W., Zhang, C., and Liu, J. (2018). Asynchronous decentralized parallel stochastic gradient descent

    2018

  31. [39]

    Littman, M. L. (1994a). Markov games as a framework for multi-agent reinforcement learning. In Proceedings of the Eleventh International Conference on Machine Learning (ICML) , pages 157--163

    1994

  32. [40]

    Littman, M. L. (1994b). Minimax-q: Learning in zero-sum stochastic games. In Proceedings of the Eleventh International Conference on Machine Learning (ICML) , pages 157--163

    1994

  33. [41]

    Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. (2017). Multi-agent actor-critic for mixed cooperative-competitive environments. arXiv preprint arXiv:1706.02275

    2017 arXiv

  34. [43]

    Mateos, G., Faria, J., and Boyd, S. (2019). Gossip algorithms for decentralized multi-agent reinforcement learning. IEEE Transactions on Signal Processing , 67(19):5015--5028

    2019

  35. [44]

    I., and Ratliff, L

    Mazumdar, E., Jordan, M. I., and Ratliff, L. J. (2020a). On gradient-based learning in continuous games. SIAM Journal on Mathematics of Data Science

    2020

  36. [45]

    I., and Ratliff, L

    Mazumdar, E., Jordan, M. I., and Ratliff, L. J. (2020b). On the convergence of gradient-based learning in continuous games. arXiv preprint

    2020

  37. [46]

    McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A. (2017). Communication-efficient learning of deep networks from decentralized data. Artificial Intelligence and Statistics , pages 1273--1282

    2017

  38. [47]

    P., Mirza, M., Graves, A., Lillicrap, T., Harley, P., Silver, D., and Kavukcuoglu, K

    Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, P., Silver, D., and Kavukcuoglu, K. (2016). Asynchronous methods for deep reinforcement learning. In International conference on machine learning , pages 1928--1937. PMLR

    2016

  39. [48]

    Munos, R., Steinke, T., and Harutyunyan, A. (2016). Safe and efficient off-policy reinforcement learning. In Advances in neural information processing systems , pages 2939--2947

    2016

  40. [49]

    Nabli, A., Belilovsky, E., and Oyallon, E. (2023). Accelerating asynchronous communication in decentralized deep learning. Advances in Neural Information Processing Systems , 36:47451--47474

    2023

  41. [50]

    Nadiger, C., Kumar, A., and Abdelhak, S. (2019). Federated reinforcement learning for fast personalization. In 2019 IEEE Second International Conference on Artificial Intelligence and Knowledge Engineering (AIKE) , pages 123--127. IEEE

    2019

  42. [51]

    Naeem, M., Rizvi, S., and Coronato, A. (2020). A gentle introduction to reinforcement learning and its application in different fields. IEEE Access , 8:209320--209344

    2020

  43. [52]

    V., Phung, D., and Le, T

    Nguyen, Q. V., Phung, D., and Le, T. (2020). Optimistic asynchronous decentralized deep reinforcement learning. arXiv preprint arXiv:2006.07485

    2020 arXiv

  44. [53]

    and Cort \'e s, J

    Nowzari, C. and Cort \'e s, J. (2019). Event-triggered reinforcement learning for networked control systems. IEEE Control Systems Letters , 4(2):243--248

    2019

  45. [54]

    J., and Van Roy, B

    Osband, I., Russo, D. J., and Van Roy, B. (2016). Deep exploration via bootstrapped dqn. Advances in Neural Information Processing Systems (NeurIPS) , 29

    2016

  46. [55]

    and Fukushima, M

    Pang, J.-S. and Fukushima, M. (2005). Quasi-variational inequalities, generalized nash equilibria, and multi-leader-follower games. Computational Management Science , 2:21--56

    2005

  47. [56]

    Parise, F., Grammatico, S., and Lygeros, J. (2016). Network aggregative games: Distributed convergence to generalized nash equilibria. IEEE Transactions on Automatic Control , 61(11):3391--3406

    2016

  48. [57]

    A., and Darrell, T

    Pathak, D., Agrawal, P., Efros, A. A., and Darrell, T. (2017). Curiosity-driven exploration by self-supervised prediction. International Conference on Machine Learning (ICML)

    2017

  49. [58]

    Pfau, D., Vinyals, O., and de Freitas, N. (2023). Meta-learning for equilibrium approximation in multi-agent systems. Journal of Artificial Intelligence Research , 76:1--24

    2023

  50. [59]

    and Sigaud, O

    Pourchot, P. and Sigaud, O. (2018). Continuous control with deep reinforcement learning and evolution strategies. Advances in Neural Information Processing Systems (NeurIPS)

    2018

  51. [60]

    Qi, J., Zhou, Q., Lei, L., and Zheng, K. (2021). Federated reinforcement learning: Techniques, applications, and open challenges. arXiv preprint arXiv:2108.11887

    2021 arXiv

  52. [61]

    Rosen, J. B. (1965). Existence and uniqueness of equilibrium points for concave n-person games. Econometrica , 33(3):520--534

    1965

  53. [62]

    P., Facchinei, F., and Pang, J.-S

    Scutari, G., Palomar, D. P., Facchinei, F., and Pang, J.-S. (2010). Convex optimization, game theory, and variational inequality theory. IEEE Signal Processing Magazine , 27(3):35--49

    2010

  54. [63]

    V., and Johansson, K

    Seyboth, G., Dimarogonas, D. V., and Johansson, K. H. (2013). Event-triggered distributed optimization. IEEE Transactions on Automatic Control , 58(2):246--262

    2013

  55. [64]

    and Leyton-Brown, K

    Shoham, Y. and Leyton-Brown, K. (2007). Multiagent Systems: Algorithmic, Game-Theoretic, and Logical Foundations . Cambridge University Press

    2007

  56. [65]

    E., and Yi, Y

    Son, K., Kim, D., Kang, W., Hostallero, D. E., and Yi, Y. (2019). Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. International Conference on Machine Learning (ICML)

    2019

  57. [66]

    Sutton, R. S. and Barto, A. G. (2018). Reinforcement Learning: An Introduction . MIT Press

    2018

  58. [67]

    Tang, Y., Lou, T., and Shi, Y. (2020). Communication topology aware multi-agent reinforcement learning. In IEEE INFOCOM 2020-IEEE Conference on Computer Communications , pages 937--945. IEEE

    2020

  59. [68]

    Truex, S., Baracaldo, N., Anwar, A., Steinke, T., Ludwig, H., Zhang, R., and Zhou, Y. (2019). A hybrid approach to privacy-preserving federated learning. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security , pages 1--11

    2019

  60. [69]

    Wang, P., Lu, Z., and Gao, Y. (2020). Tackling asymmetry via regularization in multi-agent reinforcement learning. arXiv preprint arXiv:2007.14435

    2020 arXiv

  61. [70]

    Wei, J., Huang, X., Ma, X., Jin, X., and Ren, F. (2020). Federated off-policy learning under distributed data with personalized privacy preservation. Sensors , 20(17):4883

    2020

  62. [71]

    Wen, C., Devlin, S., Hofmann, K., and Rowland, M. (2021). Multi-agent trust region policy optimization. International Conference on Autonomous Agents and Multi-Agent Systems (AAMAS)

    2021

  63. [72]

    Xu, Z., Zhu, Q., Sun, J., and Liu, H. (2024). Communication-efficient decentralized multi-agent reinforcement learning for cooperative adaptive cruise control. arXiv preprint arXiv:2403.14952

    2024 arXiv

  64. [73]

    and Wang, J

    Yang, Y. and Wang, J. (2018). Mean field multi-agent reinforcement learning. International Conference on Machine Learning (ICML)

    2018

  65. [74]

    Yang, Y., Wang, J., Luo, R., and Li, M. (2018). Mean field multi-agent reinforcement learning. International Conference on Machine Learning (ICML)

    2018

  66. [75]

    Yu, C., Zhan, X., and Yang, Y. (2021). The surprising effectiveness of MAPPO : Multi-agent proximal policy optimization. Advances in Neural Information Processing Systems (NeurIPS)

    2021

  67. [76]

    T., and Romberg, J

    Zeng, S., Doan, T. T., and Romberg, J. (2024a). A two-time-scale stochastic optimization framework with applications in control and reinforcement learning

    2024

  68. [77]

    T., and Romberg, J

    Zeng, S., Doan, T. T., and Romberg, J. (2024b). A two-time-scale stochastic optimization framework with applications in control and reinforcement learning. SIAM Journal on Optimization , 34(1):946--976

    2024

  69. [78]

    Zhang, K., Yang, Z., Liu, H., Zhang, T., and Basar, T. (2018). Fully decentralized multi-agent reinforcement learning with networked agents. In Dy, J. and Krause, A., editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Ma...

    2018

  70. [79]

    Zhang, X., Chen, K., Zhang, W., and Xu, J. (2023). Advances in federated reinforcement learning: Methods and applications. Applied Sciences , 13(6):6497

    2023

  71. [80]

    Zhao, M., Xu, W., Duan, J., and Lu, J. (2020). Event-triggered communication network with limited-bandwidth constraint for multi-agent reinforcement learning. IEEE Transactions on Neural Networks and Learning Systems , 32(12):5413--5427

    2020

  72. [81]

    H., Feng, W., Lin, Y., Xu, Q., and Yang, Q

    Zhuo, H. H., Feng, W., Lin, Y., Xu, Q., and Yang, Q. (2019). Federated deep reinforcement learning. arXiv preprint arXiv:1901.08277

    2019 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.