Mitigating Relative Over-Generalization in Multi-Agent Reinforcement Learning

Giovanni Montana; Jeremie Houssineau; Ting Zhu; Yue Jin

REVIEW 4 major objections 5 minor 45 references

Mitigating Relative Over-Generalization in Multi-Agent Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces MaxMax Q-Learning (MMQ), a decentralized multi-agent reinforcement learning update rule that counters relative over-generalization by sampling possible next states from quantile-predicted bounds and updating toward…

desk verdict MMQ's new combination of quantile-bounded sampling and double-max Bellman targets is empirically promising in RO tasks, but the theory depends on an unverified inclusion premise and needs major revision. read the letter →

arxiv 2411.11099 v1 pith:WH2PTIU4 submitted 2024-11-17 cs.LG cs.AIstat.ML

Ting Zhu , Yue Jin , Jeremie Houssineau , Giovanni Montana This is my paper · ORCID

classification cs.LGcs.AIstat.ML

keywords relativeover-generalizationdecentralizedmulti-agentreinforcementlearningmaxmaxQ-learningquantilemodelsidealtransitionscooperativetasksindependentnon-stationarity

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Decentralized cooperative agents often settle for actions that look good individually but are jointly suboptimal, a failure called relative over-generalization. This paper claims that a single change to the Q-learning update can remove that failure: after taking its own action, an agent should not update toward the state it actually observed, but toward the most promising next state among the range of states its action could produce while other agents act optimally. The proposed MaxMax Q-Learning (MMQ) predicts that range with two quantile models trained on each agent's own replay buffer, samples candidate next states from it, and applies two maximum operators—one over sampled next states, one over the agent's own next actions. The paper proves that if the best candidate next state is within epsilon of the true ideal next state, the learned Q-values stay within a bounded gap of the optimal joint Q-values, and it shows empirically that MMQ outperforms decentralized baselines across differential games, particle-environment coordination tasks, and multi-agent MuJoCo.

What carries the argument

The load-bearing mechanism is the second maximum operator in the Bellman target: the maximum over candidate next states. MMQ replaces the observed next state $s'$ in a standard Bellman update with a sampled set of possible next states $\hat{S}$, predicted by two non-parametrised quantile networks (at $\tau=0.05$ and $\tau=0.95$) plus the observed $s'$ itself, and selects the candidate with the largest value of learned reward plus discounted max-action Q. This selection step is what refines the approximation of ideal transitions: it lets an agent act as if the other agents had chosen the jointly optimal action, without observing their actions. The contraction property of this modified Bellman operator, together with the Lipschitz and order-2-maximizer assumptions used in the appendix, is what carries Theorem 4.1's bound.

What would settle it

Run an RO task such as the two-agent differential game while deliberately withholding from each agent's replay buffer every transition that moves toward the coordinated optimum; if MMQ still converges to the optimal region, the quantile bounds extrapolate beyond observed ideal transitions, whereas if it stalls as baseline independent learners do, the premise that $\hat{S}$ contains $s'^*$ is falsified. A quantitative companion check is to compute the distance between $s'^*$ and the nearest sampled candidate in $\hat{S}$ and verify whether the Q-value gap tracks $(1-\gamma)^{-1}K\epsilon$ as Theorem 4.1 predicts.

Watch

Extended reading notes

Core claim

The central claim is that relative over-generalization can be mitigated without centralized training or communication by making each agent's Bellman target optimistic over next states, not just actions. Given the agent's state-action pair $(s, a_i)$, MMQ builds candidate set $\hat{S} = \{s'\} \cup \{\text{samples from } [g^{\tau_l}_i(s,a_i), g^{\tau_u}_i(s,a_i)]\}$, where $g$'s are quantile models of next-state bounds, and updates $Q_i$ toward $Y_i(s,a_i) = \max_{\hat{s}'\in\hat{S}} \left[ R_i(s,\hat{s}') + \gamma \max_{a'_i} Q_i(\hat{s}',a'_i)\right]$. Because the ideal transition assumes other agents play the optimal joint policy, the true best next state $s'^*$ lies in the full set of possible next states $S_{s,a_i}$; if $\hat{S}$ contains $s'^*$ (or a state within $\epsilon$), the max over $\hat{S}$ recovers the ideal Bellman target. The paper's Theorem 4.1 turns this into a convergence bound: $|Q_i - Q^*_i| \leq (1-\gamma)^{-1} K \epsilon$, and Theorem 4.2 bounds the Monte Carlo sampling error in one dimension by $2u/(M+1)$. Empirically, agents using MMQ reach the optimal cooperative solution in tasks where independent Q-learning, hysteretic, and ideal-transition baselines get stuck.

Load-bearing premise

The load-bearing premise is that the set of candidate next states predicted from an agent's own past transitions contains the next state that would occur if the other agents were playing their optimal joint policy, or a state within a small distance of it; if the coordinated optimal transition never appears in the agent's experience, the max over candidates has nothing to select and the convergence bound is vacuous.

Editorial extensions

If this is right

In cooperative tasks with relative over-generalization, agents using MMQ can converge to the globally optimal joint policy using only local transitions, without a centralized critic or explicit communication.
Because the target only needs the range of possible next states, the same update rule applies in deterministic and stochastic environments and in partially observable settings; the paper reports competitive behavior on stochastic differential games and Half-Cheetah 2x3.
If the ideal next state is inside the estimated set, the learned Q-values are exactly those of the ideal-transition Bellman optimality equation, so coordination performance is tied to how well the quantile models cover the true next-state set.
Using more Monte Carlo samples $M$ shrinks the sampling error ($2u/(M+1)$ in one dimension), and the ablation shows learning accelerates with $M$, so the method can trade computation for coordination accuracy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the paper's coverage check (Figure 5b) measures whether observed transitions fall inside the quantile bounds, but the algorithm's optimism depends on unobserved ideal transitions being covered; a stronger validation would be to compute the distance from $s'^*$ to the sampled candidate set in each RO task and plot it against the Q-value gap predicted by Theorem 4.1.
Editorial extension: if the quantile box is too wide in high dimensions, the Monte Carlo error term grows, so an adaptive proposal distribution—sampling more densely where high Q-values are plausible—could make MMQ scale better than uniform sampling, but that is not tested here.
Editorial extension: in settings where coordinated optimal actions are rare in the replay buffer, MMQ may benefit from an explicit exploration bonus that occasionally forces the coordinated action, because the method's guarantees require evidence of the ideal transition or a bound that contains it.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

MMQ is a new fully decentralized update rule: each agent learns per-dimension quantile bounds on the next state, samples M candidate states from those bounds plus the observed s', and targets max over candidates of R + gamma max over its own actions. That's a genuine new combination, distinct from I2Q's ideal-transition model and from hysteretic/lenient optimism. The empirical work is solid for a MARL methods paper: eight seeds with CIs, six MPE variants, differential games, and MAMuJoCo, plus ablations on sample count, reward shifting, and quantile vs Gaussian models. Code is public. The results plausibly show that MMQ often beats I2Q, HyDDPG, and IDDPG on the tasks that exhibit relative over-generalization. On that basis the paper deserves a serious referee.

The soft spots are in the theory, and they are load-bearing for the paper's stated convergence claim. The contraction proof in Appendix B is algebraically invalid as written: Eq. (14) simply drops the reward difference term without justification, so the sup-norm contraction does not follow. Theorem 4.1's proof also has a mismatch: it introduces the best state in the estimated set under the approximate Q, then reasons as if the optimal and approximate next states came from the same transition, which is not established. More important, the central inclusion premise is explicitly disavowed by the authors: Section 4.1 states that they cannot guarantee the ideal next state belongs to the estimated set. The theorem's error bound is a function of the distance between the true best next state and the best in the estimated set, which is small only if that set contains the ideal transition or something close to it. But in an RO regime the coordinated optimal transition may be rare or absent from the replay buffer, so training the quantile models on collected transitions (Eq. 5) does not imply coverage of the ideal transition. Figure 5b checks exactly the wrong thing: it measures whether observed true next states fall inside the predicted bounds, not whether the unobserved coordinated-optimal state does. Negative reward shifting and the double-max structure can inject optimism independently, so the empirical gains do not by themselves validate the inclusion premise.

None of this makes the paper unserious. The algorithm is clearly specified, the empirical comparison is fair and reasonably thorough, and the method may well be useful as an optimistic heuristic. But the theory currently overclaims. I would send it to peer review with a request for major revision: either fix the contraction and theorem statements, or reframe the theoretical discussion as motivation and add direct evidence that the estimated next-state sets cover or approximate the ideal transition, for example, in a diagnosis environment where the ideal transition can be computed exactly and the distance from the ideal state to the best sampled state is tracked over training. If the inclusion premise turns out to be empirically supported in those settings, the paper would be much stronger.

Referee Report

4 major / 5 minor

Summary. This paper proposes MaxMax Q-Learning (MMQ), a fully decentralized MARL algorithm meant to mitigate relative over-generalization. For each agent, MMQ learns quantile models of next-state transitions (Eq. (5)), forms a set of candidate next states from the predicted bounds plus the observed next state, draws M samples, and uses the target Y_i(s,a_i)=max over the sampled states of [R_i(s,s')+gamma max_{a'_i} Q_i(s',a'_i)] (Eq. (6)). The authors argue that this max-over-next-states operator approximates ideal transitions assuming other agents play optimally, provide a convergence analysis (Theorems 4.1 and 4.2), and present experiments in differential games, MPE cooperative navigation variants, and Multi-Agent MuJoCo showing that MMQ frequently outperforms I2Q, IDDPG, and HyDDPG.

Significance. The empirical study is a strength: eight seeds with confidence intervals, several RO-prone environments, ablations, and public code. If the theoretical claim were established, MMQ would be an appealing communication-free approach to RO. However, the theoretical core is not currently reliable: the contraction proof has algebraic errors, and the main bound depends on an inclusion assumption that the paper explicitly states cannot be guaranteed and that the experiments do not test. The paper's central explanatory mechanism is therefore not established, although the empirical comparison may still be informative.

major comments (4)

[Appendix B, Theorem B.1] The proof of the contraction is not valid as written. In going from Eq. (13) to Eq. (14), the term |R(s,s'_1)-R(s,s'_2)| is dropped without justification, and Eq. (15) bounds a difference of maxima evaluated at different states s'_1 and s'_2 by the sup-norm of Q1-Q2 at a common state; this does not follow from any stated assumption. A correct proof would need a Lipschitz or bounded-reward argument controlling the state difference. Since the contraction property is used to justify convergence to the fixed point Q in the proof of Theorem 4.1, this is a load-bearing gap.
[Appendix B, proof of Theorem 4.1, Eq. (19)] The displayed equality after 'It follows that' is algebraically incorrect: the left side contains R(hat_s'*_t, hat_s'*_{t+1}) - R(hat_s'*_t, s'*_{t+1}), whereas the right side q*(hat_s'*_t, s'*_{t+1}) - q(hat_s'*_t, hat_s'*_{t+1}) contains the opposite reward difference. Hence the bound by K epsilon + gamma Delta_{t+2} is not derived. Moreover, the notation s'*_{t+1} and hat_s'*_{t+1} is introduced without a formal definition connecting these states to the assumption d(s'*, hat_s'*) <= epsilon, so the recursive application of Lemma B.2 is not justified. The theorem's conclusion therefore is not proven.
[Section 4.1 and Theorem 4.1] The premise d(s'*, hat_s'*) <= epsilon is the load-bearing assumption and it is not verified. The paper states on page 6 that 'we cannot guarantee that s'* in hat_S_{s,a_i,t} subset S_{s,a_i} holds'; Theorem 4.1 is vacuous if the estimated set does not contain a state within epsilon of the ideal next state. Figure 5b only reports coverage of observed true next states; under RO the coordinated optimal transition is rare or absent from the replay buffer used in Eq. (5), so this measurement does not address whether the unobserved ideal transition is covered. Algorithm 1's inclusion of the observed s' in hat_S does not fix the issue, because s' is generated under the current, possibly suboptimal, policies of the other agents. The central claim that Eq. (6) aligns each agent's Q-function with the ideal-transition Q* is therefore not established.
[Section 4.3 and Appendix C.1] The ablation in Figure 6 shows that negative reward shifting has a substantial effect on performance in the differential game, and the theoretical analysis in Theorem 4.1 does not model reward shifting or the double-max structure. It is therefore possible that the observed gains come from the optimistic bias induced by these components rather than from accurate coverage of the ideal next state. An experiment that isolates the ideal-state-selection mechanism, for example by checking whether the argmax over hat_S coincides with the state reached under the joint optimal policy in a small tabular instance, would be needed to support the proposed explanation.

minor comments (5)

[Section 4.3] The models are called 'non-parametrised quantile models' but they are neural networks with learned parameters; please rephrase to 'neural-network quantile models'.
[Appendix B, Assumption A.3] The stated inequality with a linear term in ||a*-a|| is not what is usually called a maximizer of order 2; the terminology and the constants c and delta should be aligned with the cited parametric optimization result.
[Theorem 4.1] The theorem statement uses |Q*_i(s,a_i)-Q_i(s,a_i)|, but the proof concludes with |Q*(s_t,a_{i,t})-Q(s_t,hat_a_{i,t})|; the action mismatch needs to be fixed.
[Section 4.2, Theorem 4.2] Theorem 4.2 is stated for S=R and hat_S=[-u,u]; this one-dimensional uniform case is a useful sanity check, but the text should not imply it quantifies the general Monte Carlo error without a discussion of how the bound degrades in higher dimensions.
[Figure 5b] The caption 'Percentage of each dim of true next states fall within the predicted quantile bound' has a subject-verb agreement error; it should read 'falls within'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: MMQ's convergence bound is conditional on an explicit, unverified coverage assumption rather than being forced by construction.

full rationale

The derivation chain in MMQ is not circular. The central theoretical result, Theorem 4.1, is an explicitly conditional statement: if the best next state in the estimated set is within epsilon of the ideal next state, then the learned Q-function is within a Lipschitz-scaled bound of the ideal Q-function. The paper itself acknowledges the key premise is not guaranteed, stating in Section 4.1 that 'we cannot guarantee that s'* in \hat{S}_{s,a_i,t} subset of S_{s,a_i,t} holds', and instead defers to a performance assessment. This makes the bound a conditional theorem resting on an unverified inclusion premise, not a result that is equivalent to its inputs by definition. The update rule in Eq. (6) maximizes over an estimated next-state set, and it is not defined in terms of the Q* it is meant to approximate; the quantile bounds are fitted to observed transitions using a standard quantile loss (Eq. 5). The empirical evidence in Figure 5b measures coverage of observed next states by the fitted quantile bounds, which is weaker support for the ideal-state inclusion premise than the paper implies, but weak evidence for an assumption is a correctness or validation concern, not circularity. No load-bearing self-citation chain, imported uniqueness theorem, or ansatz smuggled in via citation is present: the ideal-transition alignment result is cited to the external prior work of Jiang and Lu [2022], and the performance claims are benchmarked on external environments and baselines. The paper therefore contains no circular step that reduces a prediction to a fitted input or to a self-referential definition.

Assumptions & free parameters 3 free parameters · 7 assumptions · 0 invented entities

The main contribution is the algorithm; its theoretical support rests on smoothness assumptions and, pivotally, on the unverified claim that replay-buffer-trained quantile bounds cover the ideal coordinated next state. Several hand-set hyperparameters (M, tau_l, tau_u, reward-shift constant) affect the reported performance. No new physical entities are postulated.

free parameters (3)

M (number of next-state samples) = 15 in main results; 1 and 3 in ablation
The max is taken over M samples; Figure 5a shows performance varies with M. Chosen by hand, not derived.
Quantile levels tau_l, tau_u = 0.05, 0.95
Define the estimated support of possible next states. Chosen by hand; no justification for these particular values.
Negative reward shift constant = 2 (NC=2 in Fig. 7)
Subtracting a constant from all rewards is shown in Appendix C.1 to significantly improve MMQ's Q-value initialization and final performance; the value is chosen by hand.

assumptions (7)

domain assumption R(s, .) is Lipschitz continuous for each s (Assumption A.1)
Used in Lemma B.2 to convert next-state distance epsilon into reward error.
domain assumption Q* is twice continuously differentiable and Lipschitz (Assumption A.2)
Needed for the parametric optimization stability argument (Still 2018) in Lemma B.2.
domain assumption Every maximizer of Q*(s, .) is of second order (Assumption A.3)
Ensures the argmax of Q* shifts continuously with the state; a rather strong local strong-concavity condition.
domain assumption Deterministic environment transitions (Section 4.1)
The derivation of Eq. (3b) uses a delta-function encoding of f_env; stochastic transitions are only handled empirically in Appendix D.1.
ad hoc to paper The quantile-predicted set S_hat_{s,a_i} is a subset of the reachable set S_{s,a_i}
The text states 'Assuming no approximation error in the predicted bound'; the theory assumes the learned bounds do not hallucinate unreachable states.
ad hoc to paper The estimated set S_hat_{s,a_i} contains a next state within epsilon of the ideal s'* for all (s,a_i)
Precondition of Theorem 4.1; the paper states it cannot guarantee s'* in S_hat and never verifies it for the ideal transition.
ad hoc to paper Independence across state dimensions in the quantile model
Quantile bounds are predicted per dimension with separate losses (Eq. 5); the conclusion notes relaxing this assumption as future work.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mitigating Relative Over-Generalization in Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/WH2PTIU4

@misc{pith2026241111099,
  author       = {Pith},
  title        = {Pith review of: Mitigating Relative Over-Generalization in Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WH2PTIU4}},
  note         = {Machine review of arXiv:2411.11099}
}

read the original abstract

In decentralized multi-agent reinforcement learning, agents learning in isolation can lead to relative over-generalization (RO), where optimal joint actions are undervalued in favor of suboptimal ones. This hinders effective coordination in cooperative tasks, as agents tend to choose actions that are individually rational but collectively suboptimal. To address this issue, we introduce MaxMax Q-Learning (MMQ), which employs an iterative process of sampling and evaluating potential next states, selecting those with maximal Q-values for learning. This approach refines approximations of ideal state transitions, aligning more closely with the optimal joint policy of collaborating agents. We provide theoretical analysis supporting MMQ's potential and present empirical evaluations across various environments susceptible to RO. Our results demonstrate that MMQ frequently outperforms existing baselines, exhibiting enhanced convergence and sample efficiency.

Figures

Figures reproduced from arXiv: 2411.11099 by the authors.

**Figure 2.** Illustration of the set relationship among [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Task visualization. (a) Differential Game(DG): agents need to cross a wide zero-reward area to move to the center to gain the optimal reward. (b) Half-Cheetah 2x3 : the Half-Cheetah 2x3 scenario in MAmujoco domain; (c) MPE scenarios; Cooperative navigation(CN): two agents need to enter the grey area of the target together to gain the reward, the solo entry would induce a penalty. CN + More penalty: Same task as CN b… view at source ↗

Figures from the paper (7 more)

**Figure 4.** Figure 4: Performance comparison for two-agents setting in DG, MPE scenarios and Half-Cheetah [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

**Figure 5.** Figure 5: (a) Ablation study for different sample number [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: Performance comparison between reward shifting and original reward [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]

**Figure 7.** Figure 7: Comparison of Q-value for our algorithm with and without Negative reward shifting; [PITH_FULL_IMAGE:figures/full_fig_p023_7.png]

**Figure 10.** Figure 10: Learning curves in stochastic version of differential game (N=2) [PITH_FULL_IMAGE:figures/full_fig_p025_10.png]

**Figure 11.** Figure 11: Cooperative navigation and Predator-Prey environment with default reward; MMQ [PITH_FULL_IMAGE:figures/full_fig_p025_11.png]

**Figure 12.** Figure 12: Learning curves on Multi-Agent Mujoco with RO reward setting [PITH_FULL_IMAGE:figures/full_fig_p026_12.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 29 canonical work pages

[1]

Decentralized multi-agent deep reinforcement learning in swarms of drones for flood monitoring

David Baldazo, Juan Parras, and Santiago Zazo. Decentralized multi-agent deep reinforcement learning in swarms of drones for flood monitoring. In 2019 27th European Signal Processing Conference (EUSIPCO), pages 1--5. IEEE, 2019

work page 2019
[2]

Decentralized control of quadrotor swarms with end-to-end deep reinforcement learning

Sumeet Batra, Zhehui Huang, Aleksei Petrenko, Tushar Kumar, Artem Molchanov, and Gaurav S Sukhatme. Decentralized control of quadrotor swarms with end-to-end deep reinforcement learning. In Conference on Robot Learning, pages 576--586. PMLR, 2022

work page 2022
[3]

Opportunities for multiagent systems and multiagent reinforcement learning in traffic control

Ana LC Bazzan. Opportunities for multiagent systems and multiagent reinforcement learning in traffic control. Autonomous Agents and Multi-Agent Systems, 18: 0 342--375, 2009

work page 2009
[4]

Superhuman ai for heads-up no-limit poker: Libratus beats top professionals

Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359 0 (6374): 0 418--424, 2018. doi:10.1126/science.aao1733. URL https://www.science.org/doi/abs/10.1126/science.aao1733

doi:10.1126/science.aao1733 2018
[5]

Deep reinforcement learning in a handful of trials using probabilistic dynamics models

Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018

2018
[6]

Deep multi-agent reinforcement learning for decentralized continuous cooperative control

Christian Schroeder de Witt, Bei Peng, Pierre-Alexandre Kamienny, Philip Torr, Wendelin B \"o hmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXiv:2003.06709, 19, 2020

arXiv 2003
[7]

Learning and policy search in stochastic dynamical systems with bayesian neural networks

Stefan Depeweg, Jose Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Learning and policy search in stochastic dynamical systems with bayesian neural networks. arXiv preprint arXiv:1605.07127, 2016

arXiv 2016
[8]

Learning to communicate with deep multi-agent reinforcement learning

Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016

2016

Show all 45 references

[9]

Addressing function approximation error in actor-critic methods

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587--1596. PMLR, 2018

2018
[10]

Uneven: Universal value exploration for multi-agent reinforcement learning

Tarun Gupta, Anuj Mahajan, Bei Peng, Wendelin B \"o hmer, and Shimon Whiteson. Uneven: Universal value exploration for multi-agent reinforcement learning. In International Conference on Machine Learning, pages 3930--3941. PMLR, 2021

2021
[11]

A survey of learning in multiagent environments: Dealing with non-stationarity

Pablo Hernandez-Leal, Michael Kaisers, Tim Baarslag, and Enrique Munoz de Cote. A survey of learning in multiagent environments: Dealing with non-stationarity. In 2017 16th Conference on Autonomous Agents and MultiAgent Systems (AAMAS), pages 1164--1170. IFAAMAS, 2017

2017
[12]

I2q: A fully decentralized q-learning algorithm

Jiechuan Jiang and Zongqing Lu. I2q: A fully decentralized q-learning algorithm. Advances in Neural Information Processing Systems, 35: 0 20469--20481, 2022

2022
[13]

What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017

Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017

2017
[14]

Simple and scalable predictive uncertainty estimation using deep ensembles

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017

2017
[15]

An algorithm for distributed reinforcement learning in cooperative multi-agent systems

Martin Lauer and Martin A Riedmiller. An algorithm for distributed reinforcement learning in cooperative multi-agent systems. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 535--542, San Francisco, CA, USA, 2000. Morgan Kaufmann Publishers Inc

2000
[16]

Solving homogeneous and heterogeneous cooperative tasks with greedy sequential execution

Shanqi Liu, Dong Xing, Pengjie Gu, Xinrun Wang, Bo An, and Yong Liu. Solving homogeneous and heterogeneous cooperative tasks with greedy sequential execution. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=hB2hXtxIPH

2024
[17]

Multi-agent actor-critic for mixed cooperative-competitive environments

Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in neural information processing systems, 30, 2017

2017
[18]

Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams

La \"e titia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Hysteretic q-learning: an algorithm for decentralized reinforcement learning in cooperative multi-agent teams. In 2007 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 64--69. IEEE, 2007

2007
[19]

Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems

Laetitia Matignon, Guillaume J Laurent, and Nadine Le Fort-Piat. Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems. The Knowledge Engineering Review, 27 0 (1): 0 1--31, 2012

2012
[20]

Deep exploration via bootstrapped dqn

Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in neural information processing systems, pages 4026--4034, 2016

2016
[21]

Negative update intervals in deep multi-agent reinforcement learning

Gregory Palmer, Rahul Savani, and Karl Tuyls. Negative update intervals in deep multi-agent reinforcement learning. arXiv preprint arXiv:1809.05096, 2018

2018 arXiv
[22]

The analysis and design of concurrent learning algorithms for cooperative multiagent systems

Liviu Panait. The analysis and design of concurrent learning algorithms for cooperative multiagent systems. George Mason University, 2007

2007
[23]

Biasing coevolutionary search for optimal multiagent behaviors

Liviu Panait, Sean Luke, and R Paul Wiegand. Biasing coevolutionary search for optimal multiagent behaviors. IEEE Transactions on Evolutionary Computation, 10 0 (6): 0 629--645, 2006

2006
[24]

QMIX : Monotonic value function factorisation for deep multi-agent reinforcement learning

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. QMIX : Monotonic value function factorisation for deep multi-agent reinforcement learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th Internat...

2018
[25]

Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning

Tabish Rashid, Gregory Farquhar, Bei Peng, and Shimon Whiteson. Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning. Advances in neural information processing systems, 33: 0 10199--10210, 2020

2020
[26]

Monte Carlo statistical methods, volume 2

Christian P Robert, George Casella, and George Casella. Monte Carlo statistical methods, volume 2. Springer, 1999

1999
[27]

Sahraoui, M

A. Sahraoui, M. Boulmalf, and A. Tahri. Schedule-based cooperative multi-agent reinforcement learning for multi-channel communication in wireless sensor networks. Wireless Personal Communications, 120 0 (1): 0 429--447, 2021. URL https://dblp.org/rec/journals/wpc/SahraouiBT22....

2021 doi
[28]

Planning to explore via self-supervised world models

Ramanan Sekar, Oleh Rybkin, Kostas Daniilidis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In International Conference on Machine Learning, pages 8583--8592. PMLR, 2020

2020
[29]

Safe, multi-agent, reinforcement learning for autonomous driving

Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. arXiv preprint arXiv:1610.03295, 2016

2016 arXiv
[30]

Curriculum learning for relative overgeneralization

Lin Shi and Bei Peng. Curriculum learning for relative overgeneralization. arXiv preprint arXiv:2212.02733, 2022

2022 arXiv
[31]

Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning

Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International conference on machine learning, pages 5887--5896. PMLR, 2019

2019
[32]

Lectures on parametric optimization: An introduction

Georg Still. Lectures on parametric optimization: An introduction. Optimization Online, page 2, 2018

2018
[33]

A fully decentralized surrogate for multi-agent policy optimization

Kefan Su and Zongqing Lu. A fully decentralized surrogate for multi-agent policy optimization. Transactions on Machine Learning Research, 2023

2023
[34]

Exploit reward shifting in value-based deep-rl: Optimistic curiosity-based exploration and conservative exploitation via linear reward shaping

Hao Sun, Lei Han, Rui Yang, Xiaoteng Ma, Jian Guo, and Bolei Zhou. Exploit reward shifting in value-based deep-rl: Optimistic curiosity-based exploration and conservative exploitation via linear reward shaping. Advances in Neural Information Processing Systems, 35: 0 37719--37...

2022
[35]

Multi-agent reinforcement learning: Independent vs

Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the Tenth International Conference on Machine learning, pages 330--337, 1993

1993
[36]

Multi-agent deep reinforcement learning-based trajectory planning for multi-uav assisted mobile edge computing

Liang Wang, Kezhi Wang, Cunhua Pan, Wei Xu, Nauman Aslam, and Lajos Hanzo. Multi-agent deep reinforcement learning-based trajectory planning for multi-uav assisted mobile edge computing. IEEE Transactions on Cognitive Communications and Networking, 7 0 (1): 0 73--84, 2020

2020
[37]

Lenient learning in independent-learner stochastic cooperative games

Ermo Wei and Sean Luke. Lenient learning in independent-learner stochastic cooperative games. The Journal of Machine Learning Research, 17 0 (1): 0 2914--2955, 2016

2016
[38]

Multiagent soft q-learning

Ermo Wei, Drew Wicke, David Freelan, and Sean Luke. Multiagent soft q-learning. arXiv preprint arXiv:1804.09817, 2018

2018 arXiv
[39]

An analysis of cooperative coevolutionary algorithms

Rudolf Paul Wiegand. An analysis of cooperative coevolutionary algorithms. George Mason University, 2004

2004
[40]

Uncertainty weighted actor-critic for offline reinforcement learning

Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08140, 2021

2021 arXiv
[41]

Learning multi-agent coordination for enhancing target coverage in directional sensor networks

Jing Xu, Fangwei Zhong, and Yizhou Wang. Learning multi-agent coordination for enhancing target coverage in directional sensor networks. Advances in Neural Information Processing Systems, 33: 0 10053--10064, 2020

2020
[42]

The surprising effectiveness of ppo in cooperative multi-agent games

Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35: 0 24611--24624, 2022

2022
[43]

M. Zhou, X. Ma, and Y. Li. A novel multi-objective routing scheme based on cooperative multi-agent reinforcement learning for metaverse services in fixed 6g. In WOCN, 2023. URL https://dblp.org/rec/conf/wocc/ZhouML23. DOI: 10.1109/WOCC52294.2023.00029

2023
[44]

Smarts: An open-source scalable multi-agent rl training school for autonomous driving

Ming Zhou, Jun Luo, Julian Villella, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, Montgomery Alban, Iman Fadakar, Zheng Chen, Chongxi Huang, Ying Wen, Kimia Hassanzadeh, Daniel Graves, Zhengbang Zhu, Yihan Ni, Nhat Nguyen, Mohamed Elsayed, Haitham Ammar, Alexander Cowen...

2020
[45]

A survey of multi-agent reinforcement learning with communication

Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent reinforcement learning with communication. arXiv preprint arXiv:2203.08975, 2022

2022 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Decentralized multi-agent deep reinforcement learning in swarms of drones for flood monitoring

David Baldazo, Juan Parras, and Santiago Zazo. Decentralized multi-agent deep reinforcement learning in swarms of drones for flood monitoring. In 2019 27th European Signal Processing Conference (EUSIPCO), pages 1--5. IEEE, 2019

work page 2019

[2] [2]

Decentralized control of quadrotor swarms with end-to-end deep reinforcement learning

Sumeet Batra, Zhehui Huang, Aleksei Petrenko, Tushar Kumar, Artem Molchanov, and Gaurav S Sukhatme. Decentralized control of quadrotor swarms with end-to-end deep reinforcement learning. In Conference on Robot Learning, pages 576--586. PMLR, 2022

work page 2022

[3] [3]

Opportunities for multiagent systems and multiagent reinforcement learning in traffic control

Ana LC Bazzan. Opportunities for multiagent systems and multiagent reinforcement learning in traffic control. Autonomous Agents and Multi-Agent Systems, 18: 0 342--375, 2009

work page 2009

[4] [4]

Superhuman ai for heads-up no-limit poker: Libratus beats top professionals

Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359 0 (6374): 0 418--424, 2018. doi:10.1126/science.aao1733. URL https://www.science.org/doi/abs/10.1126/science.aao1733

doi:10.1126/science.aao1733 2018

[5] [5]

Deep reinforcement learning in a handful of trials using probabilistic dynamics models

Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018

2018

[6] [6]

Deep multi-agent reinforcement learning for decentralized continuous cooperative control

Christian Schroeder de Witt, Bei Peng, Pierre-Alexandre Kamienny, Philip Torr, Wendelin B \"o hmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXiv:2003.06709, 19, 2020

arXiv 2003

[7] [7]

Learning and policy search in stochastic dynamical systems with bayesian neural networks

Stefan Depeweg, Jose Miguel Hernandez-Lobato, Finale Doshi-Velez, and Steffen Udluft. Learning and policy search in stochastic dynamical systems with bayesian neural networks. arXiv preprint arXiv:1605.07127, 2016

arXiv 2016

[8] [8]

Learning to communicate with deep multi-agent reinforcement learning

Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016

2016