Pith. sign in

REVIEW 3 major objections 4 minor 42 references

Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning

T0 review · 3 major / 4 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Communication between decentralized critics inflates policy-gradient variance, and a message-dependent baseline provably reduces it.

desk verdict Theorem 1's proof uses a message-conditioned next-action distribution, contradicting the paper's decentralized actors, so the main variance result may not apply to the evaluated estimator. read the letter →

arxiv 2502.06261 v1 pith:OOIEM4CI submitted 2025-02-10 cs.LG

Changxi Zhu , Mehdi Dastani , Shihan Wang This is my paper · ORCID
classification cs.LG
keywords multi-agentdeepreinforcementlearningdecentralizedtrainingcommunicationvariancereductionpolicygradientbaselinetechniqueKLregularizationDCCDA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that in decentralized multi-agent reinforcement learning where critics exchange messages during training (the DCCDA setting), communication itself is a source of variance in policy gradients, and that a carefully chosen baseline can remove part of that variance. It proves that the DCCDA sample gradient has variance at least as large as the centralized-training (CTDE) gradient, both when messages perfectly represent sender information and when they are corrupted by noise. The proposed cure is an optimal message-dependent baseline that leaves the expected gradient unchanged but lowers variance, plus a KL-divergence regularizer that aligns the non-communicating execution policy with the policy implied by the communicating critic. Experiments on StarCraft Multi-Agent Challenge and Traffic Junction show that the two techniques together reduce gradient variance and improve win rates.

What carries the argument

The load-bearing objects are the decentralized communicating critic $Q_i(h_i,a_i,m_{-i})$, where $m_{-i}$ collects the messages from all other agents, and the message-dependent baseline $b_i(h_i,m_{-i})$. The critics are the only channel for communication during training; actors sample from $\pi_i(\cdot|h_i,\theta_i)$ without seeing messages. The baseline is the variance-minimizing centering term derived in Theorem 3, and the KL term $D_{\mathrm{KL}}(\pi_i(\cdot|h_i,\theta_i)\,\|\,\mathrm{SoftMax}(Q_i(h_i,\cdot,m_{-i})/\alpha))$ anchors the actor to the communicating critic's implied policy. The analytical mechanism is a Bellman-uniqueness argument: under ideal communication the expected communicating critic equals the joint centralized critic, turning the variance comparison into a Jensen gap.

What would settle it

Measure the variance of a single policy-gradient update in a DCCDA agent whose messages are deliberately uninformative (e.g., random noise) while the critic is fixed at its true on-policy values: the paper's inequality predicts the DCCDA gradient variance is still no smaller than the corresponding CTDE variance, so observing a strictly smaller variance at any step would contradict Theorem 1's scope.

Watch

Extended reading notes

Core claim

On the paper's own terms, under the DCCDA setting — decentralized actors that never communicate, decentralized critics that do — the single-sample policy gradient $\hat{g}_{\mathrm{DCCDA}}^i$ has variance greater than or equal to the CTDE single-sample gradient $\hat{g}_{\mathrm{CTDE}}^i$. In the idealistic case, the equality $Q^\pi(h,a)=\mathbb{E}_{m_{-i}|h,a}[Q_i^\pi(h_i,a_i,m_{-i})]$ (Lemma 1) converts the variance comparison into a Jensen gap, giving $\mathrm{Var}(\hat{g}_{\mathrm{DCCDA}}^i)\ge \mathrm{Var}(\hat{g}_{\mathrm{CTDE}}^i)$. The proof extends to non-idealistic communication through a surrogate reward function, yielding $\mathrm{Var}(\hat{g}_{\mathrm{DCCDA-noise}}^i)\ge \mathrm{Var}(\hat{g}_{\mathrm{CTDE}}^i)$. The constructive result is the optimal message-dependent baseline $b_i^*(h_i,m_{-i})=\mathbb{E}_{a_i}[Q_i(h_i,a_i,m_{-i})S]/\mathbb{E}_{a_i}[S]$, where $S=\nabla_{\theta_i}\log \pi_i(a_i|h_i,\theta_i)^\top \nabla_{\theta_i}\log \pi_i(a_i|h_i,\theta_i)$; this baseline is unbiased and cuts the variance by a non-negative squared term (Corollary 1). A KL regularizer then keeps the decentralized execution policy close to the policy suggested by the communicating critic, which the paper shows empirically improves both learning speed and final performance.

Load-bearing premise

The argument requires that received messages are perfect representations of the sender's local history and action (Assumption 1) and that learned critics converge to true on-policy values; with learned, noisy message encoders neither condition is exactly met, and the non-idealistic theorem additionally assumes a binary reward structure.

Editorial extensions

If this is right

  • The variance inequality identifies communication as a genuine cost in DCCDA training: even perfect information, once sent stochastically, makes gradient estimates noisier than a centralized critic that receives everything.
  • The optimal message-dependent baseline is model-agnostic, so it applies to any stochastic message function and can extend existing DCCDA methods without redesigning their communication protocols.
  • The KL regularizer matters empirically: removing either the baseline or the regularizer lowers win rates on most tested maps, and the two together deliver lower gradient-norm variance than the underlying DCCDA methods.
  • Because the baseline is computed from critic values and messages already stored in the replay buffer, it adds negligible compute overhead and preserves decentralized execution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the variance inequality holds in other environments, variance reduction in communicating critics should be treated as a first-class design goal rather than a side effect of better critics.
  • The ideal-communication equality suggests a stronger statement may hold for any message distribution that makes the critic conditionally independent of the sender's history, a relaxation worth testing.
  • An adaptive temperature or scaling factor for the KL term would be a natural extension, since the paper's own ablation shows the regularizer can hurt when the critic's implied policy is unreliable.
  • The binary-reward noise model in Theorem 2 could likely be generalized, but continuous rewards may require a different variance decomposition because the surrogate-reward construction relies on flipping a binary reward.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript studies variance in policy gradients for decentralized multi-agent deep reinforcement learning with communicating critics and decentralized actors (DCCDA). It claims that DCCDA sample gradients have variance at least as large as CTDE sample gradients under an idealistic perfect-communication assumption (Theorem 1) and under a binary-reward noisy communication model (Theorem 2). It then proposes a message-dependent optimal baseline (Theorem 3 and Corollary 1) plus a KL regularization term to align decentralized actors with communicating critics, and it reports improved win rates and lower gradient-norm variability on SMAC and Traffic Junction when extending GAAC and IPPO-Comm with these techniques.

Significance. If the theoretical results hold, the paper provides a useful first analysis of communication-induced variance in the DCCDA setting and a modular variance-reduction technique that transfers across algorithms. The empirical evaluation is reasonably broad, with ablations identifying the separate contributions of the baseline (OB) and the KL term, and the variance decomposition in Theorem 1 and the baseline derivation in Theorem 3 are explicit and non-circular. However, the central Lemma 1 proof currently uses a message-conditioned policy in the critic's Bellman backup even though DCCDA actors are decentralized, which means Theorem 1 may characterize a different algorithm than the one evaluated. The non-idealistic theorem also relies on a surrogate reward that depends on the unobserved true reward, limiting its reach. These issues are load-bearing for the paper's main claims, so the manuscript needs a substantive revision.

major comments (3)
  1. [Appendix A, Eq. (5) and line 7l] The Bellman equation for the communicating critic backs up with the message-conditioned policy pi_i(a'_i|hao_i,m'_{-i}), and line 7l factorizes the next-action distribution using exactly this policy. This contradicts the DCCDA definition in Section 3.2, where actors use decentralized policies pi_i(a_i|h_i) and do not communicate during training or execution. Consequently, Lemma 1, and with it Theorem 1 and Corollary 1, may only characterize a variant with communicating actors rather than the DCCDA estimator evaluated in the experiments. The authors should either re-derive the fixed-point equivalence using the decentralized backup pi_i(a'_i|hao_i) and prove that E_{m_{-i}|h,a}[Q^pi_i(hi,ai,m_{-i})] satisfies the centralized Bellman equation under that backup, or explicitly state and justify an alternative interpretation of Q^pi_i. The acknowledgement in Section 4.2 that non-communicating experience 'can mislead the training of communicating critics' does not repair the proof; it underscores the mismatch.
  2. [Appendix B, Eq. (14) and Theorem 2] The non-idealistic analysis restricts rewards to the binary set {r_+,r_-} and defines a surrogate reward hat{R}_i that depends on the true reward r_i, which is not observed by the agent once messages are noisy. The estimator hat{g}_{DCCDA-noise} built from the surrogate Q-function is therefore not the noisy DCCDA gradient that a practical agent would compute, and Theorem 2 does not establish the claimed variance ordering for the actual DCCDA estimator under noisy communication. The paper should state this limitation explicitly and, if the claim is intended to cover practical noisy communication, provide an analysis that does not require access to unobserved true rewards.
  3. [Appendix B, Theorem 2 proof, derivations (21a) and (22f)] The proof writes the noisy gradient with Q^pi_i(h,a,epsilon_i), whereas Lemma 3 and Eq. (16) define the surrogate object hat{Q}^pi_i(h,a,epsilon_i). The equality of expectations in (21c) is therefore not demonstrated as written, because the objects on the two sides do not match. Please correct the notation and verify that the variance comparison in (22) uses the same estimator throughout.
minor comments (4)
  1. [Section 5.2, Table 1] The reported quantity is the standard deviation of gradient norms across 8 seeds, which is a between-run statistic rather than the variance of the single-sample gradient estimators analyzed in Theorems 1--3; the text should clarify the relationship between this empirical measure and the theoretical variance, or report within-training gradient variance alongside it.
  2. [Section 3.2 and Algorithms 1--2] The theoretical gradient definitions use on-policy Q-values Q^pi_i, while the practical algorithms update critics by TD learning; the paper should state explicitly when Q_i is assumed to have converged to Q^pi_i, since Theorem 3 and Corollary 1 rely on that convergence.
  3. [Appendix D.4, Table 5] COMA is listed with 0.0 win rate on several maps but with nonzero standard deviations, and e.g. '0.1 (0.2)' appears for 5m_vs_6m; please check the formatting and rounding so the table is internally consistent.
  4. [Equations (2)--(3)] The KL term is written as L_KL(theta_i) = -D_KL(...), and Eq. (3) then adds beta * grad_theta_i L_KL(theta_i) to the policy gradient. Since L_KL is the negative KL divergence, the sign is correct but non-standard; a one-sentence explanation of the sign convention would help readers avoid confusion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the variance theorems follow from explicit assumptions and Jensen's inequality; the only notable issue is an internal-consistency mismatch in the Bellman backup, which is not a circular reduction.

full rationale

The paper's central derivation is not circular. Theorem 1 is obtained by proving Lemma 1 (Qπ(h,a)=E[Qπ_i]) from the stated ideal-communication Assumption 1 and the Bellman equations, then applying Jensen's inequality to the variance difference; the inequality is a mathematical consequence, not an assumed conclusion. Theorem 2 uses an analogous surrogate-reward construction (Eq. 14) and induction to establish the expectation equality, again followed by Jensen. Theorem 3 is a direct variance-minimization over the baseline bi(hi,m−i): the derivative condition yields b*_i=E[Qi S]/E[S], and Corollary 1 substitutes this value and obtains Var(DCCDA-OB)=Var(DCCDA) minus a non-negative term. None of these steps fits a fitted parameter to the target quantity or imports a uniqueness theorem from the authors' own prior work. The paper does contain a minor self-citation ([8], a survey by the same authors) but it is used only as background and is not load-bearing. The most serious potential issue, flagged by the skeptical reading, is that the Bellman backup in Eq. (5) defines Qπ_i using a message-conditioned next-action policy π_i(a'_i|hao_i,m'−i), while the DCCDA actor is stated to be decentralized; taken literally, Lemma 1 may characterize a different training policy than the one executed. This is an internal-consistency/correctness risk (and the paper itself acknowledges the actor-critic mismatch in Section 4.2, proposing the KL term to mitigate it), but it is not a circular reduction: the inequality is not assumed in the premises in the sense of 'Eq. X = Eq. Y by construction.' The empirical evaluation is benchmarked against external methods and reports actual learning curves and gradient-norm statistics, so no fitted input is relabeled as a prediction. Overall the paper is self-contained against its stated assumptions, and no circularity step is identifiable.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central results depend on the idealistic communication assumption (perfect message decoding), the convergence of learned critics to true Q-values, and a specific binary-reward noise model. The proposed methods introduce two tuned hyperparameters (alpha, beta) that are selected on the same benchmark tasks. No new physical or ontological entities are postulated.

free parameters (3)
  • alpha (KL temperature) = grid searched; values 0.5, 1, 5 in Appendix D.6
    Controls softmax sharpness in the KL regularization term; selected based on evaluation on all six tasks.
  • beta (KL scaling factor) = grid searched; values 0.0001, 0.001, 0.01 in Appendix D.6
    Scales the KL regularization term in the final gradient; selected on the same tasks used for reporting final results.
  • noise rate e = not set in experiments
    Parameter in the non-idealistic communication analysis that defines the probability of reward flipping due to noise; theoretical only.
assumptions (4)
  • ad hoc to paper Assumption 1: each received message correctly represents the sender's local information (histories and actions).
    Invoked in Lemma 1 (Appendix A) to equate the expected communicating critic with the centralized critic; required for Theorem 1.
  • domain assumption The learned critic Qi(hi, ai, m−i) converges to the true on-policy Q^π_i(hi, ai, m−i).
    Stated in Section 4.2 before Theorem 3; the optimal baseline is derived using true Q-values, not function approximations.
  • ad hoc to paper The noise model in the non-idealistic setting restricts rewards to binary values and uses a surrogate reward that depends on the true reward.
    Used in Appendix B to prove Lemma 2 and Theorem 2; the surrogate reward (Equation 14) is not computable in practice because true rewards are unknown.
  • standard math Standard Bellman operator contraction and uniqueness of Q-functions.
    Used to prove Lemma 1 and Lemma 3 following Lyu et al.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning." pith.science (2026). https://pith.science/paper/OOIEM4CI

@misc{pith2026250206261,
  author       = {Pith},
  title        = {Pith review of: Reducing Variance Caused by Communication in Decentralized Multi-agent Deep Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OOIEM4CI}},
  note         = {Machine review of arXiv:2502.06261}
}
read the original abstract

In decentralized multi-agent deep reinforcement learning (MADRL), communication can help agents to gain a better understanding of the environment to better coordinate their behaviors. Nevertheless, communication may involve uncertainty, which potentially introduces variance to the learning of decentralized agents. In this paper, we focus on a specific decentralized MADRL setting with communication and conduct a theoretical analysis to study the variance that is caused by communication in policy gradients. We propose modular techniques to reduce the variance in policy gradients during training. We adopt our modular techniques into two existing algorithms for decentralized MADRL with communication and evaluate them on multiple tasks in the StarCraft Multi-Agent Challenge and Traffic Junction domains. The results show that decentralized MADRL communication methods extended with our proposed techniques not only achieve high-performing agents but also reduce variance in policy gradients during training.

Figures

Figures reproduced from arXiv: 2502.06261 by the authors.

Figure 1
Figure 1. DCCDA methods integrated with OB and KL. [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Averaged win rate of all methods. method MAPPO outperforms other methods up to approximately 15 million steps but is ultimately surpassed by IPPO￾Comm-OB-KL. Despite this, in the remaining maps, where agents do not have different roles, IPPO-Comm-OB-KL surpasses all other methods. These results indicate that CTDE methods tend to perform better in scenarios where agents with different roles/skills need to collaborate… view at source ↗
Figure 3
Figure 3. Averaged win rate when ablating OB and KL. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: The training and execution phases for CTDE (without communication), CTDE (with communication), DTDE (without [PITH_FULL_IMAGE:figures/full_fig_p025_4.png]
Figure 5
Figure 5. Figure 5: As shown, GAAC-OB-KL and IPPO-Comm-OB-KL exhibit significantly lower variance in policy gradients [PITH_FULL_IMAGE:figures/full_fig_p028_5.png]
Figure 5
Figure 5. Figure 5: Variance in policy gradient norm of all methods. [PITH_FULL_IMAGE:figures/full_fig_p029_5.png]
Figure 6
Figure 6. Figure 6: Averaged win rate under different α and β in 4 tasks of SMAC and 2 tasks of Traffic Junction. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 39 canonical work pages

  1. [1]

    Safe, multi-agent, reinforcement learning for autonomous driving

    Shai Shalev-Shwartz, Shaked Shammah, and Amnon Shashua. Safe, multi-agent, reinforcement learning for autonomous driving. CoRR, abs/1610.03295, 2016

    arXiv 2016

  2. [2]

    Andrew Bagnell, and Jan Peters

    Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. Int. J. Robotics Res., 32(11):1238–1274, 2013

    work page 2013

  3. [3]

    Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy P. Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game of go without human knowledge. Nat., 550(7676):354–359, 2017

    work page 2017

  4. [4]

    Superhuman ai for multiplayer poker

    Noam Brown and Tuomas Sandholm. Superhuman ai for multiplayer poker. Science, 365(6456):885–890, 2019

    2019

  5. [5]

    Multi-agent deep reinforcement learning: a survey

    Sven Gronauer and Klaus Diepold. Multi-agent deep reinforcement learning: a survey. Artif. Intell. Rev. , 55(2):895–943, 2022

    work page 2022

  6. [6]

    A review of cooperative multi-agent deep reinforcement learning

    Afshin Oroojlooy and Davood Hajinezhad. A review of cooperative multi-agent deep reinforcement learning. Appl. Intell., 53(11):13677–13722, 2023

    work page 2023

  7. [7]

    An overview of multi-agent reinforcement learning from game theoretical perspective

    Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. CoRR, abs/2011.00583, 2020

    arXiv 2011

  8. [8]

    A survey of multi-agent deep reinforcement learning with communication

    Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent deep reinforcement learning with communication. Autonomous Agents and Multi-Agent Systems, 38(4), 2024

    work page 2024

Show all 42 references
  1. [9]

    Learning to communicate in multi-agent reinforcement learning : A review

    Mohamed Salah Zaïem and Etienne Bennequin. Learning to communicate in multi-agent reinforcement learning : A review. CoRR, abs/1911.05438, 2019

    1911 arXiv

  2. [10]

    Synchronizing UA V teams for timely data collection and energy transfer by deep reinforcement learning.IEEE Trans

    Omar Sami Oubbati, Mohammed Atiquzzaman, Hyotaek Lim, Abderrezak Rachedi, and Abderrahmane Lakas. Synchronizing UA V teams for timely data collection and energy transfer by deep reinforcement learning.IEEE Trans. Veh. Technol., 71(6):6682–6697, 2022

    2022

  3. [11]

    Gupta, Maxim Egorov, and Mykel J

    Jayesh K. Gupta, Maxim Egorov, and Mykel J. Kochenderfer. Cooperative multi-agent control using deep reinforcement learning. In Gita Sukthankar and Juan A. Rodríguez-Aguilar, editors, Autonomous Agents and Multiagent Systems - AAMAS 2017 Workshops, Best Papers, São Paulo, Braz...

    2017

  4. [12]

    Actor-attention-critic for multi-agent reinforcement learning

    Shariq Iqbal and Fei Sha. Actor-attention-critic for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volum...

    2019

  5. [13]

    Multi-agent game abstraction via graph attention neural network

    Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agent game abstraction via graph attention neural network. In The Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI), pages 7211–7218, 2020

    2020

  6. [14]

    Foerster, Yannis M

    Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems 29 (NIPS), pages 2137–2145, 2016

    2016

  7. [15]

    Learning attentional communication for multi-agent cooperation

    Jiechuan Jiang and Zongqing Lu. Learning attentional communication for multi-agent cooperation. In Advances in Neural Information Processing Systems 31 (NIPS), pages 7265–7275, 2018. 10 REDUCING VARIANCE CAUSED BY COMMUNICATION IN DECENTRALIZED MADRL

    2018

  8. [16]

    On centralized critics in multi-agent reinforcement learning

    Xueguang Lyu, Andrea Baisero, Yuchen Xiao, Brett Daley, and Christopher Amato. On centralized critics in multi-agent reinforcement learning. J. Artif. Intell. Res., 77:295–354, 2023

    2023

  9. [17]

    Contrasting centralized and decentralized critics in multi-agent reinforcement learning

    Xueguang Lyu, Yuchen Xiao, Brett Daley, and Christopher Amato. Contrasting centralized and decentralized critics in multi-agent reinforcement learning. In Frank Dignum, Alessio Lomuscio, Ulle Endriss, and Ann Nowé, editors, AAMAS ’21: 20th International Conference on Autonomou...

    2021

  10. [18]

    Mikayel Samvelyan, Tabish Rashid, Christian Schröder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob N. Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. In Edith Elkind, Manuela Veloso, Noa Agmon, and Mat...

    2019

  11. [19]

    Learning when to communicate at scale in multiagent cooperative and competitive tasks

    Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, 2019

    2019

  12. [20]

    Learning nearly decomposable value functions via communication minimization

    Tonghan Wang, Jianhao Wang, Chongyi Zheng, and Chongjie Zhang. Learning nearly decomposable value functions via communication minimization. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, 2020

    2020

  13. [21]

    Tarmac: Targeted multi-agent communication

    Abhishek Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pineau. Tarmac: Targeted multi-agent communication. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 1538–1546, 2019

    2019

  14. [22]

    Succinct and robust multi-agent communication with temporal message control

    Sai Qian Zhang, Qi Zhang, and Jieyu Lin. Succinct and robust multi-agent communication with temporal message control. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33...

    2020

  15. [23]

    Multi-agent incentive communication via decentralized teammate modeling

    Lei Yuan, Jianhao Wang, Fuxiang Zhang, Chenghe Wang, Zongzhang Zhang, Yang Yu, and Chongjie Zhang. Multi-agent incentive communication via decentralized teammate modeling. In Thirty-Sixth AAAI Conference on Artificial Intelligence (AAAI-22), 2022

    2022

  16. [24]

    Efficient communication via self-supervised information aggregation for online and offline multi-agent reinforcement learning

    Cong Guan, Feng Chen, Lei Yuan, Zongzhang Zhang, and Yang Yu. Efficient communication via self-supervised information aggregation for online and offline multi-agent reinforcement learning. CoRR, abs/2302.09605, 2023

    2023 arXiv

  17. [25]

    T2MAC: targeted and trusted multi-agent communication through selective engagement and evidence-driven integration

    Chuxiong Sun, Zehua Zang, Jiabao Li, Jiangmeng Li, Xiao Xu, Rui Wang, and Changwen Zheng. T2MAC: targeted and trusted multi-agent communication through selective engagement and evidence-driven integration. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editor...

    2024

  18. [26]

    Learning agent communication under limited bandwidth by message pruning

    Hangyu Mao, Zhengchao Zhang, Zhen Xiao, Zhibo Gong, and Yan Ni. Learning agent communication under limited bandwidth by message pruning. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 5142–5149, 2020

    2020

  19. [27]

    Learning individually inferred communication for multi-agent cooperation

    Ziluo Ding, Tiejun Huang, and Zongqing Lu. Learning individually inferred communication for multi-agent cooperation. In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33 ...

    2020

  20. [28]

    Scalable communication for multi-agent reinforcement learning via transformer-based email mechanism

    Xudong Guo, Daming Shi, and Wenhui Fan. Scalable communication for multi-agent reinforcement learning via transformer-based email mechanism. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI 2023, 19th-25th August 2023, Macao,...

    2023

  21. [29]

    Paleja, and Matthew C

    Yaru Niu, Rohan R. Paleja, and Matthew C. Gombolay. Multi-agent graph-attention communication and teaming. In 20th International Conference on Autonomous Agents and Multiagent Systems (AAMAS), pages 964–973, 2021

    2021

  22. [30]

    Rgmcomm: Return gap minimization via discrete communications in multi-agent reinforcement learning

    Jingdi Chen, Tian Lan, and Carlee Joe-Wong. Rgmcomm: Return gap minimization via discrete communications in multi-agent reinforcement learning. In Michael J. Wooldridge, Jennifer G. Dy, and Sriraam Natarajan, editors, Thirty-Eighth AAAI Conference on Artificial Intelligence, A...

    2024

  23. [31]

    Learning efficient multi-agent communication: An information bottleneck approach

    Rundong Wang, Xu He, Runsheng Yu, Wei Qiu, Bo An, and Zinovi Rabinovich. Learning efficient multi-agent communication: An information bottleneck approach. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learn...

    2020

  24. [32]

    Turner, Zoubin Ghahramani, and Sergey Levine

    George Tucker, Surya Bhupatiraju, Shixiang Gu, Richard E. Turner, Zoubin Ghahramani, and Sergey Levine. The mirage of action-dependent baselines in reinforcement learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 -...

    2018

  25. [33]

    Settling the variance of multi-agent policy gradients

    Jakub Grudzien Kuba, Muning Wen, Linghui Meng, Shangding Gu, Haifeng Zhang, David Mguni, Jun Wang, and Yaodong Yang. Settling the variance of multi-agent policy gradients. In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, e...

    2021

  26. [34]

    Bayen, Sham M

    Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M. Bayen, Sham M. Kakade, Igor Mordatch, and Pieter Abbeel. Variance reduction for policy gradient with action-dependent factorized baselines. In 6th International Conference on Learning Representations, ICLR 2018...

    2018

  27. [35]

    Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson

    Jakob N. Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counter- factual multi-agent policy gradients. In Sheila A. McIlraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligenc...

    2018

  28. [36]

    Oliehoek and Christopher Amato

    Frans A. Oliehoek and Christopher Amato. A Concise Introduction to Decentralized POMDPs. Springer Briefs in Intelligent Systems. Springer, 2016

    2016

  29. [37]

    How, and John Vian

    Shayegan Omidshafiei, Jason Pazis, Christopher Amato, Jonathan P. How, and John Vian. Deep decentralized multi-task multi-agent reinforcement learning under partial observability. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Ma...

    2017

  30. [38]

    Bayen, and Yi Wu

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre M. Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In NeurIPS, 2022

    2022

  31. [39]

    Reinforcement learning with perturbed rewards

    Jingkang Wang, Yang Liu, and Bo Li. Reinforcement learning with perturbed rewards. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium o...

    2020

  32. [40]

    Boltzmann exploration done right

    Nicolò Cesa-Bianchi, Claudio Gentile, Gergely Neu, and Gábor Lugosi. Boltzmann exploration done right. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V . N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processin...

    2017

  33. [41]

    Multi-agent reinforcement learning is a sequence modeling problem

    Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is a sequence modeling problem. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Informa...

    2022

  34. [42]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learnin...

    2018

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.