Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors

De Ma; Dong Xing; Gang Pan; Jiahao Lin; Lang Feng; Li Zhang

REVIEW 4 major objections 6 minor 68 references

Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read BiDist claims that training against a fictitious population updated by forward and reverse distillation lets multi-agent policies generalize to unseen co-players without storing past policies.

desk verdict A plausible new training trick with an overclaimed theory and thin experimental reporting—worth a serious look, but not as-is. read the letter →

arxiv 2505.11100 v1 pith:MJAC5SGJ submitted 2025-05-16 cs.LG cs.AI

Lang Feng , Jiahao Lin , Dong Xing , Li Zhang , De Ma , Gang Pan This is my paper · ORCID

classification cs.LGcs.AI

keywords multi-agentreinforcementlearningpopulation-populationgeneralizationunseenco-playersmixed-playframeworkknowledgedistillationself-playzero-shotcoordinationMeltingPot

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BiDist proposes a way to train a team of reinforcement-learning agents that keeps performing well when matched with co-players it has never seen. The method randomly detaches a subset of agents into a fictitious population whose policies are distilled copies of the learning policies, then updates those copies in two alternating directions: forward distillation pulls them toward the current learned policies, and reverse distillation pushes them away. The authors argue that this mix reproduces the benefit of self-play without storing a policy library, while also exploring interactions outside the historical policy space. On five Melting Pot tasks spanning cooperation, competition, and social dilemmas, BiDist attains the top normalized focal per-capita return in 22 of 25 scenarios and near-top performance in the rest, supported by a δ-cover generalization bound that ties smaller training-distribution covering radius to smaller test loss.

What carries the argument

Random fictitious population with alternating forward and reverse distillation. A fictitious population is a subset of agents randomly detached from the training population at each iteration; those agents execute distilled policies $\pi_\phi$ instead of the learning policies $\pi_\theta$ while collecting trajectories in the substrate. Every $k_d$ iterations, the distilled parameters $\phi$ are updated by one step of forward distillation ($\phi \leftarrow \phi - \eta_f \nabla_\phi L_{\mathrm{KL}}$, minimizing KL to the learning policies) or one step of reverse distillation ($\phi \leftarrow \phi + \eta_r \nabla_\phi L_{\mathrm{KL}}$, maximizing KL). The alternating schedule makes the distilled policies lag behind the learning policies, which supplies implicit self-play coverage, and then pushes them away, which supplies outside-space exploration. The theoretical carrier is Theorem 4.4: if the training distribution $P$ is a $\delta$-cover of the testing distribution $P_{\mathcal{Z}'}$ and the loss is $\lambda$-Lipschitz and zero on $P$, then $\mathbb{E}_{o_i,a^*_i\sim P_{\mathcal{Z}'}}[l(o_i,a^*_i;\Phi_P)] \le \delta\lambda(1+L|\mathcal{A}_i|) + \sqrt{\frac{L^2\log(1/\gamma)}{2n}}$ with probability at least $1-\gamma$; this ties smaller $\delta$ to smaller worst-case test loss, and BiDist's reverse phase is what reduces $\delta$ beyond self-play.

What would settle it

Train BiDist on a substrate, then test against a background population whose preferred action is the one farthest from the action the reverse-distilled fictitious population shifted toward. If the focal population's per-capita return in that scenario collapses to the MAPPO level, the reverse step has not actually covered outside-space preferences and the central generalization claim is falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that a multi-agent population can learn to interact well with unseen co-players by training against a 'fictitious population' whose policies are distilled copies of the learning agents, updated in two alternating directions. Forward distillation (minimizing KL divergence from the learning policies) recreates the coverage that self-play obtains from storing historical policies, without the storage cost; reverse distillation (maximizing KL divergence) shifts the distilled policies' preferred actions so that the learning agents encounter partners whose behavior lies outside the historical policy space. The authors show that this preference shift is the key difference from entropy maximization, randomization, and noise injection, and they give a δ-cover generalization bound in which reducing the covering radius δ of the training distribution tightens the guarantee on the testing distribution. Empirically, BiDist attains the maximum normalized focal per-capita return in 22 of the 25 tested Melting Pot scenarios and stays near the top in the remaining three, with t-SNE visualizations showing its joint action distribution spreading well beyond the clusters produced by self-play baselines.

Load-bearing premise

The proof assumes each observation has one ground-truth optimal action and that the loss is exactly zero on the training distribution; in zero-shot multi-agent tasks the best action depends on the unseen co-players' joint behavior, so that ground-truth action is not well defined for the generalization scenario.

Editorial extensions

If this is right

Zero-shot evaluation against unseen background populations is improved without storing past policies; one distilled network per agent is enough.
Any MARL algorithm that samples policies from a mixed population can carry BiDist, so the benefit is not tied to the MAPPO instantiation.
The δ-cover analysis says that whatever shrinks the covering radius of the training distribution tightens the generalization guarantee, placing BiDist in a broader family of coverage-reducing training schemes.
Reverse distillation's KL maximization is qualitatively different from entropy maximization, randomization, or noise injection because it targets the identity of the most-preferred action.
Both distillation directions are necessary: removing either forward or reverse distillation degrades performance, so implicit self-play retention and outside-space exploration each contribute.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension would apply the same forward/reverse distillation loop to human-AI coordination, where the reverse step could be steered toward plausible human biases rather than arbitrary preference shifts; the paper only evaluates pre-trained background populations.
One measurable prediction of the δ-cover argument is that the empirical covering radius of the joint action distribution should shrink as BiDist training proceeds and zero-shot returns rise, giving a cheap diagnostic that does not require running the test scenarios.
The ablation result that forward distillation alone slightly surpasses RPM hints that distillation error itself may add useful diversity; isolating a perfect-copy baseline would separate the effect of lagged policies from the effect of distillation noise.
Because reverse distillation uses a small fixed learning rate, a natural next step is a schedule for $\eta_r$ that grows the preference shift over training, potentially covering farther outside-space behaviors without generating infeasible policies.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Zhen,

This one is worth reading if you care about zero-shot co-player generalization in MARL. The core idea is simple: keep a separate 'distilled' policy network, and alternate between distilling it from the current learning policies (which acts like implicit self-play) and pushing it away by KL maximization. That gives you a co-player population without storing a history of policies. As far as I can tell, that combination is new. Prior methods like RPM, PP, or OPRE either keep explicit pools or rely on entropy-based diversity. The empirical section also does the right kind of coverage: cooperation, competition, and social dilemma tasks in Melting Pot. The ablations show that both forward and reverse directions matter, and the single-network resource story is a genuine advantage over policy-pool methods.

But there are two soft spots that keep me from taking the headline numbers at face value.

First, the experimental reporting. Table 1 reports min-max normalized returns with no error bars, no standard deviations, and no raw scores. BiDist scores 1.0 or close to it in every scenario. That could be a real and consistent win, or it could be an artifact of normalization when the absolute differences are small. The text says results are averaged over 3 seeds, but without variance we can't distinguish a robust method from a lucky run. This is fixable and necessary.

Second, the theory. Theorem 4.4 is advertised as showing that reducing the covering radius delta lowers the generalization error bound. But the proof in Appendix A.3 bounds expectation over actions sampled from the learned policy, while the theorem statement concerns the ground-truth optimal action a*_i. Nothing connects those two quantities. On top of that, in the zero-shot co-player setting the 'optimal action' is not a well-defined function of the observation alone—it depends on the background population's policy. So the theorem doesn't actually apply to this problem. The claim that BiDist reduces delta is asserted from a schematic, not proved. My take: the theory is not load-bearing. The method might still work for reasons this theorem doesn't capture, and the authors should either fix the proof or honestly label the theoretical section as intuition.

The central method itself is coherent and not circular. The hyperparameters include some hand-tuning (distillation interval chosen on one task), but that is minor.

Who's this for? Anyone working on population-based training or zero-shot coordination. It deserves a serious referee—send it to review—but I would ask for major revision: add error bars and raw returns, and either repair or demote the theoretical analysis. As it stands, the empirical claim is plausible but not yet demonstrated to the level the paper asserts.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes Bidirectional Distillation (BiDist), a training framework for population-population generalization in multi-agent reinforcement learning. During training, a random subset of agents is designated as a 'fictitious' population that uses distilled policies. Forward distillation periodically distills the learning policies into the fictitious population to emulate self-play without storing past policies; reverse distillation pushes the distilled policies away via KL-divergence maximization to induce preference shifts and explore behavior outside the historical policy space. The learning agents are trained with MAPPO against this fictitious population. The paper claims that this mixed-play procedure improves zero-shot generalization to unseen co-players, supports the claim with a theoretical analysis based on a δ-cover covering radius (Theorem 4.4), and reports experiments on five Melting Pot substrates (Pure Coordination, Coop Mining, Chicken, Coins, Prisoners Dilemma) with baselines MAPPO, RanNet, OPRE, PP, and RPM. The central empirical claim is that BiDist attains the highest min-max normalized focal per-capita return in nearly all reported scenarios.

Significance. If substantiated, BiDist is a valuable practical contribution: it replaces explicit policy pools with a single distilled network, is algorithm-agnostic (instantiated with MAPPO), and is evaluated across cooperative, competitive, and social-dilemma settings. The paper includes ablation studies (Section 5.5, Figure 8), a perturbation comparison validating the choice of KL maximization (Section 5.4), and resource-efficiency experiments (Appendix C), which are genuine strengths. The training procedure itself is not circular: the distilled policies act as legitimate co-players generated from the learning policies. However, the theoretical argument advertised in the abstract and Section 1 is not established: the proof of Theorem 4.4 does not track the quantity it claims to bound, and the paper does not prove that reverse distillation reduces δ. The empirical evidence also lacks error bars and significance tests. The core algorithmic idea is plausible and worth pursuing, but the manuscript currently overstates the support it provides.

major comments (4)

[Section 4.4; Appendix A.3, Eqs. (A.20)-(A.25)] Theorem 4.4 states a bound on E_{o_i,a*_i∼P_Z'}[l(o_i,a*_i; Φ_P)], but the proof bounds E_{a_i∼π_i(·|o_i)}[l(o_i,a_i; Φ_P)]. The step labeled Hoeffding (Eq. A.24) compares these two different expectations under different measures, which is not a valid application of Hoeffding's inequality. Moreover, the proof uses the zero-loss assumption l(o_j,a_i; Φ_P)=0 for all actions a_i for o_j∈P, while Definition 4.2 defines l only at the single ground-truth optimal action a*_i. The stronger assumption is never stated, so the proof is incomplete even for the surrogate quantity.
[Definition 4.2; Section 3] The theorem assumes a well-defined ground-truth optimal action a*_i for each observation. In the zero-shot co-player setting formalized in Section 3, the optimal action for agent i depends on the joint policy of the unseen background population g, which varies across testing scenarios Z'. The manuscript does not define a distribution over co-player policies for which P is a δ-cover of P_Z', nor does it justify that such an a*_i exists. Consequently, the application of Theorem 4.4 to BiDist is not established.
[Section 1; Section 4.4] The paper's central claim that BiDist 'efficiently reduces δ' is not proven. The discussion after Theorem 4.4 and the schematic in Figure 2 are qualitative; no formal statement connects the forward and reverse distillation updates to a decrease in the covering radius δ. Without a proof or at least a quantitative characterization of how the fictitious population generated by BiDist relates to P_Z', the theoretical mechanism advertised in the abstract remains unsupported.
[Table 1; Section 5.2] The primary empirical result reports min-max normalized focal per-capita returns without any standard deviations, confidence intervals, or significance tests. Min-max normalization within each scenario can exaggerate small raw differences, and the near-universal score of 1.00 for BiDist across scenarios may in part be an artifact of this normalization. The paper should report unnormalized returns with error bars across the three seeds mentioned in Appendix B.3, and provide statistical comparisons to the strongest baseline.

minor comments (6)

[Figure 2] The legend text appears to contain missing symbols (the markers are described as 'denotes' without visible glyphs). Please ensure the figure renders correctly in the final PDF.
[Equations (3)-(5)] The definition of the sampling distribution 𝔅(v_<i; p,N) is confusing, particularly the role of δ_i; please define the joint distribution over v explicitly.
[Algorithm 1, Line 8] The notation θ←arg max_θ J(θ) is inconsistent with the clipped PPO objective in Eq. (9); please describe the actual parameter update step.
[Table B.2] 'Distillation internal' should be 'Distillation interval'.
[Appendix C] The text contains broken Unicode glyphs in the figure captions (e.g., '0/uni...' sequences); please check the source files.
[Section 5.1] The 70%/30% sampling probabilities for the PP baseline are stated without justification; citing the source or adding a sensitivity note would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: BiDist's training procedure, empirical evaluation, and generalization-bound discussion are self-contained; the proof gap in Theorem 4.4 is a rigor issue, not a circular reduction.

full rationale

The paper does not fit any parameter to the test scenarios and then relabel it as a prediction; the distilled-policy update rules (forward KL minimization, reverse KL maximization) are defined independently of the reported normalized returns, and all baselines are evaluated on the same external Melting Pot substrates. The δ-cover generalization bound is a generic Lipschitz/covering argument cited from external references rather than an assumption that BiDist reduces δ; the statement that BiDist reduces δ is an intuitive and empirical claim supported by t-SNE visualizations and ablations, not a conclusion forced by the theorem's definition. The main formal weakness is that the proof of Theorem 4.4 bounds E_{a_i∼π_i(·|o_i)}[l(o_i,a_i;ΦP)] in Eq. (A.20)/(A.23), while the theorem states a bound on E_{o_i,a*_i∼P_Z'}[l(o_i,a*_i;ΦP)], and no argument connects l(o_i,a_i) to l(o_i,a*_i); this is a potential correctness gap in the theoretical claim, but it is not an instance of the paper deriving its conclusion from its own assumption or renaming a fitted quantity as a prediction. Self-citations are limited to method-building blocks (e.g., prior MARL algorithm work) and are not load-bearing for the central empirical claim, so they do not raise the circularity score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method relies on hand-set hyperparameters (p, k_d, eta_f, eta_r) and on theoretical assumptions that are either domain-specific or not fully justified. No new physical or conceptual entities are introduced; the fictitious population is a training construct, not a new entity in the environment.

free parameters (4)

probability p = 0.4 for small tasks, 0.2 for larger ones
Controls the fraction of agents assigned to the fictitious population; chosen per task in Section 5.1.
distillation interval k_d = 5
Selected from ablation on Pure Coordination (Appendix C.2); interval between forward and reverse distillation updates.
forward distillation learning rate eta_f = 1e-3
Hyperparameter for forward distillation updates; given in Table B.2.
reverse distillation learning rate eta_r = 1e-5
Set small to prevent infeasible or ineffective distilled policies (Section 5.1).

assumptions (4)

domain assumption A unique ground-truth optimal action a*_i exists for each observation in the zero-shot co-player task (Definition 4.2).
Used to define the loss function and the training-distribution zero-loss condition; questionable in MARL because optimal actions depend on co-player policies.
domain assumption The training distribution P is a delta-cover of the testing distribution P_Z' (Theorem 4.4).
This is the core condition for the generalization bound; the paper argues informally that BiDist reduces delta but does not prove it.
ad hoc to paper The loss function vanishes on the training distribution (l(o_i, a*_i; Phi_P) = 0 for all o_i in P).
Used in the proof of Theorem 4.4 (Appendix A.3, bounding first term); unrealistic for MARL agents that are not optimal on every training observation.
standard math Policy and loss are lambda-Lipschitz with the stated constant (Lemma 4.3).
Provides the smoothness needed to transfer errors from training to test observations; the constant is architecture-specific but derived in the appendix.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors." pith.science (2026). https://pith.science/paper/MJAC5SGJ

@misc{pith2026250511100,
  author       = {Pith},
  title        = {Pith review of: Bidirectional Distillation: A Mixed-Play Framework for Multi-Agent Generalizable Behaviors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MJAC5SGJ}},
  note         = {Machine review of arXiv:2505.11100}
}

read the original abstract

Population-population generalization is a challenging problem in multi-agent reinforcement learning (MARL), particularly when agents encounter unseen co-players. However, existing self-play-based methods are constrained by the limitation of inside-space generalization. In this study, we propose Bidirectional Distillation (BiDist), a novel mixed-play framework, to overcome this limitation in MARL. BiDist leverages knowledge distillation in two alternating directions: forward distillation, which emulates the historical policies' space and creates an implicit self-play, and reverse distillation, which systematically drives agents towards novel distributions outside the known policy space in a non-self-play manner. In addition, BiDist operates as a concise and efficient solution without the need for the complex and costly storage of past policies. We provide both theoretical analysis and empirical evidence to support BiDist's effectiveness. Our results highlight its remarkable generalization ability across a variety of cooperative, competitive, and social dilemma tasks, and reveal that BiDist significantly diversifies the policy distribution space. We also present comprehensive ablation studies to reinforce BiDist's effectiveness and key success factors. Source codes are available in the supplementary material.

Figures

Figures reproduced from arXiv: 2505.11100 by the authors.

**Figure 2.** Inside-space and outside-space generalization. [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Illustration of BiDist in an 8-agent task. Based on vector [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Illustrative instances of the testing scenarios (0-2) [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: The t-SNE results of different algorithms on [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 7.** Figure 7: Normalized joint probability distribution of dis [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

**Figure 8.** Figure 8: Ablation studies of BiDist on Pure Coordination task. RPM is for comparison. 5.5 Ablations We finally evaluate the ablations of BiDist to demonstrate the effectiveness of each component. Specifically, we examine the impact of the fictitious population by setting 𝒗 = {…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 37 canonical work pages

[1]

John P Agapiou, Alexander Sasha Vezhnevets, Edgar A Duéñez-Guzmán, Jayd Matyas, Yiran Mao, Peter Sunehag, Raphael Köster, Udari Madhushani, Kavya Kopparapu, Ramona Comanescu, et al . 2022. Melting Pot 2.0. arXiv preprint arXiv:2211.13746 (2022)

arXiv 2022
[2]

Pierre-Luc Bacon, Jean Harb, and Doina Precup. 2017. The option-critic architec- ture. In Proceedings of the AAAI conference on artificial intelligence , Vol. 31

work page 2017
[3]

Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. 2019. Emergent Tool Use From Multi-Agent Au- tocurricula. In International Conference on Learning Representations

work page 2019
[4]

Samuel Barrett and Peter Stone. 2015. Cooperating with unknown teammates in complex domains: A robot soccer case study of ad hoc teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 29

work page 2015
[5]

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. 2019. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 (2019)

arXiv 2019
[6]

Nicolas Carion, Nicolas Usunier, Gabriel Synnaeve, and Alessandro Lazaric. 2019. A structured prediction approach for generalization in cooperative multi-agent reinforcement learning. Advances in neural information processing systems 32 (2019)

work page 2019
[7]

Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems 32 (2019)

2019
[8]

Rujikorn Charakorn, Poramate Manoonpong, and Nat Dilokthanakul. 2022. Gen- erating Diverse Cooperative Agents by Learning Incompatible Policies. In The Eleventh International Conference on Learning Representations

work page 2022

Show all 68 references

[9]

Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, and John Schulman. 2019. Quantifying generalization in reinforcement learning. InInternational Conference on Machine Learning. PMLR, 1282–1289

2019
[10]

Zhongxiang Dai, Yizhou Chen, Bryan Kian Hsiang Low, Patrick Jaillet, and Teck- Hua Ho. 2020. R2-B2: Recursive reasoning-based Bayesian optimization for no-regret learning in games. In International Conference on Machine Learning . PMLR, 2291–2301

2020
[11]

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition . Ieee, 248–255

2009
[12]

Jesse Farebrother, Marlos C Machado, and Michael Bowling. 2018. Generalization and regularization in dqn. arXiv preprint arXiv:1810.00123 (2018)

2018 arXiv
[13]

Lang Feng, Dong Xing, Junru Zhang, and Gang Pan. 2023. FP3O: Enabling proximal policy optimization in multi-agent cooperation with parameter-sharing versatility. arXiv preprint arXiv:2310.05053 (2023)

2023 arXiv
[14]

Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. 2018. Learning with Opponent-Learning Awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. 122–130

2018
[15]

Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shi- mon Whiteson. 2018. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on Artificial Intelligence , Vol. 32

2018
[16]

Shani Gamrian and Yoav Goldberg. 2019. Transfer learning for related reinforce- ment learning tasks via image-to-image translation. In International conference on machine learning. PMLR, 2063–2072

2019
[17]

Pengjie Gu, Mengchen Zhao, Jianye Hao, and Bo An. 2021. Online ad hoc teamwork under partial observability. In International Conference on Learning Representations

2021
[18]

Johannes Heinrich, Marc Lanctot, and David Silver. 2015. Fictitious self-play in extensive-form games. In International conference on machine learning . PMLR, 805–813

2015
[19]

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)

2015 arXiv
[20]

other- play

Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. 2020. “other- play” for zero-shot coordination. InInternational Conference on Machine Learning. PMLR, 4399–4410

2020
[21]

Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. 2020. Composi- tionality decomposed: How do neural networks generalise? Journal of Artificial Intelligence Research 67 (2020), 757–795

2020
[22]

Shariq Iqbal and Fei Sha. 2019. Actor-attention-critic for multi-agent reinforce- ment learning. InInternational conference on machine learning. PMLR, 2961–2970

2019
[23]

Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. 2019. Human-level performance in 3D multiplayer games with population-based reinforcement learning. S...

2019
[24]

Yuhua Jiang, Qihan Liu, Xiaoteng Ma, Chenghao Li, Yiqin Yang, Jun Yang, Bin Liang, and Qianchuan Zhao. 2024. Learning Diverse Risk Preferences in Population-Based Self-Play. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 12910–12918

2024
[25]

James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of...

2017
[26]

Ilya Kostrikov, Denis Yarats, and Rob Fergus. 2020. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649 (2020)

2020 arXiv
[27]

Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. 2022. Trust Region Policy Optimisation in Multi-Agent Reinforcement Learning. In International Conference on Learning Representations

2022
[28]

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015. Deep learning. nature 521, 7553 (2015), 436–444

2015
[29]

Kimin Lee, Kibok Lee, Jinwoo Shin, and Honglak Lee. 2019. Network random- ization: A simple technique for generalization in deep reinforcement learning. arXiv preprint arXiv:1910.05396 (2019)

2019 arXiv
[30]

Joel Z Leibo, Edgar A Dueñez-Guzman, Alexander Vezhnevets, John P Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charlie Beattie, Igor Mordatch, and Thore Graepel. 2021. Scalable evaluation of multi-agent reinforcement learning with melting pot. In International conferen...

2021
[31]

Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAI Pieter Abbeel, and Igor Mordatch. 2017. Multi-agent actor-critic for mixed cooperative-competitive environments. Advances in Neural Information Processing Systems 30 (2017)

2017
[32]

Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. 2021. Trajec- tory diversity for zero-shot coordination. In International conference on machine learning. PMLR, 7204–7213

2021
[33]

Anuj Mahajan, Mikayel Samvelyan, Tarun Gupta, Benjamin Ellis, Mingfei Sun, Tim Rocktäschel, and Shimon Whiteson. 2022. Generalization in cooperative multi-agent systems. arXiv preprint arXiv:2202.00104 (2022)

2022 arXiv
[34]

Reuth Mirsky, William Macke, Andy Wang, Harel Yedidsion, and Peter Stone. 2020. A penny for your thoughts: The value of communication in ad hoc teamwork. Good Systems-Published Research (2020)

2020
[35]

Dung Nguyen, Hung Le, Kien Do, Sunil Gupta, Svetha Venkatesh, and Truyen Tran. 2024. Diversifying Training Pool Predictability for Zero-shot Coordination: A Theory of Mind Approach. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence. 166–174

2024
[36]

Frans A Oliehoek, Christopher Amato, et al . 2016. A concise introduction to decentralized POMDPs. Vol. 1. Springer

2016
[37]

Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. 2008. Optimal and approximate Q-value functions for decentralized POMDPs. Journal of Artificial Intelligence Research 32 (2008), 289–353

2008
[38]

Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. 2018. Assessing generalization in deep reinforcement learning. arXiv preprint arXiv:1810.12282 (2018)

2018 arXiv
[39]

Wei Qiu, Xiao Ma, Bo An, Svetlana Obraztsova, YAN Shuicheng, and Zhongwen Xu. 2023. RPM: Generalizable Multi-Agent Policies for Multi-Agent Reinforce- ment Learning. In The Eleventh International Conference on Learning Representa- tions

2023
[40]

Muhammad A Rahman, Niklas Hopner, Filippos Christianos, and Stefano V Albrecht. 2021. Towards open ad hoc teamwork using graph-based policy learning. In International Conference on Machine Learning . PMLR, 8776–8786

2021
[41]

Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus
[42]

Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. 2018. Qmix: Monotonic value function factori- sation for deep multi-agent reinforcement learning. In International Conference on Machine Learning. PMLR, 4295–4304

2018
[43]

Claude Sammut and Geoffrey I Webb. 2017. Encyclopedia of machine learning and data mining. Springer Publishing Company, Incorporated

2017
[44]

Bidipta Sarkar, Andy Shih, and Dorsa Sadigh. 2024. Diverse conventions for human-AI collaboration. Advances in Neural Information Processing Systems 36 (2024)

2024
[45]

Jordan, and Pieter Abbeel

John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and Pieter Abbeel. 2016. High-Dimensional Continuous Control Using Generalized Advan- tage Estimation. In 4th International Conference on Learning Representations

2016
[46]

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov
[47]

Ozan Sener and Silvio Savarese. 2018. Active Learning for Convolutional Neu- ral Networks: A Core-Set Approach. In International Conference on Learning Representations

2018
[48]

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. 2018. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science 3...

2018
[49]

Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. 2019. Qtran: Learning to factorize with transformation for cooperative multi- agent reinforcement learning. In International Conference on Machine Learning . PMLR, 5887–5896

2019
[50]

Peter Stone, Gal Kaminka, Sarit Kraus, and Jeffrey Rosenschein. 2010. Ad hoc autonomous agent teams: Collaboration without pre-coordination. InProceedings of the AAAI Conference on Artificial Intelligence , Vol. 24. 1504–1509

2010
[51]

DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett
[52]

Gerald Tesauro. 1994. TD-Gammon, a self-teaching backgammon program, achieves master-level play. Neural computation 6, 2 (1994), 215–219

1994
[53]

Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research 9, 11 (2008)

2008
[54]

Alexander Vezhnevets, Yuhuai Wu, Maria Eckstein, Rémi Leblond, and Joel Z Leibo. 2020. Options as responses: Grounding behavioural hierarchies in multi- agent reinforcement learning. In International Conference on Machine Learning . PMLR, 9733–9742

2020
[55]

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, An- drew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. nature 575, 7782 (2019), 350–354

2019
[56]

Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. 2021. QPLEX: Duplex Dueling Multi-Agent Q-Learning. In 9th International Conference on Learning Representations. OpenReview.net

2021
[57]

Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. 2020. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523 (2020)

2020 arXiv
[58]

Ying Wen, Yaodong Yang, Rui Luo, Jun Wang, and Wei Pan. 2018. Probabilistic Recursive Reasoning for Multi-Agent Reinforcement Learning. In International Conference on Learning Representations

2018
[59]

Xue Yan, Jiaxian Guo, Xingzhou Lou, Jun Wang, Haifeng Zhang, and Yali Du. 2024. An efficient end-to-end training approach for zero-shot human-AI coordination. Advances in Neural Information Processing Systems 36 (2024)

2024
[60]

Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. 2023. Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biased. In The Eleventh International Conference on Learning Representations

2023
[61]

Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu
[62]

Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. 2023. A survey of progress on cooperative multi-agent reinforcement learning in open environment. arXiv preprint arXiv:2312.01058 (2023)

2023 arXiv
[63]

Amy Zhang, Nicolas Ballas, and Joelle Pineau. 2018. A dissection of overfit- ting and generalization in continuous reinforcement learning. arXiv preprint arXiv:1806.07937 (2018)

2018 arXiv
[64]

Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. 2023. Maximum entropy population-based training for zero- shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 6145–6153. A PROOFS A...

2023
[65]

arXiv preprint arXiv:2103.01955 (2021)

The surprising effectiveness of ppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955 (2021)

2021 arXiv
[2017]

arXiv preprint arXiv:1707.06347 (2017)

Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 (2017)

2017 arXiv
[2020]

arXiv preprint arXiv:2006.12862 (2020)

Automatic data augmentation for generalization in deep reinforcement learning. arXiv preprint arXiv:2006.12862 (2020)

2020 arXiv
[2021]

Advances in Neural Information Processing Systems 34 (2021), 14502–14515

Collaborating with humans without human data. Advances in Neural Information Processing Systems 34 (2021), 14502–14515

2021

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

John P Agapiou, Alexander Sasha Vezhnevets, Edgar A Duéñez-Guzmán, Jayd Matyas, Yiran Mao, Peter Sunehag, Raphael Köster, Udari Madhushani, Kavya Kopparapu, Ramona Comanescu, et al . 2022. Melting Pot 2.0. arXiv preprint arXiv:2211.13746 (2022)

arXiv 2022

[2] [2]

Pierre-Luc Bacon, Jean Harb, and Doina Precup. 2017. The option-critic architec- ture. In Proceedings of the AAAI conference on artificial intelligence , Vol. 31

work page 2017

[3] [3]

Bowen Baker, Ingmar Kanitscheider, Todor Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. 2019. Emergent Tool Use From Multi-Agent Au- tocurricula. In International Conference on Learning Representations

work page 2019

[4] [4]

Samuel Barrett and Peter Stone. 2015. Cooperating with unknown teammates in complex domains: A robot soccer case study of ad hoc teamwork. In Proceedings of the AAAI Conference on Artificial Intelligence , Vol. 29

work page 2015

[5] [5]

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. 2019. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 (2019)

arXiv 2019

[6] [6]

Nicolas Carion, Nicolas Usunier, Gabriel Synnaeve, and Alessandro Lazaric. 2019. A structured prediction approach for generalization in cooperative multi-agent reinforcement learning. Advances in neural information processing systems 32 (2019)

work page 2019

[7] [7]

Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. 2019. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems 32 (2019)

2019

[8] [8]

Rujikorn Charakorn, Poramate Manoonpong, and Nat Dilokthanakul. 2022. Gen- erating Diverse Cooperative Agents by Learning Incompatible Policies. In The Eleventh International Conference on Learning Representations

work page 2022