Pith. sign in

REVIEW 3 major objections 7 minor 1 cited by

Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination

T0 review · 3 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Training a single self-play policy on a distribution of procedurally generated cooperative tasks yields zero-shot coordination with novel partners and novel environments, and this environment diversity outperforms population-based…

desk verdict CEC is a real empirical step for zero-shot coordination, but the novel-environment claim is undercut by reuse of five wall topologies and the abstract overstates human reward versus E3T. read the letter →

arxiv 2504.12714 v2 pith:SK5ZXHX4 submitted 2025-04-17 cs.MA cs.AIcs.LG

Kunal Jha , Wilka Carvalho , Yancheng Liang , Simon S. Du , Max Kleiman-Weiner , Natasha Jaques This is my paper · ORCID
classification cs.MAcs.AIcs.LG
keywords zero-shotcoordinationmulti-agentreinforcementlearningcross-environmentcooperationenvironmentdiversityproceduralgenerationself-playhuman-AIOvercooked
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that environment diversity can substitute for partner diversity in zero-shot coordination: a single self-play policy trained across many procedurally generated cooperative tasks learns general cooperative norms that transfer to novel partners and novel tasks. This matters because current state-of-the-art methods train against populations of partners on one task and fail when the task changes even slightly; practical agents would otherwise need retraining for every new kitchen, road, or room. The authors demonstrate the effect in a simple gridworld and in a scaled-up cooking game, and show that their agents are preferred by human participants on subjective cooperation metrics despite never seeing human data. If correct, the result offers a route toward generalist cooperative AI that does not require collecting human demonstrations or maintaining partner populations.

What carries the argument

The load-bearing mechanism is the procedural task distribution itself: a generator samples one of five base wall structures, removes objects, randomizes goal, plate, pot, and onion placements on reachable walls, randomizes agent start positions, and randomly rotates the grid, yielding billions of solvable layouts. A single recurrent (LSTM) IPPO policy trains in self-play against itself on this distribution, with the cross-environment objective forcing the policy to solve the same kinds of coordination problems in different spatial arrangements. The recurrence lets the policy adapt within an episode, which the authors argue is what lets a single frozen network act differently with different partners and in different environments. This replaces the partner population of PBT with a task distribution while using the same total compute.

What would settle it

Train a CEC agent with a generator that samples wall topologies from a disjoint held-out family never seen during training (not just new object positions on the five familiar wall structures), and measure cross-play reward on those held-out layouts; if performance falls to single-task-baseline levels, the reported environment generalization is driven by exposure to the training layout support rather than by general cooperative norms.

Watch

Extended reading notes

Core claim

The central claim is that optimizing the cross-environment objective $J(\pi_C)=\mathbb{E}_{m\sim\mathcal{M}}[S(\pi_C,\pi_C,m)]$ — self-play on a distribution of tasks rather than against a population of partners — yields a policy whose cross-play performance with novel partners is higher than population-based training on a single task, and whose behavior transfers to novel tasks and to human partners. In the toy Dual Destination game, CEC scores near an oracle cross-play bound, while self-play and fictitious co-play fail on novel layouts. In Overcooked, single-task methods receive zero reward on 100 held-out procedural layouts, whereas CEC achieves substantial cross-play reward; fine-tuning CEC on a target layout improves single-task cross-play but reduces novel-layout generalization. Human studies on two layouts show CEC outperforms FCP on cooperation score, approaches the reward trajectory of specialized agents, and is rated higher on adaptation, consistency, and enjoyability.

Load-bearing premise

The central claim depends on the held-out test layouts being genuinely novel environments, but the procedural generator builds every layout from the same five wall structures the agents train on, changing only object placements and rotations; if wall structure is what limits generalization, the advantage may shrink once test layouts contain unseen wall topologies.

Editorial extensions

If this is right

  • A single CEC policy can be reused as pretraining: fine-tuning on a target layout improves cross-play reward there beyond FCP and IPPO, at the cost of reduced generalization to novel layouts.
  • Population-based single-task methods score zero on 100 held-out procedural Overcooked layouts, so deploying them where the task varies even slightly would require retraining per layout.
  • Without any human data, CEC approaches the human-AI reward of a specialized agent that trained on the test layout, and beats all baselines on human ratings of adaptation, consistency, and enjoyability.
  • Because CEC trains one policy instead of a population, the same compute budget can be spent on many environment samples rather than many partner seeds, making the approach cheaper to scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stricter procedural split that withholds wall topologies, not just object placements, would likely shrink the reported generalization gap; part of CEC's advantage may come from having seen the same five wall structures in training rather than from truly general norms.
  • Combining partner diversity with environment diversity underperforms vanilla CEC in their experiments, but the reported learning curves suggest this is a capacity and convergence issue; longer training or larger networks may unlock both axes together.
  • The same environment-diversity principle should transfer to partially observable and communication-heavy games; the paper's partial-observation Dual Destination result supports this, but hidden-state conventions like those in Hanabi remain untested.
  • CEC's collision-avoidance behavior could be measured directly as a learned social norm (e.g., yielding frequency) and tested for transfer to a different task family, rather than relying only on human preference ratings.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces Cross-Environment Cooperation (CEC), a self-play training paradigm that replaces partner diversity with environment diversity, and evaluates it on a toy Dual Destination game and a JAX-based procedurally generated Overcooked benchmark. The authors train a single recurrent IPPO policy on a distribution of procedurally generated tasks and compare cross-play performance with single-task self-play, FCP, E3T, and a CEC-finetuned variant, in both AI-AI and human-AI settings. They report that CEC outperforms PBT baselines in cross-play on the original five layouts and on 100 generated grids, and that human participants rate CEC as more cooperative despite somewhat lower task score than E3T.

Significance. If the main result holds, the paper provides a useful counterpoint to population-based training: it suggests that increasing environment diversity during self-play can improve zero-shot coordination with unseen partners, and it contributes a fast JAX procedural generator (up to 10 million steps per minute on a single GPU, with roughly 10^17 layouts claimed) plus open source code. The simulated and human experiments are extensive, with multiple seeds, several baselines, an empirical game-theoretic analysis, and a human study with subjective cooperation metrics. The main caveat is that environment novelty is not fully tested, because the generator samples wall structures from the same five layouts that are used in training and evaluation; the central novelty claim therefore needs additional experimental support.

major comments (3)
  1. [Section 4 / Section A.1 / Algorithm 1] The claim of zero-shot coordination in novel environments is not fully supported as stated. Algorithm 1 first samples L_base uniformly from the same five wall layouts used as evaluation layouts, and the only leakage check in the post-processing step is that object positions differ from the held-out levels G_h. Consequently, the 100 'procedurally generated held-out grids' used in Figure 6 and Figure 22 share their wall-topology support with CEC's training distribution, while the five original layouts are not held out at the topology level either. To support the abstract's claim of generalization to 'many new problems', the evaluation should include wall structures that were never used as base predicates during training, for example new wall topologies or a deliberately held-out set of topologies.
  2. [Section 6, Q1 / Figure 6] The headline comparison between environment diversity and partner diversity is confounded. CEC trains on the same five wall structures on which the single-task baselines are evaluated, whereas FCP, E3T, and IPPO are trained on only one layout. The cross-play gap observed in Figure 6 may therefore partly reflect training-set coverage of the test layouts rather than a general advantage of environment diversity over partner diversity. A cleaner test would train a population-based method on the same environment distribution, or hold out entire wall structures from CEC, and compare methods with matched environment coverage.
  3. [Section 5, Evaluation Protocol / Section 6] The statement that 'we hold out those five layouts from the CEC generator' is inaccurate as written. Section A.1 says the generator samples each of the five original layouts as the base predicate uniformly at random, and the resampling condition only checks that goal, pot, plate, and onion positions differ from held-out levels. This internal inconsistency should be corrected, and the experiments should be revised so that the '5 Heldout Grids' panel is not read as evidence of generalization to never-seen wall topologies.
minor comments (7)
  1. [Abstract] The abstract says CEC 'outperforms competitive baselines quantitatively and qualitatively when collaborating with real people', but Figure 9 shows CEC's task score is lower than E3T's and only the subjective ratings favor CEC; the wording should be adjusted to match the reported results.
  2. [Section 6, Q1 heading] The heading 'Is increasing environment diversity more effective than increasing partner diversity of ZSC?' should read 'for ZSC'.
  3. [Section 4] The phrase 'single level ad-hoc single level ad-hoc collaboration performance' contains a duplicated phrase and should be rewritten.
  4. [Section A.3] The sentence about methods being 'swerve trained on' appears to contain a typo and should read 'methods were trained on'.
  5. [Section A.6] The paragraph begins with a sentence fragment: 'From.' should be removed or completed.
  6. [Section A.9] The limitations paragraph lists two open questions in one sentence; numbering them would improve readability, and the fact that CEC has not converged should be acknowledged in the main text when interpreting the quantitative comparisons.
  7. [Figure 6] The label '5 Heldout Grids' is misleading given the generator's base-topology leakage; the caption should specify exactly what was held out (object placements) and distinguish it from a true held-out-topology evaluation.

Circularity Check

1 steps flagged · score 1.0 of 10

Mild self-definitional overlap: the eval 'novel environments' are sampled from the same procedural generator, and therefore the same five wall structures, used in CEC training; the partner-generalization result remains independent.

  1. self definitional [Section 4 (Procedurally Generated Overcooked), Algorithm 1, Section 5 (Evaluation Protocol)]
    "we uniformly sample the wall structure from each of these five layouts, then randomly generate features like goals, plates, pots, and onions within the grid. [...] Sample Lbase∼U(L){Discrete uniform distribution} [...] we have the Overcooked procedural environment generator create 100 coordination challenges that neither the ST baselines nor any of the CEC agents have seen during training."

    CEC optimizes Eq. 2 by sampling training tasks m∼M from the procedural generator, and the held-out evaluation samples its 100 procedurally generated grids from the same generator. Algorithm 1 draws Lbase uniformly from the same five wall layouts used in training and rejects a grid only when object positions exactly match a held-out level, so wall topology is shared with training. Thus 'novel environment' is defined as fresh object placement on the training wall support, and the environment-novelty claim reduces to within-generator generalization; the large advantage over IPPO/FCP/E3T on those grids may partly reflect familiarity with the five wall structures rather than transfer to unseen topologies.

full rationale

This is an empirical RL paper, not a derivation, so the fitted-input and self-citation patterns do not apply: no parameter is fitted to the evaluation set and renamed as a prediction, and no load-bearing argument reduces to the authors' own prior claims. The central comparison of CEC against IPPO, FCP, and E3T is measured on independently trained baselines with matched compute, including separate cross-algorithm payoff matrices and human studies; those results stand on their own. The only substantial caveat is the one flagged above: Algorithm 1's Lbase∼U(L) sampling means the 100 'held-out' procedural grids share wall-topology support with CEC's training distribution, and the exclusion check only compares object positions, not layout structure. That weakens the strength of the 'novel environments' wording and is a distribution-leakage or evaluation-scope limitation, but it does not make the partner-generalization result equivalent to its inputs. Accordingly, a score of 1 reflects one mild self-definitional overlap in the environment-novelty claim without treating the paper's overall empirical contribution as circular.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to the target results; training hyperparameters are standard and fixed. The central claim rests on domain assumptions about task structure and the representativeness of procedural sampling, not on invented entities.

assumptions (4)
  • domain assumption All environments in the procedural generator share the same transition dynamics and reward function; only initial state distribution changes (Section 3).
    The mathematical setup restricts generalization to tasks drawn from a shared Markov Game structure, so it does not address coordination across genuinely different task mechanics.
  • domain assumption The procedural generator guarantees solvable coordination challenges by ensuring at least one reachable set of task-relevant items (Section A.1).
    Reachability is used as a proxy for solvability; actual two-agent solvability is not verified by planning or search.
  • domain assumption The five original Overcooked layouts are representative of real human-AI coordination challenges.
    The human study uses only Counter Circuit and Coordination Ring from these five, and the generator samples wall structures from all five.
  • domain assumption Different random seeds of the same algorithm simulate novel human partners in cross-play.
    This is standard ZSC methodology, but seed diversity may not capture the full space of human strategies.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination." pith.science (2026). https://pith.science/paper/SK5ZXHX4

@misc{pith2026250412714,
  author       = {Pith},
  title        = {Pith review of: Cross-environment Cooperation Enables Zero-shot Multi-agent Coordination},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SK5ZXHX4}},
  note         = {Machine review of arXiv:2504.12714}
}
read the original abstract

Zero-shot coordination (ZSC), the ability to adapt to a new partner in a cooperative task, is a critical component of human-compatible AI. While prior work has focused on training agents to cooperate on a single task, these specialized models do not generalize to new tasks, even if they are highly similar. Here, we study how reinforcement learning on a distribution of environments with a single partner enables learning general cooperative skills that support ZSC with many new partners on many new problems. We introduce two Jax-based, procedural generators that create billions of solvable coordination challenges. We develop a new paradigm called Cross-Environment Cooperation (CEC), and show that it outperforms competitive baselines quantitatively and qualitatively when collaborating with real people. Our findings suggest that learning to collaborate across many unique scenarios encourages agents to develop general norms, which prove effective for collaboration with different partners. Together, our results suggest a new route toward designing generalist cooperative agents capable of interacting with humans without requiring human data.

Figures

Figures reproduced from arXiv: 2504.12714 by the authors.

Figure 1
Figure 1. Overview of learning general coordination through Cross-environment Cooperation (CEC). By training agents in self-play on a large distribution of environments, we find that agents develop the ability to coordinate with novel partners and novel problems, contrasting with prior work which suggests self-play is insufficient for learning general norms for cooperation. on two main sources of variation during training: pa… view at source ↗
Figure 2
Figure 2. The Dual Destination Problem. In the fixed task (a), players start in opposite squares and must enter different green squares from each other to receive a reward. In the procedurally generated variation (b), the initial positions of the green goal cells and agents are randomized. training procedure provides a continuous curriculum of di￾verse experience as the model is continuously rewarded for exploiting their oppo… view at source ↗
Figure 3
Figure 3. Evaluation of IPPO and FCP baselines on the Fixed and Procedurally generated versions of the Dual Destination problem (error bars show the standard error of the mean). CEC generalizes better in both cases (p < 0.001 for t-tests comparing CEC to both FCP and IPPO). opposing green grid cells. In the basic setup, these grid cells are equidistant from each other and the agents’ original starting position, as illustrated… view at source ↗
Figures from the paper (20 more)
Figure 4
Figure 4. Figure 4: Five original Overcooked layouts. Left to right: Asym￾metric Advantages, Coordination Ring, Counter Circuit, Cramped Room, Forced Coordination [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Sample from the billions of solvable, diverse Overcooked tasks created by our procedural environment generator [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]
Figure 6
Figure 6. Figure 6: Evaluation of baselines on (left) 5 original Overcooked layouts vs. (right) 100 procedurally generated held-out layouts (standard error bars). Single-task methods and PBT struggle in both settings, while CEC agents generalize effectively. Finetuning CEC on a single gri…
Figure 8
Figure 8. Figure 8: Empirical game-theoretic evaluation of cross-algorithm play on the (a) five original and (b) 100 procedurally generated Overcooked tasks. Arrows show the gradient of the replicator dynamic on the cross-algorithm meta-game. Vectors flow towards CEC and CEC-Finetune indi…
Figure 7
Figure 7. Figure 7: CEC SP Training Performance compared to XP Perfor￾mance on 5 held-out levels. Despite the distribution sampling each layout predicate at uniform, CEC gets better at different layouts at different rates as it consistently improves across all tasks. a. 5 Original Tasks b…
Figure 9
Figure 9. Figure 9: (Top) Average success rates of algorithms cooperating with ad-hoc human partners on Counter Circuit and Coordination Ring, with standard error bars. CEC outperforms PBT methods and approaches E3T’s performance, despite only training on diverse layouts. Using a 2-sided …
Figure 11
Figure 11. Figure 11: Average number of collisions between humans and AI partners on Counter Circuit and Coordination Ring, with standard error bars shown. CEC achieves the lowest average collision rate. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_11.png]
Figure 12
Figure 12. Figure 12: Five original Overcooked layouts. The crosses on each of the layouts indicate “unreachable” locations. When sampling layouts in the procedural generation process, we make sure to sample at least one of the plate piles, onion piles, pots, and goals on walls that are re…
Figure 13
Figure 13. Figure 13: In the partially observable setting, CEC ZSC performance on the Dual Destination problem replicates findings from the fully observable case, suggesting it has promise for other games with imperfect information and the need for dynamic conventions, such as Hanabi [PIT…
Figure 14
Figure 14. Figure 14: Overview of the Multi-Task Dual Destination problem. Agents are rewarded for going to either opposite pink squares or opposite green squares [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 16
Figure 16. Figure 16: CEC with E3T exhibits lower cross play performance in the ZSC setting than any other method on the original 5 Overcooked layouts, but exhibits better generality to the 100 held out procedurally generated layouts than CEC fine-tuned on one of the original 5 [PITH_FULL…
Figure 18
Figure 18. Figure 18: Combining CEC with a partner diversity method (E3T) leads to a training curve which achieves half of the reward as CEC on its own in the same amount of training time used NiceWebRL (Carvalho, 2025), a unified tool for evaluating single-Human, Human-AI, and Human-Human…
Figure 19
Figure 19. Figure 19: In 300 million timesteps, CEC with an LSTM con￾verged to the maximum reward on the Dual Destination problem [PITH_FULL_IMAGE:figures/full_fig_p018_19.png]
Figure 21
Figure 21. Figure 21: Heatmap comparing different algorithms playing each other in the single-task setting, averaged across the original 5 layouts. Brighter yellow regions indicate better XP performance. IPPO E3T FCP CEC CEC-Finetune Algorithm IPPO E3T FCP CEC CEC-Finetune Algorithm 0.00 0…
Figure 23
Figure 23. Figure 23: Comparison of model performance across each of the original 5 Overcooked layouts, with standard error bars shown. CEC or CEC-Finetune achieve the highest mean reward in 4 out of 5 layouts. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_23.png]
Figure 26
Figure 26. Figure 26: CEC Seed on Counter Circuit which can coordinate with IPPO seeds from Figures 24 and 25. By focusing it’s attention on task-relevant objects rather than arbitrary strategies, it’s able to effec￾tively cooperate with novel partners on novel problems. Adaptive Consisten…
Figure 27
Figure 27. Figure 27: Participant assessments of different models across 7 different metrics averaged across Counter Circuit and Coordination Ring. A.12. Human-AI Collaboration Success Per Layout We plot the success of different models playing with humans in our user study in Figures 30 an…
Figure 28
Figure 28. Figure 28: Participant assessments of different models across 7 different metrics for the experiment Counter Circuit. Adaptive Consistent Human-like In My Way Frustrating Enjoyed Coordination Question 0 1 2 3 4 Rating IPPO FCP IPPO FCP E3T CEC CEC-Finetune [PITH_FULL_IMAGE:figu…
Figure 29
Figure 29. Figure 29 [PITH_FULL_IMAGE:figures/full_fig_p022_29.png]
Figure 31
Figure 31. Figure 31: Success rates of different algorithms playing Coordina￾tion Ring with humans. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_31.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Unsupervised Partner Design Enables Robust Ad-hoc Teamwork

    cs.LG 2025-08 conditional novelty 6.0 of 10

    UPD trains cooperative agents by adaptively selecting randomly generated partners with high return variance, outperforming population-based and population-free baselines in foraging, cooking, and with humans.

Reference graph

Works this paper leans on

64 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    https://nicegui.io

    Nicegui --- nicegui.io. https://nicegui.io. [Accessed 26-09-2024]

    work page 2024

  2. [2]

    Atchley, P., Pannell, H., Wofford, K., Hopkins, M., and Atchley, R. A. Human and AI collaboration in the higher education environment: opportunities and concerns. Cognitive Research: Principles and Implications, 9 0 (1): 0 20, April 2024. ISSN 2365-7464. doi:10.1186/s41235-024-00547-9. URL https://doi.org/10.1186/s41235-024-00547-9

    work page doi:10.1186/s41235-024-00547-9 2024

  3. [3]

    I., Tegegn, E., Kalloniatis, T., Mahjoub, O., Macfarlane, M., Smit, A

    Bonnet, C., Luo, D., Byrne, D., Surana, S., Abramowitz, S., Duckworth, P., Coyette, V., Midgley, L. I., Tegegn, E., Kalloniatis, T., Mahjoub, O., Macfarlane, M., Smit, A. P., Grinsztajn, N., Boige, R., Waters, C. N., Mimouni, M. A., Sob, U. A. M., de Kock, R., Singh, S., Furelos-Blanco, D., Le, V., Pretorius, A., and Laterre, A. Jumanji: a diverse suite o...

    arXiv 2024

  4. [4]

    Effects of nonverbal communication on efficiency and robustness in human-robot teamwork

    Breazeal, C., Kidd, C., Thomaz, A., Hoffman, G., and Berlin, M. Effects of nonverbal communication on efficiency and robustness in human-robot teamwork. pp.\ 708 -- 713, 09 2005. ISBN 0-7803-8912-3. doi:10.1109/IROS.2005.1545011

    work page arXiv 2005

  5. [5]

    A Structured Prediction Approach for Generalization in Cooperative Multi-Agent Reinforcement Learning

    Carion, N., Synnaeve, G., Lazaric, A., and Usunier, N. A structured prediction approach for generalization in cooperative multi-agent reinforcement learning, 2019. URL https://arxiv.org/abs/1910.08809

    work page Pith review arXiv 2019

  6. [6]

    K., Griffiths, T

    Carroll, M., Shah, R., Ho, M. K., Griffiths, T. L., Seshia, S. A., Abbeel, P., and Dragan, A. On the utility of learning about humans for human-ai coordination, 2020. URL https://arxiv.org/abs/1910.05789

    arXiv 2020

  7. [7]

    Nicewebrl: a framework for comparing humans and ai across many domains, 2025

    Carvalho, w. Nicewebrl: a framework for comparing humans and ai across many domains, 2025. URL https://github.com/wcarvalho/nicewebrl

    work page 2025

  8. [8]

    The theory of social functions: challenges for computational social science and multi-agent learning

    Castelfranchi, C. The theory of social functions: challenges for computational social science and multi-agent learning. Cognitive Systems Research, 2 0 (1): 0 5--38, 2001. ISSN 1389-0417. doi:https://doi.org/10.1016/S1389-0417(01)00013-4. URL https://www.sciencedirect.com/science/article/pii/S1389041701000134

    doi:10.1016/s1389-0417(01)00013-4 2001

Show all 64 references
  1. [9]

    Quantifying agent interaction in multi-agent reinforcement learning for cost-efficient generalization, 2023

    Chen, Y., Tang, C., Tian, R., Li, C., Li, J., Tomizuka, M., and Zhan, W. Quantifying agent interaction in multi-agent reinforcement learning for cost-efficient generalization, 2023. URL https://arxiv.org/abs/2310.07218

    2023 arXiv

  2. [10]

    Quantifying generalization in reinforcement learning

    Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J. Quantifying generalization in reinforcement learning. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning R...

    2019

  3. [11]

    Leveraging procedural generation to benchmark reinforcement learning, 2020

    Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Leveraging procedural generation to benchmark reinforcement learning, 2020. URL https://arxiv.org/abs/1912.01588

    2020 arXiv

  4. [12]

    S., Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P

    de Witt, C. S., Gupta, T., Makoviichuk, D., Makoviychuk, V., Torr, P. H. S., Sun, M., and Whiteson, S. Is independent learning all you need in the starcraft multi-agent challenge? CoRR, abs/2011.09533, 2020. URL https://arxiv.org/abs/2011.09533

    2011 arXiv

  5. [13]

    Emergent complexity and zero-shot transfer via unsupervised environment design, 2021

    Dennis, M., Jaques, N., Vinitsky, E., Bayen, A., Russell, S., Critch, A., and Levine, S. Emergent complexity and zero-shot transfer via unsupervised environment design, 2021. URL https://arxiv.org/abs/2012.02096

    2021 arXiv

  6. [14]

    Multi-agent reinforcement learning for autonomous vehicles: a survey

    Dinneweth, J., Boubezoul, A., Mandiau, R., and Espié, S. Multi-agent reinforcement learning for autonomous vehicles: a survey. Autonomous Intelligent Systems, 2 0 (1): 0 27, November 2022. ISSN 2730-616X. doi:10.1007/s43684-022-00045-z. URL https://doi.org/10.1007/s43684-022-00045-z

    2022 doi

  7. [15]

    C., Hsu, Y.-C., Zhang, Y., Tjanaka, B., and Nikolaidis, S

    Fontaine, M. C., Hsu, Y.-C., Zhang, Y., Tjanaka, B., and Nikolaidis, S. On the importance of environments in human-robot coordination, 2021. URL https://arxiv.org/abs/2106.10853

    2021 arXiv

  8. [16]

    S., Jonban, M

    Ghazimirsaeid, S. S., Jonban, M. S., Mudiyanselage, M. W., Marzband, M., Martinez, J. L. R., and Abusorrah, A. Multi-agent-based energy management of multiple grid-connected green buildings. Journal of Building Engineering, 74: 0 106866, 2023. ISSN 2352-7102. doi:https://doi.o...

    2023

  9. [17]

    V., Wiest, O., and Zhang, X

    Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., and Zhang, X. Large language model based multi-agents: A survey of progress and challenges, 2024. URL https://arxiv.org/abs/2402.01680

    2024 arXiv

  10. [18]

    J., and Norenzayan, A

    Henrich, J., Heine, S. J., and Norenzayan, A. The weirdest people in the world? Behavioral and Brain Sciences, 33 0 (2–3): 0 61–83, 2010. doi:10.1017/S0140525X0999152X

    2010 doi

  11. [19]

    O ther-play

    Hu, H., Lerer, A., Peysakhovich, A., and Foerster, J. “ O ther-play” for zero-shot coordination. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.\ 4399--4410. P...

  12. [20]

    Evolutionary robotics and the radical envelope-of-noise hypothesis

    Jakobi, N. Evolutionary robotics and the radical envelope-of-noise hypothesis. Adaptive Behavior, 6 0 (2): 0 325--368, 1997. doi:10.1177/105971239700600205. URL https://doi.org/10.1177/105971239700600205

    1997 doi

  13. [21]

    K., Austerweil, J

    Kleiman-Weiner, M., Ho, M. K., Austerweil, J. L., Michael L, L., and Tenenbaum, J. B. Coordinate to cooperate or compete: abstract goals and joint intentions in social interaction. In Proceedings of the 38th Annual Conference of the Cognitive Science Society, 2016

    2016

  14. [22]

    Generalization through diversity: Improving unsupervised environment design

    Li, W., Varakantham, P., and Li, D. Generalization through diversity: Improving unsupervised environment design. In Elkind, E. (ed.), Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23 , pp.\ 5411--5419. International Joint Con...

    2023 doi

  15. [23]

    S., and Jaques, N

    Liang, Y., Chen, D., Gupta, A., Du, S. S., and Jaques, N. Learning to cooperate with humans using generative agents. arXiv preprint arXiv:2411.13934, 2024

    2024 arXiv

  16. [24]

    A technique for the measurement of attitudes

    Likert, R. A technique for the measurement of attitudes. Archives of Psychology, 140: 0 1--55, 1932

    1932

  17. [25]

    Multi-agent actor-critic for mixed cooperative-competitive environments

    Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I. Multi-agent actor-critic for mixed cooperative-competitive environments. CoRR, abs/1706.02275, 2017. URL http://arxiv.org/abs/1706.02275

    2017 arXiv

  18. [26]

    JaxLife : An Open - Ended Agentic Simulator , volume ALIFE 2024: Proceedings of the 2024 Artificial Life Conference of Artificial Life Conference Proceedings

    Lu, C., Beukman, M., Matthews, M., and Foerster, J. JaxLife : An Open - Ended Agentic Simulator , volume ALIFE 2024: Proceedings of the 2024 Artificial Life Conference of Artificial Life Conference Proceedings . July 2024. doi:10.1162/isal_a_00770. URL https://doi.org/10.1162/...

    2024 doi

  19. [27]

    Ma, M., Liu, J., Sokota, S., Kleiman-Weiner, M., and Foerster, J. N. Learning intuitive policies using action features. In International Conference on Machine Learning, pp.\ 23358--23372. PMLR, 2023

    2023

  20. [28]

    Craftax: A lightning-fast benchmark for open-ended reinforcement learning, 2024

    Matthews, M., Beukman, M., Ellis, B., Samvelyan, M., Jackson, M., Coward, S., and Foerster, J. Craftax: A lightning-fast benchmark for open-ended reinforcement learning, 2024. URL https://arxiv.org/abs/2402.16801

    2024 arXiv

  21. [29]

    R., Leibo, J

    McKee, K. R., Leibo, J. Z., Beattie, C., and Everett, R. Quantifying the effects of environment and population diversity in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems, 36 0 (1): 0 21, March 2022. ISSN 1573-7454. doi:10.1007/s10458-022-09548-8...

    2022 doi

  22. [30]

    Stabilizing unsupervised environment design with a learned adversary

    Mediratta, I., Jiang, M., Parker-Holder, J., Dennis, M., Vinitsky, E., and Rockt\"aschel, T. Stabilizing unsupervised environment design with a learned adversary. In Chandar, S., Pascanu, R., Sedghi, H., and Precup, D. (eds.), Proceedings of The 2nd Conference on Lifelong Lear...

    2023

  23. [31]

    Learning to assist humans without inferring rewards, 2025

    Myers, V., Ellis, E., Levine, S., Eysenbach, B., and Dragan, A. Learning to assist humans without inferring rewards, 2025. URL https://arxiv.org/abs/2411.02623

    2025 arXiv

  24. [32]

    Xland-minigrid: Scalable meta-reinforcement learning environments in jax, 2024

    Nikulin, A., Kurenkov, V., Zisman, I., Agarkov, A., Sinii, V., and Kolesnikov, S. Xland-minigrid: Scalable meta-reinforcement learning environments in jax, 2024. URL https://arxiv.org/abs/2312.12044

    2024 arXiv

  25. [33]

    Personalizing reinforcement learning from human feedback with variational preference learning, 2024

    Poddar, S., Wan, Y., Ivison, H., Gupta, A., and Jaques, N. Personalizing reinforcement learning from human feedback with variational preference learning, 2024. URL https://arxiv.org/abs/2408.10075

    2024 arXiv

  26. [34]

    A., and Botvinick, M

    Rabinowitz, N., Perbet, F., Song, F., Zhang, C., Eslami, S. A., and Botvinick, M. Machine theory of mind. In International conference on machine learning, pp.\ 4218--4227. PMLR, 2018

    2018

  27. [35]

    S., Lopes, G., and Ribeiro, A

    Ribeiro, T., Gon c alves, F., Garcia, I. S., Lopes, G., and Ribeiro, A. F. Charmie: A collaborative healthcare and home service and assistant robot for elderly care. Applied Sciences, 11 0 (16): 0 7248, 2021

    2021

  28. [36]

    Multi-agent systems in epidemiology: a first step for computational biology in the study of vector-borne disease transmission

    Roche, B., Guégan, J.-F., and Bousquet, F. Multi-agent systems in epidemiology: a first step for computational biology in the study of vector-borne disease transmission. BMC Bioinformatics, 9 0 (1): 0 435, October 2008. ISSN 1471-2105. doi:10.1186/1471-2105-9-435. URL https://...

    2008 doi

  29. [37]

    The overcooked generalisation challenge

    Ruhdorfer, C., Bortoletto, M., Penzkofer, A., and Bulling, A. The overcooked generalisation challenge. 2024. URL https://arxiv.org/abs/2406.17949

    2024 arXiv

  30. [38]

    S., Souly, A., Bandyopadhyay, S., Samvelyan, M., Jiang, M., Lange, R

    Rutherford, A., Ellis, B., Gallici, M., Cook, J., Lupu, A., Ingvarsson, G., Willi, T., Khan, A., de Witt, C. S., Souly, A., Bandyopadhyay, S., Samvelyan, M., Jiang, M., Lange, R. T., Whiteson, S., Lacerda, B., Hawes, N., Rocktaschel, T., Lu, C., and Foerster, J. N. Jaxmarl: Mu...

    2023 arXiv

  31. [39]

    and Levine, S

    Sadeghi, F. and Levine, S. (cad) \^ 2 rl: Real single-image flight without a single real image. CoRR, abs/1611.04201, 2016. URL http://arxiv.org/abs/1611.04201

    2016 arXiv

  32. [41]

    Maestro: Open-ended environment design for multi-agent reinforcement learning, 2023 b

    Samvelyan, M., Khan, A., Dennis, M., Jiang, M., Parker-Holder, J., Foerster, J., Raileanu, R., and Rocktäschel, T. Maestro: Open-ended environment design for multi-agent reinforcement learning, 2023 b . URL https://arxiv.org/abs/2303.03376

    2023 arXiv

  33. [42]

    Multi-agent diagnostics for robustness via illuminated diversity

    Samvelyan, M., Paglieri, D., Jiang, M., Parker-Holder, J., and Rockt \"a schel, T. Multi-agent diagnostics for robustness via illuminated diversity. arXiv preprint arXiv:2401.13460, 2024

    2024 arXiv

  34. [43]

    Diverse conventions for human-ai collaboration, 2023

    Sarkar, B., Shih, A., and Sadigh, D. Diverse conventions for human-ai collaboration, 2023. URL https://arxiv.org/abs/2310.15414

    2023 arXiv

  35. [44]

    C., and Tenenbaum, J

    Serrino*, J., Kleiman-Weiner*, M., Parkes, D. C., and Tenenbaum, J. B. Finding friend and foe in multi-agent games. In Advances in Neural Information Processing Systems, volume 32, 2019

    2019

  36. [45]

    Sheridan, T. B. Human–robot interaction: Status and challenges. Human Factors, 58 0 (4): 0 525--532, 2016. doi:10.1177/0018720816644364. URL https://doi.org/10.1177/0018720816644364. PMID: 27098262

    2016 doi

  37. [46]

    L., and Tenenbaum, J

    Shum*, M., Kleiman-Weiner*, M., Littman, M. L., and Tenenbaum, J. B. Theory of minds: Understanding behavior in groups through inverse planning. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp.\ 6163--6170, 2019

    2019

  38. [47]

    Mastering the game of Go without human knowledge

    Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., and Hassabis, D. Mastering the game of Go without human knowledge. Nat...

    2017 doi

  39. [48]

    Ad hoc autonomous agent teams: Collaboration without pre-coordination

    Stone, P., Kaminka, G., Kraus, S., and Rosenschein, J. Ad hoc autonomous agent teams: Collaboration without pre-coordination. Proceedings of the AAAI Conference on Artificial Intelligence, 24 0 (1): 0 1504--1509, Jul. 2010 a . doi:10.1609/aaai.v24i1.7529. URL https://ojs.aaai....

    2010 doi

  40. [49]

    A., Kraus, S., and Rosenschein, J

    Stone, P., Kaminka, G. A., Kraus, S., and Rosenschein, J. S. Ad hoc autonomous agent teams: collaboration without pre-coordination. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI'10, pp.\ 1504–1509. AAAI Press, 2010 b

    2010

  41. [50]

    Learning to share and hide intentions using information regularization

    Strouse, D., Kleiman-Weiner, M., Tenenbaum, J., Botvinick, M., and Schwab, D. Learning to share and hide intentions using information regularization. In Advances in Neural Information Processing Systems, volume 31, 2018

    2018

  42. [51]

    R., Botvinick, M., Hughes, E., and Everett, R

    Strouse, D., McKee, K. R., Botvinick, M., Hughes, E., and Everett, R. Collaborating with humans without human data, 2022. URL https://arxiv.org/abs/2110.08176

    2022 arXiv

  43. [52]

    Domain randomization for transferring deep neural networks from simulation to the real world, 2017

    Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P. Domain randomization for transferring deep neural networks from simulation to the real world, 2017. URL https://arxiv.org/abs/1703.06907

    2017 arXiv

  44. [53]

    The cultural origins of human cognition

    Tomasello, M. The cultural origins of human cognition. The cultural origins of human cognition. Harvard University Press, Cambridge, MA, US, 1999. ISBN 0-674-00070-6 (Hardcover). Pages: vi, 248

    1999

  45. [54]

    Z., and Graepel, T

    Tuyls, K., P \' e rolat, J., Lanctot, M., Leibo, J. Z., and Graepel, T. A generalised method for empirical game theoretic analysis. CoRR, abs/1803.06376, 2018. URL http://arxiv.org/abs/1803.06376

    2018 arXiv

  46. [55]

    M., Mathieu, M., Dudzik, A., Chung, J., Choi, D

    Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J. P., Jaderberg, M., Vezhnevets, A. S., Leblond, R., Po...

    2019

  47. [56]

    X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J

    Wang, J. X., Kurth-Nelson, Z., Kumaran, D., Tirumala, D., Soyer, H., Leibo, J. Z., Hassabis, D., and Botvinick, M. Prefrontal cortex as a meta-reinforcement learning system. Nature Neuroscience, 21 0 (6): 0 860--868, June 2018. ISSN 1546-1726. doi:10.1038/s41593-018-0147-8. UR...

    2018 doi

  48. [57]

    Quantifying zero-shot coordination capability with behavior preferring partners, 2024

    Wang, X., Zhang, S., Zhang, W., Dong, W., Chen, J., Wen, Y., and Zhang, W. Quantifying zero-shot coordination capability with behavior preferring partners, 2024. URL https://openreview.net/forum?id=wTRpjTO3F7

    2024

  49. [58]

    P., Tuyls, K., and Greenwald, A

    Wellman, M. P., Tuyls, K., and Greenwald, A. Empirical game-theoretic analysis: A survey. arXiv preprint arXiv:2403.04018, 2024

    2024 arXiv

  50. [59]

    A., Wang*, R

    Wu*, S. A., Wang*, R. E., Evans, J. A., Tenenbaum, J. B., Parkes, D. C., and Kleiman-Weiner, M. Too many cooks: Bayesian inference for coordinating multi-agent collaboration. Topics in Cognitive Science, 13 0 (2): 0 414--432, 2021

    2021

  51. [60]

    Dota: A large-scale dataset for object detection in aerial images

    Xia, G.-S., Bai, X., Ding, J., Zhu, Z., Belongie, S., Luo, J., Datcu, M., Pelillo, M., and Zhang, L. Dota: A large-scale dataset for object detection in aerial images. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 3974--3983, 2018. doi:10.1109/CV...

    2018

  52. [61]

    An efficient end-to-end training approach for zero-shot human- AI coordination

    Yan, X., Guo, J., Lou, X., Wang, J., Zhang, H., and Du, Y. An efficient end-to-end training approach for zero-shot human- AI coordination. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=6ePsuwXUwf

    2023

  53. [62]

    B., Torralba, A., and Shu, T

    Ying, L., Jha, K., Aarya, S., Tenenbaum, J. B., Torralba, A., and Shu, T. Goma: Proactive embodied cooperative communication via goal-oriented mental alignment, 2024. URL https://arxiv.org/abs/2403.11075

    2024 arXiv

  54. [64]

    Maximum entropy population-based training for zero-shot human-ai coordination, 2022

    Zhao, R., Song, J., Yuan, Y., Haifeng, H., Gao, Y., Wu, Y., Sun, Z., and Wei, Y. Maximum entropy population-based training for zero-shot human-ai coordination, 2022. URL https://arxiv.org/abs/2112.11701

    2022 arXiv

  55. [65]

    Posterior sampling for multi-agent reinforcement learning: solving extensive games with imperfect information

    Zhou, Y., Li, J., and Zhu, J. Posterior sampling for multi-agent reinforcement learning: solving extensive games with imperfect information. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=Syg-ET4FPS

    2020

  56. [66]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.