Pith. sign in

REVIEW 4 major objections 5 minor 37 references

ColorGrid: A Multi-Agent Non-Stationary Environment for Goal Inference and Assistance

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Standard cooperative MARL, IPPO, fails to learn a partner's changing hidden goal in the new ColorGrid benchmark.

desk verdict Useful new benchmark, but the 'IPPO cannot solve it' claim rests on a confounded leader comparison and a single seed. read the letter →

arxiv 2501.10593 v1 pith:AVQ6UYNY submitted 2025-01-17 cs.AI cs.LG

Andrey Risukhin , Kavel Rao , Ben Caffee , Alan Fan This is my paper · ORCID
classification cs.AIcs.LG
keywords multi-agentreinforcementlearningnon-stationaryenvironmentgoalinferencehiddengoalsasymmetricinformationIPPObenchmarksocial
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces ColorGrid, a two-agent grid-world benchmark in which one agent (the leader) knows the current goal block color while the other (the follower) must infer it from the leader's movements, and the goal color switches over time. The paper's central claim is that IPPO, a standard cooperative multi-agent reinforcement learning algorithm, cannot solve the asymmetric, non-stationary setting: with the goal hidden from the follower, trained followers fall behind a trivial baseline that simply copies the last color the leader collected. This matters because real human-AI collaboration often requires an assistant to infer a changing, unspoken human goal, and the paper argues that current state-of-the-art MARL methods lack this capability. The paper also reports that the cost of exploration—whether random block collection has positive, neutral, or negative expected reward—strongly shapes what the follower learns, with only the neutral case producing partial goal-following behavior.

What carries the argument

The load-bearing mechanism is the combination of three design choices in ColorGrid: (1) a non-stationary goal, where the goal block color switches with probability $2/3 \cdot 1/32 \approx 2.08\%$ per time step; (2) asymmetric information, where the leader sees the goal one-hot vector but the follower receives zeros in its place; and (3) a tunable cost of exploration, set by choosing the goal reward ($+1$, $+2$, or $+4$) against a fixed $-1$ penalty for incorrect blocks. These components force the follower to infer the leader's intent from trajectory observations alone. The paper pairs the environment with an IPPO implementation that includes an LSTM for the asymmetric follower, an auxiliary cross-entropy loss for predicting the goal color, and penalty annealing that ramps the incorrect-block penalty from $0$ to $1$ between 4M and 10M training steps.

What would settle it

Train an IPPO follower in the asymmetric, neutral-expected-reward ColorGrid setting across at least ten seeds using the paper's reported hyperparameters, and compare mean final reward to the A* copying follower baseline averaged over 100 seeds; if any seed or the mean reaches or exceeds the A* follower's reward, the claim that ColorGrid is unsolved by IPPO is falsified.

Watch

Extended reading notes

Core claim

In ColorGrid's asymmetric mode, the leader always knows the goal color and receives a $+1$ reward for collecting a matching block and $-1$ for others, while the follower sees only the board and the leader's trajectory; there is no message channel. Training a follower with IPPO against a frozen IPPO-trained leader yields poor behavior across all reward structures tested: with positive expected value the follower collects every block, with negative expected value it collects none, and only with neutral expected value does it sometimes pursue the correct color. Averaged over three seeds in the symmetric setting, IPPO converges to positive reward only when the incorrect-block penalty is annealed in over training and when an auxiliary supervised goal-prediction loss is used. In the asymmetric setting, the A* copying follower—which routes to the last color picked up by the leader—significantly outperforms all IPPO-trained followers, and the paper concludes that ColorGrid with non-stationary and asymmetric goals is currently unsolved by IPPO.

Load-bearing premise

The conclusion that IPPO cannot solve ColorGrid rests on a few training runs: the headline comparisons in Figure 3 use a single seed, and Table 2 averages only three seeds with one fixed hyperparameter set; if those runs are unrepresentative, the unsolved claim could fail.

Editorial extensions

If this is right

  • If the paper's claim is right, state-of-the-art independent PPO cannot learn to track a changing hidden goal from a partner's trajectory, so MARL benchmarks for human-AI assistance must include non-stationary and asymmetric objectives to expose this gap.
  • The cost of exploration becomes a first-order design lever: a pessimistic reward setting makes IPPO followers collapse to inaction, which mirrors high-stakes human-robot settings where conservative assistants may be harmless but unhelpful.
  • Penalty annealing and a supervised goal-prediction auxiliary loss are necessary for IPPO to learn even the symmetric version of ColorGrid with sparse rewards, so these ingredients should be part of any fair comparison on this benchmark.
  • The A* copying baseline provides an upper reference point for the inference problem: the information needed to act correctly is present in the leader's trajectory, so the failure is one of learning, not of observability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension of the paper's finding is that an explicit belief-tracking module—for example an online inverse-RL head that maintains a posterior over the leader's goal—could close the gap, since the leader's trajectory provably contains the relevant information (the copy-the-leader baseline succeeds with it).
  • Because the paper's own switch-probability ablation shows difficulty persists even when the goal never switches, the IPPO failure is likely driven more by sparse rewards and the hidden-goal credit-assignment problem than by non-stationarity itself; a testable prediction is that varying block sparsity alone will reproduce the failure.
  • The paper's cost-of-exploration results suggest a transferable design principle for human-AI assistance: when wrong actions are costly, assistants trained with independent PPO become harmlessly inactive, so shaping or an explicit inference signal is needed to make them useful rather than merely safe.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ColorGrid, a PettingZoo-based 32x32 grid-world MARL environment in which a leader and a follower collect colored blocks, the goal color can switch during an episode, and the follower may be denied access to the goal color. The authors release the environment code, model checkpoints, and trajectory visualizations. They evaluate Independent PPO (IPPO) under symmetric and asymmetric goal information, with ablations over reward magnitude, penalty annealing, an auxiliary goal-prediction loss, goal concatenation location, distance and potential-field reward shaping, and goal-switch probability. Their central empirical claim is that the asymmetric setting with non-stationary hidden goals is currently unsolved by IPPO, and that an A* 'copying' follower substantially outperforms all trained IPPO followers. The paper presents this as a benchmark motivation for future MARL goal-inference algorithms.

Significance. If the central negative result is reliable, the paper would provide a useful benchmark artifact for the under-studied problem of real-time hidden-goal inference in cooperative MARL. The strongest parts of the submission are concrete: the environment is implemented and released, the state representation and reward structure are clearly specified, the neural architecture and training details are described in enough detail to reproduce, and several ablations (penalty annealing, auxiliary loss, goal concatenation position) go beyond a single environment demonstration. However, the load-bearing claim that IPPO is insufficient for goal inference rests on a small number of seeds and on an unmatched comparison between different leader policies; the manuscript itself acknowledges in Section 5.1 the need for 'running more experiments with different seeds and determining optimal hyperparameter configurations.' The benchmark contribution is sound as an environment release, but the paper's main negative result needs stronger statistical support and a deconfounded experimental design before it can support the stated conclusions.

major comments (4)
  1. [§4.2, Figure 3] The central evidence that the IPPO follower fails to infer the leader's goal is reported for a single seed. The Figure 3 caption states 'We use seed 0 for these comparisons,' while the A* baseline scores are averaged over 100 seeds. Without error bars, confidence intervals, or multiple seeds for the IPPO curves, the claim that IPPO is 'insufficient to learn to infer the leader's goal' cannot be distinguished from a seed-dependent training outcome. This is load-bearing because the abstract and Section 5 both present the negative result as the paper's main empirical finding.
  2. [§4.2, Figure 3] The comparison between the trained IPPO follower and the A* copying follower confounds follower inference ability with leader expressiveness. The IPPO follower is trained against a frozen IPPO leader, while the A* copying baseline is paired with an A* leader. The text notes that the two leaders have similar cumulative reward, but similar reward does not imply similar trajectory statistics; the IPPO leader may switch goals more slowly, wander, or take non-shortest paths, making the hidden goal harder or easier to infer. The claim 'IPPO is insufficient to learn to infer the leader's goal' requires holding the leader policy fixed across follower conditions, or at least controlling the information content of the leader's trajectories. As written, the reported gap conflates the follower's inference capability with the leader policy used to generate the demonstrations.
  3. [§4, Table 2; Appendix A.2] The quantitative support for the effect of penalty annealing and the auxiliary loss is based on Table 2, which averages converged rewards over only 3 seeds and reports no variance, confidence intervals, or per-seed values. Since the paper's broad claim is that ColorGrid is 'unsolved by IPPO,' the absence of any uncertainty quantification is a material gap: with three seeds and no error bars, the reader cannot assess whether the differences between the four rows (e.g., 48.8 vs. 32.8) are real effects or training noise. The paper should either provide many more seeds with variance reporting or substantially soften the scope of the claim.
  4. [§4, §5.1] The 'unsolved by IPPO' conclusion is stated despite the use of a single fixed hyperparameter configuration (Appendix A.2) and no systematic hyperparameter search. Section 5.1 explicitly lists 'determining optimal hyperparameter configurations' as future work, which is appropriate, but the abstract and Section 4.2 nevertheless assert that the environment is 'currently unsolved by IPPO.' That assertion is stronger than the evidence supports: without a hyperparameter sensitivity analysis or at least a demonstration that reasonable variations do not change the outcome, the negative result could be an artifact of the chosen learning rate, entropy coefficient, network size, or other fixed settings. The authors should either provide such a sensitivity analysis or rephrase the claim as 'unsolved under the tested default configuration.'
minor comments (5)
  1. [§3.1] In the paragraph on customizability, the sentence 'The is also customizable' is incomplete and should be finished or removed.
  2. [§3.1] The text says 'see 3.1 for the computation explanation' of the goal-switch probability, but the derivation appears in Appendix A.3; the cross-reference should point to the appendix.
  3. [§3.1, Appendix A.3] The default goal-switch probability is given as 2.08% in the main text and Appendix A.3, but Section 4 states 'goal switch probability of 2%' when describing the default settings; these numbers should be made consistent.
  4. [§3.2.1, Eq. (1)] Equation (1) defines c_i but does not define the predicted probability \hat c_i; please add a sentence clarifying that \hat c_i is the auxiliary network's softmax output for color i.
  5. [§4.1, footnote 3] The statement that A* with an admissible heuristic 'is guaranteed to perform optimally' is imprecise for graph search unless the heuristic is also consistent; for this grid setting the intended meaning is clear, but the wording should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular dependency: the paper's negative result is an empirical benchmark finding, not a derivation from its inputs.

full rationale

ColorGrid does not claim to derive a prediction from first principles; it reports an empirical negative result (IPPO fails on the asymmetric non-stationary task) and releases a benchmark environment. The environment's reward values, switch probability, penalty annealing, and shaping terms are design choices, not outputs of a derivation, so tuning them until IPPO struggles is benchmark construction rather than circular reasoning. The auxiliary goal-prediction loss (Eq. 1) uses the true goal label during training, but the paper does not claim this label is absent from training; the claim is that the follower receives no explicit goal at execution, which is consistent with the architecture. The A* copying follower baseline is a hand-coded heuristic evaluated against a different leader than the IPPO follower, which is a potential experimental confound in the comparison, but it is not a case where the conclusion is equivalent to the input by construction. No load-bearing self-citations or imported uniqueness theorems appear: the cited architecture from Ndousse et al. is prior external work, and the central claims do not rest on it as a proof. The stated limitations (few seeds, fixed hyperparameters) weaken external validity but do not make the argument circular. Therefore no circular step meets the standard of quote-plus-reduction.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The paper introduces an environment, not a physical entity, so the invented-entities ledger is empty. The free parameters are environment configuration and training choices that shape the negative result. The axioms are the modeling assumptions underlying the benchmark and the IPPO evaluation.

free parameters (7)
  • goal switch probability = 0.02 (2/3 * 1/32)
    Default environment parameter; defines non-stationarity and is hand-chosen so the follower has enough time to reach a new goal. The '2/3' and '1/32' are justifications, not fitted values.
  • reward values for positive/neutral/negative EV cases = +4/-1, +2/-1, +1/-1 for goal/incorrect blocks
    Chosen to make random block collection have positive, zero, or negative expected value; drives the cost-of-exploration finding.
  • block density = 10% of grid cells
    Default block coverage; sparsity affects whether sparse rewards cause collapse.
  • penalty annealing schedule = penalty coefficient linear from 0 to 1 between 4M and 10M timesteps
    Hand-designed to let agents explore early; without it IPPO collapses to inaction, so the central training protocol depends on it.
  • auxiliary loss coefficient kappa = 0.2 (0.4 ablated)
    Weight of the goal-prediction cross-entropy term; tuned after observing collapse, affects stability.
  • distance reward shaping = threshold=10, penalty 0.25 or 0.5, for 20M-40M timesteps
    Used in warmstarting experiments to prevent leader from stealing blocks; part of the asymmetric training recipe.
  • IPPO hyperparameters = LR 1e-4, rollout 128, gamma 0.99, GAE 0.95, clip 0.2, entropy 0.01, etc.
    Fixed as 'standard' without schedule; the unsolved conclusion is conditional on this set.
assumptions (6)
  • domain assumption The interaction is modeled as a (partially observable) Markov decision process and IPPO with actor-critic networks is an appropriate algorithm to evaluate.
    The paper frames ColorGrid as an MARL benchmark and immediately evaluates with IPPO; if the task is better solved by model-based or off-policy methods, the negative result is less meaningful.
  • domain assumption Blocks respawn uniformly at random in empty cells at constant density.
    Environment definition in Section 3.1; the benchmark's difficulty depends on this respawn rule.
  • domain assumption Goal color switches at each step with probability 0.02 independent of agent actions.
    Section 3.1 and Appendix A.3; this defines the non-stationarity and is not derived from data.
  • standard math A* with shortest-path cost is an admissible heuristic and thus optimal for the single-agent navigation subproblem.
    Footnote 3 in Section 4.1; used as baseline and as the copying follower.
  • domain assumption The follower receives no goal information in asymmetric mode (zeros concatenated), and the state representation with 5 channels plus one-hot is sufficient.
    Section 3.2.1; if some leak exists via the one-hot insertion, the hidden-goal claim changes.
  • domain assumption The auxiliary supervised loss can be trained with the true goal label during learning (cheating by labels).
    Eq. (1) uses ground-truth goal color as supervised label, so the follower gets goal information during training via the auxiliary task; this is a design choice that may make the benchmark easier, not harder.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ColorGrid: A Multi-Agent Non-Stationary Environment for Goal Inference and Assistance." pith.science (2026). https://pith.science/paper/AVQ6UYNY

@misc{pith2026250110593,
  author       = {Pith},
  title        = {Pith review of: ColorGrid: A Multi-Agent Non-Stationary Environment for Goal Inference and Assistance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AVQ6UYNY}},
  note         = {Machine review of arXiv:2501.10593}
}
read the original abstract

Autonomous agents' interactions with humans are increasingly focused on adapting to their changing preferences in order to improve assistance in real-world tasks. Effective agents must learn to accurately infer human goals, which are often hidden, to collaborate well. However, existing Multi-Agent Reinforcement Learning (MARL) environments lack the necessary attributes required to rigorously evaluate these agents' learning capabilities. To this end, we introduce ColorGrid, a novel MARL environment with customizable non-stationarity, asymmetry, and reward structure. We investigate the performance of Independent Proximal Policy Optimization (IPPO), a state-of-the-art (SOTA) MARL algorithm, in ColorGrid and find through extensive ablations that, particularly with simultaneous non-stationary and asymmetric goals between a ``leader'' agent representing a human and a ``follower'' assistant agent, ColorGrid is unsolved by IPPO. To support benchmarking future MARL algorithms, we release our environment code, model checkpoints, and trajectory visualizations at https://github.com/andreyrisukhin/ColorGrid.

Figures

Figures reproduced from arXiv: 2501.10593 by the authors.

Figure 3
Figure 3. Using a frozen expert leader trained with IPPO, we train a cold-started follower varying [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 25 canonical work pages

  1. [1]

    Basis for intentions: Efficient inverse reinforcement learning using past experience, 2022

    Marwa Abdulhai, Natasha Jaques, and Sergey Levine. Basis for intentions: Efficient inverse reinforcement learning using past experience, 2022

    work page 2022

  2. [2]

    Albrecht and Subramanian Ramamoorthy

    Stefano V . Albrecht and Subramanian Ramamoorthy. A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems, 2015

    work page 2015

  3. [3]

    Ho, Thomas L

    Micah Carroll, Rohin Shah, Mark K. Ho, Thomas L. Griffiths, Sanjit A. Seshia, Pieter Abbeel, and Anca D. Dragan. On the utility of learning about humans for human-ai coordination. CoRR, abs/1910.05789, 2019. URL http://arxiv.org/abs/1910.05789

    arXiv 1910

  4. [4]

    Christian Schroeder de Witt, Tarun Gupta, Denys Makoviichuk, Viktor Makoviychuk, Philip H. S. Torr, Mingfei Sun, and Shimon Whiteson. Is independent learning all you need in the starcraft multi-agent challenge?, 2020. URL https://arxiv.org/abs/2011.09533

    arXiv 2020

  5. [5]

    Emergent complexity and zero-shot transfer via unsupervised environment design, 2021

    Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent complexity and zero-shot transfer via unsupervised environment design, 2021

    work page 2021

  6. [6]

    Counterfactual multi-agent policy gradients, 2017

    Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients, 2017

    work page 2017

  7. [7]

    Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018

    Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018

    2018

  8. [8]

    The evolution of cultural evolution

    Joseph Henrich and Richard McElreath. The evolution of cultural evolution. Evolutionary Anthropology: Issues, News, and Reviews: Issues, News, and Reviews , 12(3):123–135, 2003

    work page 2003

Show all 37 references
  1. [9]

    Pablo Hernandez-Leal, Bilal Kartal, and Matthew E. Taylor. Agent modeling as auxiliary task for deep reinforcement learning, 2019

    2019

  2. [10]

    Reinforcement learning with unsupervised auxiliary tasks, 2016

    Max Jaderberg, V olodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks, 2016

    2016

  3. [11]

    Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garc´ıa Casta˜neda, Charlie Beattie, Neil C

    Max Jaderberg, Wojciech M. Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garc´ıa Casta˜neda, Charlie Beattie, Neil C. Rabinowitz, Ari S. Morcos, Avraham Ruderman, Nicolas Sonnerat, Tim Green, Louise Deason, Joel Z. Leibo, David Silver, Demis Hassabis, Koray Kavukcuo...

    2018

  4. [12]

    Recursive bayesian human intent recognition in shared-control robotics

    Siddarth Jain and Brenna Argall. Recursive bayesian human intent recognition in shared-control robotics. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 3905–3912, 2018. doi: 10.1109/IROS.2018.8593766

    2018

  5. [13]

    Losey, and Dorsa Sadigh

    Hong Jun Jeon, Dylan P. Losey, and Dorsa Sadigh. Shared autonomy with learned latent actions, 2020

    2020

  6. [14]

    Learning dynamics model in reinforcement learning by incorporating the long term future, 2019

    Nan Rosemary Ke, Amanpreet Singh, Ahmed Touati, Anirudh Goyal, Yoshua Bengio, Devi Parikh, and Dhruv Batra. Learning dynamics model in reinforcement learning by incorporating the long term future, 2019

    2019

  7. [15]

    Multi-agent reinforcement learning with multi- step generative models, 2019

    Orr Krupnik, Igor Mordatch, and Aviv Tamar. Multi-agent reinforcement learning with multi- step generative models, 2019

    2019

  8. [16]

    Social learning strategies

    Kevin N Laland. Social learning strategies. Animal Learning & Behavior , 32(1):4–14, 2004

    2004

  9. [17]

    Generalization and network design strategies

    Yann LeCun. Generalization and network design strategies. 1989. URL https://api. semanticscholar.org/CorpusID:59861896

    1989

  10. [18]

    Rectifier nonlinearities improve neural network acoustic models

    Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlinearities improve neural network acoustic models. In Proc. icml, volume 30, page 3. Atlanta, GA, 2013. 11

    2013

  11. [19]

    Emergence of grounded compositional language in multi- agent populations, 2018

    Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi- agent populations, 2018

    2018

  12. [20]

    Emergent social learning via multi-agent reinforcement learning, 2021

    Kamal Ndousse, Douglas Eck, Sergey Levine, and Natasha Jaques. Emergent social learning via multi-agent reinforcement learning, 2021

    2021

  13. [21]

    Srinivasa

    Stefanos Nikolaidis, David Hsu, and Siddhartha S. Srinivasa. Human-robot mutual adaptation in collaborative tasks: Models and experiments. The International Journal of Robotics Research , 36:618 – 634, 2017. URL https://api.semanticscholar.org/CorpusID:7274323

    2017

  14. [22]

    Albrecht

    Georgios Papoudakis, Filippos Christianos, Lukas Sch¨afer, and Stefano V . Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS) ,

  15. [23]

    Tenenbaum, Sanja Fidler, and Antonio Torralba

    Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B. Tenenbaum, Sanja Fidler, and Antonio Torralba. Watch-and-help: A challenge for social perception and human-ai collaboration, 2021

    2021

  16. [24]

    Modeling others using oneself in multi-agent reinforcement learning, 2018

    Roberta Raileanu, Emily Denton, Arthur Szlam, and Rob Fergus. Modeling others using oneself in multi-agent reinforcement learning, 2018

    2018

  17. [25]

    Proximal policy optimization algorithms, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017

    2017

  18. [26]

    Loss is its own reward: Self-supervision for reinforcement learning, 2017

    Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss is its own reward: Self-supervision for reinforcement learning, 2017

    2017

  19. [27]

    Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, and Chelsea Finn

    Lucy Xiaoyang Shi, Zheyuan Hu, Tony Z. Zhao, Archit Sharma, Karl Pertsch, Jianlan Luo, Sergey Levine, and Chelsea Finn. Yell at your robot: Improving on-the-fly from language corrections, 2024

    2024

  20. [28]

    Message-passing approach for threshold models of behavior in networks

    Munik Shrestha and Cristopher Moore. Message-passing approach for threshold models of behavior in networks. Physical Review E , 89(2), February 2014. ISSN 1550-2376. doi: 10.1103/physreve.89.022805. URL http://dx.doi.org/10.1103/PhysRevE.89.022805

    2014 doi

  21. [29]

    Mastering chess and shogi by self-play with a general reinforcement learning algorithm, 2017

    David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforce...

    2017

  22. [30]

    Smallwood and Edward J

    Richard D. Smallwood and Edward J. Sondik. The optimal control of partially observable markov processes over a finite horizon. Oper . Res., 21:1071–1088, 1973. URL https: //api.semanticscholar.org/CorpusID:43604344

    1973

  23. [31]

    McKee, Matt Botvinick, Edward Hughes, and Richard Everett

    DJ Strouse, Kevin R. McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collabo- rating with humans without human data, 2022

    2022

  24. [32]

    Pettingzoo: Gym for multi-agent reinforcement learning

    J Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente, et al. Pettingzoo: Gym for multi-agent reinforcement learning. Advances in Neural Information Processing Systems...

    2021

  25. [33]

    Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O. Stanley. Paired open-ended trailblazer (POET): endlessly generating increasingly complex and diverse learning environments and their solutions. CoRR, abs/1901.01753, 2019. URL http://arxiv.org/abs/1901.01753

    1901 arXiv

  26. [34]

    Th´eophane Weber, S ´ebastien Racani `ere, David P. Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adria Puigdom `enech Badia, Oriol Vinyals, Nicolas Heess, Yu- jia Li, Razvan Pascanu, Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. Imagination-...

    2018

  27. [35]

    Towards generalizability of multi-agent reinforcement learning in graphs with recurrent message passing, 2024

    Jannis Weil, Zhenghua Bao, Osama Abboud, and Tobias Meuser. Towards generalizability of multi-agent reinforcement learning in graphs with recurrent message passing, 2024

    2024

  28. [36]

    The surprising effectiveness of ppo in cooperative, multi-agent games, 2021

    Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative, multi-agent games, 2021. 12 A Appendix A.1 Cost of Exploration Reward Values Below are the reward values for the three cases of COLOR GRI...

    2021

  29. [2021]

    URL http://arxiv.org/abs/2006.07869

    2006 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.