Pith. sign in

REVIEW 4 major objections 5 minor 52 references

Transformer World Model for Sample Efficient Multi-Agent Reinforcement Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MATWM's transformer world model lets cooperative agents reach near-optimal performance in as few as 50,000 environment steps, including from raw images.

desk verdict A genuine engineering contribution with clear sample-efficiency gains on image-based tasks, but the 'state-of-the-art' and 'first image-based' claims run ahead of the evidence; worth a serious peer review. read the letter →

arxiv 2506.18537 v1 pith:GE57OBIP submitted 2025-06-23 cs.LG cs.MA

Azad Deihim , Eduardo Alonso , Dimitra Apostolopoulou This is my paper
classification cs.LGcs.MA
keywords multi-agentreinforcementlearningworldmodelstransformerssampleefficiencyteammatepredictionprioritizedexperiencereplaypartialobservabilityimage-basedobservations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a single transformer-based world model can serve as the imagination engine for teams of cooperative agents, letting them learn near-optimal policies with as few as 50,000 real environment steps. If true, this would close much of the sample-efficiency gap that keeps multi-agent reinforcement learning out of real-world deployment, where collecting millions of interactions is expensive or risky. The method is the first multi-agent world model the authors know of that handles image observations, so it applies to visually rich environments that earlier vector-only world models could not touch. A sympathetic reader should take the central claim as: decentralized imagination plus explicit teammate modeling plus recency-weighted replay is enough to make model-based multi-agent training both fast and stable.

What carries the argument

The mechanism is MATWM, a transformer-based world model trained with a categorical variational autoencoder (32 classes per 32 latent variables). A context window of real observations and actions is encoded into latents, an action mixer merges each latent with an action, and a vanilla transformer produces hidden states. From each hidden state, separate heads predict the next latent, reward, continuation, action mask, and, via a stop-gradient variant, the teammate action distribution; the model is trained with a composite loss combining reconstruction, symlog two-hot reward, binary cross-entropy continuation and mask, cross-entropy teammate, and KL-based dynamics and representation terms with free bits. Agents are trained with a semi-centralized actor-critic: each agent's state concatenates its own latent and hidden state with the teammate predictor's logits, and its critic is local, with an EMA regularizer and percentile-normalized returns. The full pipeline repeats collecting real experience, training the world model on 64-length sequences sampled with a recency bias (priority decay 0.9998), and updating policies on imagined rollouts of length 16, or 12 or 8 for larger teams.

What would settle it

Train MATWM on the 3s_vs_5z SMAC map, and every 5,000 environment steps freeze the current policy and world model, then compare 16-step imagined rollouts against real environment rollouts from the same starting states; if the predicted-reward or next-latent disagreement grows while win rate rises, the imagined experience is increasingly stale and the sample-efficiency claim is undermined.

Watch

Extended reading notes

Core claim

The central discovery is that a decentralized transformer world model can generate useful imagined trajectories for multiple agents without a centralized model of the joint state. Each agent's world model predicts its own next latent state, reward, and continuation, with a learned teammate predictor supplying the likely actions of all other agents, so the focal agent can plan as if the others were part of the environment. Action scaling (shifting each agent's action indices into disjoint ranges) lets the shared model tell agents apart without identity embeddings. Prioritized replay, with an exponential decay on sampling weights favoring recent experience, is presented as the fix for non-stationarity: it keeps the world model aligned with the agents' evolving policies. On SMAC, PettingZoo, and MeltingPot, the authors report state-of-the-art or near-state-of-the-art results within 50K steps for easy maps and 200K for hard maps, with ablations showing that removing the teammate predictor, the prioritized replay, or the action scaling each degrades performance.

Load-bearing premise

The whole approach relies on the imagined teammates staying close to the real teammates' current behavior, and the only safeguard is a replay buffer that mildly favors recent experience; if that heuristic fails, the imagination drifts and the claimed sample efficiency should collapse.

Editorial extensions

If this is right

  • If the 50K-step results hold, model-based MARL becomes practical for domains like robotics and autonomous driving where real experience is expensive.
  • Handling image observations means the same architecture transfers from vector-based simulators to visually rich environments without specialized encoders.
  • Ablations suggest that explicit teammate prediction is a main driver of coordination gains, pointing to a lightweight, communication-free way to achieve teamwork.
  • Prioritized replay with recency weights offers a general recipe for keeping world models synchronized with evolving policies in non-stationary multi-agent settings.
  • The architecture extends in principle to competitive and mixed cooperative-competitive settings because the teammate predictor treats other agents as predictable entities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the world model can reinterpret old experiences under new dynamics, as the authors argue, the same mechanism could be used for continual learning, where an agent's own policy changes long after a transition was stored.
  • The recency-prioritized replay could be made adaptive by measuring world-model prediction error on fresh real transitions and adjusting the decay rate, which might stabilize the performance dips the paper observes.
  • The teammate predictor's near-constant cost with agent count suggests MATWM could scale to dozens of agents where centralized aggregation and joint-value critics become intractable, though the paper itself demonstrates only up to 20 agents.
  • The view of teammates as non-deterministic environmental entities implies the same machinery could model moving obstacles or human traffic, broadening the method beyond cooperative games.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MATWM, a transformer-based decentralized world model for cooperative multi-agent reinforcement learning. It extends the single-agent STORM architecture with a teammate predictor, prioritized replay sampling, action-scaling for agent disambiguation, and a semi-centralized actor-critic trained on imagined rollouts. The authors evaluate MATWM on SMAC (vector observations) and on PettingZoo Butterfly and MeltingPot (image observations), comparing it to model-free and world-model baselines. The central claims are that MATWM achieves state-of-the-art performance and strong sample efficiency, reaching near-optimal behavior in as few as 50K environment steps, and that it is the first multi-agent world model capable of learning from image-based observations.

Significance. If the claims are substantiated, MATWM would be a meaningful advance in sample-efficient MARL: it combines several techniques that are established in single-agent world models but rarely used in multi-agent settings, and it demonstrates a concrete path to low-budget cooperative learning in both vector and visual domains. The paper also provides ablations for the teammate predictor, prioritized replay, and action scaling, and shares a code link, which are useful for reproducibility. However, the evidence as presented overstates some results: the hard-map SMAC comparison is not favorable on three of six hard/easy cases, the image-based comparison lacks any world-model baseline, and the mechanism claimed to control non-stationarity is not validated beyond a single prioritized-versus-uniform ablation. These gaps do not invalidate the core idea, but they need to be addressed before the 'state-of-the-art' and 'first image-based' claims can be accepted.

major comments (4)
  1. [3.2 and Table C.6] The non-stationarity concern is explicitly acknowledged in Section 3.2, where the authors state that the decentralized structure 'could exacerbate the non-stationarity issue' and that prioritized replay 'alleviates' it. The only supporting evidence is the -PER ablation in Table 5, which compares prioritized sampling against uniform sampling. That ablation does not establish that the specific decay of 0.9998 is sufficient: with weight = decay^age, the half-life is about 3,465 steps, and after 20,000 steps a sample retains roughly 1.8% of its original weight. Since policies update every environment step and the buffer holds up to 50,000 transitions, the world model is trained on a substantial fraction of stale teammate behavior. The paper reports no sweep over the decay rate, no measurement of world-model prediction error over time, and no diagnostic comparing imagined rollouts against real multi-agent dynamics. I would like to see either a decay sweep, a direct error plot for dynamics/teammate prediction, or a stated bound justifying why this decay is sufficient.
  2. [4.1, Table 2] The abstract and Section 4.1 claim that MATWM 'outperforms' MARIE by 'decisive margins' and is more sample-efficient overall. The numbers in Table 2 do not support that phrasing on the hard maps: on 2c_vs_64zg, MARIE has median 14 vs MATWM 7; on 3s_vs_5z, MARIE has 66 vs MATWM 64; on 8m (easy), MARIE has 72 vs MATWM 67. The mean over all 12 maps is 61.1 for MATWM and 57.1 for MARIE, a difference of 4 points with no significance testing or confidence intervals. The paper should either soften the claim to 'comparable or better on most maps' or provide statistical support and an explicit sample-efficiency curve (e.g., area under the win-rate versus steps curve) for the claimed superiority.
  3. [3.1, Algorithm 1 and Equation (8)] The teammate predictor is trained on ground-truth teammate actions from real latent states, but during imagination it is queried on latent states produced by the world-model rollout. If the imagined latent states drift from the real distribution, the teammate logits become unreliable, and the policy conditions on these unreliable logits. This is a standard exposure-bias problem, and it is directly relevant to the paper's claim that imagination enables coordination. The paper does not report the teammate predictor's accuracy on real vs imagined latent states, nor does it quantify how much of the final performance depends on the predictor being accurate at the imagination horizon used (up to 16 steps). A simple diagnostic (e.g., teammate prediction accuracy on held-out imagined rollouts) would make the mechanism much more credible.
  4. [4.2, Tables 3 and 4] The image-based experiments compare MATWM only against model-free baselines, and those baselines often perform near random under the 50K budget. The claim that MATWM is the first multi-agent world model capable of learning from image-based observations is therefore not tested against any existing multi-agent world model, since MARIE, MAMBA, and MBVD are not run or adapted for PettingZoo/MeltingPot. To support the 'first' and 'state-of-the-art' claims, the paper should either include an adapted world-model baseline (even a simple DreamerV3-style single-agent baseline per agent) or explicitly reframe the contribution as a demonstration that a world-model approach can learn on these visual tasks at 50K steps, without claiming comparative superiority over other world models in this setting.
minor comments (5)
  1. [Equation (8) and notation] The index range for the teammate predictor is inconsistent: the text says 'N non-focal agents' but the sum runs over i = 0 to N, while the description says 'For each teammate i ∈ 0,...,N'. Please clarify whether N is the number of non-focal agents or the total number of agents, and use matching bounds.
  2. [Section 3.1, action scaling] The phrase 'mutually orthogonal' is misleading: offsetting discrete action indices (e.g., {0,1,2} to {3,4,5}) does not make the actions orthogonal in any formal sense unless a specific embedding is defined. I suggest rewording to 'non-overlapping action indices' or specifying the embedding.
  3. [Table 2 formatting] The map name 'so many baneling' appears as 'somanybaneling' in the table, and the column header 'MA TWM' has an unintended space. Please fix these formatting issues for readability.
  4. [Section 4, evaluation protocol] The paper reports median and standard deviation over four seeds, but no significance tests or bootstrap intervals are provided. Given that several differences are small (e.g., 64 vs 66 on 3s_vs_5z), adding a simple pairwise significance test or effect-size measure would strengthen the claims.
  5. [Algorithm 2, line 17] The algorithm trains the world model on a combined batch from all replay buffers, but the text in Section 3 says each agent has its own replay buffer and the world model is trained using an equal distribution of experiences from each agent. Please state explicitly how the combined batch is formed (equal number of transitions per agent, or proportional to buffer size).

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: MATWM's reported performance is derived from external benchmarks and supervised world-model training, not from its own definitions or fitted predictions.

full rationale

The paper's derivation chain is self-contained and empirically grounded. The world model is trained with supervised losses (Eqs. 3-9) on real transitions, with the teammate predictor supervised by ground-truth teammate actions via cross-entropy (Eq. 8). The actor and critic are trained on imagined rollouts (Eqs. 10-14), but the reported metrics are real-environment win rates and rewards evaluated on external benchmarks (SMAC, PettingZoo, MeltingPot), so no reported quantity is defined in terms of the quantity it is said to predict. The prioritized replay mechanism with decay 0.9998 is a design choice addressing an explicitly acknowledged non-stationarity concern, and the absence of a decay sweep is a robustness gap, not a circular step: the world model's predictions are not forced to match the reported results by construction. The only self-citation is Ref. [30] (IRIS, Micheli, Alonso, Fleuret), cited in the Related Work as one of several transformer-based single-agent world models; it is not load-bearing for MATWM's architecture or results, since MATWM builds on STORM [14] and DreamerV3 [16], which are independent external works. Therefore no significant circularity is present.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical. The listed hyperparameters and modeling assumptions are what the reported performance rests on, and the paper does not derive them from theory. No new physical entities are introduced; the teammate predictor is a neural module trained with supervision.

free parameters (6)
  • Latent representation size = 32 categorical variables, 32 classes each
    Taken from DreamerV2/STORM; controls how much observation information can be retained and is not derived from the benchmark.
  • Imagination horizon = 16; 12 for 4-6 agents and 2c_vs_64z; 8 for 7+ agents
    Hand-set by agent count; directly determines length of imagined rollouts and interacts with sample-efficiency results.
  • Replay priority decay = 0.9998
    Hand-chosen exponential decay that controls recency weighting for world-model training; central to the non-stationarity mitigation claim.
  • KL loss weights beta1 and beta2 = 0.5 and 0.1
    Hand-chosen weights balancing dynamics and representation losses; not justified by theory.
  • World model and agent batch sizes = 16 and 512; 768 or 1024 for larger agent counts
    Adjusted by agent count during experiments; affect update frequency and stability.
  • Entropy regularization coefficient eta = Not reported
    The actor loss in Equation 11 includes eta H(pi), but the value of eta is not listed in Table C.6, leaving an unidentified setting.
assumptions (5)
  • standard math Backpropagation through straight-through categorical sampling and transformer sequence modeling is valid for this training setup.
    Relies on standard deep learning background; not proven in the paper.
  • domain assumption The learned world model produces imagined rollouts accurate enough to serve as a training signal for policy learning.
    Central to all world-model methods; only evaluated empirically, not guaranteed for partially observable multi-agent settings.
  • domain assumption Prioritized replay with exponential decay keeps the world model aligned with evolving teammate policies.
    The authors explicitly hypothesize this and cite it as the remedy for non-stationarity; no formal guarantee is given.
  • domain assumption Teammate predictor outputs, trained with stop-gradient, are useful state features for the semi-centralized actor and critic.
    Required for the semi-centralized critic to work without direct access to other agents' states.
  • ad hoc to paper Action scaling by offsetting action indices distinguishes agents sufficiently for the world model.
    This design trick is introduced specifically for MATWM; its effectiveness is tested only through ablation, not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transformer World Model for Sample Efficient Multi-Agent Reinforcement Learning." pith.science (2026). https://pith.science/paper/GE57OBIP

@misc{pith2026250618537,
  author       = {Pith},
  title        = {Pith review of: Transformer World Model for Sample Efficient Multi-Agent Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GE57OBIP}},
  note         = {Machine review of arXiv:2506.18537}
}
read the original abstract

We present the Multi-Agent Transformer World Model (MATWM), a novel transformer-based world model designed for multi-agent reinforcement learning in both vector- and image-based environments. MATWM combines a decentralized imagination framework with a semi-centralized critic and a teammate prediction module, enabling agents to model and anticipate the behavior of others under partial observability. To address non-stationarity, we incorporate a prioritized replay mechanism that trains the world model on recent experiences, allowing it to adapt to agents' evolving policies. We evaluated MATWM on a broad suite of benchmarks, including the StarCraft Multi-Agent Challenge, PettingZoo, and MeltingPot. MATWM achieves state-of-the-art performance, outperforming both model-free and prior world model approaches, while demonstrating strong sample efficiency, achieving near-optimal performance in as few as 50K environment interactions. Ablation studies confirm the impact of each component, with substantial gains in coordination-heavy tasks.

Figures

Figures reproduced from arXiv: 2506.18537 by the authors.

Figure 1
Figure 1. Overview of the MATWM architecture. The figure illustrates how real context (past observations [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Curves of evaluation win rates for each method on three representative SMAC maps. Curves [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Curves of average reward obtained across all agents for each method on three representative [PITH_FULL_IMAGE:figures/full_fig_p014_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 34 canonical work pages

  1. [1]

    Foerster, G

    J. Foerster, G. Farquhar, T. Afouras, N. Nardelli, S. Whiteson, Counterfactual multi- agent policy gradients, Proceedings of the AAAI Conference on Artificial Intelligence 32 (1) (2018). doi:10.1609/aaai.v32i1.11794. URL https://ojs.aaai.org/index.php/AAAI/article/view/11794

    doi:10.1609/aaai.v32i1.11794 2018

  2. [2]

    R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, I. Mordatch, Multi-agent actor-critic for mixed cooperative-competitive environments, in: Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS), NeurIPS ’17, Curran Associates Inc., Red Hook, NY, USA, 2017, pp. 6382–6393

    work page 2017

  3. [3]

    C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, Y. Wu, The surprising effectiveness of ppo in cooperative multi-agent games, in: Proceedings of the 36th In- ternational Conference on Neural Information Processing Systems (NeurIPS), NeurIPS ’22’, Curran Associates Inc., Red Hook, NY, USA, 2022, p. 1787

    work page 2022

  4. [4]

    Rashid, M

    T. Rashid, M. Samvelyan, C. Schroeder de Witt, G. Farquhar, J. Foerster, S. Whiteson, Monotonic value function factorisation for deep multi-agent reinforcement learning, Journal of Machine Learning Research 21 (1) (2020) 178:1–178:51

    work page 2020

  5. [5]

    J. Wang, Z. Ren, T. Liu, Y. Yu, C. Zhang, Qplex: Duplex dueling multi-agent q- learning, in: Proceedings of the 9th International Conference on Learning Representa- tions (ICLR), 2021. URL https://openreview.net/forum?id=Rcmk0xxIQV

    work page 2021

  6. [6]

    K. Son, D. Kim, W. J. Kang, D. E. Hostallero, Y. Yi, Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning, in: Proceedings of the 36th International Conference on Machine Learning (ICML), PMLR, 2019, pp. 5887–5896

    work page 2019

  7. [7]

    Sunehag, G

    P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, T. Graepel, Value-decomposition net- works for cooperative multi-agent learning based on team reward, in: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), AAMAS ’18, Internationa...

    work page 2018

  8. [8]

    Hernandez-Leal, B

    P. Hernandez-Leal, B. Kartal, M. E. Taylor, A very condensed survey and critique of multiagent deep reinforcement learning, in: Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems (AAMAS), AAMAS ’20, International Foundation for Autonomous Agents and Multiagent Systems, Richland, SC, 2020, pp. 2146–2148

    work page 2020

Show all 52 references
  1. [9]

    J. Ding, Y. Zhang, Y. Shang, Y. Zhang, Z. Zong, J. Feng, Y. Yuan, H. Su, N. Li, N. Sukiennik, F. Xu, Y. Li, Understanding world or predicting future? a comprehensive 19 survey of world models (2024). arXiv:2411.14499. URL https://arxiv.org/abs/2411.14499

    2024

  2. [10]

    Z. Xu, B. Zhang, Y. Zhan, Y. Baiia, G. Fan, et al., Mingling foresight with imagination: Model-based cooperative multi-agent reinforcement learning, in: Advances in Neural Information Processing Systems, Vol. 35, 2022, pp. 11327–11340

    2022

  3. [11]

    Egorov, A

    V. Egorov, A. Shpilman, Scalable multi-agent model-based reinforcement learning, in: Proceedings of the 21st International Conference on Autonomous Agents and Mul- tiagent Systems (AAMAS), AAMAS ’22, International Foundation for Autonomous Agents and Multiagent Systems, Richla...

    2022

  4. [12]

    Zhang, C

    Y. Zhang, C. Bai, B. Zhao, J. Yan, X. Li, X. Li, Decentralized transformers with centralized aggregation are sample-efficient multi-agent world models (2024). arXiv: 2406.15836. URL https://arxiv.org/abs/2406.15836

    2024 arXiv

  5. [13]

    W. H. Guss, B. Houghton, N. Topin, A. Velu, S. Codel, M. Alfredo, S. T. Iqbal, D. Dey, S. Wong, K. Gopalakrishnan, et al., Minerl: A large-scale dataset of minecraft demonstrations, in: Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), 2019

    2019

  6. [14]

    Zhang, G

    W. Zhang, G. Wang, J. Sun, Y. Yuan, G. Huang, Storm: Efficient stochastic trans- former based world models for reinforcement learning, Advances in Neural Information Processing Systems 36 (2023) 27147–27166

    2023

  7. [15]

    Hafner, T

    D. Hafner, T. Lillicrap, M. Norouzi, J. Ba, Mastering atari with discrete world models (2020). arXiv:2010.02193. URL https://arxiv.org/abs/2010.02193

    2020 arXiv

  8. [16]

    Hafner, J

    D. Hafner, J. Pasukonis, J. Ba, T. Lillicrap, Mastering diverse domains through world models (2024). arXiv:2301.04104. URL https://arxiv.org/abs/2301.04104

    2024 arXiv

  9. [17]

    Robine, M

    J. Robine, M. H¨ oftmann, T. Uelwer, S. Harmeling, Transformer-based world models are happy with 100k interactions, in: Proceedings of the Eleventh International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=TdBaDGCpjly

    2023

  10. [18]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in Neural Information Processing Systems, Vol. 30, Curran Associates, Inc., 2017, pp. 5998–6008. URL https://proceedings.neurips.cc/paper_f...

    2017

  11. [19]

    D. P. Kingma, T. Salimans, R. Jozefowicz, X. Chen, I. Sutskever, M. Welling, Improved variational inference with inverse autoregressive flow, Advances in neural information processing systems 29 (2016)

    2016

  12. [20]

    T. Feng, W. Wang, Y. Yang, A survey of world models for autonomous driving (2025). arXiv:2501.11260. URL https://arxiv.org/abs/2501.11260

    2025 arXiv

  13. [21]

    F.-M. Luo, T. Xu, H. Lai, X.-H. Chen, W. Zhang, Y. Yu, A survey on model-based reinforcement learning, Science China Information Sciences 67 (2) (2024) 121101

    2024

  14. [22]

    J. I. Kim, Y. J. Lee, J. Heo, J. Park, J. Kim, S. R. Lim, J. Jeong, S. B. Kim, Sample- efficient multi-agent reinforcement learning with masked reconstruction, PLOS ONE 18 (9) (2023) e0291545

    2023

  15. [23]

    X. Liu, Y. Chen, H. Li, D. Zhao, Learning future representation with synthetic obser- vations for sample-efficient reinforcement learning, Science China Information Sciences 68 (5) (2025) 150202

    2025

  16. [24]

    G. Ma, L. Zhang, H. Wang, L. Li, Z. Wang, Z. Wang, L. Shen, X. Wang, D. Tao, Learn- ing better with less: Effective augmentation for sample-efficient visual reinforcement learning, in: Proceedings of the Thirty-Seventh Conference on Neural Information Pro- cessing Systems (Neu...

    2023

  17. [25]

    S. Liu, X. S. Zhang, Y. Li, Y. Zhang, J. Cheng, On the data-efficiency with contrastive image transformation in reinforcement learning, in: The Eleventh International Con- ference on Learning Representations, 2023

    2023

  18. [26]

    D. Ha, J. Schmidhuber, World models, Zenodo (2018). doi:10.5281/ZENODO.1207631. URL https://zenodo.org/record/1207631

    2018

  19. [27]

    D. E. Rumelhart, G. E. Hinton, R. J. Williams, Learning internal representations by error propagation, in: D. E. Rumelhart, J. L. McClelland, the PDP Research Group (Eds.), Parallel Distributed Processing: Explorations in the Microstructure of Cogni- tion. Volume 1: Foundation...

    1986

  20. [28]

    Kaiser, M

    L. Kaiser, M. Babaeizadeh, P. Milos, B. Osinski, R. H. Campbell, K. Czechowski, D. Erhan, C. Finn, P. Kozakowski, S. Levine, A. Mohiuddin, R. Sepassi, G. Tucker, H. Michalewski, Model-based reinforcement learning for atari, in: Proceedings of the 8th International Conference o...

    2020

  21. [29]

    Hafner, T

    D. Hafner, T. Lillicrap, J. Ba, M. Norouzi, Dream to control: Learning behaviors by latent imagination (2019). arXiv:1912.01603. URL https://arxiv.org/abs/1912.01603 21

    2019 arXiv

  22. [30]

    Micheli, E

    V. Micheli, E. Alonso, F. Fleuret, Transformers are sample-efficient world models, in: Proceedings of the Eleventh International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=vhFu1Acb0xb

    2023

  23. [31]

    C. Chen, J. Yoon, Y.-F. Wu, S. Ahn, Transdreamer: Reinforcement learning with transformer world models, arXiv preprint, under review at ICLR (2022). URL https://openreview.net/forum?id=s3K0arSRl4d

    2022

  24. [32]

    K. Cho, B. van Merri¨ enboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning phrase representations using RNN encoder–decoder for statistical machine translation, in: A. Moschitti, B. Pang, W. Daelemans (Eds.), Proceedings of the 2014 Conference on Empi...

    2014 doi

  25. [33]

    Q. Liu, J. Ye, X. Ma, J. Yang, B. Liang, C. Zhang, Efficient multi-agent reinforce- ment learning by planning, in: Proceedings of the Twelfth International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=CpnKq3UJwp

    2024

  26. [34]

    Schrittwieser, I

    J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, D. Silver, Mastering atari, go, chess and shogi by planning with a learned model, Nature 588 (2020) 604–609. doi: 10.1038/s41586-020-03051-4

    2020 doi

  27. [35]

    Jaegle, F

    A. Jaegle, F. Gimeno, A. Brock, O. Vinyals, A. Zisserman, J. Carreira, Perceiver: General perception with iterative attention, in: M. Meila, T. Zhang (Eds.), Proceedings of the 38th International Conference on Machine Learning, Vol. 139 of Proceedings of Machine Learning Resea...

    2021

  28. [36]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy opti- mization algorithms (2017). arXiv:1707.06347. URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  29. [37]

    Kaplanis, M

    C. Kaplanis, M. Shanahan, C. Clopath, Policy consolidation for continual reinforcement learning (2019). arXiv:1902.00255. URL https://arxiv.org/abs/1902.00255

    2019 arXiv

  30. [38]

    W. Ye, S. Liu, T. Kurutach, P. Abbeel, Y. Gao, Mastering atari games with limited data, in: Advances in Neural Information Processing Systems, Vol. 34, Curran Associates, Inc., 2021, pp. 25476–25488. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ d5eca8dc3820c...

    2021

  31. [39]

    D. P. Kingma, M. Welling, Auto-encoding variational bayes (2013). arXiv:1312.6114. URL https://arxiv.org/abs/1312.6114

    2013 arXiv

  32. [40]

    van den Oord, O

    A. van den Oord, O. Vinyals, K. Kavukcuoglu, Neural discrete representation learning, in: Proceedings of the 31st International Conference on Neural Information Processing Systems (NeurIPS), NeurIPS’17, Curran Associates Inc., Long Beach, California, USA, 2017, pp. 6309–6318

    2017

  33. [41]

    Beattie, J

    C. Beattie, J. Z. Leibo, D. Teplyashin, T. Ward, M. Wainwright, H. K¨ uttler, A. Lefrancq, S. Green, V. Vald´ es, A. Sadik, J. Schrittwieser, K. Anderson, S. York, M. Cant, A. Cain, A. Bolton, S. Gaffney, H. King, D. Hassabis, S. Legg, S. Petersen, Deepmind lab (2016). arXiv:1...

    2016 arXiv

  34. [42]

    E. J. Meyer, A. White, M. C. Machado, Harnessing discrete representations for con- tinual reinforcement learning, Reinforcement Learning Journal 2 (2024) 606–628

    2024

  35. [43]

    E. Jang, S. Gu, B. Poole, Categorical reparameterization with gumbel-softmax, in: Proceedings of the 5th International Conference on Learning Representations (ICLR), 2017. URL https://arxiv.org/abs/1611.01144

    2017 arXiv

  36. [44]

    LeCun, B

    Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, L. D. Jackel, Backpropagation applied to handwritten zip code recognition, Neural Compu- tation 1 (4) (1989) 541–551. doi:10.1162/neco.1989.1.4.541. URL https://doi.org/10.1162/neco.1989.1.4.541

    1989 doi

  37. [45]

    Bengio, N

    Y. Bengio, N. L´ eonard, A. Courville, Estimating or propagating gradients through stochastic neurons for conditional computation (2013). arXiv:1308.3432. URL https://arxiv.org/abs/1308.3432

    2013 arXiv

  38. [46]

    R. S. Sutton, A. G. Barto, Reinforcement Learning: An Introduction, MIT Press, Cambridge, MA, USA, 1998

    1998

  39. [47]

    Samvelyan, T

    M. Samvelyan, T. Rashid, C. Schroeder de Witt, G. Farquhar, N. Nardelli, T. G. J. Rudner, C.-M. Hung, P. H. S. Torr, J. Foerster, S. Whiteson, The starcraft multi- agent challenge, in: Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems...

    2019

  40. [48]

    J. K. Terry, B. Black, N. Grammel, M. Jayakumar, A. Hari, R. Sullivan, L. Santos, R. Perez, C. Horsch, C. Dieffendahl, N. L. Williams, Y. Lokesh, P. Ravi, Pettingzoo: A standard api for multi-agent reinforcement learning, in: Proceedings of the 35th Inter- national Conference ...

    2021

  41. [49]

    J. P. Agapiou, A. S. Vezhnevets, E. A. Du´ e˜ nez-Guzm´ an, J. Matyas, Y. Mao, P. Sune- hag, R. K¨ oster, U. Madhushani, K. Kopparapu, R. Comanescu, D. Strouse, M. B. Johanson, S. Singh, J. Haas, I. Mordatch, D. Mobbs, J. Z. Leibo, Melting pot 2.0 (2023). arXiv:2211.13746. URL...

    2023 arXiv

  42. [50]

    Papoudakis, F

    G. Papoudakis, F. Christianos, L. Sch¨ afer, S. V. Albrecht, Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks, in: Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2021. URL https://arxiv.o...

    2021 arXiv

  43. [51]

    Papadopoulos, A

    G. Papadopoulos, A. Kontogiannis, F. Papadopoulou, C. Poulianou, I. Koumentis, G. Vouros, An extended benchmarking of multi-agent reinforcement learning algo- rithms in complex fully cooperative tasks (2025). arXiv:2502.04773. URL https://arxiv.org/abs/2502.04773

    2025 arXiv

  44. [52]

    Formanek, A

    C. Formanek, A. Jeewa, J. Shock, A. Pretorius, Off-the-grid marl: Datasets and base- lines for offline multi-agent reinforcement learning, in: Proceedings of the 2023 Interna- tional Conference on Autonomous Agents and Multiagent Systems (AAMAS), AAMAS ’23, International Found...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.