Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

No Press Diplomacy: Modeling Multi-Agent Gameplay

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Pith's one-line read A neural policy trained on 150,000 human games beats hand-crafted rule-based bots in the No Press variant of Diplomacy, the paper reports.

desk verdict A solid benchmark paper: the first large-scale neural result in No Press Diplomacy, and the claim that DipNet beats rule-based bots holds up despite a few evaluation caveats. read the letter →

arxiv 1909.02128 v2 pith:4FFAXM4S submitted 2019-09-04 cs.AI cs.LGcs.MA

classification cs.AIcs.LGcs.MA
keywords DiplomacyNoPressmulti-agentgameplaysequentialsocialdilemmasgraphconvolutionalnetworksupervisedlearningreinforcementTrueSkill
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a neural network can outperform hand-coded rule-based agents in the No Press version of Diplomacy, a seven-player game of trust and betrayal with no dice and no direct communication channel. To do so it introduces DipNet, a graph-convolution network that reads the board and previous orders and generates one order per unit, trained first by supervised learning on more than 150,000 human games and then refined by reinforcement learning through self-play. In a TrueSkill tournament, both the supervised and self-play versions outrank Albert, the strongest rule-based baseline. If that ranking holds, Diplomacy becomes a tractable data-driven benchmark for complex sequential social dilemmas in which cooperation must be inferred from actions rather than messages.

What carries the argument

The machinery is DipNet: a graph convolutional network over the 81-province map with FiLM conditioning on player power and season, an LSTM decoder that generates unit orders one at a time in a top-left to bottom-right ordering with a masked softmax over valid orders, and a second input stream encoding previous-phase orders. The graph convolution aggregates neighbor information over 16 layers so the network can track long-range unit coordination; FiLM injects whose turn it is and what season; the sequential masked decoder ensures only legal orders are produced. This object carries the argument because the claimed performance and the support-coordination analysis both depend on it: ablations that remove FiLM, the board state, or the location-aware decoder drop support accuracy, especially for the 16th unit in the order sequence, and reduce effective cross-power support.

What would settle it

Re-run the mixed tournament with starting powers forced to be balanced across agents and recompute TrueSkill; if Albert reaches or passes DipNet's score under that protocol, the claimed state-of-the-art performance is an artifact of the original unbalanced draw.

Watch

Extended reading notes

Core claim

The paper's central claim is that a fully end-to-end neural policy can achieve state-of-the-art No Press Diplomacy performance without an explicit communication channel: DipNet, trained by supervised learning from expert human trajectories and then fine-tuned by self-play reinforcement learning, beats popular rule-based bots, including Albert. In the 1-vs-6 tournament, SL DipNet wins 28.9% of games against six Albert copies and is defeated in 23.1% of games, with a TrueSkill of 28.1 against Albert's 24.5; RL DipNet scores 27.4. The paper also claims that the architecture's support-order prediction accuracy and its effective cross-power support ratio show it can coordinate units and cooperate with other powers, and that this cooperation learned from human data partially deteriorates during self-play.

Load-bearing premise

The ranking rests on the tournament's TrueSkill scores ignoring Diplomacy's starting-power imbalance, so the comparison is only trustworthy if every agent drew each of the seven powers equally often across the 1,378 games.

Editorial extensions

If this is right

  • If the TrueSkill ranking holds, DipNet is the first data-driven agent to beat the strongest rule-based No Press bot, making Diplomacy a viable target for learning-based multi-agent research rather than purely hand-crafted strategies.
  • The 150,000-game human dataset and the released game engine give other agents a shared evaluation platform, so future No Press results can be compared against the same rule-based bots and the same rating procedure.
  • The support-order ablation shows the full architecture keeps coordination accuracy at the 16th decoded unit (32.2% vs 25.9% for the board-state-only variant), indicating that the map encoder and decoder jointly sustain multi-unit tactics.
  • The coalition analysis shows SL DipNet has an effective cross-power support ratio of 10.2% versus 5.3% for RL DipNet, so the paper's self-play recipe improves overall strength without preserving the human-learned cooperative behavior.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the authors leave implicit is that the power-imbalance caveat could be probed directly: a balanced-power rematch between DipNet and Albert, conditioned on starting country, would show whether the claimed edge is robust or driven by favorable draws.
  • A natural extension is to use DipNet's previous-order stream as a controlled channel: feeding the agent games with randomized or current-phase orders would test whether cross-power support is genuinely coordinated signaling or a side effect of the board-state encoding.
  • The deterioration of effective cooperation under self-play suggests an experiment the authors point toward but do not run: retrain the RL agent with an opponent-aware or inequity-averse objective and compare its effective cross-power support ratio to the SL baseline; if the ratio recovers, the known failure mode of naive policy gradients in social dilemmas would be confirmed in Diplomacy's richer
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper introduces DipNet, a neural-network policy for the No Press variant of Diplomacy. The model is trained by supervised learning on a newly collected dataset of 156,468 human games, then fine-tuned with A2C self-play. The central claim is that both the supervised (SL) and reinforcement-learned (RL) versions beat popular rule-based bots, with TrueSkill scores of 28.1 and 27.4 respectively, compared with 24.5 for the strongest baseline, Albert. The paper also presents architecture ablations, support-order prediction accuracy, and a coalition analysis based on cross-power support ratios, and it releases the dataset, game engine, and code.

Significance. The work is a significant empirical contribution: it is, to my knowledge, the first large-scale data-driven agent for No Press Diplomacy, and the reported TrueSkill gap over rule-based bots is a meaningful demonstration that learned policies can outperform hand-crafted bots in a complex deterministic social dilemma. The 156k-game dataset, the DAIDE-compatible engine, and the public code are valuable community assets. The ablation study and the coalition analysis are useful, and the observation that self-play training reduces effective cross-power support is an interesting and non-obvious finding. The evaluation caveats discussed below are important but do not invalidate the contribution.

major comments (2)
  1. [Appendix A; Section 6.2] The headline claim that DipNet 'beats popular rule-based bots' rests on TrueSkill ratings computed without accounting for starting-power imbalance, as the authors acknowledge in the final paragraph of Appendix A, and reported without confidence intervals or per-power breakdowns. Given that human win rates range from 3.6% for Italy to 7.2% for Turkey (Table 1), the aggregate TrueSkill gap could in principle be driven by a subset of starting powers. Random power assignment balances in expectation, so this is not a claim of overt bias, but it limits the support for the claim as stated. Please add a power-stratified analysis (for example, per-power TrueSkill or win rate conditional on assigned power) and either confidence intervals or a significance test for the SL-vs-Albert and RL-vs-Albert gaps.
  2. [Section 6.1] The supervised test set is defined as the last 5% of games sorted by game id in alphabetical order. If game ids are assigned chronologically or correlate with map variant or player population, this split may not be representative and could yield optimistic accuracy numbers. Please use a random split, or alternatively demonstrate that the last-5% subset has the same distribution over powers, game lengths, and game outcomes as the full dataset, and report standard errors or confidence intervals for the accuracy numbers.
minor comments (5)
  1. [Section 4.3] The phrase 'attention based on the location' is misleading because the decoder described in Eq. (2) concatenates the current location's embedding rather than computing attention over locations; please rephrase to match the actual mechanism.
  2. [Table 4] Please define in the caption whether 'Win%' means solo victories only, and clarify the relationship between the 1-vs-6 tournament and the TrueSkill tournament; the 'TrueSkill A-B' column appears to be the global rating difference rather than a rating computed from the adjacent 1-vs-6 games, and this should be stated explicitly.
  3. [Appendix B; Section 1] There are typos in which 'affect' should be 'effect'; please correct them throughout.
  4. [Table 1] The 'Total' row mixes win/draw/defeated percentages with survival-rate columns, which is confusing; please split the row or add a note explaining the columns.
  5. [Equation (1)] Please specify the encoding of the conditioning variables p and s (e.g., one-hot vectors) and the dimensions of the FiLM linear layer f_l, since the vector [p;s] is not defined in the text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: DipNet's central claim is an empirical tournament comparison against independent rule-based baselines, with no fitted parameter defining the target metric.

full rationale

The paper's central claim is that the SL and RL versions of DipNet 'demonstrate state-of-the-art No Press performance by beating popular rule-based bots' (Abstract; Section 6.2). This claim is supported by TrueSkill ratings computed from a tournament in which DipNet variants and the baselines (Random, GreedyBot, Dumbbot, Albert) are all evaluated under the same protocol (Appendix A). No parameter is fitted to the evaluation metric: the SL model is trained on held-out human trajectories, the RL model is trained by self-play with hand-set reward coefficients, and the TrueSkill rating is an external ranking computed from game outcomes. The ablation studies compare genuine architectural variants, and the coalition analysis uses independently defined metrics. The only self-citations in the paper are to FiLM conditioning layers (references [29] and [31], with overlapping authorship), but FiLM is used as a general architectural tool and is not load-bearing for the central empirical result. Appendix A explicitly acknowledges a limitation: 'we do not consider the existing power imbalance in the game, e.g., winning as Austria is harder than winning as France.' This is a validity concern about the ranking protocol, not a circularity: it does not make DipNet's victory definitional, and the manuscript itself flags the issue as future work. Because the central derivation is self-contained and the evaluation is against independent baselines, no circular step is present.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical; the paper introduces no new formal or physical entities. The hand-set quantities are reward-shaping coefficients, an n-step horizon, and architecture hyperparameters. The main domain assumptions are the sufficiency of the last movement phase's orders for inferring relationships, the representativeness of the webDiplomacy dataset, and the validity of the TrueSkill evaluation protocol.

free parameters (4)
  • Local reward coefficient = +1/-1 per supply center gained/lost
    Hand-set reward shaping for the RL variant; combined by averaging with the terminal reward.
  • Terminal reward scale = 34 points for solo win, proportional by supply centers for draws
    Hand-set to map supply centers to points; averaging gives equal weight to local and terminal rewards.
  • n-step return horizon = 15
    Chosen for the A2C updates; not reported as tuned.
  • Number of GCN layers = 16
    Chosen by the authors; Table 6 shows 8 layers achieve nearly identical accuracy, so this is not load-bearing.
assumptions (4)
  • domain assumption Orders from the last movement phase are sufficient to infer current relationships between powers.
    Stated in Section 4.1; this justifies the input representation of previous orders. If long-term reputation matters, the model cannot capture it.
  • domain assumption The webDiplomacy dataset is a valid source of expert-level No Press play.
    The training and test data come from anonymous online games; no expert-rating filter is described.
  • domain assumption TrueSkill ratings computed from 1,378 games with ranks by elimination order and supply centers are a reliable agent ranking.
    Appendix A; the authors acknowledge power imbalance is ignored.
  • standard math Standard stochastic gradient training and backpropagation are reliable for optimizing the architecture.
    Assumed throughout the training pipeline.

how reviews work

0 comments
Cite this review

Pith. "Pith review of No Press Diplomacy: Modeling Multi-Agent Gameplay." pith.science (2026). https://pith.science/paper/4FFAXM4S

@misc{pith2026190902128,
  author       = {Pith},
  title        = {Pith review of: No Press Diplomacy: Modeling Multi-Agent Gameplay},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4FFAXM4S}},
  note         = {Machine review of arXiv:1909.02128}
}
read the original abstract

Diplomacy is a seven-player non-stochastic, non-cooperative game, where agents acquire resources through a mix of teamwork and betrayal. Reliance on trust and coordination makes Diplomacy the first non-cooperative multi-agent benchmark for complex sequential social dilemmas in a rich environment. In this work, we focus on training an agent that learns to play the No Press version of Diplomacy where there is no dedicated communication channel between players. We present DipNet, a neural-network-based policy model for No Press Diplomacy. The model was trained on a new dataset of more than 150,000 human games. Our model is trained by supervised learning (SL) from expert trajectories, which is then used to initialize a reinforcement learning (RL) agent trained through self-play. Both the SL and RL agents demonstrate state-of-the-art No Press performance by beating popular rule-based bots.

Figures

Figures reproduced from arXiv: 1909.02128 by the authors.

Figure 1
Figure 1. The standard Diplomacy map rely on chance to win [2, 3, 4, 5, 6]. Instead, players must put their faith in other players and not in the game’s mechanics (e.g. having a player role a critical hit). Diplomacy is also one of the first SSD games to feature a rich environment. A single player may have up to 34 units, with each unit having an average of 26 possible actions. This astronomical action space makes planning an… view at source ↗
Figure 2
Figure 2. Encoding of the board state and previous orders. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. DipNet architecture 5 Datasets and Game Engine Our dataset is generated by aggregating 156,468 anonymized human games. We also develop an open source game engine for this dataset to standardize its format and rule out invalid orders. The dataset contains 33,279 No Press games, 1,290 Public Press games, 105,266 Press games (messages are not included), and 16,633 games not played on the standard map. We are going to r… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Cognitive Dark Matter: Measuring What AI Misses

    q-bio.NC 2026-03 conditional novelty 6.0 of 10

    AI's patchy performance stems from under-measured hidden cognitive processes, called cognitive dark matter, which could be made visible by collecting process-tracing and neural-behavioral data at scale.

Reference graph

Works this paper leans on

37 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Multi-agent reinforcement learning in sequential social dilemmas

    Joel Z Leibo, Vinicius Zambaldi, Marc Lanctot, Janusz Marecki, and Thore Graepel. Multi-agent reinforcement learning in sequential social dilemmas. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgent Systems, pages 464–473. International Foundation for Autonomous Agents and Multiagent Systems, 2017

  2. [2]

    The hanabi challenge: A new frontier for ai research

    Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. arXiv preprint arXiv:1902.00506, 2019

  3. [3]

    Superhuman ai for heads-up no-limit poker: Libratus beats top professionals

    Noam Brown and Tuomas Sandholm. Superhuman ai for heads-up no-limit poker: Libratus beats top professionals. Science, 359(6374):418–424, 2018

  4. [4]

    Deepstack: Expert-level artificial intelligence in heads-up no-limit poker

    Matej Moravˇcík, Martin Schmid, Neil Burch, Viliam Lis`y, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):508–513, 2017

  5. [5]

    Openai five

    OpenAI. Openai five. https://blog.openai.com/openai-five/, 2018

  6. [6]

    Oriol Vinyals, Igor Babuschkin, Junyoung Chung, Michael Mathieu, Max Jaderberg, Wo- jciech M. Czarnecki, Andrew Dudzik, Aja Huang, Petko Georgiev, Richard Powell, Timo Ewalds, Dan Horgan, Manuel Kroiss, Ivo Danihelka, John Agapiou, Junhyuk Oh, Valentin Dalibard, David Choi, Laurent Sifre, Yury Sulsky, Sasha Vezhnevets, James Molloy, Trevor Cai, David Budd...

  7. [7]

    Dp w1995a: Communication in no-press diplomacy

    Simon Szykman. Dp w1995a: Communication in no-press diplomacy. http://uk.diplom. org/pouch/Zine/W1995A/Szykman/Syntax.html, 1995. Accessed: 2019-05-01

  8. [8]

    Daide - diplomacy artificial intelligence development environment

    David Norman. Daide - diplomacy artificial intelligence development environment. http: //www.daide.org.uk/, 2013. Accessed: 2019-05-01. 9

Show all 37 references
  1. [9]

    Diplomacy ai - albert

    Jason van Hal. Diplomacy ai - albert. https://sites.google.com/site/diplomacyai/,

  2. [10]

    TrueskillTM: a bayesian skill rating system

    Ralf Herbrich, Tom Minka, and Thore Graepel. TrueskillTM: a bayesian skill rating system. In Advances in neural information processing systems, pages 569–576, 2007

  3. [11]

    A player rating system for diplomacy

    Tony Nichols. A player rating system for diplomacy. http://www.stabbeurfou.org/docs/ articles/en/DP_S1998R_Diplomacys_New_Rating_System.html, 1998. Accessed: 2019-05-01

  4. [12]

    Ghost-ratings explained

    WebDiplomacy. Ghost-ratings explained. https://sites.google.com/view/ webdipinfo/ghost-ratings/ghost-ratings-explained , 2019. Accessed: 2019-05-01

  5. [13]

    Site scoring system

    super dipsy. Site scoring system. https://www.playdiplomacy.com/forum/viewtopic. php?f=565&t=34913, 2013. Accessed: 2019-05-01

  6. [14]

    Asynchronous methods for deep reinforcement learning

    V olodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lilli- crap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928–1937, 2016

  7. [15]

    Mastering the game of go with deep neural networks and tree search

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484, 2016

  8. [16]

    Mastering the game of go without human knowledge

    David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. Nature, 550(7676):354, 2017

  9. [17]

    Human-level performance in first-person multiplayer games with population-based deep rein- forcement learning

    Max Jaderberg, Wojciech M Czarnecki, Iain Dunning, Luke Marris, Guy Lever, Antonio Garcia Castaneda, Charles Beattie, Neil C Rabinowitz, Ari S Morcos, Avraham Ruderman, et al. Human-level performance in first-person multiplayer games with population-based deep rein- forcement l...

  10. [18]

    Dipblue: A diplomacy agent with strategic and trust reasoning

    André Ferreira, Henrique Lopes Cardoso, and Luis Paulo Reis. Dipblue: A diplomacy agent with strategic and trust reasoning. In ICAART 2015-7th International Conference on Agents and Artificial Intelligence, Proceedings, 2015

  11. [19]

    Dipgame: A testbed for multiagent systems

    Angela Fabregues, David Navarro, Alejandro Serrano, and Carles Sierra. Dipgame: A testbed for multiagent systems. In Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems: volume 1-Volume 1, pages 1619–1620. International Foundation for Au...

  12. [20]

    Negotiations over large agreement spaces, 2015

    Dave Jonge and Jordi González Sabaté. Negotiations over large agreement spaces, 2015

  13. [21]

    Learning a game strategy using pattern-weights and self-play

    Ari Shapiro, Gil Fuchs, and Robert Levinson. Learning a game strategy using pattern-weights and self-play. In International Conference on Computers and Games, pages 42–60. Springer, 2002

  14. [22]

    The evolution of cooperation

    Robert Axelrod and William D Hamilton. The evolution of cooperation. Science, 211(4489):1390–1396, 1981

  15. [23]

    Learning to communicate with deep multi-agent reinforcement learning

    Jakob Foerster, Ioannis Alexandros Assael, Nando de Freitas, and Shimon Whiteson. Learning to communicate with deep multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, pages 2137–2145, 2016

  16. [24]

    Intrinsic social motivation via causal influence in multi-agent rl

    Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro A Ortega, DJ Strouse, Joel Z Leibo, and Nando de Freitas. Intrinsic social motivation via causal influence in multi-agent rl. arXiv preprint arXiv:1810.08647, 2018

  17. [25]

    Inequity aversion improves cooperation in intertemporal social dilemmas

    Edward Hughes, Joel Z Leibo, Matthew Phillips, Karl Tuyls, Edgar Dueñez-Guzman, Anto- nio García Castañeda, Iain Dunning, Tina Zhu, Kevin McKee, Raphael Koster, et al. Inequity aversion improves cooperation in intertemporal social dilemmas. In Advances in Neural Infor- mation ...

  18. [26]

    Consequentialist conditional cooperation in social dilemmas with imperfect information

    Alexander Peysakhovich and Adam Lerer. Consequentialist conditional cooperation in social dilemmas with imperfect information. arXiv preprint arXiv:1710.06975, 2017

  19. [27]

    Behavioural game theory: thinking, learning and teaching

    Colin F Camerer, Teck-Hua Ho, and Juin Kuan Chong. Behavioural game theory: thinking, learning and teaching. In Advances in Understanding Strategic Behaviour , pages 120–180. Springer, 2004. 10

  20. [28]

    Semi-supervised classification with graph convolutional networks

    Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907, 2016

  21. [29]

    Film: Visual reasoning with a general conditioning layer

    Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reasoning with a general conditioning layer. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018

  22. [30]

    Weight normalization: A simple reparameterization to accelerate training of deep neural networks

    Tim Salimans and Durk P Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pages 901–909, 2016

  23. [31]

    Feature-wise transformations

    Vincent Dumoulin, Ethan Perez, Nathan Schucher, Florian Strub, Harm de Vries, Aaron Courville, and Yoshua Bengio. Feature-wise transformations. Distill, 3(7):e11, 2018

  24. [32]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016

  25. [33]

    Daide - clients

    David Norman. Daide - clients. http://www.daide.org.uk/clients.html, 2013. Ac- cessed: 2019-05-01

  26. [34]

    Emergent communication through negotiation

    Kris Cao, Angeliki Lazaridou, Marc Lanctot, Joel Z Leibo, Karl Tuyls, and Stephen Clark. Emergent communication through negotiation. arXiv preprint arXiv:1804.03980, 2018

  27. [35]

    Strategic information transmission

    Vincent P Crawford and Joel Sobel. Strategic information transmission. Econometrica: Journal of the Econometric Society, pages 1431–1451, 1982

  28. [36]

    Learning with opponent-learning awareness

    Jakob Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch. Learning with opponent-learning awareness. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems , pages 122–130. International Foun...

  29. [2013]

    Accessed: 2019-05-01

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.