Pith. sign in

REVIEW 2 major objections 5 minor 13 references

Artificial Generals Intelligence: Mastering Generals.io with Reinforcement Learning

T0 review · 2 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A reinforcement-learning agent trained for 36 hours reaches the top 25 human players on the Generals.io 1v1 ladder.

desk verdict A solid environment paper with a credible head-to-head result; the leaderboard claim needs data before it can be trusted. read the letter →

arxiv 2507.06825 v2 pith:MAFNDXAF submitted 2025-07-09 cs.LG cs.AI

Matej Straka , Martin Schmid This is my paper · ORCID
classification cs.LGcs.AI
keywords reinforcementlearningGenerals.ioreal-timestrategymulti-agentself-playrewardshapingpartialobservabilityfogofwar
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces a lightweight, open real-time strategy environment based on Generals.io and a reference agent that demonstrates the environment's reach. The agent, trained with supervised behavior cloning followed by self-play PPO fine-tuning, reached the top 0.003% of the human 1v1 leaderboard after 36 hours on a single H100 GPU. In a direct comparison it beat the previous best bot, Human.exe, with a 54.82% win-rate over 529 games. The authors argue that the combination of potential-based reward shaping and hand-crafted memory features is what keeps the self-play agent from collapsing into an aggressive but exploitable style. If right, this makes a fog-of-war RTS with thousands of active human players an accessible benchmark for multi-agent reinforcement learning.

What carries the argument

The load-bearing mechanism is the training pipeline: a U-Net policy network (the same architecture used for the board game Stratego) processes an observation tensor augmented by a hand-built memory stack, and is trained by behavior cloning then self-play with Proximal Policy Optimization. The component that prevents degenerate aggressive play is potential-based reward shaping, where the shaped reward is $r_{\text{shaped}}(s,a,s') = r_{\text{original}}(s,a,s') + \gamma\varphi(s') - \varphi(s)$, with potential $\varphi(s) = 0.3\varphi_{\text{land}}(s) + 0.3\varphi_{\text{army}}(s) + 0.4\varphi_{\text{castle}}(s)$ and each sub-potential the log-ratio of the agent's material to the enemy's, normalized to $[-1,1]$. This shaping preserves the optimal policy while steering learning toward building material advantage. The memory stack encodes revealed castle and general positions, explored cells, cells the opponent has seen, and the last seven moves of each side, giving the feed-forward network a limited form of memory under fog of war.

What would settle it

Run the trained agent under a fixed nickname for an entire Generals.io 1v1 season, recording every opponent rating and game timestamp; the claim fails if its final percentile is outside the top 0.003% or if its win rate against a pre-registered panel of top-50 human players is at or below 50%.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that a reinforcement-learning agent can reach human expert level in Generals.io with modest compute: after behavior cloning on 16,320 filtered replays and 36 hours of self-play with PPO on one H100 GPU, the final population-trained agent played 4,700 online games under two nicknames and consistently ranked in the top 0.003% of the human 1v1 leaderboard, placing among the top 25 players. It also achieved a 54.82% win-rate against the prior state-of-the-art bot Human.exe across 529 games, with a 95% Wilson confidence interval of 50.56% to 59.01%, and the paper constructs an Elo table placing this agent above that bot. The same section reports that reward shaping is essential: a naive self-play agent is beaten 71.9% of the time by the shaped agent. The authors present the environment itself as a second contribution, emphasizing its speed (about 3,500 frames per second on a 12-core CPU with 12 parallel environments) and compatibility with standard multi-agent interfaces.

Load-bearing premise

The top-25 leaderboard claim rests on the official Generals.io matchmaking and rating system being a fair measure of skill and on the 4,700 online games being a representative, uncherry-picked sample.

Editorial extensions

If this is right

  • A fog-of-war real-time strategy game can serve as a practical MARL benchmark: the environment runs thousands of frames per second and trains a top-ladder agent in 36 hours on one H100 GPU.
  • Policy invariance of potential-based reward shaping transfers cleanly to this setting, preventing the self-play collapse to pure aggression and improving win-rates by large margins in pairwise tests.
  • Because the trained policy can be deployed directly to official Generals.io servers, agent strength can be measured against a large active human population rather than only against other bots.
  • The agent's emergent sidesteps, snowballing, and backdooring provide concrete behavioral evidence that the strategic depth of the game survives in the learned policy.
  • The pairwise Elo table implies the reference agent sits above the prior heuristics-only state of the art, establishing a baseline that future agents must beat.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors leave implicit that the same recipe, behavior cloning from human replays, self-play, and potential-based shaping, may transfer to other fog-of-war RTS games, since none of the components is specific to Generals.io.
  • Because the paper reports no rating values, opponent rating distribution, or temporal window for the 4,700 online games, the top-25 claim is only as reliable as the unofficial matchmaking sample; a controlled season-long evaluation against a fixed panel of top-50 humans would make the skill claim directly testable.
  • Replacing the hand-crafted memory stack with a recurrent or transformer memory would likely improve play further, because the agent currently does not remember when it revealed map regions or what the army counts were at that time.
  • The environment's speed and API compatibility could make it a convenient testbed for studying general-sum dynamics and multi-team play once free-for-all and 2v2 modes are added, which the paper lists as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper presents a new reinforcement learning environment based on the browser game Generals.io, implemented with Gymnasium and PettingZoo compatibility and capable of running thousands of frames per second on commodity hardware. The authors also train a reference agent using behavior cloning followed by PPO self-play with potential-based reward shaping and hand-crafted memory features. This agent reportedly reached the top 0.003% of the human 1v1 leaderboard, and in a head-to-head evaluation defeated the prior state-of-the-art bot Human.exe with a 54.82% win rate over 529 games (95% Wilson CI 50.56%–59.01%). The paper further provides an ablation study with 2,000-game pairwise comparisons and an Elo table among existing bots.

Significance. The environment is a valuable contribution to the multi-agent RL benchmark ecosystem: it is lightweight, fast, and offers strategic complexity comparable to larger RTS games, while being far more accessible computationally. The evaluation methodology is generally careful, with Wilson confidence intervals for the headline head-to-head result and a substantial sample size for the ablation study. If the leaderboard claim can be substantiated, this would be a notable demonstration of a relatively low-cost RL system reaching human-expert level in a competitive game. Strengths of the work include the public release of code, tutorials, and replay examples, as well as explicit acknowledgment of limitations such as the need for recurrent memory.

major comments (2)
  1. [Section 6, first paragraph (and Abstract)] The claim that the agent 'consistently ranked within the top 0.003% on the human 1v1 leaderboard, placing it among the top 25 players' is not supported by the evidence presented. The paper provides no rating values, no distribution of opponent ratings, no game-selection criterion, and no temporal window for the 4,700 online games. Furthermore, rank 25 at the 0.003rd percentile implies a leaderboard of roughly 833,000 accounts, which is inconsistent with the paper's own statement that Generals.io has 'thousands of weekly active players' unless the denominator is all registered accounts rather than active players. The authors should clarify the denominator, report the actual leaderboard position and rating, and document how the 4,700 games were selected. This claim is load-bearing for the paper's central 'top-tier human-level' conclusion, so the manuscript should provide the missing data or substantially temper the claim.
  2. [Section 6, second paragraph] The head-to-head result against Human.exe is credible and the Wilson CI is appropriately reported, but the paper does not specify the conditions under which the 529 games were played—for example, whether they were played on the official Generals.io server or in the local environment, the map sizes and time controls, and whether both sides operated under identical computational constraints. Without this information, the result is difficult to reproduce or compare across future work.
minor comments (5)
  1. [Figure 2 caption] The caption lists '(3) point of view of the blue player' but should be '(c) point of view of the blue player' to match the panel labels (a), (b), (c).
  2. [Section 5, Behavior Cloning] The sentence 'We further resort to games played on the newest patch of the game' uses 'resort' where 'restrict' is intended.
  3. [Section 5, Self-Play] The replacement criterion 'a 45 % win-rate versus the current pool (approximately 55 % with both sides using arg max)' is confusing; please clarify which side is stochastic in the 45% figure and which is using arg max.
  4. [Section 6, Emergent Behaviors] The replay references [GR:1] through [GR:12] are not explained in the text; please provide a footnote or link explaining how readers can access these replays.
  5. [Section 3, Movement and Combat] The statement that 'each half-turn lasts 500ms online' is ambiguous; please specify whether this is the interval between a player's move submissions, the game's turn duration, or the duration of a full round.

Circularity Check

0 steps flagged · score 1.0 of 10

Agent performance is measured externally against humans and prior bots; no load-bearing circular step found, only hand-tuned shaping weights and selected emergent examples that do not drive the central claim.

full rationale

The paper's central claims are (a) a benchmark environment with open source code, (b) a reference agent trained by behavior cloning plus self-play, and (c) empirical performance against human opponents on the official Generals.io leaderboard and against existing bots (Human.exe, Flobot, HASP). The win-rate claims against Human.exe (54.82%, Wilson CI 50.56–59.01) and the leaderboard percentile (top 0.003%, top 25) are external measurements, not quantities derived from the paper's own fitted parameters. Behavior cloning uses external replays filtered by human star ratings; self-play win-rate thresholds (45% vs. pool) are training criteria, and the final agent's wins are measured against external humans, so the evaluation is not statistically forced by the training setup. The potential-based reward shaping (phi = 0.3 land + 0.3 army + 0.4 castle) is theoretically motivated by Ng et al. (1999) policy invariance, and the specific weights are hand-chosen; while hand-tuning shaping to make training work is a mild concern for the generality of the ablation results, it does not make the external performance measurements circular, because the training signal and the evaluation signal are distinct (shaping vs. win/lose against humans). The emergent behaviors (feints, snowballing, backdooring) are selected examples rather than systematic measurements, but they are presented as qualitative showcases, not as the load-bearing evidence for the top-tier claim. No self-citation chain is load-bearing: the paper cites Perolat et al. (2022) for the U-Net architecture, which is an external architectural choice, not an imported uniqueness theorem. The related-work comparison to HASP (Xu et al., 2018) is used only as a baseline for the Elo table, not as justification for the paper's method. The only vaguely self-referential element is the Elo table anchored by Flobot at 1500, which is a standard calibration choice and is arithmetically derived from measured pairwise win-rates; it does not reduce a prediction to an input. One might note that the 'top 0.003%' claim is arithmetically fragile (rank 25 / 0.00003 = ~833k accounts) and lacks opponent rating distribution, but that is a reporting/correctness concern about unverified matchmaking, not circularity. Overall, the derivation chain is self-contained against external benchmarks, so the circularity score is 1 (small residual due to hand-tuned shaping and selective examples).

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central performance claim depends on a small number of hand-chosen parameters and unverified assumptions about the official game server and dataset quality. These are listed so readers can see what is supplied by the paper versus what is pulled from the environment.

free parameters (6)
  • Reward shaping potential weights (phi_land, phi_army, phi_castle) = 0.3, 0.3, 0.4
    Hand-chosen weights for the potential function in Section 5; no sensitivity analysis is provided, and they directly shape the training signal.
  • max_ratio normalization constant in reward shaping = Unspecified
    Used to bound each log-ratio in [-1,1]; the value is not stated, making exact reproduction harder.
  • Self-play opponent pool size N = N=3
    Selected as the pool size for the top model; no systematic study of this hyperparameter is reported.
  • Candidate replacement win-rate threshold = 45%
    Threshold for replacing oldest model in the pool; chosen without reported sensitivity analysis.
  • Behavior cloning dataset filters: max game length and minimum player rating = 500 turns (1000 moves), 70-star rating
    Post hoc filters on the replay dataset; the paper asserts long games are suboptimal without quantitative justification.
  • GAE lambda = 0.95
    Standard GAE hyperparameter, listed as a design choice.
assumptions (4)
  • domain assumption The official Generals.io rating system and leaderboard are a valid measure of skill, and the 4,700 match results are representative of human performance.
    The top 0.003% claim in Section 6 relies on the leaderboard without providing rating curves or opponent distributions.
  • ad hoc to paper Games longer than 500 turns are atypical and often suboptimal, so excluding them improves training data quality.
    Stated in Section 5 Behavior Cloning with qualitative justification only; this filtering changes the training distribution.
  • domain assumption The game's fog-of-war and official server interactions were modeled faithfully enough that trained policies transfer directly to official servers.
    Section 4 claims deployment to official servers; this is central to the leaderboard and Human.exe evaluations.
  • standard math Potential-based reward shaping preserves the optimal policy set as per Ng et al. (1999).
    Standard result invoked in Section 5 Reward Shaping.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Artificial Generals Intelligence: Mastering Generals.io with Reinforcement Learning." pith.science (2026). https://pith.science/paper/MAFNDXAF

@misc{pith2026250706825,
  author       = {Pith},
  title        = {Pith review of: Artificial Generals Intelligence: Mastering Generals.io with Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MAFNDXAF}},
  note         = {Machine review of arXiv:2507.06825}
}
read the original abstract

We introduce a real-time strategy game environment based on Generals.io, a game with thousands of weekly active players. Our environment is fully compatible with Gymnasium and PettingZoo and is capable of running thousands of frames per second on commodity hardware. We also present a reference agent, trained with supervised pre-training and self-play, which reached the top 0.003% of the 1v1 human leaderboard after only 36 hours on a single H100 GPU. To accelerate learning, we incorporate potential-based reward shaping and memory features. Our contributions of a modular RTS benchmark and a competitive baseline agent provide an accessible yet challenging platform for advancing multi-agent reinforcement learning research. The documented code, together with examples and tutorials, is available at https://github.com/strakam/generals-bots.

Figures

Figures reproduced from arXiv: 2507.06825 by the authors.

Figure 1
Figure 1. Three zoom levels of the same game state are shown. Cells of different colors belong to [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Three views of the same game state: (a) perfect information view; (b) point of view of the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The environment produces game observation encoded as a 3D tensor. This observation [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 linked inside Pith

  1. [7]

    DOI: 10.1126/science.add4679

    ISSN 1095-9203. DOI: 10.1126/science.add4679. URL http://dx.doi.org/10.1126/science.add4679. Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim G. J. Rudner, Chia-Man Hung, Philip H. S. Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge,

    doi:10.1126/science.add4679

  2. [10]

    URL https: //arxiv.org/abs/1506.02438. David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. Mastering chess and shogi by self-play with a general reinforce- ment learning algorithm,

    arXiv

  3. [11]

    URL https://arxiv.org/abs/1712.01815. J. K. Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis Santos, Rodrigo Perez, Caroline Horsch, Clemens Dieffendahl, Niall L. Williams, Yashas Lokesh, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learning,

    arXiv

  4. [13]

    org/abs/2407.17032

    URL https://arxiv. org/abs/2407.17032. Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782):350–354,

    arXiv

  5. [2013]

    Matej Morav ˇcík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling

    URL https://arxiv.org/abs/1312.5602. Matej Morav ˇcík, Martin Schmid, Neil Burch, Viliam Lisý, Dustin Morrill, Nolan Bard, Trevor Davis, Kevin Waugh, Michael Johanson, and Michael Bowling. Deepstack: Expert-level artificial intelligence in heads-up no-limit poker. Science, 356(6337):508–513, May

    arXiv

  6. [2017]

    John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel

    URL https://arxiv.org/abs/1707.06347. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation,

    arXiv

  7. [2018]

    URL https://www.science.org/doi/abs/10.1126/science.aao1733

    DOI: 10.1126/science.aao1733. URL https://www.science.org/doi/abs/10.1126/science.aao1733. Wenzhe Li, Zihan Ding, Seth Karten, and Chi Jin. Fightladder: A benchmark for competitive multi- agent reinforcement learning,

    doi:10.1126/science.aao1733

  8. [2019]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov

    URL https://arxiv.org/abs/ 1902.04043. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms,

    arXiv 1902

Show all 13 references
  1. [2020]

    org/abs/1909.07528

    URL https://arxiv. org/abs/1909.07528. Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław D˛ ebiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv...

    1909 arXiv

  2. [2021]

    9 Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U

    URL https://arxiv.org/abs/2009.14471. 9 Mark Towers, Ariel Kwiatkowski, Jordan Terry, John U. Balis, Gianluca De Cola, Tristan Deleu, Manuel Goulão, Andreas Kallinteris, Markus Krimmel, Arjun KG, Rodrigo Perez-Vicente, An- drea Pierré, Sander Schulhoff, Jun Jet Tai, Hannah Tan...

    2009 arXiv

  3. [2022]

    URL https://www.science.org/ doi/abs/10.1126/science.ade9097

    DOI: 10.1126/science.ade9097. URL https://www.science.org/ doi/abs/10.1126/science.ade9097. V olodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning,

    doi

  4. [2024]

    Meta Fundamental AI Research Diplomacy Team

    URL https://arxiv.org/abs/2406.02081. Meta Fundamental AI Research Diplomacy Team. Human-level play in the game of <i>diplomacy</i> by combining language models with strategic reasoning. Science, 378(6624): 1067–1074,

    arXiv

  5. [9203]

    URL http://dx.doi.org/10.1126/science

    DOI: 10.1126/science.aam6960. URL http://dx.doi.org/10.1126/science. aam6960. Andrew Y Ng, Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Icml, volume 99, pp. 278–287. Citeseer,

    doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.