Pith. sign in

REVIEW 5 major objections 5 minor 16 references

PolicyEvolve: Evolving Programmatic Policies by LLMs for multi-player games via Population-Based Training

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PolicyEvolve claims that an LLM-driven population-based loop can evolve interpretable rule-based policies for two-player zero-sum games, beating prompt-only baselines while using very few environment interactions.

desk verdict PolicyEvolve is a plausible extension of LLM-generated programmatic RL to multi-agent self-play, but the evaluation is too thin to support the headline claims. read the letter →

arxiv 2509.06053 v1 pith:IICL7AUI submitted 2025-09-07 cs.LG cs.AI

Mingrui Lv , Hangzhi Liu , Zhi Luo , Hongjie Zhang , Jie Ou This is my paper · ORCID
classification cs.LGcs.AI
keywords programmaticreinforcementlearningLLMcodegenerationpopulation-basedtrainingmulti-agentgamesself-playinterpretablepoliciespolicyevolutionzero-sum
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PolicyEvolve addresses a gap left by programmatic reinforcement learning: prior LLM-based policy-code methods target single-agent tasks with static environments, while multi-player games require policies that keep adapting to changing opponents. The paper's claim is that a population-based training loop—an elite Global Pool, a temporary Local Pool, a Policy Planner that writes and revises Python policy code, and a Trajectory Critic that reads match logs and proposes fixes—can produce interpretable code policies for a two-player zero-sum game with minimal environmental interaction. If true, this means multi-agent strategies can be obtained as white-box, executable rules rather than opaque neural networks, with sample counts orders of magnitude below typical MARL. Experiments on a wrestling-style game with several LLMs show the evolved policy's ELO rising across 20 iterations and beating naive, chain-of-thought, and ReAct-style prompting baselines in head-to-head matches. The paper also reports ablations showing that auxiliary environment hints, reflection-based iteration, and persistent reflection memory each contribute to win rate.

What carries the argument

The central mechanism is a population-based self-play loop organised around two policy pools. The Global Pool keeps the accumulated elite programmatic policies, ranks them by ELO from pairwise matches, and supplies opponents; the Local Pool stores the current round's candidate policies and their match data. The Policy Planner combines the three highest-ELO policies, environment descriptions, and stored reflection memories to generate and debug code, while the Trajectory Critic converts collected trajectories into structured flaw-location-suggestion reflections and periodically summarises the reflection memory. The 60% win-rate threshold, together with the ELO update rule, converts noisy matc

What would settle it

Independent replication with fixed random seeds and match-count sweeps: if running PolicyEvolve several times on the same task yields final policies whose ELO and pairwise win rates against baselines vary widely, or if promotion decisions change materially when the evaluation is recomputed with 100 matches instead of 10, then the observed stable improvement is an artifact of evaluation noise rather than genuine policy evolution.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that an LLM can serve as both the policy generator and the critic in an evolutionary loop for multi-agent games. Starting from a Global Pool seeded with a random policy, the Policy Planner writes an initial rule-based policy from an environment description; the policy plays against opponents sampled from the pool; the Trajectory Critic compresses the match data into concrete failure analyses and code-level improvement suggestions; and the Policy Planner rewrites the code accordingly. This local loop repeats until the policy's average win rate over 10 matches against sampled pool opponents exceeds 60%, at which point the policy enters the Glo

Load-bearing premise

The whole evolutionary ladder rests on the assumption that a policy's average win rate over 10 random matches against a handful of sampled opponents is a trustworthy measure of its true strength relative to the Global Pool.

Editorial extensions

If this is right

  • If the claims hold, interpretable programmatic policies become a practical alternative to neural MARL in zero-sum games, since the policies are code that can be inspected and audited.
  • The approach promises a dramatic cut in sample requirements: instead of millions of environment steps, evolutionary pressure is applied through a few dozen or hundred matches per iteration.
  • The framework should transfer to other two-player zero-sum tasks where an LLM can be given a natural-language description and a Python action interface, because no task-specific neural training is needed.
  • Because each promotion adds a policy to the Global Pool, the pool itself is a growing curriculum of historical opponents, which should make the final policy resilient to diverse play styles.
  • The reported 100% pairwise win rates against baselines suggest that the evolutionary loop, not just the underlying LLM, is what produces the performance gain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The win-rate signal is computed from only 10 matches per opponent set; a natural extension is to make the number of matches adaptive, increasing the count when the estimate is close to the 60% threshold, which would reduce the risk that noisy evaluations promote or discard policies by luck.
  • The reflection-memory simplification step is essentially a token-budget compression technique; it could be carried over to other LLM agent loops where long histories cause context and cost to grow.
  • The same dual-pool architecture may extend to cooperative or mixed-motive games, where the 'win rate' objective would be replaced by a team payoff or social welfare signal, though the paper does not test this.
  • One testable prediction is that the ELO-versus-iteration curve will be monotone in expectation even with weaker LLMs; if that holds, the framework's gains are structural rather than dependent on a particular model.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. PolicyEvolve is a framework for generating programmatic (code-based) policies for multi-player games by iteratively prompting LLMs. It maintains a Global Pool of elite policies, a Local Pool of candidate policies, a Policy Planner that generates/debugs code, and a Trajectory Critic that analyzes match logs and proposes improvements. Candidates play L=10 matches against sampled opponents; those exceeding a 60% win-rate threshold are promoted. Experiments are conducted on the Wrestle task with three LLMs (Hunyuan, Qwen, DeepSeek) and compared against Random, Naive, CoT, and React baselines. The paper reports that the evolved policy achieves steadily increasing ELO, the highest pairwise win rate, and better sample efficiency, with ablations on prompt components and a toy example.

Significance. If the empirical claims were well supported, PolicyEvolve would be a useful contribution: it produces interpretable rule-based policies with a self-play-style training loop driven by LLMs, and it targets a genuine gap (LLM-generated programmatic policies for multi-agent, not single-agent, settings). The framework is modular and the ablation study addresses relevant prompt-design questions. However, the current evidence is not sufficient to establish the central claims. The evaluation rests on very noisy win-rate estimates, lacks seeds/error bars, uses a single environment, and reports an internally inconsistent summary table. With code and data release and additional experiments, the idea could be compelling, but as submitted the empirical support is too thin for the strength of the claims made in the abstract and Section 1.

major comments (5)
  1. [§3.1, Eq. (3); §2.1, §2.4, §3.2, §3.4] The promotion decision and all downstream results are driven by win rates computed from L=10 matches. With a true 60% win probability, the standard error of the estimated win rate is sqrt(0.6*0.4/10) ≈ 0.155; a truly 50% policy has roughly a 38% chance of winning 6 or more of 10 matches. Thus the 60% promotion threshold and the ELO/softmax sampling in Eqs. (1)–(2) can be dominated by sampling noise. Also, Eq. (3) defines WR as the average score over s∈{-1,0,1}, so the text's '60% win rate' is ambiguous: literal average score >0.6 corresponds to ~80% wins with no draws, whereas a 60% win fraction would be a different threshold. This ambiguity and the noise affect Fig. 7, Table 1, and the 'clear improvement' from 0.47 to 0.55 in §3.4, which is within sampling error. Please report multiple seeds, confidence intervals, and ideally a larger L or paired statistical tests.
  2. [Table 1] Table 1 is internally inconsistent. The caption says each cell is the win rate of the column policy, but the Random row appears to give a value of 0.54 for Random vs. Random; a policy should have win rate 0.5 against itself. Moreover, the matrix is asymmetric but no match counts or confidence intervals are given, so entries such as 1.00 cannot be distinguished from, e.g., 0.95 with L=10. The table should be corrected and reported with uncertainty or at least the number of matches.
  3. [Abstract, §1, §3.1] The abstract and Section 1 claim 'extensive experiments on multi-player game tasks' and a general framework, but the entire evaluation is on a single task (Wrestle). No second environment, game, or task is reported, so the generality claim is unsupported. Please either add additional environments or substantially temper the claims regarding multi-player games.
  4. [Abstract, §1, §3.2] The paper repeatedly claims superior sample efficiency and 'minimal environmental interactions', but no measure of environmental interactions is reported: no number of episodes, steps, or matches per iteration, and no comparison of total interaction counts against baselines or MARL methods. This claim is therefore not supported by the presented data. Please define and report the relevant metric.
  5. [§3.1, §3.2] Reproducibility is insufficient. No code or data are provided, no seeds are stated, LLM versions are not specified (only 'Hunyuan, Qwen, DeepSeek'), and baseline details are incomplete—for example, React is described as pitting a generated policy against Random, but the number of iterations, temperature, prompt templates, and opponent sampling are not specified. The comparison may conflate the proposed method's advantages with confounded baseline implementations. Please provide code/data, seeds, model identifiers, and full baseline settings.
minor comments (5)
  1. [Throughout] There are numerous typos and grammatical errors, e.g., 'polcies', 'Wrestletask', 'thier', and inconsistent capitalization. A careful proofread is needed.
  2. [§2.2, §3.1] The hyperparameters n (number of policies sampled), t (number of evaluations), and the ELO K-factor are mentioned but not given concrete values or tuned. Please report all hyperparameters explicitly.
  3. [§3.1] The 'JIDI platform' and the Wrestle environment are not cited or described with a public reference; if the environment is not publicly available, this further limits reproducibility.
  4. [Fig. 7, Fig. 8, Fig. 9] None of the figures show error bars or confidence bands. Given that all win-rate estimates come from L=10 matches, the visual claims of 'stable improvement' and 'ablation benefit' cannot be assessed.
  5. [Contributions, §1] The claim that PolicyEvolve is 'the first programmatic reinforcement learning framework specifically designed for multi-agent tasks' is a strong priority claim and is not supported by a systematic literature comparison. Please soften or substantiate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical comparisons, not derivations that reduce to their own inputs.

full rationale

PolicyEvolve's central claim is that its evolution loop produces policies that outperform prompt-based baselines in the Wrestle environment. This is an empirical result: policies are admitted based on measured win rates (Eq. 3) and compared in head-to-head ELO evaluations (Eq. 1, Fig. 7/8, Table 1) against baselines generated independently by Naive/CoT/React prompts. There is no equation in the paper that defines a predicted quantity in terms of an input and then presents that definition as a discovery; the win rate is a measurement, and the 60% promotion threshold is a selection rule, not an output to be validated. The paper contains no self-citations, no imported uniqueness theorem, and no ansatz smuggled in via citation; references [11]-[16] are external and none is load-bearing in a way that makes the central result depend on the authors' own prior claims. The ELO improvement across iterations (Fig. 7) is partly a designed consequence of the promotion rule, since only policies with at least 60% win rate enter the Global Pool, but that is a property of population-based selection, not circular reasoning: the final claim is judged against independent baselines. The main weaknesses, such as L=10 matches per win-rate estimate, missing variance/seeds, and single-environment evaluation, are evidence-quality concerns that affect reliability but do not make the derivation circular.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

No new physical entities are postulated. All parameters are hyperparameters chosen by the authors; none are fitted by an external benchmark. The framework's components (Global Pool, Local Pool, Reflection Memory) are software modules, not invented physical entities.

free parameters (7)
  • Win-rate evaluation matches L = 10
    Number of random matches per policy comparison reported in Eq. 3; controls the reliability of all reported win rates.
  • Promotion threshold = 60%
    Section 2.1: policy promoted to Global Pool when win rate exceeds 60%; affects pool composition and final policy quality.
  • ELO initial score = 1200
    Section 2.2: newly admitted policies are assigned initial ELO 1200; ranking and opponent sampling depend on this choice.
  • ELO adjustment factor K = 32
    Section 2.2: K=32 controls rating update magnitude; not reported as tuned.
  • Global Pool sample size n = not reported
    Section 2.2: 'randomly samples n policies' for evaluation; n determines evaluation cost and coverage.
  • Opponent count k = not reported
    Section 2.5: softmax sampling of k opponents from Global Pool; k affects diversity of training.
  • Evolution iterations = 20
    Section 3.2: results after 20 improvement rounds; no convergence analysis or sensitivity.
assumptions (5)
  • domain assumption LLMs can generate executable, effective policy code from natural language environment descriptions and code feedback.
    The entire framework depends on this capability; no formal guarantee, only empirical results.
  • domain assumption The Wrestle task is representative of multi-player games.
    General claim of a 'general framework for multi-player games' rests on a single zero-sum environment; generalizability is assumed.
  • domain assumption Win rate over L=10 matches is a reliable estimator of policy strength.
    Used for promotion decisions and ELO updates; no confidence intervals reported.
  • domain assumption Self-play against the Global Pool transfers to unseen opponents.
    Final evaluation includes baselines, but training uses only Global Pool opponents; transfer is assumed.
  • standard math ELO rating computed via Eq. 1 is a valid ranking for comparing policies.
    Standard ELO, but its validity depends on match volume and stationary opponents, which is not checked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PolicyEvolve: Evolving Programmatic Policies by LLMs for multi-player games via Population-Based Training." pith.science (2026). https://pith.science/paper/IICL7AUI

@misc{pith2026250906053,
  author       = {Pith},
  title        = {Pith review of: PolicyEvolve: Evolving Programmatic Policies by LLMs for multi-player games via Population-Based Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IICL7AUI}},
  note         = {Machine review of arXiv:2509.06053}
}
read the original abstract

Multi-agent reinforcement learning (MARL) has achieved significant progress in solving complex multi-player games through self-play. However, training effective adversarial policies requires millions of experience samples and substantial computational resources. Moreover, these policies lack interpretability, hindering their practical deployment. Recently, researchers have successfully leveraged Large Language Models (LLMs) to generate programmatic policies for single-agent tasks, transforming neural network-based policies into interpretable rule-based code with high execution efficiency. Inspired by this, we propose PolicyEvolve, a general framework for generating programmatic policies in multi-player games. PolicyEvolve significantly reduces reliance on manually crafted policy code, achieving high-performance policies with minimal environmental interactions. The framework comprises four modules: Global Pool, Local Pool, Policy Planner, and Trajectory Critic. The Global Pool preserves elite policies accumulated during iterative training. The Local Pool stores temporary policies for the current iteration; only sufficiently high-performing policies from this pool are promoted to the Global Pool. The Policy Planner serves as the core policy generation module. It samples the top three policies from the Global Pool, generates an initial policy for the current iteration based on environmental information, and refines this policy using feedback from the Trajectory Critic. Refined policies are then deposited into the Local Pool. This iterative process continues until the policy achieves a sufficiently high average win rate against the Global Pool, at which point it is integrated into the Global Pool. The Trajectory Critic analyzes interaction data from the current policy, identifies vulnerabilities, and proposes directional improvements to guide the Policy Planner

Figures

Figures reproduced from arXiv: 2509.06053 by the authors.

Figure 1
Figure 1. The differences in using PRL to solve single-agent versus multi-agent tasks. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. shows a high-level overview of the PolicyEvolve framework. Specifically, PolicyEvolve comprises four modules: Global Pool, Local Pool, Policy Planner, and Trajectory Critic. Centered around population-based training, the Global Pool preserves elite populations. It continuously samples multiple policies for adversarial evaluations, ranking them via an ELO scoring mechanism to guide subsequent opponent sampling and ne… view at source ↗
Figure 3
Figure 3. Comparison of DRL and PRL in utilizing empirical data for policy updates. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Initial policy generation process and prompts. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Reflection Memory structure for the wrestle task before and after simplification. [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: The global view, agent’s local view, and action space for the wrestle task. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: The ELO score gradually improves with policy iteration. [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Comparison of five policies in head-to-head battles, showing ELO scores and code sizes. [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]
Figure 9
Figure 9. Figure 9: Win Rate Comparison with different prompt tactics [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 13 canonical work pages

  1. [1]

    Reinforcement learning in robotics: A survey.The International Journal of Robotics Research, 32(11):1238–1274, 2013

    Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey.The International Journal of Robotics Research, 32(11):1238–1274, 2013

    2013

  2. [2]

    Sim-to-real robot learning from pixels with progressive nets

    Andrei A Rusu, Matej Veˇcerík, Thomas Rothörl, Nicolas Heess, Razvan Pascanu, and Raia Hadsell. Sim-to-real robot learning from pixels with progressive nets. InConference on robot learning, pages 262–270. PMLR, 2017

    work page 2017

  3. [3]

    Google research football: A novel reinforcement learning environment

    Karol Kurach, Anton Raichuk, Piotr Sta´nczyk, Michał Zaj ˛ ac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 4501–4510, 2020. 13 APREPRINT- SEP...

    work page 2020

  4. [4]

    A comprehensive review of multi-agent reinforcement learning in video games.IEEE Transactions on Games, 2025

    Zhengyang Li, Qijin Ji, Xinghong Ling, and Quan Liu. A comprehensive review of multi-agent reinforcement learning in video games.IEEE Transactions on Games, 2025

    work page 2025

  5. [5]

    Robustness and sample complexity of model-based marl for general-sum markov games.Dynamic Games and Applications, 13(1):56–88, 2023

    Jayakumar Subramanian, Amit Sinha, and Aditya Mahajan. Robustness and sample complexity of model-based marl for general-sum markov games.Dynamic Games and Applications, 13(1):56–88, 2023

    work page 2023

  6. [6]

    Multi-agent reinforcement learning for autonomous driving: A survey.CoRR, 2024

    Ruiqi Zhang, Jing Hou, Florian Walter, Shangding Gu, Jiayi Guan, Florian Röhrbein, Yali Du, Panpan Cai, Guang Chen, and Alois Knoll. Multi-agent reinforcement learning for autonomous driving: A survey.CoRR, 2024

    work page 2024

  7. [7]

    Deep reinforcement learning for autonomous driving: A survey.IEEE transactions on intelligent transportation systems, 23(6):4909–4926, 2021

    B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey.IEEE transactions on intelligent transportation systems, 23(6):4909–4926, 2021

    2021

  8. [8]

    Equilibrium selection for multi-agent reinforcement learning: A unified framework.arXiv preprint arXiv:2406.08844, 2024

    Runyu Zhang, Jeff Shamma, and Na Li. Equilibrium selection for multi-agent reinforcement learning: A unified framework.arXiv preprint arXiv:2406.08844, 2024

    work page arXiv 2024

Show all 16 references
  1. [9]

    Emergent reciprocity and team formation from randomized uncertain social preferences.Advances in neural information processing systems, 33:15786–15799, 2020

    Bowen Baker. Emergent reciprocity and team formation from randomized uncertain social preferences.Advances in neural information processing systems, 33:15786–15799, 2020

    2020

  2. [10]

    Taxai: A dynamic economic simulator and benchmark for multi-agent reinforcement learning

    Q Mi, S Xia, Y Song, H Zhang, S Zhu, and J Wang. Taxai: A dynamic economic simulator and benchmark for multi-agent reinforcement learning. InProceedings of the International Joint Conference on Autonomous Agents and Multiagent Systems, AAMAS, pages 1390–1399, 2024

    2024

  3. [11]

    A new approach to solving smac task: Generating decision tree code from large language models.arXiv e-prints, pages arXiv–2410, 2024

    Yue Deng, Weiyu Ma, Yuxin Fan, Yin Zhang, Haifeng Zhang, and Jian Zhao. A new approach to solving smac task: Generating decision tree code from large language models.arXiv e-prints, pages arXiv–2410, 2024

    2024

  4. [12]

    Adrd: Llm-driven autonomous driving based on rule-based decision systems.arXiv preprint arXiv:2506.14299, 2025

    Fanzhi Zeng, Siqi Wang, Chuzhao Zhu, and Li Li. Adrd: Llm-driven autonomous driving based on rule-based decision systems.arXiv preprint arXiv:2506.14299, 2025

    2025 arXiv

  5. [13]

    Language models speed up local search for finding programmatic policies.Transactions on Machine Learning Research, 20(X), 2024

    Quazi Asif Sadmine, Hendrik Baier, and Levi Lelis. Language models speed up local search for finding programmatic policies.Transactions on Machine Learning Research, 20(X), 2024

    2024

  6. [14]

    Synthesizing programmatic reinforcement learning policies with large language model guided search

    Max Liu, Chan-Hung Yu, Wei-Hsu Lee, Cheng-Wei Hung, Yen-Chun Chen, and Shao-Hua Sun. Synthesizing programmatic reinforcement learning policies with large language model guided search. InThe Thirteenth International Conference on Learning Representations

  7. [15]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  8. [16]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023. 14

    2023

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.