REVIEW 4 major objections 5 minor 1 cited by
Learning Game-Playing Agents with Generative Code Optimization
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read An LLM-guided optimizer learns Atari policies as Python programs, reaching scores competitive with deep RL baselines in a fraction of the training time.
desk verdict A clear demonstration of LLM-refined programmatic policies on Atari, but the headline efficiency claim rests on an apples-to-oranges comparison: the agent sees object coordinates while the baselines see pixels. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two components carry the argument. Object-centric Atari representations, provided by OCAtari, replace raw pixels with named game objects carrying position, size, velocity, and status fields, so the LLM operates on compact semantics instead of thousands of pixels. Execution traces and the generative optimizer, provided by the Trace framework and OptoPrime, record a rollout's dataflow and rewrite the functions decorated as trainable, using both a scalar reward from a full ~4000-step evaluation and staged natural-language feedback as the optimization signal. The feedback is deliberately staged by performance tier so that short context-limited rollouts still receive long-horizon guidance.
What would settle it
Rerun CleanRL DQN and PPO on the same machine and evaluation protocol used in the paper, and compare wall-clock time to reach the Table 2 scores. If standard RL reaches Pong 21, Breakout 353, and Space Invaders 1200 in times comparable to or shorter than 36–91 minutes, the central efficiency claim collapses; if not, it holds.
Extended reading notes
Core claim
The central claim is that decision-making policies can be optimized as source code rather than as tensors: the agent is a modular Python class whose action-selection functions are marked trainable, and each training iteration rolls out a traced episode, computes a scalar reward plus natural-language feedback from a longer evaluation run, and lets an LLM optimizer rewrite the code. In Pong the learned program reaches 21 (against 19 PPO and 20 DQN), in Breakout it reaches 353 (against 443 PPO and 302 DQN), and in Space Invaders it reaches 1200 (against 939 PPO and 1383 DQN). Within the scope of the three games tested, this establishes that code-based policies can be competitive with neural policies on long-horizon tasks while using far fewer environment interactions.
Load-bearing premise
The time-efficiency claim rests on the assumption that the public baseline times in Table 2 are fair representatives; the paper did not rerun the baselines on its own hardware, and its appendix admits an optimized A2C implementation solves Breakout in 33 minutes, faster than its reported 1h31m.
Editorial extensions
If this is right
- The same recipe—trace a rollout, evaluate long, feed staged language feedback, then rewrite code—can be ported to other domains with compact state descriptions and hand-codable tactics.
- Final policies are readable Python, so debugging, testing, and verification of agent behavior become code review tasks rather than network inspection tasks.
- Training with one environment and about twenty optimizer iterations means practitioners can iterate on feedback design faster than on reward shaping.
- The approach broadens LLM-based code optimization from improving runtime efficiency to improving decision quality in closed-loop control.
Reading between the lines
- A fair test of the headline efficiency claim is to rerun the same baselines on the same hardware; without that, the raw scores stand but the time comparison may overstate the gap.
- The documented inference of the right wall at x = 152 suggests the same optimizer could be asked to infer transition rules or reward structure, effectively turning execution traces into an environment-model learner.
- Games whose winning strategies are not easily expressed in a few if-statements, such as fast-twitch or fine continuous control tasks, may resist this representation; that boundary is untested.
- If staged feedback can be generated automatically from game mechanics rather than hand-written per game, the method becomes nearly intervention-free.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes a generative optimization framework, built on the Trace/OptoPrime LLM optimizer, for learning Atari policies represented as Python programs. Policies consume OCAtari object-centric observations (coordinates, velocities, sizes) and are refined over 20 iterations using execution traces and staged natural-language feedback. The authors report final scores of 21 (Pong), 353 (Breakout), and 1200 (Space Invaders) with wall-clock times of 43m, 1h31m, and 36m, and compare these to CleanRL DQN/PPO baselines and human scores. They claim competitive performance with up to 98% less training time and far fewer environment interactions, plus an ablation showing full-game staged feedback outperforms rollout-only feedback on Pong, and an analysis of code complexity growth.
Significance. If substantiated, the result would be notable: demonstrating that LLM-driven programmatic policies can handle long-horizon sparse-reward Atari tasks with transparent code would be a meaningful step toward interpretable RL. The paper is also among the first to apply execution-trace-based code optimization to Atari, and it includes an honest ablation and full policy code in the appendix, which aids reproducibility. However, the headline comparison is currently confounded by an observation-space mismatch and by non-matched baseline timings, so the significance of the efficiency claim cannot be assessed from the reported evidence.
major comments (4)
- [Section 3.1 and Table 2] The central efficiency/competitiveness claim is confounded by the observation-space mismatch. The learned policies (Figures A.5-A.16) branch directly on obs['Ball']['x'], obs['Ball']['dx'], and similar object-centric quantities from OCAtari, whereas the DQN/PPO baselines in Table 2 are standard pixel-observation NoFrameskip-v4 agents. Tracking a ball from exact coordinates and velocities is a substantially easier control problem than estimating state from raw frames, so the reported 43m/1h31m/36m and low interaction counts cannot be attributed to the code representation or the LLM optimizer. To support the headline claim, the authors need either object-centric RL baselines trained on the same OCAtari observations, or a version of their method consuming raw pixels; this issue alone undermines the abstract's claim of being competitive while using significantly less training time and much fewer environment interactions.
- [Appendix E and Table 2] The baseline timings are not controlled. The DQN/PPO numbers are taken from CleanRL and public W&B logs, not re-run on the same hardware with the same protocol; RL uses 8 parallel environments versus 1 for the proposed agent; and Appendix E itself notes an optimized A2C implementation solves Breakout in 33 minutes, which is faster than the reported 1h31m for the learned agent. As presented, the time-savings claim is therefore not robust. The authors should either run all baselines under matched conditions (same machine, same environment wrapper, same number of parallel environments) or explicitly reframe the paper's claim to 'competitive scores on this particular set of runs' without a general time-efficiency comparison.
- [Section 4.1, Table 2, and Table A.1] No variance or multiple seeds are reported; the table shows a single run per game, and the best-performing policy iteration appears to be selected from the optimization trajectory. This makes it impossible to assess whether the reported scores are typical. Please report mean and standard deviation over at least 3-5 independent optimization runs, and disclose the selection rule (e.g., best evaluation score across iterations) used to produce Table 2.
- [Section 4.4 and Tables 1/A.2/A.3] The staged-feedback thresholds (Reward>=19, >=300, >=1000, etc.) and the rollout horizons (15/300/400 steps) are hand-tuned design choices, and the ablation only compares rollout-only feedback against rollout-plus-full-game feedback on Pong. There is no sensitivity analysis for the thresholds or horizons, so the generality of the method and the claim of 'minimal human intervention' are not yet established. The authors should ablate at least one threshold and one horizon, or explicitly state that these are per-game hyperparameters the user must tune.
minor comments (5)
- [Figure A.4] The code snippet in Figure A.4 uses 'tace.Module' rather than 'trace.Module'; this typo should be corrected.
- [Introduction and Section 4.3] There are typographical errors that should be fixed, including 'generlization' in the Introduction and 'structual complexity' in Section 4.3.
- [Appendix E] The phrase 'See Appendix E' appears inside Appendix E itself, and the statement that A2C can solve Breakout in 33 minutes relies on this self-reference; please provide a proper citation and specify the hardware and environment configuration used for that number.
- [Section 3.2] The relationship between Trace and OptoPrime is not explained clearly; the text first says the policy is optimized 'within the Trace framework' and later refers to OptoPrime as the optimizer, so please clarify how they relate.
- [Abstract] The phrase 'our game-playing Python program' is vague; please name the agent or method explicitly in the abstract.
Circularity Check
No significant circularity: the headline scores are empirical measurements against external baselines; the only self-citation (the authors' own OptoPrime/Trace optimizer) is present but not load-bearing, and the OCAtari-versus-pixels issue is a fairness confound rather than a circular step.
full rationale
The paper's central claims are empirical measurements rather than derivations. The Atari scores in Table 2 are obtained by executing the learned Python policies in the environment, and the baseline numbers are taken from open-source CleanRL runs and public Weights & Biases logs (Appendix E). There is no fitted parameter later renamed as a prediction, no uniqueness theorem imported from prior work, and no equation whose output is its input by construction. The only self-citation is the Trace framework and the OptoPrime optimizer (Cheng et al., 2024), which includes a co-author of this paper (Allen Nie); the paper uses OptoPrime as a software tool, and the optimizer's game-playing capability is evidenced directly by this same paper's rollouts (e.g., Figure A.4's recovery of the right-wall constant x=152 from observed trajectory data, and the measured scores in Table 2). Per the reviewing rules, a cited result that is code-based and externally falsifiable does not raise the circularity score. Two genuine concerns appear in the manuscript but belong to correctness/fairness rather than circularity: (1) Section 3.1 states the agent consumes OCAtari object-centric observations (x, y, w, h, dx, dy) while the Deep RL baselines use raw pixel frames, so the 'competitive with deep RL baselines' claim is confounded by an observation-space mismatch; and (2) Appendix E admits that a highly optimized A2C implementation can solve Breakout in 33 minutes, which undercuts the claimed time advantage for Breakout (1h31m) but is a baseline-comparability problem, not a circular derivation. Section 5 similarly concedes that 'performance depends on carefully crafted prompts,' which reflects human-authored feedback and prompts in the loop but does not amount to circularity. Because no claim reduces to its inputs or to a self-citation chain, the circularity burden is minimal.
Assumptions & free parameters
free parameters (5)
- Staged feedback thresholds =
Pong: High >= 19, Medium > 0; Breakout: High >= 300, Medium > 0; SI: High >= 1000, Medium > 500
- Rollout horizon per game =
Pong 400, Breakout 300, Space Invaders 15 steps
- Optimization iterations =
20
- Optimizer memory size =
5
- LLM backend =
Claude-3-5-sonnet-20241022-v2:0
assumptions (4)
- domain assumption OCAtari object-centric observations contain sufficient state (positions, velocities, sizes) for an LLM to write effective Atari policies.
- domain assumption The Trace/OptoPrime optimizer can reliably translate execution traces and natural language feedback into code edits that improve performance.
- domain assumption A short rollout (400/300/15 steps) plus a full evaluation episode (~4000 steps) provides enough signal for iterative improvement.
- ad hoc to paper The LLM's inferred game constants (e.g., right wall x=152) arise from causal reasoning over trajectories, not from memorized Atari knowledge.
Cite this review
Pith. "Pith review of Learning Game-Playing Agents with Generative Code Optimization." pith.science (2026). https://pith.science/paper/3GMEITB4
@misc{pith2026250819506,
author = {Pith},
title = {Pith review of: Learning Game-Playing Agents with Generative Code Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/3GMEITB4}},
note = {Machine review of arXiv:2508.19506}
}
read the original abstract
We present a generative optimization approach for learning game-playing agents, where policies are represented as Python programs and refined using large language models (LLMs). Our method treats decision-making policies as self-evolving code, with current observation as input and an in-game action as output, enabling agents to self-improve through execution traces and natural language feedback with minimal human intervention. Applied to Atari games, our game-playing Python program achieves performance competitive with deep reinforcement learning (RL) baselines while using significantly less training time and much fewer environment interactions. This work highlights the promise of programmatic policy representations for building efficient, adaptable agents capable of complex, long-horizon reasoning.
Figures
Forward citations
Cited by 1 Pith paper
-
Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombination
HERO optimizes programs by generating atomic edits without score feedback and selecting the highest-scoring subset of those edits, avoiding the 'weakest-link' failure of accepting or rejecting whole edit bundles.
Reference graph
Works this paper leans on
-
[1]
Can You Improve My Code? Optimizing Programs with Local Search
Abdollahi, F., Ameen, S., Taylor, M. E., and Lelis, L. H. Can you improve my code? optimizing programs with local search. arXiv preprint arXiv:2307.05603,
-
[8]
Huang, S., Dossa, R. F. J., Raffin, A., Kanervisto, A., and Wang, W. The 37 implementation details of proximal policy optimization. The ICLR Blog Track 2023, 2022a. Huang, S., Dossa, R. F. J., Ye, C., Braga, J., Chakraborty, D., Mehta, K., and Ara ˜Aˇsjo, J. G. Cleanrl: High-quality single-file implementations of deep reinforcement learn- ing algorithms. ...
work page 2023
-
[9]
Ishida, S., Corrado, G., Fedoseev, G., Yeo, H., Russell, L., Shotton, J., Henriques, J. F., and Hu, A. Langprop: A code optimization framework using large language mod- els applied to driving. arXiv preprint arXiv:2401.10314,
-
[10]
Playing nethack with llms: Potential & limitations as zero-shot agents
5 Learning Game-Playing Agents with Generative Code Optimization Jeurissen, D., Perez-Liebana, D., Gow, J., Cakmak, D., and Kwan, J. Playing nethack with llms: Potential & limitations as zero-shot agents. In 2024 IEEE Conference on Games (CoG), pp. 1–8. IEEE,
work page 2024
-
[11]
H., Czechowski, K., Erhan, D., Finn, C., Koza- kowski, P., Levine, S., et al
Kaiser, L., Babaeizadeh, M., Milos, P., Osinski, B., Camp- bell, R. H., Czechowski, K., Erhan, D., Finn, C., Koza- kowski, P., Levine, S., et al. Model-based reinforce- ment learning for atari. arXiv preprint arXiv:1903.00374,
arXiv 1903
-
[13]
Crafting papers on machine learning
Langley, P. Crafting papers on machine learning. In Langley, P. (ed.),Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp. 1207–1216, Stan- ford, CA,
2000
-
[16]
Proximal policy optimization algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
-
[17]
B., Darvish, K., Aspuru-Guzik, A., Shkurti, F., and Garg, A
Skreta, M., Yoshikawa, N., Arellano-Rubach, S., Ji, Z., Kristensen, L. B., Darvish, K., Aspuru-Guzik, A., Shkurti, F., and Garg, A. Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting. arXiv preprint arXiv:2303.14100,
Show all 21 references
-
[18]
and Walsh, T
Vidler, A. and Walsh, T. Playing games with large lan- guage models: Randomness and strategy. arXiv preprint arXiv:2503.02582,
-
[20]
S., Wei, Y ., and Zhang, L
Xia, C. S., Wei, Y ., and Zhang, L. Automated program repair in the era of large pre-trained language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pp. 1482–1494. IEEE,
2023
-
[21]
Atari Game Setup Pong In Pong, the player controls a paddle on the right side of the screen to deflect the ball into the enemy’s goal
6 Learning Game-Playing Agents with Generative Code Optimization A. Atari Game Setup Pong In Pong, the player controls a paddle on the right side of the screen to deflect the ball into the enemy’s goal. The player scores a point if the enemy misses the ball. The game ends when...
2025
-
[1976]
Mnih, V ., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M
doi: 10.1109/TSE.1976.233837. Mnih, V ., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602,
1976
-
[2017]
De- cision transformer: Reinforcement learning via sequence modeling
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084–15097, 2021a. Chen, M., Tworek, J., J...
-
[2018]
Hua, W., Liu, O., Li, L., Amayuelas, A., Chen, J., Jiang, L., Jin, M., Fan, L., Sun, F., Wang, W., Wang, X., and Zhang, Y
URLhttps: //openreview.net/forum?id=H1Dy---0Z . Hua, W., Liu, O., Li, L., Amayuelas, A., Chen, J., Jiang, L., Jin, M., Fan, L., Sun, F., Wang, W., Wang, X., and Zhang, Y . Game-theoretic llm: Agent workflow for negotiation games. arXiv preprint arXiv:2411.05990,
-
[2019]
L., and Jin, C
Karten, S., Nguyen, A. L., and Jin, C. Pok \’echamp: an expert-level minimax language agent. arXiv preprint arXiv:2503.04094,
-
[2020]
Distributed prior- itized experience replay
Horgan, D., Quan, J., Budden, D., Barth-Maron, G., Hessel, M., van Hasselt, H., and Silver, D. Distributed prior- itized experience replay. In 6th International Confer- ence on Learning Representations, ICLR 2018, Vancou- ver, BC, Canada, April 30 - May 3, 2018, Conference Tra...
2018
-
[2021]
Mas- tering atari with discrete world models
Hafner, D., Lillicrap, T., Norouzi, M., and Ba, J. Mas- tering atari with discrete world models. arXiv preprint arXiv:2010.02193,
2010 arXiv
-
[2022]
E., Adi, Y ., Liu, J., Sauvestre, R., Remez, T., et al
Roziere, B., Gehring, J., Gloeckle, F., Sootla, S., Gat, I., Tan, X. E., Adi, Y ., Liu, J., Sauvestre, R., Remez, T., et al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950,
-
[2023]
Op- timizing code runtime performance through context- aware retrieval-augmented generation
Acharya, M., Zhang, Y ., Leach, K., and Huang, Y . Op- timizing code runtime performance through context- aware retrieval-augmented generation. arXiv preprint arXiv:2501.16692,
-
[2024]
Cui, C., Wang, W., Zhang, M., Chen, G., Luo, Z., and Ooi, B. C. Alphaevolve: A learning framework to discover novel alphas in quantitative investment. In Proceedings of the 2021 International conference on management of data, pp. 2208–2216,
2021
-
[2025]
V oyager: An open- ended embodied agent with large language models
Wang, G., Xie, Y ., Jiang, Y ., Mandlekar, A., Xiao, C., Zhu, Y ., Fan, L., and Anandkumar, A. V oyager: An open- ended embodied agent with large language models. arXiv preprint arXiv:2305.16291,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.