{"id":"8df1a45c-5e1a-474e-9817-eeaba8e99b1d","arxiv_id":"2508.19506","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LLM-guided refinement of Python policies reaches competitive Atari scores with fewer environment interactions than standard deep RL, on Pong, Breakout, and Space Invaders.","lead":"The paper uses an LLM to iteratively improve Python programs that control agents in three Atari games, reaching scores close to deep reinforcement learning agents. A generalist might read it as a test of whether interpretable code can beat opaque neural nets on classic game benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central efficiency/competitiveness claim is confounded: the programmatic agent gets OCAtari object-level state (coordinates and velocities) while the deep RL baselines use raw pixels, so the comparison is not apples-to-apples.","rationale":"The reader's weakest assumption was that the baseline comparison is unfair because numbers come from public logs and an optimized A2C solves Breakout faster than the reported time. I agree the comparison is the load-bearing point, but the more fundamental confound is observation modality: the proposed agent uses OCAtari ground-truth coordinates and velocities while CleanRL baselines use raw pixels. This makes the task easier before any code optimization occurs and is not addressed by re-running the same baselines on the same hardware. A same-observation control (PPO on OCAtari, or the programmatic agent on raw frames) would settle the claim. In fairness, the paper is transparent about its setup, includes an ablation for Pong, and publishes the final policy code in the appendix; the raw scores are plausible as demonstrations. However, without an observation-matched baseline, the central comparison cannot support the strength of the abstract's claim. I keep the reader's CONDITIONAL verdict, but the primary condition changes from 'release code and add seeds' to 'demonstrate the result under matched observation spaces and report variance across runs.'","tokens_in":15648,"tokens_out":8432,"duration_ms":81375,"concrete_test":"Train a standard CleanRL PPO agent on Pong using the same OCAtari object-centric observation (including dx/dy) under the same action-repeat, single-environment, and wall-clock or step budget as the programmatic agent. If PPO reaches a perfect 21 score within the reported 43 minutes, the efficiency advantage is an artifact of the observation space; if it does not, the central claim survives this specific check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 states that policies consume OCAtari object-centric observations, with per-object (x, y, w, h, dx, dy); the learned Pong/Breakout/Space Invaders code in Figures A.6-A.16 branches directly on obs['Ball']['dx'], obs['Ball']['y'], etc. The CleanRL DQN/PPO numbers in Table 2 are for standard pixel-observation NoFrameskip-v4 environments. This changes task difficulty before any learning happens: tracking a ball from its exact coordinates and velocity is a much simpler control problem than estimating state from raw frames. The reported 43m/1h31m and low interaction counts are therefore attributable to the privileged object-centric representation, not to the code representation or the LLM optimizer. Appendix E's admission that optimized A2C solves Breakout in 33 minutes is a second fairness issue, but it is secondary to the observation-space mismatch. The lack of seeds/error bars and selection of the best-performing policy iteration compound the uncertainty, but the observation-space confound alone is sufficient to undermine the headline claim of being 'competitive with deep RL baselines while using significantly less training time and much fewer environment interactions.'","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a generative optimization framework, built on the Trace/OptoPrime LLM optimizer, for learning Atari policies represented as Python programs. Policies consume OCAtari object-centric observations (coordinates, velocities, sizes) and are refined over 20 iterations using execution traces and staged natural-language feedback. The authors report final scores of 21 (Pong), 353 (Breakout), and 1200 (Space Invaders) with wall-clock times of 43m, 1h31m, and 36m, and compare these to CleanRL DQN/PPO baselines and human scores. They claim competitive performance with up to 98% less training time and far fewer environment interactions, plus an ablation showing full-game staged feedback outperforms rollout-only feedback on Pong, and an analysis of code complexity growth.","tokens_in":15911,"tokens_out":5144,"duration_ms":46855,"significance":"If substantiated, the result would be notable: demonstrating that LLM-driven programmatic policies can handle long-horizon sparse-reward Atari tasks with transparent code would be a meaningful step toward interpretable RL. The paper is also among the first to apply execution-trace-based code optimization to Atari, and it includes an honest ablation and full policy code in the appendix, which aids reproducibility. However, the headline comparison is currently confounded by an observation-space mismatch and by non-matched baseline timings, so the significance of the efficiency claim cannot be assessed from the reported evidence.","major_comments":[{"comment":"The central efficiency/competitiveness claim is confounded by the observation-space mismatch. The learned policies (Figures A.5-A.16) branch directly on obs['Ball']['x'], obs['Ball']['dx'], and similar object-centric quantities from OCAtari, whereas the DQN/PPO baselines in Table 2 are standard pixel-observation NoFrameskip-v4 agents. Tracking a ball from exact coordinates and velocities is a substantially easier control problem than estimating state from raw frames, so the reported 43m/1h31m/36m and low interaction counts cannot be attributed to the code representation or the LLM optimizer. To support the headline claim, the authors need either object-centric RL baselines trained on the same OCAtari observations, or a version of their method consuming raw pixels; this issue alone undermines the abstract's claim of being competitive while using significantly less training time and much fewer environment interactions.","section":"Section 3.1 and Table 2"},{"comment":"The baseline timings are not controlled. The DQN/PPO numbers are taken from CleanRL and public W&B logs, not re-run on the same hardware with the same protocol; RL uses 8 parallel environments versus 1 for the proposed agent; and Appendix E itself notes an optimized A2C implementation solves Breakout in 33 minutes, which is faster than the reported 1h31m for the learned agent. As presented, the time-savings claim is therefore not robust. The authors should either run all baselines under matched conditions (same machine, same environment wrapper, same number of parallel environments) or explicitly reframe the paper's claim to 'competitive scores on this particular set of runs' without a general time-efficiency comparison.","section":"Appendix E and Table 2"},{"comment":"No variance or multiple seeds are reported; the table shows a single run per game, and the best-performing policy iteration appears to be selected from the optimization trajectory. This makes it impossible to assess whether the reported scores are typical. Please report mean and standard deviation over at least 3-5 independent optimization runs, and disclose the selection rule (e.g., best evaluation score across iterations) used to produce Table 2.","section":"Section 4.1, Table 2, and Table A.1"},{"comment":"The staged-feedback thresholds (Reward>=19, >=300, >=1000, etc.) and the rollout horizons (15/300/400 steps) are hand-tuned design choices, and the ablation only compares rollout-only feedback against rollout-plus-full-game feedback on Pong. There is no sensitivity analysis for the thresholds or horizons, so the generality of the method and the claim of 'minimal human intervention' are not yet established. The authors should ablate at least one threshold and one horizon, or explicitly state that these are per-game hyperparameters the user must tune.","section":"Section 4.4 and Tables 1/A.2/A.3"}],"minor_comments":[{"comment":"The code snippet in Figure A.4 uses 'tace.Module' rather than 'trace.Module'; this typo should be corrected.","section":"Figure A.4"},{"comment":"There are typographical errors that should be fixed, including 'generlization' in the Introduction and 'structual complexity' in Section 4.3.","section":"Introduction and Section 4.3"},{"comment":"The phrase 'See Appendix E' appears inside Appendix E itself, and the statement that A2C can solve Breakout in 33 minutes relies on this self-reference; please provide a proper citation and specify the hardware and environment configuration used for that number.","section":"Appendix E"},{"comment":"The relationship between Trace and OptoPrime is not explained clearly; the text first says the policy is optimized 'within the Trace framework' and later refers to OptoPrime as the optimizer, so please clarify how they relate.","section":"Section 3.2"},{"comment":"The phrase 'our game-playing Python program' is vague; please name the agent or method explicitly in the abstract.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The observation-space confound between OCAtari object-centric inputs and pixel-based RL baselines is the central problem; I would not recommend acceptance without a re-run of baselines under matched observation spaces or an explicit reframing of the claims. The authors should also be asked to disclose the relationship between their Trace/OptoPrime optimizer and their own prior work, since the optimizer is developed by the same group; this is not misconduct but matters for assessing how 'off-the-shelf' the contribution is."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing you should know: this is a workshop-style demonstration that an LLM can iteratively refine small Python programs into decent Atari policies, but the central claim of being 'competitive with deep RL while using significantly less training time' is confounded by the observation space. The agent gets OCAtari's object-centric state (exact x, y, dx, dy) while the CleanRL/PPO baselines are trained on raw pixels. That is a privileged state, and it makes the efficiency comparison unfair before any learning happens. The stress-test note is right, and it remains the main thing to keep in mind.\n\nWhat is actually new: the application of the Trace/OptoPrime framework to full Atari games with long horizons, plus the staged feedback design. The machinery is prior work, including the authors' own framework, but the measurements (Pong 21, Breakout 353, Space Invaders 1200 with code policies) are new. The paper is clear: it includes the learned code, a code-complexity analysis, and an ablation showing full-game staged feedback beats rollout-only feedback on Pong. The anecdote about the optimizer inferring the wall position from trajectory data is striking and reproducible.\n\nWhere the soft spots are: the observation-space mismatch is load-bearing, not a side issue. It alone undermines any clean 'competitive' claim against pixel-based RL. The baseline times are copied from public logs, not re-run on matched hardware; the appendix admits an optimized A2C solves Breakout in 33 minutes, which is already faster than their reported 1h31m. There are no error bars or seeds; the staged feedback thresholds are hand-tuned. These are real limitations, but they don't kill the value of the demonstration itself.\n\nThe paper is for readers interested in programmatic policies and LLM-based code optimization; it is a useful existence proof, not a rigorous benchmark comparison. It deserves a serious referee—someone should check whether the claims survive with matched baselines—but a referee should probably recommend major revision before acceptance, including re-running or conditioning the comparisons and releasing code.\n\nI'd bring it up in reading group as a case study in benchmark fairness, and I'd cite it with caveats if writing about programmatic agents.","headline":"A clear demonstration of LLM-refined programmatic policies on Atari, but the headline efficiency claim rests on an apples-to-oranges comparison: the agent sees object coordinates while the baselines see pixels.","tokens_in":16418,"tokens_out":2685,"would_cite":true,"duration_ms":24041,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM-guided optimizer learns Atari policies as Python programs, reaching scores competitive with deep RL baselines in a fraction of the training time.","keywords":["generative code optimization","programmatic policies","large language models","Atari","reinforcement learning","execution traces","LLM-based optimization","interpretable agents"],"falsifier":"Rerun CleanRL DQN and PPO on the same machine and evaluation protocol used in the paper, and compare wall-clock time to reach the Table 2 scores. If standard RL reaches Pong 21, Breakout 353, and Space Invaders 1200 in times comparable to or shorter than 36–91 minutes, the central efficiency claim collapses; if not, it holds.","tokens_in":15448,"feed_emoji":"🕹️","tokens_out":5749,"duration_ms":49754,"temperature":0.7,"pith_summary":"The paper tries to establish that a game-playing policy need not be a neural network: it can be a modular Python program, and a large language model can improve that program iteratively the way gradient descent improves weights. On Pong, Breakout, and Space Invaders, the resulting code-based agents reach scores competitive with published DQN and PPO baselines while using a single environment instance and reported training times 52–98% shorter. The significance is that programmatic policies are inspectable and testable, so an agent learned this way is a human-readable artifact rather than an opaque weight matrix. The authors position this as first evidence that generative code optimization can handle long-horizon, sparse-reward sequential decisions, not just short control tasks.","feed_headline":"Coded Atari agents match deep RL at a fraction of the time","feed_subtitle":"An optimizer rewrites Python game policies until they hit Pong 21 and Breakout 353, in under 90 minutes.","key_machinery":"Two components carry the argument. Object-centric Atari representations, provided by OCAtari, replace raw pixels with named game objects carrying position, size, velocity, and status fields, so the LLM operates on compact semantics instead of thousands of pixels. Execution traces and the generative optimizer, provided by the Trace framework and OptoPrime, record a rollout's dataflow and rewrite the functions decorated as trainable, using both a scalar reward from a full ~4000-step evaluation and staged natural-language feedback as the optimization signal. The feedback is deliberately staged by performance tier so that short context-limited rollouts still receive long-horizon guidance.","core_discovery":"The central claim is that decision-making policies can be optimized as source code rather than as tensors: the agent is a modular Python class whose action-selection functions are marked trainable, and each training iteration rolls out a traced episode, computes a scalar reward plus natural-language feedback from a longer evaluation run, and lets an LLM optimizer rewrite the code. In Pong the learned program reaches 21 (against 19 PPO and 20 DQN), in Breakout it reaches 353 (against 443 PPO and 302 DQN), and in Space Invaders it reaches 1200 (against 939 PPO and 1383 DQN). Within the scope of the three games tested, this establishes that code-based policies can be competitive with neural policies on long-horizon tasks while using far fewer environment interactions.","pith_inferences":["A fair test of the headline efficiency claim is to rerun the same baselines on the same hardware; without that, the raw scores stand but the time comparison may overstate the gap.","The documented inference of the right wall at x = 152 suggests the same optimizer could be asked to infer transition rules or reward structure, effectively turning execution traces into an environment-model learner.","Games whose winning strategies are not easily expressed in a few if-statements, such as fast-twitch or fine continuous control tasks, may resist this representation; that boundary is untested.","If staged feedback can be generated automatically from game mechanics rather than hand-written per game, the method becomes nearly intervention-free."],"forward_implications":["The same recipe—trace a rollout, evaluate long, feed staged language feedback, then rewrite code—can be ported to other domains with compact state descriptions and hand-codable tactics.","Final policies are readable Python, so debugging, testing, and verification of agent behavior become code review tasks rather than network inspection tasks.","Training with one environment and about twenty optimizer iterations means practitioners can iterate on feedback design faster than on reward shaping.","The approach broadens LLM-based code optimization from improving runtime efficiency to improving decision quality in closed-loop control."],"supporting_citations":[{"why":"Supplies the Trace execution-trace machinery and the OptoPrime generative optimizer that perform the iterative code rewrites.","marker":"Cheng et al., 2024"},{"why":"Supplies the OCAtari object-centric Atari environments that convert pixels into named object states for the policies.","marker":"Delfosse et al., 2024"},{"why":"Defines the Arcade Learning Environment used as the game platform.","marker":"Bellemare et al., 2013"},{"why":"Provides the CleanRL implementations whose benchmark scores and runtimes are the main DQN/PPO comparison.","marker":"Huang et al., 2022b"},{"why":"Supplies the published PPO implementation details and baseline numbers used in Table 2.","marker":"Huang et al., 2022a"},{"why":"Defines the DQN baseline whose Pong, Breakout, and Space Invaders scores are compared against.","marker":"Mnih et al., 2013"},{"why":"Defines the PPO baseline compared in Table 2.","marker":"Schulman et al., 2017"}],"fun_headline_variants":["LLM-rewritten Python matches deep RL on Atari","Code as policy: LLM trains Atari agents faster","Agents as code: LLM-optimized programs rival deep RL","Rewrite, run, win: LLM-optimized game code","Python policies match neural nets on Atari with less data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The time-efficiency claim rests on the assumption that the public baseline times in Table 2 are fair representatives; the paper did not rerun the baselines on its own hardware, and its appendix admits an optimized A2C implementation solves Breakout in 33 minutes, faster than its reported 1h31m.","fun_headline_variants_meta":{"raw":{"variants":["LLM-rewritten Python matches deep RL on Atari","Code as policy: LLM trains Atari agents faster","Agents as code: LLM-optimized programs rival deep RL","Rewrite, run, win: LLM-optimized game code","Python policies match neural nets on Atari with less data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000344,"raw_usage":{"total_tokens":1825,"prompt_tokens":816,"completion_tokens":1009,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":432,"completion_tokens_details":{"reasoning_tokens":922}},"tokens_in":432,"tokens_out":1009,"duration_ms":8489,"temperature":1.0,"reasoning_tokens":922,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:51:14.662061+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun CleanRL DQN and PPO on the same machine and evaluation protocol used in the paper, and compare wall-clock time to reach the Table 2 scores. If standard RL reaches Pong 21, Breakout 353, and Space Invaders 1200 in times comparable to or shorter than 36–91 minutes, the central efficiency claim collapses; if not, it holds.","supporting_citations":[],"review_version":2}