{"id":"f761deab-3c89-4427-af9f-b39c968cb38b","arxiv_id":"2505.03947","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Reasoning LLMs given object coordinates can reach mid-level Frogger scores, and preloading their short game traces into a DQN's replay buffer yields higher early training rewards than an empty-buffer DQN.","lead":"This paper asks whether modern reasoning language models can play the Atari game Frogger when given object positions instead of pixels, and whether their play traces can make a classic reinforcement learning agent learn faster. The authors report moderate zero-shot scores, gains from showing past rewards at high reasoning effort, and a 35% reward improvement for an LLM-seeded DQN, but with single-run evidence and a weak baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 35.3% sample-efficiency claim is not numerically reproducible from the paper's own §3.8 final averages (15 vs 24 = 60%) and rests on a single unseeded run, so the central quantitative result is unverified.","rationale":"The reader's weakest assumption (undertuned/multi-seed baseline) is part of the problem, but the most direct load-bearing issue is that the paper's own numbers do not reproduce its headline. §3.8 gives final means 15 and 24, implying 60%, while the conclusion says 35.3%; without a precise averaging window or per-seed data, the central quantitative result is undefined. The absence of error bars on a single DQN run means the 35.3% could be seed noise; RL agents are notoriously seed-sensitive at 5,000 episodes with an 8,000-step replay buffer. The missing random-demonstration control also means the gain cannot be attributed specifically to LLM reasoning. A multi-seed rerun with raw logs and two averaging definitions would settle the question. The open-source repository is a real asset and makes the proposed check feasible. This does not change the reader's CONDITIONAL verdict; it strengthens the conditions required.","tokens_in":17310,"tokens_out":8054,"duration_ms":84467,"concrete_test":"Re-run both agents with the open-source implementation using at least 10 random seeds under the exact hyperparameters of §3.6–3.7. From the raw per-episode reward logs, compute (a) mean reward over the last 100 episodes and (b) mean over all 5,000 episodes for each seed. Verify whether the 35.3% figure matches either definition and whether the difference is significant under a paired test. As a secondary check, preload 1,657 transitions from a random policy into the replay buffer with priority 5.0; if this control matches the LLM-guided result, the improvement is not attributable to LLM demonstrations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in §3.8 and the conclusion is that LLM-guided DQN achieves 35.3% higher episodic reward after 5,000 episodes. But §3.8 reports final average rewards of 15 for standard DQN and 24 for LLM-guided DQN, which is 60% higher (24/15 = 1.60). No definition is given for whether 35.3% is a final-window mean, a whole-training mean, or a median; no raw learning curves or per-seed values are shown. The comparison is a single run of each agent with no error bars, yet DQN training on Atari is high-variance across seeds. The reader's concern about an undertuned baseline is plausible, but the more immediate problem is that the headline number cannot be reconciled with the reported endpoint means, and even if reconciled, a single run cannot establish a 35.3% effect. The paper also does not include a control preloading an equal number of random or scripted transitions, so the observed gain is not isolated to LLM-generated content. These gaps leave the central claim unsupported as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether reasoning large language models can play the Atari game Frogger, first in a zero-shot setting using object-centric state descriptions from OCAtari, then with in-context rewards, exploratory and reflective prompting, and finally as a source of demonstrations to bootstrap a Deep Q-Network trained with prioritized experience replay. The central quantitative claim is that preloading the replay buffer with 10 LLM-played episodes (1,657 transitions) yields 35.3% higher average episodic reward after 5,000 training episodes than the same DQN with an empty buffer. The paper also reports that o3-mini can reach a score of 32 zero-shot with object-centric inputs, that providing past rewards can more than double episodic reward at high reasoning effort, and that QwQ-32B achieves a score of 17 under a standard three-past-steps configuration. The implementation and LLM trajectories are released publicly.","tokens_in":17528,"tokens_out":3180,"duration_ms":33182,"significance":"If the empirical claims were adequately supported, the paper would make a useful contribution to two active research directions: assessing reasoning LLMs as low-level game-playing policies, and using LLM-generated demonstrations to improve the sample efficiency of traditional RL agents. The LLM demonstration result is potentially actionable because it proposes a cheap way to seed replay buffers for pixel-based DQN agents. The paper has concrete strengths: it releases open-source code and HuggingFace trajectory datasets, it systematically varies reasoning effort and context length, and it reports qualitative reasoning traces that help explain failure modes. However, the main quantitative result is currently not verifiable from the paper as written, and the experimental design lacks the statistical controls needed to support a 35.3% effect size.","major_comments":[{"comment":"The headline number '35.3% higher rewards after 5,000 episodes' cannot be reconciled with the reported final average rewards of 15 for standard DQN and 24 for LLM-guided DQN, since 24/15 = 1.60, i.e., 60% higher. No definition is given for whether 35.3% is a final-window average, a whole-training average, a median, or some other statistic, and no learning curves or per-run values are shown. This claim is load-bearing for the paper's practical contribution and must be made reproducible.","section":"Section 3.8"},{"comment":"The DQN comparison is based on a single unseeded run of each condition. DQN training on Atari is known to have high variance across seeds, so a single run cannot establish a 35.3% improvement or support the statement that 'the performance gap emerged before 1,000 episodes and continued to widen.' The paper should report at least 3-5 seeds with error bars or individual runs, and should state the random seed policy.","section":"Sections 3.6 and 3.8"},{"comment":"The experiment lacks a control condition that preloads the same number of transitions from a non-LLM source, such as random actions or a scripted heuristic policy. Without this control, the observed gain cannot be attributed to the LLM-generated content specifically; it could result from any curated initialization of the replay buffer. Adding this ablation is necessary to support the claim that LLM demonstrations, rather than simply having preloaded data, improve sample efficiency.","section":"Sections 3.7 and 3.8"},{"comment":"The DQN baseline is described as a 'simplified training setup' with 5,000 training episodes and an 8,000-step replay buffer, but no hyperparameter details are given for the network architecture, optimizer, learning rate, target network updates, exploration schedule, or discount factor. Without these details and without evidence that this configuration is a representative DQN baseline, the reader cannot assess whether the 35.3% gap is due to the LLM demonstrations or to an undertuned baseline. The paper should either provide full hyperparameters and a standard reference configuration, or justify the simplified setup with a sensitivity analysis.","section":"Section 3.6"},{"comment":"The zero-shot LLM results are obtained with OCAtari object coordinates, while the DQN baseline observes raw pixels. The paper argues this is fair by citing OCAtari's Freeway result, but that does not establish comparability for Frogger. This issue does not invalidate the demonstration-bootstrapping claim, since the stored transitions are full environment frames, but it should be acknowledged more directly when claiming LLM agents 'play Frogger' in a way that is comparable to standard RL agents.","section":"Sections 3.2 and 3.3"}],"minor_comments":[{"comment":"The word 'maticulously' should be 'meticulously'.","section":"Section 2.2"},{"comment":"Section numbering is inconsistent: '3.9 Future Research Directions' appears with no content, followed by '3.10 RL fine-tuning of Reasoning LLMs' and then '3.10.1 LLM-Guided Monte Carlo Tree Search'. Please renumber the sections coherently.","section":"Section 3.9 / 3.10"},{"comment":"In the reflection example, the LLM response describes 'At step 58' while the provided prompt is for step 56, and the JSON quotes in the response are mismatched. This appears to be a copy-paste error and should be corrected.","section":"Supplement 5.7"},{"comment":"There is a stray closing parenthesis in 'See Supplement section 5.8 for LLM outputs at the end of each life)'.","section":"Section 3.4.2"},{"comment":"The paper says '3 past steps exactly mirrors the common 4 framestack configurations' but a framestack of 4 typically includes the current frame plus three previous frames; the text should clarify whether 'past steps: 3' includes the current step or only previous steps, to avoid ambiguity.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper's empirical core is interesting but the current statistical evidence does not support the headline claim. The 35.3% figure is unreproducible from the reported numbers, and the lack of multiple seeds and controls is a serious gap for an RL paper. I believe this is fixable within the manuscript's scope by adding multi-seed runs, learning curves, and control preload experiments, so I recommend major revision rather than rejection. The LLM-played trajectories and code release are valuable assets that should be preserved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: the paper's most actionable claim—LLM demonstrations give DQN a 35.3% episodic reward boost after 5,000 episodes—does not survive contact with the paper's own numbers. §3.8 reports final averages of 15 for standard DQN and 24 for the LLM-guided version; that's 60% higher, not 35.3%. No definition of the metric is given, and the comparison is a single run of each agent with no error bars. On Atari, that's not enough to establish an effect.\n\nThat said, the paper is worth a serious referee. The idea is a good one: take reasoning-LLM play (which is cheap to generate, even if imperfect) and preload those transitions into a prioritized replay buffer to bootstrap DQN, following DQfD. The authors actually shipped code and data, and the supplement is honest about the reflective agent hurting performance, the DFS baseline being brittle under sticky actions, and the exploration prompt not helping. That kind of reporting is rare.\n\nWhat's genuinely new are the specific measurements: zero-shot Frogger scores for o3-mini and QwQ-32B on OCAtari object-centric states, the in-context reward effect (showing past rewards helps at high reasoning effort), and the finding that more past steps hurts reasoning. The combination of known components isn't groundbreaking, but the empirical mapping is real and useful, especially the negative results.\n\nThe soft spots are real and one is load-bearing. First, the 35.3% vs 60% inconsistency. Second, single runs, no seeds, no error bars—DQN on Atari is notoriously high-variance, and this baseline is further simplified (5,000 episodes, 8,000-step buffer). Third, there's no control preloading random or scripted transitions, so the observed gain is not isolated to LLM-generated content—it could be any demonstration. Fourth, the paper doesn't specify how the object-centric LLM demonstrations are converted into the pixel-based (84x84 grayscale) transitions that the DQN replays. That's a nontrivial step. Fifth, the 'zero-shot' framing is generous: the LLM receives OCAtari object coordinates, not pixels, which is a privileged observation. The authors acknowledge this in §3.2, but the abstract doesn't.\n\nAlso, the supplement has duplicated LLM outputs for different configurations, which looks like copy-paste and undercuts confidence in the trace data.\n\nWho this is for: researchers in LLM-guided RL, demonstration-driven RL, and sample efficiency. The central idea is worth pursuing even if this specific number turns out to be an artifact. I'd send it to peer review with a request for seeds, error bars, a random-demo control, and a corrected metric. The qualitative findings are worth publishing after those fixes.","headline":"Promising idea, but the 35.3% claim is inconsistent with the paper's own reported means and rests on single unseeded runs; still worth refereeing for the direction, not the number.","tokens_in":18071,"tokens_out":4751,"would_cite":false,"duration_ms":43190,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reasoning LLMs can play Frogger zero-shot, and ten of their playthroughs raise a DQN's average reward by 35.3%.","keywords":["reasoning LLMs","zero-shot game playing","object-centric state representation","in-context reinforcement learning","prioritized experience replay","Deep Q-Network","Frogger Atari","sample efficiency"],"falsifier":"Run a controlled comparison in which the same DQN receives, instead of the ten LLM episodes, 1,657 transitions collected from random actions or from the DQN's own early training; if the reward at 5,000 episodes rises by the same 35.3% margin, the boost is not specific to LLM demonstrations.","tokens_in":17080,"feed_emoji":"🐸","tokens_out":9288,"duration_ms":81620,"temperature":0.7,"pith_summary":"The paper tries to establish that a reasoning large language model (LLM) can play the Atari game Frogger without any game-specific training when each decision is prompted with a structured list of on-screen objects, reaching the tenth of twelve lanes in its best zero-shot run. It further claims that showing the LLM the rewards from past actions, under high reasoning effort, more than doubles its episodic reward, which is evidence that in-context reward feedback is a usable learning signal for LLM agents. The most actionable claim is that seeding a standard Deep Q-Network's replay buffer with ten LLM-played episodes—1,657 transitions in total—raises the network's average episodic reward by 35.3% after 5,000 training episodes compared with the same DQN starting from an empty buffer. If these claims hold, cheap, imperfect LLM demonstrations can substitute for a large amount of exploratory interaction in sparse-reward games, which is why the result matters.","feed_headline":"Frogger DQN jumps 35% when seeded with ten LLM playthroughs","feed_subtitle":"A reasoning LLM plays Frogger with no training, and ten of its episodes make a standard RL agent learn faster.","key_machinery":"The mechanism that carries the LLM experiments is object-centric state representation: each frame is reduced to a list of objects, each with a category, coordinates, width, and height (for example, 'Frog at (79,171) size (7,7)'), so the model never has to estimate distances from pixels. The mechanism that carries the sample-efficiency result is prioritized experience replay, a replay buffer that samples past transitions according to how much the network still has to learn from them. The ten LLM episodes are inserted at an initial priority of 5.0 so they are sampled frequently at the start, and their priorities decay as the network learns, eventually being replaced by the agent's own experience.","core_discovery":"On the paper's own terms, the central discovery is that reasoning LLMs can act as low-level policies for Frogger once the pixel frame is replaced by an object list, and that their trajectories can be turned into training data for a traditional RL agent. The best zero-shot o3-mini configuration scores 32 and crosses to the tenth lane out of twelve; with high reasoning effort and past rewards shown, the episodic reward roughly doubles relative to the same setup without rewards. The load-bearing result is the DQN experiment: ten LLM-played episodes with rewards ranging from 12 to 35, totalling 1,657 transitions, are preloaded at high priority into the replay buffer, and after 5,000 episodes the seeded DQN achieves 35.3% higher average episodic reward than the identical DQN with an empty buffer, with the gap already visible before 1,000 episodes.","pith_inferences":["Editorial inference: because the headline 35.3% gap is a single-run comparison, the first check I would run is a multi-seed version with matched hyperparameters; if the gap shrinks under averaging, the conclusion should be softened to 'demonstrations help the early phase,' not 'demonstrations improve final performance.'","Editorial inference: the object-list prompt plays to Frogger's separable horizontal lanes, so I would not expect the same zero-shot scores to transfer to games with dense or overlapping visuals; a testable extension is to apply the identical prompt to another lane-based Atari game, such as Freeway, where object-centric RL is already known to match pixel-based RL.","Editorial inference: the reflective-agent drop from 45 to 21 hints that verbal reflection is brittle when it omits low-level timing and visibility rules; a cheap experiment would add the car speeds and the 'objects on the current lane disappear' rule to the reflection prompt and measure whether round-2 performance recovers."],"forward_implications":["An empty-buffer DQN can be jump-started by injecting a small set of LLM trajectories, so the training budget needed to reach a given Frogger score shrinks when ten LLM episodes are available.","The zero-shot results show that no replay of past steps is better than replaying three or all steps, which is the opposite of the frame-stacking intuition from DQN; the extra context appears to bias the LLM towards frequent past actions.","Under high reasoning effort, showing past rewards in the prompt more than doubles episodic reward (22 to 45 with no past steps, and 15 to 35 with three past steps), so in-context reward feedback is a working learning signal for these models.","A recursive search baseline reaches only scores of 8–9 and becomes impractical or fragile beyond that, which supports the idea that LLM-guided exploration is a more scalable route than exhaustive state search in this game."],"supporting_citations":[{"why":"Supplies the DQN architecture, frame preprocessing, and training setup that the paper simplifies for its baseline.","marker":"Mnih et al., 2015"},{"why":"Provides prioritized experience replay, the mechanism used to preload and sample LLM demonstrations.","marker":"Schaul et al., 2016"},{"why":"Establishes the demonstration-bootstrapping idea that the LLM-seeded DQN adapts.","marker":"Hester et al., 2017"},{"why":"Supplies the object-centric Atari state extraction that lets the LLM and DQN act from object lists.","marker":"Delfosse et al., 2024"},{"why":"Defines the sticky-action and evaluation-protocol context, and the DFS 'Brute' baseline the paper contrasts with its agents.","marker":"Machado et al., 2017"},{"why":"Gives prior evidence and the subtask decomposition for why multimodal LLMs struggle with low-level Atari control.","marker":"Waytowich et al., 2024"}],"fun_headline_variants":["Zero-shot LLM plays Frogger, then trains DQN for 35% gain","LLM's ten Frogger demos give DQN a 35% edge","Frogger RL agent learns 35% better from LLM play logs","Reasoning LLM, zero-shot, seeds DQN for 35% better Frogger","Ten LLM episodes make Frogger DQN 35% stronger"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the simplified DQN baseline—5,000 training episodes, an 8,000-step replay buffer, and a single run—fairly represents traditional RL on Frogger, because if a properly tuned or multi-seed DQN would close the gap, the 35.3% improvement attributed to LLM demonstrations would be an artifact of the baseline's hyperparameters rather than a real sample-efficiency gain.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLM plays Frogger, then trains DQN for 35% gain","LLM's ten Frogger demos give DQN a 35% edge","Frogger RL agent learns 35% better from LLM play logs","Reasoning LLM, zero-shot, seeds DQN for 35% better Frogger","Ten LLM episodes make Frogger DQN 35% stronger"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001696,"raw_usage":{"total_tokens":6673,"prompt_tokens":855,"completion_tokens":5818,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":471,"completion_tokens_details":{"reasoning_tokens":5709}},"tokens_in":471,"tokens_out":5818,"duration_ms":36880,"temperature":1.0,"reasoning_tokens":5709,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:41:55.402869+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled comparison in which the same DQN receives, instead of the ten LLM episodes, 1,657 transitions collected from random actions or from the DQN's own early training; if the reward at 5,000 episodes rises by the same 35.3% margin, the boost is not specific to LLM demonstrations.","supporting_citations":[],"review_version":1}