Pith. sign in

REVIEW 5 major objections 5 minor 9 references

Frog Soup: Zero-Shot, In-Context, and Sample-Efficient Frogger Agents

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reasoning LLMs can play Frogger zero-shot, and ten of their playthroughs raise a DQN's average reward by 35.3%.

desk verdict Promising idea, but the 35.3% claim is inconsistent with the paper's own reported means and rests on single unseeded runs; still worth refereeing for the direction, not the number. read the letter →

arxiv 2505.03947 v1 pith:G3MPSI4I submitted 2025-05-06 cs.AI

Xiang Li , Yiyang Hao , Doug Fulop This is my paper · ORCID
classification cs.AI
keywords reasoningLLMszero-shotgameplayingobject-centricstaterepresentationin-contextreinforcementlearningprioritizedexperiencereplayDeepQ-NetworkFroggerAtarisampleefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a reasoning large language model (LLM) can play the Atari game Frogger without any game-specific training when each decision is prompted with a structured list of on-screen objects, reaching the tenth of twelve lanes in its best zero-shot run. It further claims that showing the LLM the rewards from past actions, under high reasoning effort, more than doubles its episodic reward, which is evidence that in-context reward feedback is a usable learning signal for LLM agents. The most actionable claim is that seeding a standard Deep Q-Network's replay buffer with ten LLM-played episodes—1,657 transitions in total—raises the network's average episodic reward by 35.3% after 5,000 training episodes compared with the same DQN starting from an empty buffer. If these claims hold, cheap, imperfect LLM demonstrations can substitute for a large amount of exploratory interaction in sparse-reward games, which is why the result matters.

What carries the argument

The mechanism that carries the LLM experiments is object-centric state representation: each frame is reduced to a list of objects, each with a category, coordinates, width, and height (for example, 'Frog at (79,171) size (7,7)'), so the model never has to estimate distances from pixels. The mechanism that carries the sample-efficiency result is prioritized experience replay, a replay buffer that samples past transitions according to how much the network still has to learn from them. The ten LLM episodes are inserted at an initial priority of 5.0 so they are sampled frequently at the start, and their priorities decay as the network learns, eventually being replaced by the agent's own experience.

What would settle it

Run a controlled comparison in which the same DQN receives, instead of the ten LLM episodes, 1,657 transitions collected from random actions or from the DQN's own early training; if the reward at 5,000 episodes rises by the same 35.3% margin, the boost is not specific to LLM demonstrations.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that reasoning LLMs can act as low-level policies for Frogger once the pixel frame is replaced by an object list, and that their trajectories can be turned into training data for a traditional RL agent. The best zero-shot o3-mini configuration scores 32 and crosses to the tenth lane out of twelve; with high reasoning effort and past rewards shown, the episodic reward roughly doubles relative to the same setup without rewards. The load-bearing result is the DQN experiment: ten LLM-played episodes with rewards ranging from 12 to 35, totalling 1,657 transitions, are preloaded at high priority into the replay buffer, and after 5,000 episodes the seeded DQN achieves 35.3% higher average episodic reward than the identical DQN with an empty buffer, with the gap already visible before 1,000 episodes.

Load-bearing premise

The load-bearing premise is that the simplified DQN baseline—5,000 training episodes, an 8,000-step replay buffer, and a single run—fairly represents traditional RL on Frogger, because if a properly tuned or multi-seed DQN would close the gap, the 35.3% improvement attributed to LLM demonstrations would be an artifact of the baseline's hyperparameters rather than a real sample-efficiency gain.

Editorial extensions

If this is right

  • An empty-buffer DQN can be jump-started by injecting a small set of LLM trajectories, so the training budget needed to reach a given Frogger score shrinks when ten LLM episodes are available.
  • The zero-shot results show that no replay of past steps is better than replaying three or all steps, which is the opposite of the frame-stacking intuition from DQN; the extra context appears to bias the LLM towards frequent past actions.
  • Under high reasoning effort, showing past rewards in the prompt more than doubles episodic reward (22 to 45 with no past steps, and 15 to 35 with three past steps), so in-context reward feedback is a working learning signal for these models.
  • A recursive search baseline reaches only scores of 8–9 and becomes impractical or fragile beyond that, which supports the idea that LLM-guided exploration is a more scalable route than exhaustive state search in this game.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: because the headline 35.3% gap is a single-run comparison, the first check I would run is a multi-seed version with matched hyperparameters; if the gap shrinks under averaging, the conclusion should be softened to 'demonstrations help the early phase,' not 'demonstrations improve final performance.'
  • Editorial inference: the object-list prompt plays to Frogger's separable horizontal lanes, so I would not expect the same zero-shot scores to transfer to games with dense or overlapping visuals; a testable extension is to apply the identical prompt to another lane-based Atari game, such as Freeway, where object-centric RL is already known to match pixel-based RL.
  • Editorial inference: the reflective-agent drop from 45 to 21 hints that verbal reflection is brittle when it omits low-level timing and visibility rules; a cheap experiment would add the car speeds and the 'objects on the current lane disappear' rule to the reflection prompt and measure whether round-2 performance recovers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper investigates whether reasoning large language models can play the Atari game Frogger, first in a zero-shot setting using object-centric state descriptions from OCAtari, then with in-context rewards, exploratory and reflective prompting, and finally as a source of demonstrations to bootstrap a Deep Q-Network trained with prioritized experience replay. The central quantitative claim is that preloading the replay buffer with 10 LLM-played episodes (1,657 transitions) yields 35.3% higher average episodic reward after 5,000 training episodes than the same DQN with an empty buffer. The paper also reports that o3-mini can reach a score of 32 zero-shot with object-centric inputs, that providing past rewards can more than double episodic reward at high reasoning effort, and that QwQ-32B achieves a score of 17 under a standard three-past-steps configuration. The implementation and LLM trajectories are released publicly.

Significance. If the empirical claims were adequately supported, the paper would make a useful contribution to two active research directions: assessing reasoning LLMs as low-level game-playing policies, and using LLM-generated demonstrations to improve the sample efficiency of traditional RL agents. The LLM demonstration result is potentially actionable because it proposes a cheap way to seed replay buffers for pixel-based DQN agents. The paper has concrete strengths: it releases open-source code and HuggingFace trajectory datasets, it systematically varies reasoning effort and context length, and it reports qualitative reasoning traces that help explain failure modes. However, the main quantitative result is currently not verifiable from the paper as written, and the experimental design lacks the statistical controls needed to support a 35.3% effect size.

major comments (5)
  1. [Section 3.8] The headline number '35.3% higher rewards after 5,000 episodes' cannot be reconciled with the reported final average rewards of 15 for standard DQN and 24 for LLM-guided DQN, since 24/15 = 1.60, i.e., 60% higher. No definition is given for whether 35.3% is a final-window average, a whole-training average, a median, or some other statistic, and no learning curves or per-run values are shown. This claim is load-bearing for the paper's practical contribution and must be made reproducible.
  2. [Sections 3.6 and 3.8] The DQN comparison is based on a single unseeded run of each condition. DQN training on Atari is known to have high variance across seeds, so a single run cannot establish a 35.3% improvement or support the statement that 'the performance gap emerged before 1,000 episodes and continued to widen.' The paper should report at least 3-5 seeds with error bars or individual runs, and should state the random seed policy.
  3. [Sections 3.7 and 3.8] The experiment lacks a control condition that preloads the same number of transitions from a non-LLM source, such as random actions or a scripted heuristic policy. Without this control, the observed gain cannot be attributed to the LLM-generated content specifically; it could result from any curated initialization of the replay buffer. Adding this ablation is necessary to support the claim that LLM demonstrations, rather than simply having preloaded data, improve sample efficiency.
  4. [Section 3.6] The DQN baseline is described as a 'simplified training setup' with 5,000 training episodes and an 8,000-step replay buffer, but no hyperparameter details are given for the network architecture, optimizer, learning rate, target network updates, exploration schedule, or discount factor. Without these details and without evidence that this configuration is a representative DQN baseline, the reader cannot assess whether the 35.3% gap is due to the LLM demonstrations or to an undertuned baseline. The paper should either provide full hyperparameters and a standard reference configuration, or justify the simplified setup with a sensitivity analysis.
  5. [Sections 3.2 and 3.3] The zero-shot LLM results are obtained with OCAtari object coordinates, while the DQN baseline observes raw pixels. The paper argues this is fair by citing OCAtari's Freeway result, but that does not establish comparability for Frogger. This issue does not invalidate the demonstration-bootstrapping claim, since the stored transitions are full environment frames, but it should be acknowledged more directly when claiming LLM agents 'play Frogger' in a way that is comparable to standard RL agents.
minor comments (5)
  1. [Section 2.2] The word 'maticulously' should be 'meticulously'.
  2. [Section 3.9 / 3.10] Section numbering is inconsistent: '3.9 Future Research Directions' appears with no content, followed by '3.10 RL fine-tuning of Reasoning LLMs' and then '3.10.1 LLM-Guided Monte Carlo Tree Search'. Please renumber the sections coherently.
  3. [Supplement 5.7] In the reflection example, the LLM response describes 'At step 58' while the provided prompt is for step 56, and the JSON quotes in the response are mismatched. This appears to be a copy-paste error and should be corrected.
  4. [Section 3.4.2] There is a stray closing parenthesis in 'See Supplement section 5.8 for LLM outputs at the end of each life)'.
  5. [Section 3.3] The paper says '3 past steps exactly mirrors the common 4 framestack configurations' but a framestack of 4 typically includes the current frame plus three previous frames; the text should clarify whether 'past steps: 3' includes the current step or only previous steps, to avoid ambiguity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the central DQN sample-efficiency claim is an empirically measured comparison against an independent baseline, not a derivation that reduces to its inputs.

full rationale

This is an empirical study whose load-bearing claim is a measured difference in training outcomes: a standard DQN with an empty prioritized replay buffer versus the same DQN preloaded with 1,657 LLM-generated transitions. The reported 35.3% improvement is an observed result of environment interaction, not the output of an equation that contains the claim as an input. The LLM demonstrations are external data inserted into the replay buffer; they are not fitted parameters, and the comparison baseline is independently implemented. Citations to DQfD, PER, OCAtari, and the Arcade Learning Environment are background methods or external frameworks, none of which are self-citations doing load-bearing definitional work. The only self-referential aspect, using LLM demonstrations to show that LLMs can help RL, is an experimental design choice rather than hidden circularity: the measured signal comes from the Frogger environment and the DQN's own TD-error updates. The paper contains no equation defining a result in terms of the result, no fitted quantity renamed as a prediction, and no invoked uniqueness theorem from the authors' prior work. The discrepancy between the claimed 35.3% improvement and the §3.8 endpoint means (15 vs 24, which is 60%), the lack of error bars from a single unseeded run, and the absence of a non-LLM demonstration control are substantive reproducibility and experimental-design concerns, but they are not circularity under the defined criteria.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central empirical claims rest on several hand-set hyperparameters and domain assumptions. Most importantly, the LLM agent is evaluated on OCAtari object coordinates rather than raw pixels, and the DQN comparison uses a simplified, single-seed baseline. The compatibility of LLM demonstrations with the DQN observation space is not fully specified.

free parameters (4)
  • expert_priority = 5.0
    Initial PER priority for all LLM expert transitions, chosen by hand in Section 3.7; no sensitivity analysis is reported.
  • demo_trajectories = 10 episodes / 1,657 steps
    Number and size of LLM demonstrations loaded into the replay buffer; no ablation of this quantity is provided.
  • replay_buffer_size = 8,000
    Replay buffer capacity fixed by RAM constraints in Section 3.6; this small buffer shapes the DQN comparison.
  • object_size_filter = 5 pixels
    Objects smaller than 5 pixels are removed during binary segmentation in Section 3.7; this hand-set threshold affects the object list given to the LLM.
assumptions (3)
  • domain assumption OCAtari's object list is a sufficient and accurate state representation for playing Frogger.
    Section 3.2 relies on OCAtari's extracted objects; the paper acknowledges missing information such as the river/road divide, home bases, lives, time bar, and score, so the assumption is partial.
  • domain assumption The simplified DQN with an 8,000-step replay buffer and 5,000 training episodes is a meaningful baseline for traditional RL on Frogger.
    Sections 3.6 and 3.8 compare LLM-guided DQN against this baseline; the choice is driven by Colab RAM limits and practical time constraints.
  • ad hoc to paper The LLM demonstration trajectories are compatible with the pixel-based DQN replay buffer.
    Section 3.7 preloads (s,a,r,s',d) transitions from LLM play into DQN without specifying how the object-centric LLM states map to the DQN's 84x84 grayscale frame input.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Frog Soup: Zero-Shot, In-Context, and Sample-Efficient Frogger Agents." pith.science (2026). https://pith.science/paper/G3MPSI4I

@misc{pith2026250503947,
  author       = {Pith},
  title        = {Pith review of: Frog Soup: Zero-Shot, In-Context, and Sample-Efficient Frogger Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/G3MPSI4I}},
  note         = {Machine review of arXiv:2505.03947}
}
read the original abstract

One of the primary aspirations in reinforcement learning research is developing general-purpose agents capable of rapidly adapting to and mastering novel tasks. While RL gaming agents have mastered many Atari games, they remain slow and costly to train for each game. In this work, we demonstrate that latest reasoning LLMs with out-of-domain RL post-training can play a challenging Atari game called Frogger under a zero-shot setting. We then investigate the effect of in-context learning and the amount of reasoning effort on LLM performance. Lastly, we demonstrate a way to bootstrap traditional RL method with LLM demonstrations, which significantly improves their performance and sample efficiency. Our implementation is open sourced at https://github.com/AlienKevin/frogger.

Figures

Figures reproduced from arXiv: 2505.03947 by the authors.

Figure 1
Figure 1. States, actions, and rewards of the Atari 2600 Frogger game environment [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Zero-Shot/In-Context RL with o3-mini on Object-Centric Representation of Frogger [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Episodic rewards vs number of completion to [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Enhance DQN with LLM-generated demonstrations [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Example of one episode by DQN vs LLM￾Guided DQN The LLM-guided DQN consistently outperformed the standard DQN in total rewards throughout training. The performance gap emerged before 1,000 episodes and continued to widen as training progressed. While tra￾ditional DQN m…
Figure 5
Figure 5. Figure 5: LLM-Guided DQN 3.8. Experiment results of LLM guided DQN Over the course of 5,000 episodes, we observed a clear up￾ward trend in both agents’ average rewards, indicating pro￾gressive learning. Initially, when playing randomly, the average reward was 8-10, meaning that …
Figure 7
Figure 7. Figure 7: Monte-Carlo Tree Search with LLM Branch Pruning 4. Conclusion In this work, we have demonstrated that reasoning LLMs can achieve reasonable performance on the challenging Frogger game when given only the previous states, ac￾tions, and rewards. We found that increased r…
Figure 8
Figure 8. Figure 8: gpt-4o-2024-11-20 (210x160) gameplay frames [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: gpt-4o-2024-11-20 (1024x1024) gameplay frames [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Claude-3.7-sonnet (210x160) gameplay frames [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Claude-3.7-sonnet (1024x1024) gameplay frames [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 6 linked inside Pith

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Ocatari: Object-centric atari 2600 reinforcement learning environments, 2024

    Delfosse, Q., Blüml, J., Gregori, B., Sztwiertnia, S., and Kersting, K. Ocatari: Object-centric atari 2600 reinforcement learning environments, 2024. URL https://arxiv.org/abs/2306.08649

    arXiv 2024

  3. [3]

    Z., and Gruslys, A

    Hester, T., Vecerik, M., Pietquin, O., Lanctot, M., Schaul, T., Piot, B., Horgan, D., Quan, J., Sendonaris, A., Dulac-Arnold, G., Osband, I., Agapiou, J., Leibo, J. Z., and Gruslys, A. Deep q-learning from demonstrations, 2017. URL https://arxiv.org/abs/1704.03732

    arXiv 2017

  4. [4]

    C., Bellemare, M

    Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M., and Bowling, M. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents, 2017. URL https://arxiv.org/abs/1709.06009

    arXiv 2017

  5. [5]

    Playing atari with deep reinforcement learning, 2013

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning, 2013

    2013

  6. [6]

    A., Veness, J., Bellemare, M

    Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015

    2015

  7. [7]

    Prioritized experience replay, 2016

    Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay, 2016. URL https://arxiv.org/abs/1511.05952

    arXiv 2016

  8. [8]

    Reflexion: Language agents with verbal reinforcement learning, 2023

    Shinn, N., Cassano, F., Berman, E., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv.org/abs/2303.11366

    arXiv 2023

Show all 9 references
  1. [9]

    R., White, D., Sunbeam, M., and Goecks, V

    Waytowich, N. R., White, D., Sunbeam, M., and Goecks, V. G. Atari-gpt: Investigating the capabilities of multimodal large language models as low-level policies for atari games, 2024. URL https://arxiv.org/abs/2408.15950

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.