Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

GameDevBench, the first benchmark for game-development agents, shows that even the best model solves only 54.5% of tasks, and that simple visual feedback consistently improves performance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 00:11 UTC pith:6PWTYSPO

load-bearing objection A useful, publicly released benchmark with a serious internal-consistency problem and a multimodal claim that is weaker than the construction pipeline lets on. the 3 major comments →

arxiv 2602.11103 v2 pith:6PWTYSPO submitted 2026-02-11 cs.AI cs.CLcs.SE

GameDevBench: Evaluating Agentic Capabilities Through Game Development

Wayne Chi , Yixiong Fang , Arnav Yayavaram , Siddharth Yayavaram , Seth Karten , Qiuhong Anna Wei , Runkun Chen , Alexander Wang
show 3 more authors
Valerie Chen Ameet Talwalkar Chris Donahue
This is my paper
classification cs.AI cs.CLcs.SE
keywords game developmentmultimodal agentsbenchmarkGodotvisual feedbackagentic AILLM agentstutorial-based tasks
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces GameDevBench, the first benchmark that evaluates AI agents on game-development tasks inside a modern game engine. It consists of 132 tasks derived from web and video tutorials, each requiring agents to edit code, scenes, and visual assets to produce a working game feature. The paper reports that the best agent solves only 54.5% of tasks, with success dropping from 46.9% on gameplay logic to 31.6% on 2D graphics tasks. It also shows that giving agents screenshots of the editor or recordings of the running game consistently improves performance. A sympathetic reader would take this as evidence that game development is a hard, multimodal testbed where visual feedback is a cheap and effective lever.

Core claim

The paper's central claim is that GameDevBench is a valid and useful benchmark for measuring agentic game-development ability: tasks are complex (average solutions touch 5 files and over 100 lines across 3.4 file types, triple typical software benchmarks), multimodal, and deterministically verifiable through Godot's test framework, which checks assets, node trees, animation states, and physics. Using it, the paper finds that current agents still fail most tasks, that performance depends heavily on both model and agent framework, and that two simple feedback mechanisms—an MCP server returning editor screenshots and instructions for recording gameplay video—raise success rates across nearly al

What carries the argument

The load-bearing machinery is the benchmark's construction pipeline: tutorials (YouTube transcripts and text recipes) are converted by an LLM agent into tasks, each with a starting project, a ground-truth solution, and a GDScript test that deterministically verifies the result. The multimodal feedback methods are the second mechanism: an MCP server that screenshots the Godot editor and a built-in video-recording command that lets the agent watch the running game. The tests and the feedback loop together are what let the paper convert game development—normally a visual, subjective activity—into measurable agent performance.

Load-bearing premise

The result rests on the assumption that GameDevBench's deterministic tests capture the tutorial's intended outcome—rather than exact node names, property values, or structural choices—so that a passing test means the game feature actually works as a human would understand it.

What would settle it

Audit a random sample of 20 GameDevBench tasks: have a human expert create a functionally correct solution that deliberately differs from the ground-truth implementation (e.g., different node names, different but equivalent animation frames, alternative scene structure that still meets the tutorial's visual goal). Run the task's test suite on those solutions. If a substantial share of functionally correct but structurally different solutions fail, the success rates overstate agentic game-development ability and instead measure how well agents guess the reference implementation's exact wiring.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the reported success rates hold, game development becomes a standard, challenging testbed for multimodal agent research, alongside web navigation and software engineering.
  • Simple visual feedback (screenshots or runtime video) is a practical, low-cost way to improve agent performance in code-and-asset tasks, and combining both methods is often more cost-effective than either alone.
  • The sharp performance gap between frontier and smaller models suggests multimodal game understanding remains a bottleneck, pointing to training data or architecture changes rather than better prompting alone.
  • The finding that the same model performs quite differently when run through different agent frameworks means benchmark results must be reported with framework details, and framework compatibility is itself a significant variable.
  • The deterministic, tutorial-derived pipeline is renewable: the benchmark can be extended to new tutorials and engines, tracking agent capability over time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • One could extend the benchmark to other game engines to test whether the observed multimodal gap generalizes or is specific to this engine.
  • The visual-feedback result suggests a broader recipe: in any agent domain where the agent's actions change a visual state, providing a cheap rendering of that state may beat more elaborate reward or critique schemes.
  • The benchmark's reliance on tutorial-derived ground truth implies a ceiling: agents can only be measured on patterns tutorial authors chose, so near-perfect scores might reflect overfitting to common Godot idioms rather than general game-development skill.
  • A testable extension: measure how much of the multimodal gap disappears when agents are pretrained or fine-tuned on game-engine data, using GameDevBench as a longitudinal metric.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. GameDevBench introduces a benchmark of Godot game-development tasks distilled from web/video tutorials. The paper reports 132 tasks (the abstract says 333), each with a starting repository, an instruction, hidden Godot tests, and a verified reference solution. The benchmark is designed to require multimodal understanding (sprites, shaders, animations, scenes), while still allowing deterministic pass/fail evaluation via Godot's test framework. The paper evaluates several frontier and open-weight models in native CLI agentic frameworks and OpenHands, with and without two multimodal feedback mechanisms: an editor-screenshot MCP server and runtime video capture. The main reported findings are that agents solve slightly over half of tasks at best (54.5%), that performance is substantially lower on 2D graphics than on gameplay tasks, and that image/video feedback improves performance for most models. The authors claim this is the first benchmark for agentic game development.

Significance. GameDevBench is a plausible and potentially useful resource for the agentic-coding community. Its strengths include a publicly released task suite with deterministic tests, reference solutions, a documented construction pipeline, and human annotation. The tasks are indeed larger in scope than many SWE-Bench instances, and the paper provides a useful cost/performance analysis across models and frameworks. If the benchmark truly required visual and temporal understanding, the reported gap between gameplay and graphics tasks and the gains from video feedback would be valuable evidence about multimodal agent capabilities. However, the central claim that the benchmark evaluates multimodal game-development capability is not adequately supported: the construction pipeline and representative tasks show that nearly every quantity needed to pass the tests is explicitly supplied in the instruction text as exact node names, property values, coordinates, and resource paths. As a result, current success rates may reflect exact-value transcription and Godot file-format fluency more than visual/temporal reasoning. The benchmark still has value as a hard, multi-file, deterministic agentic coding te

major comments (3)
  1. [Abstract vs. §2, §4, Table 2] The headline numbers are internally inconsistent, which undermines the reliability of every downstream claim. The abstract states 333 tasks and a best success rate of 53.8%, while the body (§3.2, Table 2) states 132 tasks and 54.5%. The abstract reports a visual-feedback gain for 'GPT-5.4' from 41.1% to 52.0%, but Table 2 contains no GPT-5.4 and reports GPT-5.1 Codex improving from 34.1% to 41.7% with video. The introduction also says the best agent without additional support solves 47.0%, yet Table 2 gives Gemini 3 Pro a 46.2% baseline and Gemini 3 Flash a 47.0% baseline. These discrepancies must be reconciled before the paper's empirical claims can be assessed.
  2. [§3.2, Appendices A/B/E] The claim that GameDevBench provides 'deterministic verification of multimodal solutions' is not established by the construction pipeline. Appendix B requires exact node names, property values, and paths to appear verbatim in instructions, and the example tasks in Appendix E specify exact positions, sizes, layer/mask values, camera transforms, and region rectangles. For instance, E.1 gives the exact CollisionShape2D position and size, E.2 gives the exact camera transform and sphere count, and E.4 gives exact layer/mask values, radii, and region_rect. If every success-relevant value is in the instruction text, a deterministic test can be passed without the agent ever inspecting a sprite, animation, or rendered scene. The human-annotation step in §2.4 eliminates ambiguity and overly strict tests, but it does not verify that a task genuinely requires visual or temporal understanding. The au
  3. [§2.2, §2.4, Table 2] There is a partial task-solver overlap risk: Stage 2 uses a Codex agent from the GPT-5 family to construct tasks, and §4 evaluates GPT-5.1 Codex on the resulting benchmark. Human annotation fixes many syntactic and ambiguity issues but does not remove the possibility that the construction model family has implicit priors about the test format. The paper should explicitly discuss this risk and provide evidence about its magnitude. The current results are somewhat reassuring because the best model is Gemini 3 Pro, not GPT-5.1 Codex, but the possibility of task-wiring leakage remains a concern for the benchmark's validity as a general multimodal evaluation, and it should be addressed in the text.
minor comments (5)
  1. [§4.2] The sentence describing 'largest models from three different commercial model families' with baselines of 34.1%, 39.4%, and 46.2% is confusing because Table 2 also reports Gemini 3 Flash at 47.0%. Clarify whether the text refers to a specific framework or to a different subset.
  2. [§3.1] The skill-category annotation is obtained from GPT-5-mini and then reviewed by game developers, but no inter-annotator agreement or per-category validation statistics are reported. A brief reliability analysis would strengthen the claim that the categories are meaningful.
  3. [Table 3 / Figure 4] The column header 'Mean (3σ)' is unconventional; the text says it is the mean after excluding values beyond 3 standard deviations, but a clearer name such as 'Trimmed mean' or 'Mean (outliers excluded)' would avoid confusion.
  4. [Appendix B] The checklist item 'If there are multiple solutions to the problem, the tests in test.gd are flexible to allow multiple solutions' conflicts with the preceding requirements that exact strings, node names, and property values appear in the instruction. The paper should clarify what counts as an acceptable alternative solution in this framework.
  5. [References] Reference [16] lists an access date of 'January 28, 20,' which appears to be truncated. Also, reference [39] is for SWE-Bench Multimodal and may be confused with reference [13]; ensure these are cited distinctly.

Circularity Check

0 steps flagged

No significant circularity: results are empirical pass/fail measurements against deterministic Godot tests; self-citations are not load-bearing.

full rationale

GameDevBench is an empirical benchmark paper rather than a derivation with fitted parameters, so most circularity patterns do not apply. Tasks are constructed from external web/YouTube tutorials and reference repositories, then judged by deterministic Godot unit tests; the headline results (best pass@1 of 54.5%, skill-category gaps, and gains from visual feedback) are empirical measurements. No equation reduces a reported success rate to an input of the construction pipeline by definition. The only same-author citation, [7] (EditBench), is used for the peripheral analogy that "the agent was not able to perfectly create tasks and tests"; the paper independently performs its own preliminary study on 41 tasks and a full human annotation stage, so this citation is not load-bearing. The use of GPT-5-family Codex in task construction while also evaluating GPT-5.1 is a potential contamination/validity concern, but it is not circular in the technical sense: tasks are grounded in tutorial transcripts and repositories, tests are deterministic, and the reported results do not favor the constructor model (GPT-5.1 Codex baseline is 34.1%, while Gemini 3 Flash achieves 47.0%, and Gemini 3 Pro achieves the best overall 54.5%). The observation that instructions and tests are deliberately over-specified (Appendix B, Stage 3 checklist) raises a benchmark-validity question about whether tasks measure multimodal understanding or exact-value transcription, but that is a correctness threat, not a circular reduction of the benchmark's output to its input. No load-bearing argument in the paper is equivalent to its own inputs by construction, and the self-citation does not support the central claim. Thus the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

GameDevBench is a measurement instrument rather than a theoretical construction, so it does not introduce new particles, forces, or conserved quantities. The main implicit load-bearing assumptions are that Godot tests can capture multimodal correctness, that tutorial-derived tasks are representative, and that the LLM-assisted construction pipeline yields unbiased tasks.

axioms (4)
  • domain assumption Godot's test framework can deterministically verify whether a multimodal game-development task has been completed correctly.
    Section 3.2 claims that 'tests are repeatable and verifiable similar to software benchmarks while testing multimodal problems.' This assumes visual and temporal correctness reduces to code-assertable conditions.
  • domain assumption Tasks derived from web and video tutorials are representative of general game-development work.
    Section 2.1 selects tutorials based on view count and community listing. The benchmark's breadth therefore depends on this selection being representative enough to support the paper's conclusions about 'game development' broadly.
  • domain assumption LLM-generated tasks and tests are correct after the hybrid refinement and human annotation stages.
    Section 2.3 reports that only 43% of a preliminary task subset was issue-free, so the pipeline relies on automated refinement and human review to fix the remaining issues. If undetected errors remain, downstream success rates are distorted.
  • domain assumption The evaluated agents' pass@1 results on a single run are stable enough to support comparisons across models and feedback conditions.
    Table 2 reports pass@1 without confidence intervals or number of runs. The cross-model and cross-condition comparisons implicitly assume run-to-run variance is small relative to the observed gaps.

pith-pipeline@v1.3.0-alltime-deepseek · 19691 in / 7493 out tokens · 85577 ms · 2026-08-03T00:11:11.273996+00:00 · methodology

0 comments
read the original abstract

Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. In game development, agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 333 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex: the average solution requires over three times the lines of code and file changes compared to prior software development benchmarks. Agents struggle with game development, with the best agent and method solving only 53.8% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with average success rate dropping from 51.4% on gameplay-oriented tasks to 33.0% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image- and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, increasing GPT-5.4's performance from 41.1% to 52.0% when given visual feedback.

Figures

Figures reproduced from arXiv: 2602.11103 by Alexander Wang, Ameet Talwalkar, Arnav Yayavaram, Chris Donahue, Qiuhong Anna Wei, Runkun Chen, Seth Karten, Siddharth Yayavaram, Valerie Chen, Wayne Chi, Yixiong Fang.

Figure 1
Figure 1. Figure 1: We present GameDevBench, a benchmark for evaluating an agent’s ability to solve complex and multimodal game development tasks in a modern game engine. developing games within a game engine), most likely because it seemed inconceivable until recently. As LM agent capabilities continue to improve, it seems natural to ask: can agents develop video games? Game development combines many desirable characteristic… view at source ↗
Figure 2
Figure 2. Figure 2: This is an example task from GameDevBench that requests for the creation of a UI minimap. Top is the visual GUI representation and highlighted points of interest. Bottom is the same scenes and files represented in code. Tasks can be solved via the editor or entirely through code although either method requires understanding multimodal assets. Game development tasks are complex and require editing dense fil… view at source ↗
Figure 3
Figure 3. Figure 3: Types of editors in Godot. Top-left is the scene editor. Top-right is the script editor. The [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: GameDevBench features a diverse amount of filetypes (27 different types, left). The vast majority of tasks contain either images, resources (e.g., Shaders), or multiple asset types (middle). Each task contains multiple scripts and scenes, both of which are context-rich and require a significant amount of tokens to process (right). Continually Renewable. While not unique to our benchmark, our pipeline is re… view at source ↗
Figure 5
Figure 5. Figure 5: In general, agents perform better on tasks that require skills focusing on gameplay [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: We capture the trade-off between performance and cost. In general, using multimodal [PITH_FULL_IMAGE:figures/full_fig_p011_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: An example task from GameDevBench. In this example, the goal is to add physical collision and animation to the character. This can be achieved through either taking actions directly in the editor or editing code files. Each action in the editor is equivalent to specific modifications within the code files. Matching steps are denoted with the same numbers in our figure. 26 [PITH_FULL_IMAGE:figures/full_fig… view at source ↗
Figure 8
Figure 8. Figure 8: An example task from GameDevBench. In this example, the goal is to populate an empty 3D scene with a water depth visualization, including environment lighting, shader-driven water plane, background spheres, and a camera. This can be achieved through either taking actions directly in the editor or editing the scene file (main.tscn). Each action in the editor is equivalent to specific modifications within th… view at source ↗
Figure 9
Figure 9. Figure 9: An example task from GameDevBench. In this example, the goal is to build a complete three-screen menu system (Launch, Pause, and Restart) with styled buttons, title labels, a shader￾driven transition overlay, and signal connections to the menu handler script. This can be achieved through either taking actions directly in the editor or editing the scene file (menus.tscn). Each action in the editor is equiva… view at source ↗
Figure 10
Figure 10. Figure 10: An example task from GameDevBench. In this example, the goal is to build a reusable RTS unit with a sprite, collision shapes, a detection area for neighbor avoidance, and an aura shader that highlights the unit when selected. Unlike purely scene-based tasks, this task requires both editing the scene file (player.tscn) and implementing gameplay logic in a script file (unit.gd). Each action in the editor is… view at source ↗
Figure 11
Figure 11. Figure 11: Example of Godot common game development task. GPT-Codex-5.1-Max places [PITH_FULL_IMAGE:figures/full_fig_p031_11.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. GameCraft-Bench: Can Agents Build Playable Games End-to-End in a Real Game Engine?

    cs.CL 2026-06 unverdicted novelty 7.0

    GameCraft-Bench evaluates frontier coding agents on 140 Godot tasks across 15 game families and finds the strongest agent succeeds on only 41.46% of end-to-end game generation attempts.

  2. The Verifier is the Curriculum: Execution-Gated Self-Distillation for Cross-Family Game Generation

    cs.AI 2026-06 conditional novelty 6.5

    Strict-launch-gated rejection-sampling self-distillation compounds cross-family clean Godot generation from 8.8% to 42.2% and full best-of-K coverage, while gold duplication and a lenient BUILD filter erase the gain.

Reference graph

Works this paper leans on

65 extracted references · 7 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Introducing the model context protocol

    Anthropic. Introducing the model context protocol. https://www.anthropic.com/news/ model-context-protocol, November 2024

    2024

  2. [2]

    Sima 2: A generalist embodied agent for virtual worlds.arXiv preprint arXiv:2512.04797, 2025

    Adrian Bolton, Alexander Lerchner, Alexandra Cordell, Alexandre Moufarek, Andrew Bolt, Andrew Lampinen, Anna Mitenkova, Arne Olav Hallingstad, Bojan Vujatovic, Bonnie Li, et al. Sima 2: A generalist embodied agent for virtual worlds.arXiv preprint arXiv:2512.04797, 2025

    arXiv 2025

  3. [3]

    Genie: Generative interactive environments

    Jake Bruce, Michael D Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps, et al. Genie: Generative interactive environments. InForty-first International Conference on Machine Learning, 2024

    2024

  4. [4]

    Deep blue.Artificial intelligence, 134(1-2):57–83, 2002

    Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep blue.Artificial intelligence, 134(1-2):57–83, 2002

    2002

  5. [5]

    Mle-bench: Evaluating machine learning agents on machine learning engineering, 2025

    Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander Mądry. Mle-bench: Evaluating machine learning agents on machine learning engineering, 2025

    2025

  6. [6]

    Slidechat: A large vision-language assistant for whole-slide pathology image understanding, 2025

    Ying Chen, Guoan Wang, Yuanfeng Ji, Yanjun Li, Jin Ye, Tianbin Li, Ming Hu, Rongshan Yu, Yu Qiao, and Junjun He. Slidechat: A large vision-language assistant for whole-slide pathology image understanding, 2025

    2025

  7. [7]

    Edit-bench: Evaluating llm abilities to perform real-world instructed code edits, 2025

    Wayne Chi, Valerie Chen, Ryan Shar, Aditya Mittal, Jenny Liang, Wei-Lin Chiang, Anasta- sios Nikolas Angelopoulos, Ion Stoica, Graham Neubig, Ameet Talwalkar, and Chris Donahue. Edit-bench: Evaluating llm abilities to perform real-world instructed code edits, 2025

    2025

  8. [8]

    Gemini 2.5: Pushing the 13 frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the 13 frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Human-level play in the game of diplomacy by combining language models with strategic reasoning.Science, 378(6624):1067–1074, 2022

    Meta Fundamental AI Research Diplomacy Team (FAIR)†, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning.Science, 378(6624):1067–1074, 2022

    2022

  10. [10]

    The role of artificial intelligence in video game development.Kultura polisa, 20(3):50–67, 2023

    Aleksandar Filipović. The role of artificial intelligence in video game development.Kultura polisa, 20(3):50–67, 2023

    2023

  11. [11]

    Large language models and games: A survey and roadmap.IEEE Transactions on Games, 2024

    Roberto Gallotta, Graham Todd, Marvin Zammit, Sam Earle, Antonios Liapis, Julian Togelius, and Georgios N Yannakakis. Large language models and games: A survey and roadmap.IEEE Transactions on Games, 2024

    2024

  12. [12]

    Artificial intelligence usage in game development

    Dhanamma Jagli, Subhashchandra Nalla, Srinivasrao Danikonda, and Laxmi Nakirekanti. Artificial intelligence usage in game development. 2024

    2024

  13. [13]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024

    2024

  14. [14]

    The pokeagent challenge: Competitive and long-context learning at scale

    Seth Karten, Jake Grigsby, Stephanie Milani, Kiran Vodrahalli, Amy Zhang, Fei Fang, Yuke Zhu, and Chi Jin. The pokeagent challenge: Competitive and long-context learning at scale. NeurIPS Competition Track, 2025

    2025

  15. [15]

    Pokéchamp: an expert-level minimax language agent.arXiv preprint arXiv:2503.04094, 2025

    Seth Karten, Andy Luu Nguyen, and Chi Jin. Pokéchamp: an expert-level minimax language agent.arXiv preprint arXiv:2503.04094, 2025

    Pith/arXiv arXiv 2025

  16. [16]

    Godot Recipes

    KidsCanCode. Godot Recipes. https://kidscancode.org/godot_recipes/4.x/. Version 4.x, accessed January 28, 20

  17. [17]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks, 2024

    2024

  18. [18]

    Benchmarking cognitive biases in large language models as evaluators, 2024

    Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. Benchmarking cognitive biases in large language models as evaluators, 2024

    2024

  19. [19]

    Merrill, Alexander G

    Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, Boxuan Li, Harsh Raj, Ivan Bercovich, Lin Shi, Jeong Yeon Shin, Thomas Walshe, E. Kelly Buchanan, Junhong Shen, Guanghao Ye, Haowei Lin, Jason Poulos, Maoyu Wang, Marianna Nezhurina, Jenia Jitsev, Di Lu, Orfeas Menis Mastromichalakis, Zhiwei Xu, Zizhao Chen, Yue Liu, Robert Zhang, Leon Liangyu Chen, An...

    2026

  20. [20]

    Beating the world record in pokémon emerald: An AI agent case study.https: //nunu.ai/case-studies/pokemon-emerald, 2024

    Nunu AI. Beating the world record in pokémon emerald: An AI agent case study.https: //nunu.ai/case-studies/pokemon-emerald, 2024

    2024

  21. [21]

    Lewis, and Satinder Singh

    Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, and Satinder Singh. Action- conditional video prediction using deep networks in atari games. InNeural Information Processing Systems, 2015

    2015

  22. [22]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  23. [23]

    Hubert, K

    Julian Schrittwieser, Ioannis Antonoglou, T. Hubert, K. Simonyan, L. Sifre, Simon Schmitt, A. Guez, Edward Lockhart, D. Hassabis, T. Graepel, T. Lillicrap, and David Silver. Mastering atari, go, chess and shogi by planning with a learned model.Nature, 588:604 – 609, 2019

    2019

  24. [24]

    Procedural content generation in games

    Noor Shaker, Julian Togelius, and Mark J Nelson. Procedural content generation in games. 2016

    2016

  25. [25]

    Design2code: How far are we from automating front-end engineering?ArXiv, abs/2403.03163, 2024

    Chenglei Si, Yanzhe Zhang, Ryan Li, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: How far are we from automating front-end engineering?ArXiv, abs/2403.03163, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Mastering the game of go with deep neural networks and tree search.nature, 529(7587):484–489, 2016

    David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driess- che, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search.nature, 529(7587):484–489, 2016

    2016

  27. [27]

    Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, A

    David Silver, T. Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, A. Guez, Marc Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362:1140 – 1144, 2018

    2018

  28. [28]

    Mariogpt: Open-ended text2level generation through large language models, 2023

    Shyam Sudhakaran, Miguel González-Duque, Claire Glanois, Matthias Freiberger, Elias Najarro, and Sebastian Risi. Mariogpt: Open-ended text2level generation through large language models, 2023

    2023

  29. [29]

    Hoover, Aaron Isaksen, Andy Nealen, and Julian Togelius

    Adam Summerville, Sam Snodgrass, Matthew Guzdial, Christoffer Holmgård, Amy K. Hoover, Aaron Isaksen, Andy Nealen, and Julian Togelius. Procedural content generation via machine learning (pcgml), 2018

    2018

  30. [30]

    Diffusion models are real-time game engines.ArXiv, abs/2408.14837, 2024

    Dani Valevski, Yaniv Leviathan, Moab Arar, and Shlomi Fruchter. Diffusion models are real-time game engines.ArXiv, abs/2408.14837, 2024

    Pith/arXiv arXiv 2024

  31. [31]

    Generative agent-based modeling with actions grounded in physical, social, or digital space using concordia.arXiv preprint arXiv:2312.03664, 2023

    Alexander Sasha Vezhnevets, John P Agapiou, Avia Aharon, Ron Ziv, Jayd Matyas, Edgar A Duéñez-Guzmán, William A Cunningham, Simon Osindero, Danny Karmon, and Joel Z Leibo. Generative agent-based modeling with actions grounded in physical, social, or digital space using concordia.arXiv preprint arXiv:2312.03664, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Multi-actor generative artificial intelligence as a game engine.arXiv preprint arXiv:2507.08892, 2025

    Alexander Sasha Vezhnevets, Jayd Matyas, Logan Cross, Davide Paglieri, Minsuk Chang, William A Cunningham, Simon Osindero, William S Isaac, and Joel Z Leibo. Multi-actor generative artificial intelligence as a game engine.arXiv preprint arXiv:2507.08892, 2025. 15

    Pith/arXiv arXiv 2025

  33. [33]

    Vinyals, Igor Babuschkin, Wojciech M

    O. Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, A. Dudzik, Junyoung Chung, David Choi, Richard Powell, T. Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, M. Kroiss, Ivo Danihelka, Aja Huang, L. Sifre, Trevor Cai, J. Agapiou, Max Jaderberg, A. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, D. Budden, Yury Sulsky, James Mol...

    2019

  34. [34]

    Large language models are not fair evaluators, 2023

    Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are not fair evaluators, 2023

    2023

  35. [35]

    Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for ai soft...

    2025

  36. [36]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, Yitao Liu, Yiheng Xu, Shuyan Zhou, Silvio Savarese, Caiming Xiong, Victor Zhong, and Tao Yu. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments, 2024

    2024

  37. [37]

    Analysis of development of artificial intelligence in the game industry

    Sayid Adli Yakan. Analysis of development of artificial intelligence in the game industry. International Journal of Cyber and IT Service Management, 2(2):111–116, 2022

    2022

  38. [38]

    SWE-agent: Agent-computer interfaces enable automated software engineering

    John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    2024

  39. [39]

    Jimenez, Alex L

    John Yang, Carlos E. Jimenez, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik R. Narasimhan, Diyi Yang, Sida I. Wang, and Ofir Press. Swe-bench multimodal: Do ai systems generalize to visual software domains?, 2024

    2024

  40. [40]

    Jimenez, Ofir Press, Ludwig Schmidt, and Diyi Yang

    John Yang, Kilian Lieret, Joyce Yang, Carlos E. Jimenez, Ofir Press, Ludwig Schmidt, and Diyi Yang. Codeclash: Benchmarking goal-oriented software engineering, 2025

    2025

  41. [41]

    Black, Trevor Darrell, Angjoo Kanazawa, and Haiwen Feng

    Shaofeng Yin, Jiaxin Ge, Zora Zhiruo Wang, Xiuyu Li, Michael J. Black, Trevor Darrell, Angjoo Kanazawa, and Haiwen Feng. Vision-as-inverse-graphics agent via interleaved multimodal reasoning, 2026

    2026

  42. [42]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023

    2023

  43. [43]

    Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig

    Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. Webarena: A realistic web environment for building autonomous agents, 2024

    2024

  44. [44]

    Frontendbench: A benchmark for evaluating llms on front-end development via automatic evaluation.ArXiv, abs/2506.13832, 2025

    Hongda Zhu, Yiwen Zhang, Bing Zhao, Jingzhe Ding, Siyao Liu, Tong Liu, Dandan Wang, Yanan Liu, and Zhaojian Li. Frontendbench: A benchmark for evaluating llms on front-end development via automatic evaluation.ArXiv, abs/2506.13832, 2025. 16 A Task Construction Prompt Below is the full prompt provided to the Codex agent for automatic task construction from...

    Pith/arXiv arXiv 2025

  45. [45]

    why" before

    Transcript Context: Tutorials often explain "why" before "what" - look for action verbs

  46. [46]

    GitHub is Ground Truth: When transcript is unclear, GitHub repo shows what actually works

  47. [47]

    Simplify Complexity: If tutorial covers multiple concepts, break into multiple tasks

  48. [48]

    Test Repository First: Clone and run GitHub repo to understand expected behavior

  49. [49]

    Match Repo Structure: Use similar node names and organization as the repo

  50. [50]

    what" but transcript explains

    License Compliance: All repos already filtered for MIT/Apache-2.0/CC0-1.0 ### Common Pitfalls - Copying GitHub Repo Verbatim: Adapt to GameDevBench structure, don’t just copy - Ignoring Transcript: GitHub shows "what" but transcript explains "why" and learning objective - Overly Broad Tasks: Focus on one specific learning objective per task - Missing Asse...

  51. [51]

    All analysis happens in the tutorial folder first - Clone repo to repo/ subdirectory - Create analysis_progress.md for documentation - Complete all analysis before creating tasks

  52. [52]

    Document everything as you go - Update analysis_progress.md after each step - Include transcript quotes, repo structure, task ideas - Track what works and what doesn’t

  53. [53]

    Test the GitHub repo before extracting tasks - Run godot --import-all --quit - Verify it’s a working Godot project - Check for missing assets or dependencies

  54. [54]

    Navigate to GameDevBench root for task creation - Don’t create tasks inside the tutorial folder - Copy assets from tutorial’s repo/ to task directories 17

  55. [55]

    create a CharacterBody2D

    Return to tutorial folder for final documentation - Update analysis_progress.md with completion status - Note which tasks were created - Record any issues for future reference ## Phase 1: Setup ### Step 1: Check for Godot 4. We only want to operate on Godot 4 tutorials. If the tutorial folder / github repo is for a Godot 3 project, stop and report that. #...

  56. [56]

    Read and analyze the transcript

  57. [57]

    Read and examine the GitHub repository

  58. [58]

    Read and examine the task created

  59. [59]

    Document your progress

  60. [60]

    Format incremental and timer step text

    Validate whether the task satisfies each criterion 5a. Each criterion must have evidence for validation documented Copy the checklist template into the task starting point folder and fill it out as you validate. --- # Key Checklist - [ ] The task starting point runs with ‘uv run gamedevbench validate $TASK_NAME‘ and successfully outputs a test failure. - ...

  61. [61]

    Read the task instruction

    Look at the task (base and ground truth versions) in the editor. Read the task instruction. See if it looks reasonable (multimodality) or if it’s clearly a scripting oriented task. What you’re looking for is something that just makes sense. Run this before to ensure everything loads properly: godot --path /path/to/folder --editor

  62. [62]

    Ask your agent of choice to solve the task

    Change directory to the task. Ask your agent of choice to solve the task

  63. [63]

    See if each test matches the instruction

    While the agent is solving the task, take a look at test.gd. See if each test matches the instruction. If not, fix. You can usually catch some easy errors, such as named node/function tests

  64. [64]

    See if it passes / fails

    After the agent finishes, run validation. See if it passes / fails

  65. [65]

    Look at test.gd. Was there anything you missed? Did you miss it due to your mistake or were the instructions unclear?

    If the agent failed, pass the test.gd in. Ask the agent if it missed anything. Ask the agent if the things it missed are due to its own error or due to ambiguous instruction/overly strict tests. This is my usual prompt: "Look at test.gd. Was there anything you missed? Did you miss it due to your mistake or were the instructions unclear?" The agent is some...

    1929