REVIEW 3 major objections 5 minor 24 references
ScriptDoctor: Automatic Generation of PuzzleScript Games via Large Language Models and Tree Search
T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read An LLM feedback loop can autonomously generate complete, playable PuzzleScript games.
desk verdict A useful, honest pipeline demo for LLM-based game generation, but its headline numbers rest on a success metric the authors themselves admit can be inflated by broken mechanics. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the iterative feedback loop around the PuzzleScript engine. Each cycle, the LLM sees its previous script, the compiler's errors and warnings, syntax diagnostics produced by parsing the script against a context-free grammar, and BFS playtest results (whether each level is solvable, how many nodes were expanded, and solution length), then emits a revised script. The breadth-first search plays each level as a reachability problem, expanding up to one million nodes, and supplies the objective 'playability' signal that lets the pipeline run without a human in the loop.
What would settle it
Re-run the model comparison (GPT-4o vs. o1 vs. o3-mini, with and without few-shot examples) under alternative success definitions—for instance, requiring any solution at all, or solutions longer than thirty moves, or a different BFS node cap—and check whether the ordering and the few-shot gains persist.
Extended reading notes
Core claim
The paper's central claim is that a closed loop of LLM code generation, compiler feedback, grammar-based repair, and search-based playtesting can produce complete PuzzleScript games without human intervention. A script is deemed successful when it compiles and every level admits a breadth-first-search solution of length greater than ten moves, with the LLM allowed ten repair iterations. In head-to-head trials, few-shot prompting with randomly sampled human-authored games raised GPT-4o's compile rate from 30% to 70% (without chain-of-thought) and improved solvability, and the reasoning models o1 and o3-mini produced the highest rates of fully solvable games and the most complex solutions. The authors also report that the most complex generated games are often solvable because of unintended or broken mechanics, which they take as evidence that solvability alone is an incomplete feedback signal.
Load-bearing premise
A generated game counts as successful only if it compiles and every level has a solution longer than ten moves, found by a breadth-first search capped at one million nodes, so the reported success rates and model rankings could shift if those thresholds were changed.
Editorial extensions
If this is right
- Generated and playtested PuzzleScript games form a verified dataset that could be used to fine-tune smaller, more accessible language models for constrained code generation.
- The automatic metrics (compilability, solvability, solution complexity) can be plugged into a novelty-seeking or evolutionary search over generated games, a direction the paper names for future work.
- Reasoning-oriented prompting and reasoning models are the practical route to functional output in highly constrained domain-specific languages like PuzzleScript.
- Because PuzzleScript's rules are pattern rewrites, the same playtesting signals could be accelerated by GPU convolution implementations, making open-ended search over games much cheaper.
- Increasing few-shot context beyond about 30,000 tokens gives diminishing returns, so further gains will need richer feedback signals rather than more examples.
Reading between the lines
- The chosen success threshold (solution length > 10, BFS cap of 1M nodes) is arbitrary; a pipeline with a different threshold might rank models differently, so the quantitative comparisons should be read as threshold-dependent rather than absolute.
- The same compiler-error-plus-solver loop should transfer to other constrained game-description languages that have a compiler and a cheap solver, provided few-shot examples are drawn from the target language.
- The paper's finding that the most complex solutions often arise from broken mechanics suggests that solvability feedback selects for exploits, so an added critique signal—like rule-trace summaries or vision-language feedback on gameplay frames—could move selection toward intended design.
- Because few-shot examples are drawn at random from a 610-game archive, the few-shot gains may depend on which examples happen to appear; deliberately curating diverse or high-quality examples is a directly testable extension.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. ScriptDoctor is an LLM-based pipeline for automatically generating complete PuzzleScript games. The system writes PuzzleScript code, receives feedback from the PuzzleScript compiler, a Lark context-free grammar parser, and a breadth-first search player agent, and iterates for up to ten attempts. A trial is considered successful when the generated script compiles and every level admits a BFS solution of length greater than ten. The paper reports comparisons of zero-shot vs. few-shot prompting, with and without chain-of-thought, across three LLMs (GPT-4o, o1, o3-mini) and four context lengths, measuring compile rate, solvability rate, and BFS node-expansion count as a proxy for solution complexity. The authors conclude that few-shot prompting and reasoning models improve the functionality of generated games and that context-length gains diminish after 30,000 tokens.
Significance. If the central demonstration holds, ScriptDoctor is a useful proof of concept for closed-loop LLM game generation with automated playtesting: it combines compiler feedback, grammar-based repair, and search-based verification in a single pipeline, and the evaluation uses external oracles (the PuzzleScript compiler and BFS player) rather than a model fitted to the system's outputs, so there is no circularity in the loop. The authors are also honest about a key limitation in Section VI, conceding that the most complex generated games are often solvable because of broken mechanics rather than because of intended rule behavior. However, the paper's quantitative evidence is thin — all tables rest on 10–20 trials without confidence intervals or significance tests — and the operational success metric conflates BFS solvability with design quality. As a result, the comparative claims and the Conclusion's characterization of the system as generating 'concise but expressive games' are stronger than the current evidence supports.
major comments (3)
- [§III, §IV, §VI, Figure 3] The operational definition of success — compilation plus a BFS solution of length >10 in every level — does not establish that the generated rules behave as intended, yet it is used both as feedback in the repair loop and as the basis for Tables I–III and the Conclusion's 'expressive games' claim. Section VI explicitly states that the most complex games tend to be solvable 'in spite of or even as a result of their broken mechanics,' and Figure 3 illustrates a teleport/remove exploit that inflates solvability and complexity. This is the load-bearing weak point of the evaluation. I would like either (a) a small human evaluation or a rule-fidelity check on a sample of successful outputs, or (b) a revised claim that limits 'functionality' to compilability and BFS solvability and refrains from inferring expressive quality from the BFS node count.
- [§IV, Tables I–III] All quantitative comparisons are based on 10–20 trials per condition and are reported without confidence intervals, error bars, or significance tests. The solution-complexity means have enormous standard deviations (e.g., o1 in Table II: 22,771 ± 84,485), and differences such as the 70% vs. 80% compile rate in Table I or the 40% vs. 100% 'Any Solvable' rates across context lengths in Table III are not distinguishable from noise at these sample sizes. Please report per-trial results or intervals, and either add more trials or explicitly label the comparisons as exploratory and hypothesis-generating.
- [§III, §V] The paper defines 'Any Solvable' with a threshold of >1 move but 'All Solvable' with a threshold of >10 moves, so the two metrics are not a single monotone hierarchy and cannot be read as progress toward the success criterion. Section V interprets them as an ordered progression from compiling to 'any solvable' to 'all solvable,' which is potentially misleading because the thresholds differ. Please define both thresholds prominently at first use and, if the intent is to measure closeness to success, also report the fraction of games in which every level admits a solution of length >10 alongside the 'any level' rate at the same threshold.
minor comments (5)
- [Table I] The column header layout is confusing: the two boolean setting columns (Fewshot, CoT) are not clearly separated from the result columns. Label each row condition explicitly (e.g., 'Fewshot: off, CoT: off') to avoid ambiguity.
- [§IV] The success threshold of >10 moves is arbitrary, and the authors themselves note that many human-authored games have short solutions. A sensitivity analysis with a few alternative thresholds (e.g., >5, >20 moves) would help establish that the reported model rankings are not an artifact of this one cutoff.
- [§III] The contribution of the context-free-grammar repair channel is not isolated in any experiment. Because grammar feedback is one of three feedback mechanisms (alongside compiler errors and solver feedback), a small ablation would clarify how much of the performance comes from this component versus the other two.
- [§I, §VI] The Introduction states that game evaluation 'requires human evaluation,' but the paper does not include any human evaluation. This is acceptable if the claims are scoped to automated solvability, but the Conclusion should not implicitly claim design quality without such a study.
- [Figure 3] The caption for Figure 3 does not mention that the sprites may have been replaced for visualization, as Figure 2's caption does. Please add the same disclosure so readers do not infer that the generated game contains those exact visual assets.
Circularity Check
No significant circularity: ScriptDoctor's reported results rest on external compiler and BFS oracles, not on fitted inputs or load-bearing self-citations.
full rationale
The paper's central empirical chain is: an LLM generates PuzzleScript code; the PuzzleScript compiler and a Lark-based CFG parser produce error feedback; a BFS player agent returns solvability, node-expansion, and solution-length feedback; and the LLM iterates up to 10 times. Success is operationally defined as compiling and having every level admit a BFS solution longer than 10 moves, and Tables I-III report compilation, solvability, and BFS complexity rates under that definition. Every load-bearing oracle (PuzzleScript engine, BFS solver) is external to the system and is not fitted to or derived from the generated outputs. No parameter is fit to a subset of data and then renamed as a prediction; the success metric is a stipulated operationalization rather than a quantity constructed from the model's own outputs. The self-citations, including [18], [23], and [24], are used for framing, related work, and future-work context, not as proof of the reported empirical comparisons, and no uniqueness theorem or prior author result is invoked to forbid alternative interpretations. The paper itself flags the main validity concern in Section VI: BFS-based solvability/complexity can be inflated by broken mechanics, as in Figure 3, where a wizard teleport/remove exploit makes a level solvable. That is a construct-validity limitation about what 'functional' measures, not a circular reduction: the metric is not equivalent by construction to the model's inputs, and the authors explicitly acknowledge the shortfall. Under the required standard of exhibiting a specific reduction of a claimed result to its own inputs, no circular step can be identified.
Assumptions & free parameters
free parameters (3)
- success threshold: minimum solution length =
10 moves
- BFS node expansion cap =
1,000,000 nodes
- maximum generation attempts =
10
assumptions (3)
- domain assumption PuzzleScript compiler and the BFS solver provide valid, sufficient evaluation of game functionality and quality.
- domain assumption The 610-game human-authored dataset is representative and its random inclusion up to context length is a meaningful grounding signal.
- domain assumption BFS expansion count correlates with human-perceived solution complexity.
Cite this review
Pith. "Pith review of ScriptDoctor: Automatic Generation of PuzzleScript Games via Large Language Models and Tree Search." pith.science (2026). https://pith.science/paper/RERBMPKE
@misc{pith2026250606524,
author = {Pith},
title = {Pith review of: ScriptDoctor: Automatic Generation of PuzzleScript Games via Large Language Models and Tree Search},
year = {2026},
howpublished = {\url{https://pith.science/paper/RERBMPKE}},
note = {Machine review of arXiv:2506.06524}
}
read the original abstract
There is much interest in using large pre-trained models in Automatic Game Design (AGD), whether via the generation of code, assets, or more abstract conceptualization of design ideas. But so far this interest largely stems from the ad hoc use of such generative models under persistent human supervision. Much work remains to show how these tools can be integrated into longer-time-horizon AGD pipelines, in which systems interface with game engines to test generated content autonomously. To this end, we introduce ScriptDoctor, a Large Language Model (LLM)-driven system for automatically generating and testing games in PuzzleScript, an expressive but highly constrained description language for turn-based puzzle games over 2D gridworlds. ScriptDoctor generates and tests game design ideas in an iterative loop, where human-authored examples are used to ground the system's output, compilation errors from the PuzzleScript engine are used to elicit functional code, and search-based agents play-test generated games. ScriptDoctor serves as a concrete example of the potential of automated, open-ended LLM-based workflows in generating novel game content.
Figures
Reference graph
Works this paper leans on
-
[1]
G. N. Yannakakis and J. Togelius,Artificial intelligence and games. Springer, 2018, vol. 2
work page 2018
-
[2]
Search- based procedural content generation: A taxonomy and survey,
J. Togelius, G. N. Yannakakis, K. O. Stanley, and C. Browne, “Search- based procedural content generation: A taxonomy and survey,”Trans- actions on Computational Intelligence and AI in Games, vol. 3, no. 3, 2011
work page 2011
-
[3]
Automatic generation and evaluation of recombination games,
C. B. Browne, “Automatic generation and evaluation of recombination games,” Ph.D. dissertation, Queensland University of Technology, 2008
work page 2008
-
[4]
An experiment in automatic game design,
J. Togelius and J. Schmidhuber, “An experiment in automatic game design,” inSymposium On Computational Intelligence and Games. IEEE, 2008
work page 2008
-
[5]
Game-o-matic: Generating videogames that represent ideas,
M. Treanor, B. Blackford, M. Mateas, and I. Bogost, “Game-o-matic: Generating videogames that represent ideas,” inProcedural Content Generation in Games Workshop, 2012
work page 2012
-
[6]
Mechanic miner: Reflection- driven game mechanic discovery and level design,
M. Cook, S. Colton, A. Raad, and J. Gow, “Mechanic miner: Reflection- driven game mechanic discovery and level design,” inEvoApplications, EvoStar Conference. Springer, 2013
work page 2013
-
[7]
Towards generating arcade game rules with vgdl,
T. S. Nielsen, G. A. Barros, J. Togelius, and M. J. Nelson, “Towards generating arcade game rules with vgdl,” inComputational Intelligence and Games Conference. IEEE, 2015
work page 2015
-
[8]
General video game rule generation,
A. Khalifa, M. C. Green, D. Perez-Liebana, and J. Togelius, “General video game rule generation,” inComputational Intelligence and Games Conference. IEEE, 2017
work page 2017
Show all 24 references
-
[9]
Mechanic maker 2.0: reinforcement learning for evaluating generated rules,
J. J. Gonzalez, S. Cooper, and M. Guzdial, “Mechanic maker 2.0: reinforcement learning for evaluating generated rules,” inArtificial Intelligence and Interactive Digital Entertainment Conference, vol. 19, no. 1, 2023
2023
-
[10]
Automatic puzzle level generation: A general approach using a description language
A. Khalifa and M. Fayek, “Automatic puzzle level generation: A general approach using a description language.” IEEE, 2015
2015
-
[11]
Orchestrating game generation,
A. Liapis, G. N. Yannakakis, M. J. Nelson, M. Preuss, and R. Bidarra, “Orchestrating game generation,”Transactions on Games, vol. 11, no. 1, 2018
2018
-
[12]
A survey on cooperative co-evolutionary algorithms,
X. Ma, X. Li, Q. Zhang, K. Tang, Z. Liang, W. Xie, and Z. Zhu, “A survey on cooperative co-evolutionary algorithms,”Transactions on Evolutionary Computation, vol. 23, no. 3, 2018
2018
-
[13]
A survey of large language models,
W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Donget al., “A survey of large language models,”arXiv preprint arXiv:2303.18223, 2023
2023 arXiv
-
[14]
Large language models and games: A survey and roadmap,
R. Gallotta, G. Todd, M. Zammit, S. Earle, A. Liapis, J. Togelius, and G. N. Yannakakis, “Large language models and games: A survey and roadmap,”arXiv preprint arXiv:2402.18659, 2024
2024 arXiv
-
[15]
Game generation via large language models,
C. Hu, Y . Zhao, and J. Liu, “Game generation via large language models,” inConference on Games. IEEE, 2024
2024
-
[16]
General video game ai: A multitrack framework for evaluating agents, games, and content generation algorithms,
D. Perez-Liebana, J. Liu, A. Khalifa, R. D. Gaina, J. Togelius, and S. M. Lucas, “General video game ai: A multitrack framework for evaluating agents, games, and content generation algorithms,”Transactions on Games, vol. 11, no. 3, 2019
2019
-
[17]
Can ai chatbots create new games?
K. Cho, “Can ai chatbots create new games?” https://abagames.github. io/joys-of-small-game-development-en/generation/can ai chatbot create game.html, 2024, accessed on: October 23, 2024
2024
-
[18]
Gavel: Generating games via evolution and language models,
G. Todd, A. Padula, M. Stephenson, ´E. Piette, D. J. Soemers, and J. Togelius, “Gavel: Generating games via evolution and language models,”arXiv preprint arXiv:2407.09388, 2024
2024 arXiv
-
[19]
Puzzlescript games database,
Pedro, “Puzzlescript games database,” 2019, accessed: March 14, 2025. [Online]. Available: https://pedros.works/puzzlescript/database
2019
-
[20]
Hello gpt-4o,
OpenAI, “Hello gpt-4o,” May 2024, accessed: 2024-09-11. [Online]. Available: https://openai.com/index/hello-gpt-4o/
2024
-
[21]
Openai o1 system card,
A. Jaech, A. Kalai, A. Lerer, A. Richardson, A. El-Kishky, A. Low, A. Helyar, A. Madry, A. Beutel, A. Carneyet al., “Openai o1 system card,”arXiv preprint arXiv:2412.16720, 2024
2024 arXiv
-
[22]
Openai o3-mini system card,
OpenAI, “Openai o3-mini system card,” January 2025, accessed: March 15, 2025. [Online]. Available: https://openai.com/index/ o3-mini-system-card/
2025
-
[23]
Mariogpt: Open-ended text2level generation through large language models,
S. Sudhakaran, M. Gonz ´alez-Duque, M. Freiberger, C. Glanois, E. Na- jarro, and S. Risi, “Mariogpt: Open-ended text2level generation through large language models,”Advances in Neural Information Processing Systems, vol. 36, 2024
2024
-
[24]
Level generation through large language models,
G. Todd, S. Earle, M. U. Nasir, M. C. Green, and J. Togelius, “Level generation through large language models,” inFoundations of Digital Games Conference, 2023
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.