{"id":"63a9a733-979b-426b-9675-c096fc82786f","arxiv_id":"2506.06524","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An LLM pipeline with compiler feedback, grammar repair, and breadth-first search playtesting can generate PuzzleScript games that compile and are partially solvable.","lead":"ScriptDoctor is a new system that uses large language models to generate complete PuzzleScript games, with compiler errors and a search-based player agent providing automatic feedback in an iterative loop. It is a demonstration that LLM-driven game design can be made to work without constant human supervision, though the games are often simple and many remain unsolvable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The BFS-based 'functionality' metric is confounded with broken mechanics, so the headline claim of autonomous generation of expressive games is not yet established.","rationale":"The reader identified threshold/cap sensitivity as the weakest assumption, which is a valid parameter-sensitivity concern. My concern is more fundamental: even with fixed thresholds, the metric's construct validity is undermined by the paper's own admission that broken mechanics inflate complexity and solvability. The BFS node count and the 'solution >10' criterion do not measure whether a generated game is functional in the sense of implementing intended rules, which is necessary for the Conclusion's claim of an 'automatic game designer' producing 'concise but expressive games'. This goes beyond statistical significance: larger samples would not fix an invalid dependent variable. The concrete test of a blinded human mechanics audit would settle whether the reported successes are genuine or partly artifacts of buggy rule implementations. I therefore keep the reader's CONDITIONAL verdict, but add an explicit condition that such a validation be performed before the comparative claims are accepted.","tokens_in":5881,"tokens_out":3664,"duration_ms":43319,"concrete_test":"Select a stratified random sample of games classified as 'All Solvable' across the Table I-III conditions (e.g., 20 games). Have two PuzzleScript-experienced annotators, blind to model and condition, inspect each script and determine: (a) whether each rule's semantics match the apparent design intent, with no unused or broken mechanics, and (b) whether the level requires deliberate puzzle solving rather than an unintended exploit. Recompute per-condition success rates and the solution-complexity ranking using only games that pass both checks. If the corrected rates differ materially from the reported tables, or if model rankings reverse, the headline claim and comparative conclusions need revision.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section III defines a generated script as successful when it compiles and each level admits a BFS solution of length >10, and Section IV treats the BFS node count as 'solution complexity'. Section VI then concedes that 'the most complex games tend to be solvable or complex in spite of or even as a result of their broken mechanics (e.g. Figure 3)'. This is the load-bearing weak point: the only quantitative success signal, used both as solver feedback in the repair loop and as the basis for model comparisons in Tables I-III, can be inflated by rules that are buggy in interesting ways. The Figure 3 example shows a wizard teleport/remove exploit that makes a level solvable and complex, not because the intended mechanics work but because they are broken. Consequently, 'functional' as operationally defined does not establish that rules behave as intended or that games are expressive in the sense claimed in the Conclusion. The central demonstration may still hold partially, but the reported success rates and comparative conclusions are not evidence about game quality or rule fidelity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"ScriptDoctor is an LLM-based pipeline for automatically generating complete PuzzleScript games. The system writes PuzzleScript code, receives feedback from the PuzzleScript compiler, a Lark context-free grammar parser, and a breadth-first search player agent, and iterates for up to ten attempts. A trial is considered successful when the generated script compiles and every level admits a BFS solution of length greater than ten. The paper reports comparisons of zero-shot vs. few-shot prompting, with and without chain-of-thought, across three LLMs (GPT-4o, o1, o3-mini) and four context lengths, measuring compile rate, solvability rate, and BFS node-expansion count as a proxy for solution complexity. The authors conclude that few-shot prompting and reasoning models improve the functionality of generated games and that context-length gains diminish after 30,000 tokens.","tokens_in":6056,"tokens_out":5190,"duration_ms":55955,"significance":"If the central demonstration holds, ScriptDoctor is a useful proof of concept for closed-loop LLM game generation with automated playtesting: it combines compiler feedback, grammar-based repair, and search-based verification in a single pipeline, and the evaluation uses external oracles (the PuzzleScript compiler and BFS player) rather than a model fitted to the system's outputs, so there is no circularity in the loop. The authors are also honest about a key limitation in Section VI, conceding that the most complex generated games are often solvable because of broken mechanics rather than because of intended rule behavior. However, the paper's quantitative evidence is thin — all tables rest on 10–20 trials without confidence intervals or significance tests — and the operational success metric conflates BFS solvability with design quality. As a result, the comparative claims and the Conclusion's characterization of the system as generating 'concise but expressive games' are stronger than the current evidence supports.","major_comments":[{"comment":"The operational definition of success — compilation plus a BFS solution of length >10 in every level — does not establish that the generated rules behave as intended, yet it is used both as feedback in the repair loop and as the basis for Tables I–III and the Conclusion's 'expressive games' claim. Section VI explicitly states that the most complex games tend to be solvable 'in spite of or even as a result of their broken mechanics,' and Figure 3 illustrates a teleport/remove exploit that inflates solvability and complexity. This is the load-bearing weak point of the evaluation. I would like either (a) a small human evaluation or a rule-fidelity check on a sample of successful outputs, or (b) a revised claim that limits 'functionality' to compilability and BFS solvability and refrains from inferring expressive quality from the BFS node count.","section":"§III, §IV, §VI, Figure 3"},{"comment":"All quantitative comparisons are based on 10–20 trials per condition and are reported without confidence intervals, error bars, or significance tests. The solution-complexity means have enormous standard deviations (e.g., o1 in Table II: 22,771 ± 84,485), and differences such as the 70% vs. 80% compile rate in Table I or the 40% vs. 100% 'Any Solvable' rates across context lengths in Table III are not distinguishable from noise at these sample sizes. Please report per-trial results or intervals, and either add more trials or explicitly label the comparisons as exploratory and hypothesis-generating.","section":"§IV, Tables I–III"},{"comment":"The paper defines 'Any Solvable' with a threshold of >1 move but 'All Solvable' with a threshold of >10 moves, so the two metrics are not a single monotone hierarchy and cannot be read as progress toward the success criterion. Section V interprets them as an ordered progression from compiling to 'any solvable' to 'all solvable,' which is potentially misleading because the thresholds differ. Please define both thresholds prominently at first use and, if the intent is to measure closeness to success, also report the fraction of games in which every level admits a solution of length >10 alongside the 'any level' rate at the same threshold.","section":"§III, §V"}],"minor_comments":[{"comment":"The column header layout is confusing: the two boolean setting columns (Fewshot, CoT) are not clearly separated from the result columns. Label each row condition explicitly (e.g., 'Fewshot: off, CoT: off') to avoid ambiguity.","section":"Table I"},{"comment":"The success threshold of >10 moves is arbitrary, and the authors themselves note that many human-authored games have short solutions. A sensitivity analysis with a few alternative thresholds (e.g., >5, >20 moves) would help establish that the reported model rankings are not an artifact of this one cutoff.","section":"§IV"},{"comment":"The contribution of the context-free-grammar repair channel is not isolated in any experiment. Because grammar feedback is one of three feedback mechanisms (alongside compiler errors and solver feedback), a small ablation would clarify how much of the performance comes from this component versus the other two.","section":"§III"},{"comment":"The Introduction states that game evaluation 'requires human evaluation,' but the paper does not include any human evaluation. This is acceptable if the claims are scoped to automated solvability, but the Conclusion should not implicitly claim design quality without such a study.","section":"§I, §VI"},{"comment":"The caption for Figure 3 does not mention that the sprites may have been replaced for visualization, as Figure 2's caption does. Please add the same disclosure so readers do not infer that the generated game contains those exact visual assets.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The principal technical idea is plausible and the pipeline demonstration is valuable, but the evidence base is under-powered and the central success metric is overinterpreted. The authors' own Section VI concession about broken mechanics confirms the main reviewer concern, so this needs either additional evaluation or carefully narrowed claims. I do not see grounds for rejection: the system does run end to end and the paper is transparent about its limitations. The manuscript would fit a venue that values AGD system demonstrations and empirical studies of LLM-based content generation, but it needs another round of revision before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ScriptDoctor is a genuine first: a complete LLM-driven loop that generates full PuzzleScript games, pokes them through the compiler, parses feedback, and playtests with BFS, all without a human in the loop. The few-shot effect is large and the model comparisons are useful for the community. The paper is also unusually honest about its own metric's weakness.\n\nThe soft spot is the one the stress test flags. 'Functional' means compiles and BFS finds a solution longer than 10 moves. Section VI concedes that the most complex games are often solvable because of broken mechanics, citing the wizard-teleport example in Figure 3. So the same metric used both as repair feedback and as the basis for Tables I-III does not measure what the conclusion claims ('concise but expressive games'). The quantitative comparisons are real but they are not evidence about game quality or rule fidelity; they only show which conditions produce compilable, BFS-solvable code. To be fair, the authors say this themselves, but the Results and Conclusion still lean on those numbers.\n\nOther weaknesses are proportional: n=10-20 per condition, no error bars or significance tests, and the variances are huge (solution complexity SDs in the thousands). Some prose claims sit oddly next to the tables—the 'diminishing returns after 30k tokens' gloss sits next to a 30k-to-50k drop in any-solvable from 80% to 40%. No code or data is released, which makes the numbers hard to verify.\n\nWho is this for? People working on LLM-based PCG, pipeline integration, or automatic evaluation of generative systems. It won't reshape the field, but it is a concrete data point and a good discussion piece about evaluation pitfalls.\n\nRecommendation: send it to peer review. It deserves a serious referee and likely a revision that either fixes the success metric or recalibrates the claims to match what the metric actually measures.","headline":"A useful, honest pipeline demo for LLM-based game generation, but its headline numbers rest on a success metric the authors themselves admit can be inflated by broken mechanics.","tokens_in":6598,"tokens_out":2635,"would_cite":true,"duration_ms":26081,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM feedback loop can autonomously generate complete, playable PuzzleScript games.","keywords":["PuzzleScript","automatic game design","large language models","few-shot prompting","breadth-first search playtesting","procedural content generation","tree search","context-free grammar repair"],"falsifier":"Re-run the model comparison (GPT-4o vs. o1 vs. o3-mini, with and without few-shot examples) under alternative success definitions—for instance, requiring any solution at all, or solutions longer than thirty moves, or a different BFS node cap—and check whether the ordering and the few-shot gains persist.","tokens_in":5696,"feed_emoji":"🎮","tokens_out":7510,"duration_ms":67532,"temperature":0.7,"pith_summary":"ScriptDoctor is a pipeline that generates complete PuzzleScript games automatically, without human intervention. It runs an LLM through ten iterations of writing or repairing code, with the PuzzleScript compiler, a context-free-grammar parser, and a breadth-first search playtester supplying feedback. The paper reports that adding a few human-authored example games to the prompt sharply increases the fraction of scripts that compile and are solvable, that reasoning models o1 and o3-mini generate more fully solvable games than GPT-4o, and that adding more examples beyond roughly 30,000 tokens of context gives diminishing returns. If these results hold, ScriptDoctor is a concrete demonstration that LLMs can drive longer-horizon, self-evaluating game design rather than merely producing one-off code snippets.","feed_headline":"Autonomous pipeline gets LLMs to generate playable PuzzleScript games","feed_subtitle":"Compiler errors and search-based playtesting close the loop, and reasoning models yield the most solvable games.","key_machinery":"The mechanism is the iterative feedback loop around the PuzzleScript engine. Each cycle, the LLM sees its previous script, the compiler's errors and warnings, syntax diagnostics produced by parsing the script against a context-free grammar, and BFS playtest results (whether each level is solvable, how many nodes were expanded, and solution length), then emits a revised script. The breadth-first search plays each level as a reachability problem, expanding up to one million nodes, and supplies the objective 'playability' signal that lets the pipeline run without a human in the loop.","core_discovery":"The paper's central claim is that a closed loop of LLM code generation, compiler feedback, grammar-based repair, and search-based playtesting can produce complete PuzzleScript games without human intervention. A script is deemed successful when it compiles and every level admits a breadth-first-search solution of length greater than ten moves, with the LLM allowed ten repair iterations. In head-to-head trials, few-shot prompting with randomly sampled human-authored games raised GPT-4o's compile rate from 30% to 70% (without chain-of-thought) and improved solvability, and the reasoning models o1 and o3-mini produced the highest rates of fully solvable games and the most complex solutions. The authors also report that the most complex generated games are often solvable because of unintended or broken mechanics, which they take as evidence that solvability alone is an incomplete feedback signal.","pith_inferences":["The chosen success threshold (solution length > 10, BFS cap of 1M nodes) is arbitrary; a pipeline with a different threshold might rank models differently, so the quantitative comparisons should be read as threshold-dependent rather than absolute.","The same compiler-error-plus-solver loop should transfer to other constrained game-description languages that have a compiler and a cheap solver, provided few-shot examples are drawn from the target language.","The paper's finding that the most complex solutions often arise from broken mechanics suggests that solvability feedback selects for exploits, so an added critique signal—like rule-trace summaries or vision-language feedback on gameplay frames—could move selection toward intended design.","Because few-shot examples are drawn at random from a 610-game archive, the few-shot gains may depend on which examples happen to appear; deliberately curating diverse or high-quality examples is a directly testable extension."],"forward_implications":["Generated and playtested PuzzleScript games form a verified dataset that could be used to fine-tune smaller, more accessible language models for constrained code generation.","The automatic metrics (compilability, solvability, solution complexity) can be plugged into a novelty-seeking or evolutionary search over generated games, a direction the paper names for future work.","Reasoning-oriented prompting and reasoning models are the practical route to functional output in highly constrained domain-specific languages like PuzzleScript.","Because PuzzleScript's rules are pattern rewrites, the same playtesting signals could be accelerated by GPU convolution implementations, making open-ended search over games much cheaper.","Increasing few-shot context beyond about 30,000 tokens gives diminishing returns, so further gains will need richer feedback signals rather than more examples."],"supporting_citations":[{"why":"The prior PuzzleScript generation system using a genetic algorithm and constructive level generation; it defines the baseline that ScriptDoctor extends by generating full games rather than just rule subsets.","marker":"[10]"},{"why":"The scraped archive of 610 human-authored PuzzleScript games that supplies the few-shot examples in the prompt.","marker":"[19]"},{"why":"GPT-4o, the base model tested across few-shot, chain-of-thought, and context-length ablations.","marker":"[20]"},{"why":"o1, a reasoning model whose generated games appear in the comparison and case study.","marker":"[21]"},{"why":"o3-mini, the reasoning model that produced the highest all-solvable rate in the model comparison.","marker":"[22]"},{"why":"GA VEL, a fine-tuned LLM approach for game generation that the authors cite as the route for future fine-tuning of ScriptDoctor-generated data.","marker":"[18]"}],"fun_headline_variants":["Closed-loop LLM pipeline autonomously generates playable PuzzleScript games","LLMs plus compiler feedback and playtesting create PuzzleScript games","Reasoning models lead in autonomous PuzzleScript game generation","Self-correcting LLM loop produces solvable PuzzleScript games"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"A generated game counts as successful only if it compiles and every level has a solution longer than ten moves, found by a breadth-first search capped at one million nodes, so the reported success rates and model rankings could shift if those thresholds were changed.","fun_headline_variants_meta":{"raw":{"variants":["Closed-loop LLM pipeline autonomously generates playable PuzzleScript games","LLMs plus compiler feedback and playtesting create PuzzleScript games","Reasoning models lead in autonomous PuzzleScript game generation","Self-correcting LLM loop produces solvable PuzzleScript games"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1630,"prompt_tokens":909,"completion_tokens":721,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":650}},"tokens_in":525,"tokens_out":721,"duration_ms":5703,"temperature":1.0,"reasoning_tokens":650,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:54:21.477063+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the model comparison (GPT-4o vs. o1 vs. o3-mini, with and without few-shot examples) under alternative success definitions—for instance, requiring any solution at all, or solutions longer than thirty moves, or a different BFS node cap—and check whether the ordering and the few-shot gains persist.","supporting_citations":[{"cited_title":"Automatic puzzle level generation: A general approach using a description language","cited_arxiv_id":null,"evidence_quote":"The prior PuzzleScript generation system using a genetic algorithm and constructive level generation; it defines the baseline that ScriptDoctor extends by generating full games rather than just rule subsets."},{"cited_title":"Puzzlescript games database,","cited_arxiv_id":null,"evidence_quote":"The scraped archive of 610 human-authored PuzzleScript games that supplies the few-shot examples in the prompt."},{"cited_title":"Hello gpt-4o,","cited_arxiv_id":null,"evidence_quote":"GPT-4o, the base model tested across few-shot, chain-of-thought, and context-length ablations."},{"cited_title":"Openai o3-mini system card,","cited_arxiv_id":null,"evidence_quote":"o3-mini, the reasoning model that produced the highest all-solvable rate in the model comparison."},{"cited_title":"GAVEL: Generating Games Via Evolution and Language Models","cited_arxiv_id":"2407.09388","evidence_quote":"GA VEL, a fine-tuned LLM approach for game generation that the authors cite as the route for future fine-tuning of ScriptDoctor-generated data."}],"review_version":1}