Pith. sign in

REVIEW 2 major objections 5 minor 52 references

In 10,400 single-pass tries, open-weight LLMs never produced a compiling Unity game scene; the failure is missing engine knowledge, and different goal patterns break in systematically different ways.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-14 13:39 UTC pith:DX4GUH3W

load-bearing objection A clean single-pass census: 10,400 Unity C# generations, zero runnable scenes, and a reusable Grounding/Hygiene error map that actually differs by goal pattern. the 2 major comments →

arxiv 2607.10187 v1 pith:DX4GUH3W submitted 2026-07-11 cs.LG cs.SE

Knowledge-Conditioned, Single-Pass LLM Synthesis of Executable Unity Game Scenes: A Compiler Error Census across 26 Goal Playable Concepts

Hugh Xuechen Liu , K{\i}van\c{c} Tatar This is my paper
classification cs.LG cs.SE
keywords error taxonomycode generationlarge language modelsUnitygameplay design patternsgoal playable conceptssingle-pass evaluationcompiler errors
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks what large language models can actually write for Unity game scenes when they get only one shot—no compiler feedback, no repair loop, no human fixes. Across 10,400 generations of playable counterparts to 26 classic gameplay goals, every first draft failed to compile into a runnable scene. The authors sort the resulting 90,673 compiler errors into two kinds: Grounding errors (invented or misused Unity types and APIs) and Hygiene errors (plain structural C# mistakes that need no engine knowledge). That split changes sharply by goal: perception- and physics-heavy goals like Stealth fail mostly on invented engine references, while state-tracking goals like Capture fail mostly on syntax and structure. Larger models, stricter intermediate schemas, and editor versus runtime generation modes only move where the errors land; they never produce a compiling scene. The census therefore ranks goal patterns by how much engine-specific knowledge their Unity form demands, and tells designers where unaided single-pass generation breaks.

Core claim

Under a strict single-pass condition that isolates parametric knowledge, no open-weight model in the 7B–30B range produces a compiling, runnable Unity scene for any of 26 Goal Playable Concepts. The bottleneck is missing engine-specific knowledge, not scale, generation mode, or structural prompting. Failures separate cleanly into Grounding (wrong or invented Unity APIs and types) versus Hygiene (domain-independent structural defects), and the Grounding share varies by goal pattern from near zero to near one, ordering those patterns by engine-knowledge demand.

What carries the argument

The Grounding/Hygiene error taxonomy: 18 Grounding codes (need Unity type/API knowledge to resolve) versus 81 Hygiene codes (lexical/syntactic defects needing none). Applied as a census over 90,673 error occurrences from 10,400 single-pass generations of Goal Playable Concepts (playable counterparts of Björk and Holopainen’s 26 goal patterns), it turns compiler diagnostics into an ordering of design concepts by engine-knowledge demand.

Load-bearing premise

That the one current 2D Unity scene per goal pattern—built from the same primitives and prompted the same way—is representative enough that differences in error profiles can be read as differences in the abstract goals rather than quirks of those particular implementations.

What would settle it

A single-pass generation under the same harness that compiles and exposes a valid entry point for any of the 26 patterns—or a replication with alternate Unity instantiations of the same goals that erases or reverses the Grounding/Hygiene ordering—would overturn the central claim that engine knowledge is the hard bottleneck and that the census orders the patterns themselves.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Designers can treat high-Grounding goals (Stealth, Rescue, Survive, Exploration) as needing external Unity knowledge (retrieval, fine-tuning, or hand-authoring), not better generic prompts.
  • Low-Grounding goals (e.g., Capture) are better candidates for syntax repair, sanitizers, or grammar-constrained decoding because their failures are mostly structural.
  • Strict schema conditioning on sub-20B models is largely wasted: most records die at the format gate before any compiler evidence appears.
  • Under single-pass constraints, choosing a model is choosing which failure layer to face (syntax versus engine APIs), not buying a runnable scene within this open-weight tier.
  • An iterative repair loop would have to supply real Unity API knowledge for high-Grounding patterns; for low-Grounding ones a local fix may suffice.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The near-zero compile floor may generalize beyond these 26 goals to any engine-coupled generation where the model must invent scene structure and bind to live APIs rather than fill a fixed library call.
  • If Grounding share predicts repair difficulty, multi-turn repair studies could use this census as a prior: high-Grounding patterns should need more external knowledge per round.
  • The same taxonomy could be ported to other engines (Unreal, Godot) to test whether the ordering tracks perception/physics coupling or Unity-specific training data gaps.
  • Training or retrieval that injects only real Unity type and member names—without full docs—might shift mass from Grounding to Hygiene and make structural tools newly useful.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper evaluates single-pass (no compiler-feedback repair) generation of executable Unity C# scenes for 26 Goal Playable Concepts drawn from Björk and Holopainen’s goal patterns. Across a fully crossed grid of 10,400 records (five [model, generation mode] pairings spanning 7B–30B open-weight models; four IR conditioning levels; 26 patterns; 20 seeds), no script both compiles and exposes a valid entry point. From 90,673 compiler-error occurrences over 99 C# codes the authors induce a Grounding/Hygiene taxonomy (18 vs 81 codes), report per-pattern composition profiles, and argue that the bottleneck is missing engine-specific knowledge rather than scale, schema structure, or Editor vs Runtime generation mode. The census is offered as an empirical ordering of the current Unity instantiations by engine-knowledge demand.

Significance. If the zero-compilation result and the Grounding/Hygiene census hold under the stated single-pass protocol, the paper supplies a rare, large-scale negative result that cleanly isolates parametric knowledge from iterative repair—an evaluation design that most LLM game-generation demos conflate. Strengths include a fully reported factor grid, mutually exclusive outcome taxonomy, deterministic mapping from standard C# diagnostics to categories, open-weight reproducibility, and an explicit all-fail corpus that removes survivorship bias. The per-pattern profiles and design-semantics reading give practitioners a concrete map of where unaided generation breaks (perception/physics-coupled concepts vs plain state logic). That combination is useful both for PCG evaluation at the game-design layer and for diagnosing interventions (retrieval/fine-tuning vs syntactic repair).

major comments (2)
  1. [§IV-A2, §III-D, Table V] §IV-A2 and the Stealth worked example (§III-D): CS0246 is treated as the flagship Grounding code (12,297 occurrences) because invented names such as GuardAI/EnemyAI “exist in neither the engine nor the project.” Resolving CS0246 for a user-defined type can be done by emitting the missing class definition in the same compilation unit, which requires no Unity API knowledge; Runtime prompts even instruct the model to define all gameplay classes in-file. The diagnostic therefore conflates (i) hallucinated engine/project types with (ii) incomplete emission of types the model itself introduced. Given that CS0246 dominates the Grounding share and underpins the claim that the bottleneck is “missing engine-specific knowledge” and the per-pattern ordering by engine demand, the manuscript should either subdivide CS0246 (e.g., UnityEngine.*/project assets vs model-invented identifiers; defined-in-fi
  2. [Abstract; §V(a); §VI] Abstract vs §V(a)/§VI: the abstract states that “the census orders goal patterns by that demand,” while Limitation (a) and the conclusion correctly restrict the ordering to the 26 current 2D Unity instantiations (one per pattern, same primitives and prompt template). Because cross-pattern differences are the main design takeaway sold to readers, the abstract and early contribution list should carry the same qualifier as §VI (“their current specific Unity instantiations”) so the interpretive claim is not stronger than the controlled implementation set supports. The zero-compilation and composition results do not depend on the stronger reading; the ordering claim does.
minor comments (5)
  1. [Table III, §IV-A1, Fig. 3] Table III outcome labels vs body text: results prose refers to timeout_with_cs while the table uses Comp. err / TO/noCS; align naming so the five mutually exclusive outcomes are identically labeled in text, table, and Figure 3.
  2. [Title page / affiliations] Author names and place names show encoding artifacts in the compiled text (e.g., “Kıvanc ¸ Tatar”, “G ¨oteborg”, “Bj ¨ork”). Clean UTF-8 / LaTeX accents before production.
  3. [Fig. 5] Figure 5 panel counts (117/86/82/83) and the “below one error per seed” shading are important for reading selection; a one-sentence reminder in the caption that markers are conditional on compiler output would help skimmers.
  4. [§IV-A3, §V-B] §IV-A3 notes 22B-Codestral contributes 76% of raw occurrences via verbose/markdown bleed; the paper already switches to per-seed counts and shares, but a short explicit “do not rank models by raw error totals” callout in §V-B would further protect misreading.
  5. [Contributions; Table II] Generation-mode contrast is only at 30B (acknowledged). A single clarifying sentence in the contributions list that mode effects are not claimed below 30B would match the experimental design in Table II/Fig. 2.

Circularity Check

1 steps flagged

Empirical single-pass census with no circular derivation; only a non-load-bearing companion self-citation.

specific steps
  1. self citation load bearing [Section II-B / Ref. [29]; also Abstract and Section V-D]
    "Closest to our setting, the contemporaneous Mage benchmark [29] evaluates LLM-generated executable game scenes along multiple quality axes on a related class of goal-pattern tasks. The two efforts are complementary and differ in their unit of analysis. Mage measures how well generation succeeds under benchmark metrics, whereas the present study measures how generation fails at the granularity of individual compiler error occurrences."

    Same-author companion paper. Not circular in the load-bearing sense: the zero-compilation result and the Grounding/Hygiene census are obtained from the present 10,400-record pipeline and do not depend on Mage's metrics or scenes. Cited only as complementary context. Score contribution is therefore minor (self-citation present but not premise-forcing).

full rationale

The paper's load-bearing claims are observational: zero runnable scenes in 10,400 single-pass records, 90,673 compiler-error occurrences across 99 codes, and a Grounding/Hygiene split induced from the compiler's own diagnostic semantics (CS0246 etc. require external Unity types; CS1002/CS1003 do not). Success is defined by the external Unity/C# compiler and entry-point check, not by any quantity fitted from the same data. Goal patterns and GPCs are taken from independent prior literature (Björk & Holopainen; Lyu et al.). The IR schema was frozen offline before generation and is not tuned to outcomes. No parameter is fitted and then re-presented as a prediction; no uniqueness theorem is imported; no ansatz is smuggled via self-citation. The sole self-citation of note is the contemporaneous Mage benchmark by the same authors, explicitly framed as complementary (different unit of analysis) rather than as a premise that forces the zero-compilation or per-pattern results. That citation is therefore not load-bearing. Limitation (a) correctly bounds the stronger reading of the census as an ordering of abstract patterns rather than of the 26 concrete 2D instantiations; that is a scope caveat, not circularity. The derivation chain is self-contained against the compiler oracle.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 2 invented entities

The paper is an empirical census; it introduces almost no free parameters that the central claim depends on, and its main conceptual inventions are the evaluation protocol and the Grounding/Hygiene labeling of existing C# diagnostics. Background axioms are standard (compiler diagnostics are reliable, open-weight models store parametric knowledge in weights, Björk–Holopainen patterns are a stable design vocabulary).

free parameters (2)
  • compilation watchdog timeout = 120 s
    Fixed at 120 s uniformly; absolute value is a design choice that could affect which diagnostics are captured, though comparisons remain protected by uniformity.
  • generation temperature / top-p / token budgets = 0.2 / 0.95 / 2k–8k
    Set to common-practice values (0.2 / 0.95; 2048 or 8192 tokens) rather than tuned to maximize success; still experimental knobs that shape the observed error distribution.
axioms (4)
  • domain assumption A C# compiler diagnostic is a reliable, objective signal of the failure mode of a generated script.
    Used throughout Sections III–IV to treat compile success/failure and error codes as the primary measurement.
  • domain assumption Single-pass generation (no compiler-error feedback) isolates the model’s parametric knowledge from repair skill.
    Stated in Introduction and Section III-A as the justification for the experimental design.
  • domain assumption The 26 goal patterns of Björk & Holopainen, realized as the current GPC Unity instantiations, form a theoretically grounded and comparable evaluation target.
    Adopted from prior work [8,9]; the paper’s ordering claim rests on this representativeness.
  • ad hoc to paper An error code is Grounding precisely when resolving its diagnostic requires type/member/namespace information external to the script (i.e., supplied only by Unity/project).
    Definition introduced in Section IV-A2; the entire taxonomy and per-pattern profiles depend on it.
invented entities (2)
  • Grounding / Hygiene error taxonomy for Unity C# no independent evidence
    purpose: Partition the 99 observed compiler error codes into engine-knowledge failures versus domain-independent structural defects so that per-pattern profiles become interpretable.
    Induced from the observed codes rather than taken from prior literature; independent_evidence is false because the partition is defined for this study’s diagnostic set.
  • Single-pass IR-conditioned generation pipeline with four schema levels no independent evidence
    purpose: Vary structural conditioning while keeping the no-repair constraint, exposing how schema tightness relocates rather than removes failure.
    Protocol constructed and frozen by the authors (Appendix A); not an external entity with independent existence.

pith-pipeline@v1.1.0-grok45 · 33344 in / 3095 out tokens · 30321 ms · 2026-07-14T13:39:06.455048+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) write Unity C\# for game scenes. Yet nearly all demonstrations rest on an iterative repair loop that regenerates code until it compiles, conflating what the model writes with what the loop fixes. We remove the loop and evaluate a single pass, where the first draft is final. This isolates the model's parametric knowledge, the most stringent test of unaided generation. Models instantiate Goal Playable Concepts, playable counterparts of goal patterns, across 10,400 generations (four open-weight models, 7B--30B; two generation modes; four intermediate-representation (IR) conditioning levels; 26 goal patterns; 20 seeds). None compiled into a runnable scene, leaving no survivorship bias. To understand how the generated C\# scripts fail, we categorize the 99 error codes behind 90{,}673 compiler-error occurrences as Grounding (invented or misused Unity types and APIs) or Hygiene (structural defects needing no Unity knowledge). The split differs sharply by goal pattern (e.g., Stealth fails mostly on invented engine references; Capture on plain C\# structure). Larger models, stricter IRs, and different generation modes move the errors but never yield a compiling scene. The bottleneck is missing engine-specific knowledge. The census orders goal patterns by that demand, showing designers where single-pass generation breaks.

Figures

Figures reproduced from arXiv: 2607.10187 by Hugh Xuechen Liu, K{\i}van\c{c} Tatar.

Figure 1
Figure 1. Figure 1: Baseline and IR-conditioned pipelines. Data (white), LLM (light grey), [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The five [model, generation mode] and the shared factor grid. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Outcome routing for a single record. Each record enters at the top [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Reference Unity instantiation of the Stealth goal pattern. The player [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Error composition by IR conditioning. Each panel is one conditioning level. Each [[model, generation mode], pattern] appears as a marker, placed by [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Per-pattern error composition across IR conditioning levels, read [PITH_FULL_IMAGE:figures/full_fig_p010_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 7 linked inside Pith

  1. [1]

    Formal abstract design tools,

    D. Church, “Formal abstract design tools,”Game Developer, vol. 6, no. 8, pp. 44–50, 1999

    1999

  2. [2]

    I have no words & i must design: Toward a critical vocab- ulary for games,

    G. Costikyan, “I have no words & i must design: Toward a critical vocab- ulary for games,” inComputer Games and Digital Cultures Conference Proceedings, 2002

    2002

  3. [3]

    The case for game design patterns,

    B. Kreimeier, “The case for game design patterns,” 2002

    2002

  4. [4]

    Future design of accessibility in games: A design vocabulary,

    P. Cairns, C. Power, M. Barlet, and G. Haynes, “Future design of accessibility in games: A design vocabulary,”International Journal of Human-Computer Studies, vol. 131, pp. 64–71, 2019

    2019

  5. [5]

    Why video game genres fail: A classificatory analysis,

    R. I. Clarke, J. H. Lee, and N. Clark, “Why video game genres fail: A classificatory analysis,”Games and Culture, vol. 12, no. 5, pp. 445–465, 2017

    2017

  6. [6]

    Video game genre, evolution and innovation,

    D. Arsenault, “Video game genre, evolution and innovation,”Eludamos: Journal for computer game culture, vol. 3, no. 2, pp. 149–176, 2009

    2009

  7. [7]

    Juul,Half-real: Video games between real rules and fictional worlds

    J. Juul,Half-real: Video games between real rules and fictional worlds. MIT press, 2011

    2011

  8. [8]

    Goal playable concepts coupling gameplay design patterns with playable concepts,

    Z. Lyu, J. Holopainen, and S. Bj ¨ork, “Goal playable concepts coupling gameplay design patterns with playable concepts,” inProceedings of the 26th International Academic Mindtrek Conference, 2023, pp. 57–66

    2023

  9. [9]

    Bjork and J

    S. Bjork and J. Holopainen,Patterns in game design. Charles River Media Hingham, 2005, vol. 11

    2005

  10. [10]

    Game research methods: An overview,

    P. Lankoski and S. Bj ¨ork, “Game research methods: An overview,” 2015

    2015

  11. [11]

    Evaluating large language models trained on code,

    M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockmanet al., “Evaluating large language models trained on code,”arXiv preprint arXiv:2107.03374, 2021

    Pith/arXiv arXiv 2021

  12. [12]

    Program synthesis with large language models,

    J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Leet al., “Program synthesis with large language models,”arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  13. [13]

    Beyond synthetic benchmarks: Evaluating llm performance on real-world class-level code generation,

    M. Rahman, S. Khatoonabadi, and E. Shihab, “Beyond synthetic benchmarks: Evaluating llm performance on real-world class-level code generation,”arXiv preprint arXiv:2510.26130, 2025

    arXiv 2025

  14. [14]

    A survey on large language models for code generation,

    J. Jiang, F. Wang, J. Shen, S. Kim, and S. Kim, “A survey on large language models for code generation,”ACM Transactions on Software Engineering and Methodology, vol. 35, no. 2, pp. 1–72, 2026

    2026

  15. [15]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” vol. 2024, pp. 54 107–54 157, 2024

    2024

  16. [16]

    Shaker, J

    N. Shaker, J. Togelius, and M. J. Nelson,Procedural content generation in games. Springer, 2016

    2016

  17. [17]

    Search- based procedural content generation: A taxonomy and survey,

    J. Togelius, G. N. Yannakakis, K. O. Stanley, and C. Browne, “Search- based procedural content generation: A taxonomy and survey,”IEEE Transactions on Computational Intelligence and AI in Games, vol. 3, no. 3, pp. 172–186, 2011

    2011

  18. [18]

    Procedural content generation via machine learning (pcgml),

    A. Summerville, S. Snodgrass, M. Guzdial, C. Holmg ˚ard, A. K. Hoover, A. Isaksen, A. Nealen, and J. Togelius, “Procedural content generation via machine learning (pcgml),”IEEE Transactions on Games, vol. 10, no. 3, pp. 257–270, 2018

    2018

  19. [19]

    Large language models and games: A survey and roadmap,

    R. Gallotta, G. Todd, M. Zammit, S. Earle, A. Liapis, J. Togelius, and G. N. Yannakakis, “Large language models and games: A survey and roadmap,”IEEE Transactions on Games, 2024

    2024

  20. [20]

    Level generation through large language models,

    G. Todd, S. Earle, M. U. Nasir, M. C. Green, and J. Togelius, “Level generation through large language models,” inProceedings of the 18th International Conference on the Foundations of Digital Games, 2023, pp. 1–8

    2023

  21. [21]

    Mariogpt: Open-ended text2level generation through large language models,

    S. Sudhakaran, M. Gonz ´alez-Duque, M. Freiberger, C. Glanois, E. Na- jarro, and S. Risi, “Mariogpt: Open-ended text2level generation through large language models,”Advances in Neural Information Processing Systems, vol. 36, pp. 54 213–54 227, 2023

    2023

  22. [22]

    Procedural content generation for games: A survey,

    M. Hendrikx, S. Meijer, J. Van Der Velden, and A. Iosup, “Procedural content generation for games: A survey,”ACM Transactions on Multi- media Computing, Communications, and Applications (TOMM), vol. 9, no. 1, pp. 1–22, 2013

    2013

  23. [23]

    Barney,Pattern language for game design

    C. Barney,Pattern language for game design. CRC Press, 2020

    2020

  24. [24]

    Towards an ontological language for game analysis,

    J. P. Zagal, M. Mateas, C. Fern ´andez-Vara, B. Hochhalter, and N. Lichti, “Towards an ontological language for game analysis,”Worlds in play: International perspectives on digital games research, vol. 21, p. 21, 2007

    2007

  25. [25]

    An ontology for videogame interop- erability,

    J. Parkkila, F. Radulovic, D. Garijo, M. Poveda-Villal ´on, J. Ikonen, J. Porras, and A. G ´omez-P´erez, “An ontology for videogame interop- erability,”Multimedia tools and applications, vol. 76, no. 4, pp. 4981– 5000, 2017

    2017

  26. [26]

    Modeling the video game environment: the videowl ontology

    S. De Martino, M. N. Asmundo, S. A. Rizzo, and D. F. Santamaria, “Modeling the video game environment: the videowl ontology.” inWOA, 2023, pp. 191–205

    2023

  27. [27]

    Towards a video game description language,

    M. Ebner, J. Levine, S. M. Lucas, T. Schaul, T. Thompson, and J. Togelius, “Towards a video game description language,” 2013

    2013

  28. [28]

    General video game ai: A multitrack framework for evaluating agents, games, and content generation algorithms,

    D. Perez-Liebana, J. Liu, A. Khalifa, R. D. Gaina, J. Togelius, and S. M. Lucas, “General video game ai: A multitrack framework for evaluating agents, games, and content generation algorithms,”IEEE Transactions on Games, vol. 11, no. 3, pp. 195–214, 2019

    2019

  29. [29]

    Mage: Multi-axis evaluation of llm-generated executable game scenes beyond compile-pass rate,

    H. X. Liu and K. Tatar, “Mage: Multi-axis evaluation of llm-generated executable game scenes beyond compile-pass rate,” 2026. [Online]. Available: https://arxiv.org/abs/2605.07342

    Pith/arXiv arXiv 2026

  30. [30]

    37 million compilations: Investigating novice programming mistakes in large-scale student data,

    A. Altadmri and N. C. Brown, “37 million compilations: Investigating novice programming mistakes in large-scale student data,” inPro- ceedings of the 46th ACM technical symposium on computer science education, 2015, pp. 522–527

    2015

  31. [31]

    Do enhanced compiler error mes- sages help students? results inconclusive

    R. S. Pettit, J. Homer, and R. Gee, “Do enhanced compiler error mes- sages help students? results inconclusive.” inProceedings of the 2017 ACM SIGCSE Technical Symposium on Computer Science Education, 2017, pp. 465–470

    2017

  32. [32]

    Bugs in large language models generated code: An empirical study,

    F. Tambon, A. Moradi-Dakhel, A. Nikanjam, F. Khomh, M. C. Des- marais, and G. Antoniol, “Bugs in large language models generated code: An empirical study,”Empirical Software Engineering, vol. 30, no. 3, p. 65, 2025

    2025

  33. [33]

    What is wrong with your code generated by large language models? an extensive study,

    S. Dou, H. Jia, S. Wu, H. Zheng, M. Wu, Y . Tao, M. Zhang, M. Chai, J. Fan, Z. Xiet al., “What is wrong with your code generated by large language models? an extensive study,”Science China Information Sciences, vol. 69, no. 1, p. 112107, 2026

    2026

  34. [34]

    A review on nlp zero-shot and few-shot learning: methods and applications,

    G. Ramesh, M. Sahil, S. A. Palan, D. Bhandary, T. A. Ashok, J. Shreyas, and N. Sowjanya, “A review on nlp zero-shot and few-shot learning: methods and applications,”Discover Applied Sciences, vol. 7, no. 966, 2025

    2025

  35. [35]

    Qwen2. 5-coder technical report,

    B. Hui, J. Yang, Z. Cui, J. Yang, D. Liu, L. Zhang, T. Liu, J. Zhang, B. Yu, K. Luet al., “Qwen2. 5-coder technical report,”arXiv preprint arXiv:2409.12186, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,

    Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y . Wu, Y . Li, H. Gao, S. Maet al., “Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence,”arXiv preprint arXiv:2406.11931, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,” 2025

    2025

  38. [38]

    Hygienic macro expansion,

    E. Kohlbecker, D. P. Friedman, M. Felleisen, and B. Duba, “Hygienic macro expansion,” inProceedings of the 1986 ACM Conference on LISP and Functional Programming, 1986, pp. 151–161

    1986

  39. [39]

    Washing behind your ears: Principles of software hygiene

    D. M. Tilbrook and J. McMullen, “Washing behind your ears: Principles of software hygiene.”

  40. [40]

    Let me speak freely? a study on the impact of format restrictions on performance of large language models,

    Z. R. Tam, C.-K. Wu, Y .-L. Tsai, C.-Y . Lin, H.-y. Lee, and Y .- N. Chen, “Let me speak freely? a study on the impact of format restrictions on performance of large language models,”arXiv preprint arXiv:2408.02442, 2024

    Pith/arXiv arXiv 2024

  41. [41]

    Jsonschemabench: A rigorous benchmark of structured outputs for language models,

    S. Geng, H. Cooper, M. Moskal, S. Jenkins, J. Berman, N. Ranchin, R. West, E. Horvitz, and H. Nori, “Jsonschemabench: A rigorous benchmark of structured outputs for language models,”arXiv preprint arXiv:2501.10868, 2025

    Pith/arXiv arXiv 2025

  42. [42]

    Docprompting: Generating code by retrieving the docs,

    S. Zhou, U. Alon, F. F. Xu, Z. Jiang, and G. Neubig, “Docprompting: Generating code by retrieving the docs,” inThe Eleventh International Conference on Learning Representations, 2022

    2022

  43. [43]

    Self-planning code generation with large language models,

    X. Jiang, Y . Dong, L. Wang, Z. Fang, Q. Shang, G. Li, Z. Jin, and W. Jiao, “Self-planning code generation with large language models,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 7, pp. 1–30, 2024

    2024

  44. [44]

    Grammar prompting for domain-specific language generation with large language models,

    B. Wang, Z. Wang, X. Wang, Y . Cao, R. A Saurous, and Y . Kim, “Grammar prompting for domain-specific language generation with large language models,” vol. 36, 2023, pp. 65 030–65 055. 15 TABLE VII IRSCHEMA ITERATION HISTORY FROM INITIAL DRAFT TO FROZEN V0.2-RUNTIME-EVIDENCE. # Iteration What Changed Why Impact 1v0 static draft Defined six top-level field...

    2023

  45. [45]

    scripts[].object_id must reference objects[].id

  46. [46]

    scripts are per-instance (no sharing across objects)

  47. [47]

    no implicit aggregate placeholders

  48. [48]

    scene" | string

    rules[].evidence_type is required, in { direct_code, scene_override, inferred } APPENDIXB FULLERRORCODECENSUS 16 TABLE VIII: All 99 observed C# compiler error codes, categorized as Grounding (G) or Hygiene (H), with total occurrence count and message template. Quoted identifiers are shown asX; for Grounding codes, a representative identifier from the logs...

  49. [49]

    Every scripts[].object_id MUST reference a real objects[].id (no dangling refs)

  50. [50]

    Scripts are per-instance; no shared script entries across objects

  51. [51]

    Every entity must be listed explicitly in objects (no aggregate placeholders)

  52. [52]

    direct_code

    Every rules[] entry MUST include evidence_type in { "direct_code", "scene_override", "inferred" }. <PATTERN_MD>