Pith. sign in

REVIEW 3 major objections 5 minor 30 references

For LLM game agents, thinking mode and 10-step planning raise win rates while cutting per-step latency, making multi-step planning the practical lever.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 10:53 UTC pith:5644WYU7

load-bearing objection Solid benchmark and win-rate gains, but the latency trade-off claim is an artifact of the metric and the stats are too thin. the 3 major comments →

arxiv 2607.22732 v1 pith:5644WYU7 submitted 2026-07-22 cs.AI

Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step Planning

Mohit Jiwatode , Ronja Fuchs , Robin Schm\"ocker , Bodo Rosenhahn , Alexander Dockhorn This is my paper
classification cs.AI
keywords large language modelsspatial reasoninggame playingmulti-step planningplanning horizoncausal promptingself-localizationsymbolic grid navigation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that the poor performance of large-language-model game agents on spatial tasks is partly addressable without changing the model: enabling the model's internal thinking mode and asking it to plan several moves ahead both raise win rates, and multi-step planning also lowers the average time per step. To test this, it builds a controlled benchmark of three navigation games, each with five difficulty levels, using symbolic grid observations, and evaluates open-source models of four sizes with and without causal prompt context. The central finding is that thinking mode improves average win rates from about 8% to 43% but greatly increases per-step latency, while increasing the planning horizon from 1 to 10 steps improves win rate from about 18% to 31% and cuts mean per-step time from about 53 to 30 seconds. Causal prompt context yields marginal overall gains but modestly cuts latency and helps the largest model. The smaller models largely fail at coordinate self-localization, suggesting that spatial competence scales with model size.

Core claim

The paper claims that a large-language-model agent solving symbolic maze navigation in a general video-game environment benefits most from reasoning depth and action batching: thinking mode raises the aggregate win rate from 7.8% to 43.3%, and planning horizons of 5 and 10 steps outperform single-step decisions on every metric, improving win rate, completion rate, and mean per-step time. Causal prompt augmentation—supplying a structured description of game dynamics—improves win rate only marginally overall (24.6% to 25.0%), though it cuts per-step latency from about 46 seconds to 37 seconds and gives the largest model a relative win-rate gain of about 5.35%. The paper also shows that spatial

What carries the argument

The central mechanism is the planning-horizon prompt: instead of asking the model for one action per observation, the prompt asks for the next H actions (H=5 or 10) in a strict format, so the model pre-computes a trajectory and the environment executes it in batches. This amortizes the expensive thinking-mode latency over several moves. The supporting machinery is the custom spatial benchmark—three Video Game Description Language mazes of increasing logical complexity (pathfinding, key-gated exit, two-color key/door sequence) at five grid sizes—plus an optional causal-model prompt that describes entities, state variables, and transition rules as a structural equation model blueprint.

Load-bearing premise

The benchmark's win rates are treated as measuring spatial reasoning, but the tasks also require strict output formatting, symbolic state parsing, and instruction following; if those dominate, the spatial-reasoning conclusions do not follow.

What would settle it

Run the same prompt and horizon comparisons on a non-spatial control task with identical grid-and-format structure but no navigation requirement, such as matching a color sequence. If thinking mode and longer horizons produce the same win-rate and latency patterns, the observed gains come from format and instruction compliance rather than spatial reasoning. Alternatively, randomize the mapping between grid symbols and actions: if win rates do not collapse, the agent is not using the spatial layout.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • With thinking mode enabled, average win rate rises from about 8% to 43%, so reasoning depth is the single largest performance lever among the tested interventions.
  • Raising the planning horizon from 1 to 10 steps raises win rate from about 18% to 31%, completion from about 77% to 83%, and lowers mean per-step time from about 53s to 30s, so action batching recovers latency while improving success.
  • Causal context does not change overall win rate materially but cuts per-step latency, and for the largest model produces about a 5.35% relative win-rate improvement; smaller models do not benefit.
  • Win-rate degradation with level difficulty (about 40% at level 0 to 12% at level 4) validates the benchmark as a scaling test for spatial reasoning.
  • Smaller models fail at exact self-localization, so the practical path to spatial competence points to larger models or better spatial grounding, not prompt tweaks alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The combined win-rate gain from thinking mode and the latency recovery from multi-step planning suggest a deployment recipe for real-time LLM agents: use a reasoning-capable model, turn thinking on, and batch 5–10 actions to amortize cost—something the paper hints at but does not frame as a recipe.
  • Because agents navigate reasonably despite poor exact-coordinate output, they may rely on local relative cues; a test that removes global position information or demands long detours would show whether exact self-localization is actually required for robust navigation.
  • The paper's re-prompting of unparseable outputs leaves open whether the reported latency and win rates would hold under a strict fallback policy; a robustness variant with random or fallback actions could separate reasoning skill from output-format luck.
  • The causal context result is measured only as static prompt text in deterministic grids; in dynamic games with hazards that invalidate open-loop plans, a causal model that triggers replanning might show larger gains than the marginal effect seen here.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces a controlled GVGAI benchmark of three custom spatial navigation games with five difficulty levels and evaluates four open-source Qwen3 model sizes (0.6B–8B) under combinations of causal prompt augmentation, thinking mode, and planning horizon H ∈ {1,5,10}. A separate self-localization experiment tests the models' ability to report their own coordinates. The reported results are that win rates decrease with level complexity, thinking mode and longer planning horizons improve win rates, causal context gives a small improvement (mainly for the 8B model), and multi-step planning reduces mean per-step response time, which the authors interpret as a practical trade-off between reasoning depth and execution speed.

Significance. If the central claims survive scrutiny, the paper provides useful, reproducible evidence about LLM game agents on a focused spatial benchmark, with the strength of using open-source models and a clear experimental grid. The benchmark design and the per-model/per-level breakdowns are valuable assets. However, the headline latency claim is currently supported by a metric that mechanically favors longer horizons, and the paper lacks any uncertainty quantification despite using only two trials per configuration. Both issues are fixable, but they are load-bearing for RQ3 and for the practical trade-off statement in the abstract.

major comments (3)
  1. [Section IV, M1.3; Table I; Section V-C] The mean-time-per-step metric M1.3 is defined as total execution time divided by number of steps. For planning horizon H, the agent generates one response containing H actions, so the first executable action is available only after the full response is generated. Consequently, for H>1, M1.3 = T_generation/H, which mechanically decreases with H. Table I reports ≈53 s/step at H=1 and ≈30 s/step at H=10; this implies roughly 300 s of generation before the first action at H=10 versus ≈53 s at H=1. The metric therefore hides the time-to-first-action and makes longer horizons look faster even when the agent's first decision is much slower. RQ3 and the abstract's 'practical trade-off' claim are not supported by the reported metric. The authors should report time-to-first-action, episode wall-clock time, and per-output generation time separately.
  2. [Section IV-F; Table I; Section V-C] The paper uses exactly two independent trials per configuration and reports no confidence intervals, error bars, or significance tests. Yet the abstract and Section V-C use 'significantly improve' and 'the most beneficial intervention.' The causal contrast in Table I is 0.250 vs 0.246, a difference that is almost certainly within sampling noise; the horizon differences (0.178 vs 0.262 vs 0.310) are larger but still have no quantified uncertainty. The authors should add bootstrapped confidence intervals, proportion tests, or at least per-cell counts and the raw data so the reader can assess effect sizes.
  3. [Section VII-A; Section IV-F] The construct-validity limitation concedes that the tasks 'combine spatial formatting, symbolic state representation and interpretation, and action formatting.' Because unparseable outputs are re-prompted rather than falling back to a default action, and because no parse-failure or re-prompt counts are reported by condition, the observed win-rate differences may partly reflect formatting compliance and instruction-following rather than spatial reasoning. This is central to RQ1's claim of isolating spatial reasoning. The paper should report parse/reprompt rates per model, thinking mode, and horizon, and ideally analyze outcomes conditional on first-parse success.
minor comments (5)
  1. [Section IV-D, item 5; Figure 3] The text refers to Figure 3 for the planning prompt, but the figure is not included in the manuscript. The prompt template should be shown or placed in an appendix, since the output format is central to the multi-step planning results.
  2. [Section IV, metrics M2.1, M3.1, M3.2] These 'impact' metrics are not defined as absolute or relative deltas. The conclusion in Section VIII mentions a 'relatively improved' win rate of ≈5.35% for the 8B model, but the metric definitions do not say whether deltas are relative, absolute, or averaged over games/levels. Please specify.
  3. [Section V-C; Appendix D] The claim 'All the metrics show improvement for longer planning horizons' is only true for the aggregated average. Per-model results in Appendix D show non-monotonic behavior, e.g., qwen3-0.6b win rate at H=5 is 0.000 versus 0.008 at H=1, and qwen3-4b mean time at H=10 is higher than at H=5. The summary statement should be qualified as an average effect.
  4. [Section V-C; Table I] The mean time per step for thinking=off (0.544 s) versus thinking=on (85.776 s) is extremely large and likely reflects the length of generated reasoning traces. Reporting token counts or generation-length distributions would make this difference interpretable and would strengthen the latency analysis.
  5. [Figures 16–18] These figures are labeled 'Completion' but show win rate and mean normalized reward. The y-axis titles are misleading; please relabel to match the plotted quantities. Also, 'Mean Normalized Reward' is never defined in the metrics section.

Circularity Check

1 steps flagged

Multi-step planning latency claim reduces to metric M1.3 by construction; other empirical results are independent.

specific steps
  1. self definitional [Section IV (M1.3), Section IV-D (planning prompt), Section V-C, Table I, Abstract]
    "M1.3 Mean Time per Step: The total execution time of a run divided by the number of steps taken. ... Optionally, in case of planning tasks, a prompt requiring the LLM to plan the next x actions ... All the metrics show improvement for longer planning horizons, making it the most beneficial intervention in our study. ... multi-step planning further reduces mean per-step response times, offering a practical trade-off between reasoning depth and execution speed."

    M1.3 divides total run time by the number of game steps. With planning horizon H, one LLM response emits H actions, so an episode of S steps requires about S/H responses; hence M1.3 approximates per-response latency divided by H. The reported decrease from 53.1s (H=1) to 29.9s (H=10) is therefore forced by the metric's arithmetic, independent of any genuine speed improvement. Time-to-first-action, the latency the agent actually experiences at decision time, increases with H, so the abstract's 'practical trade-off' claim relies on a definitional artifact rather than on measured action-level latency.

full rationale

The only circular step is the planning-horizon latency claim. M1.3 is defined as total execution time divided by the number of steps, and planning batches H actions into a single LLM response; thus mean time per step mechanically falls as H grows. This makes the abstract's claim that multi-step planning 'reduces mean per-step response times' and Section V-C's conclusion that 'all metrics show improvement for longer planning horizons' partially equivalent to the metric definition. The other central results--win-rate improvements from thinking mode, difficulty scaling, and causal-prompt deltas--are independent empirical measurements not fitted to the conclusions. The only self-citation is the SCM blueprint from Jiwatode et al. [10] used to generate causal prompts; this is method reuse, not a load-bearing result that presupposes the outcome, and no uniqueness theorem or fitted parameter is invoked. Score 6 reflects partial circularity of one central latency claim, not full circularity of the study.

Axiom & Free-Parameter Ledger

0 free parameters · 5 axioms · 0 invented entities

The paper is an empirical study, not a derivation, so there are no fitted constants. The claims rest on the benchmark isolating spatial reasoning, on the LLM-generated causal model being correct, on deterministic environments making open-loop plans valid, on two-trial sampling being representative, and on the re-prompting policy not biasing completion and latency.

axioms (5)
  • domain assumption The three custom GVGAI games isolate spatial navigation from other game mechanics.
    Stated in Section IV-B and Section VII-A; if false, RQ1/RQ2 spatial conclusions are confounded by formatting and instruction-following.
  • domain assumption The LLM-generated causal model (using the SCM blueprint from Jiwatode et al. [10]) is a correct and useful representation of game dynamics.
    Invoked in Section IV-D and Figure 4; the paper does not verify the generated causal model's correctness, so RQ2 conclusions depend on it.
  • ad hoc to paper Re-prompting unparseable model outputs, rather than using a fallback action, does not systematically bias completion and latency comparisons.
    Section VI states unparseable outputs were re-prompted, but no retry counts or costs are reported; this affects thinking-mode completion rates and time-per-step.
  • domain assumption Two independent trials per configuration are sufficient to estimate win rates for comparison.
    Section IV-F; no error bars or statistical tests, so reported deltas (e.g., causal on/off) may be sampling noise.
  • domain assumption The GVGAI environment is deterministic, so open-loop fixed-horizon plans remain valid.
    Section VII-B; if there were stochasticity, multi-step planning gains could be due to replanning or chance.

pith-pipeline@v1.3.0-alltime-deepseek · 15711 in / 14610 out tokens · 125196 ms · 2026-08-01T10:53:29.713577+00:00 · methodology

0 comments
read the original abstract

LLM-based game agents often perform poorly on more complex tasks. This work examines whether these failures are linked to limited spatial reasoning and evaluates whether causal prompt augmentation and multi-step planning can improve win-rates while managing response latency. Using the open-source Qwen3 model family, we conduct experiments across varying model scales, reasoning modes, and planning horizons. We further introduce a focused GVGAI benchmark consisting of three custom games with five difficulty levels to isolate spatial navigation. The evaluation follows two paradigms: an initial ``positioning experiment'' to test an agent's ability to find its exact coordinates, and a study of game-play success. Our results show that while larger models with an enabled thinking mode identify their positions more accurately, overall performance in coordinate matching remains limited for smaller models. Win rates decrease as game levels and layout complexity increase, validating the benchmark's difficulty scaling. Integrating causal context into the prompts tends to improve the agents' success rates, particularly for bigger models. While enabling thinking mode and longer planning horizons significantly improve performance, multi-step planning further reduces mean per-step response times, offering a practical trade-off between reasoning depth and execution speed.

Figures

Figures reproduced from arXiv: 2607.22732 by Alexander Dockhorn, Bodo Rosenhahn, Mohit Jiwatode, Robin Schm\"ocker, Ronja Fuchs.

Figure 1
Figure 1. Figure 1: Overview of all the levels in the spatial games. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: An example of the current state provided to the LLM as an input. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prompt used to obtain next “x” actions from the LLMs [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 5
Figure 5. Figure 5: Win-rate heatmap for the spatial benchmark over games and levels. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Win rate by game, split by thinking mode. [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Model-wise Exact Position Match Rate by Thinking Mode. The 8B [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Win-rates vs. planning horizon for the models studied. Lines [PITH_FULL_IMAGE:figures/full_fig_p006_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Completion rates vs. planning horizon for the models studied. Lines [PITH_FULL_IMAGE:figures/full_fig_p006_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Mean time per step vs. planning horizon for the models studied. [PITH_FULL_IMAGE:figures/full_fig_p007_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Prompt used to translate VGDL into natural language [PITH_FULL_IMAGE:figures/full_fig_p009_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: An example of the natural language description of the game generated [PITH_FULL_IMAGE:figures/full_fig_p009_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Win-rate heatmap comparing causal vs non-causal prompts across [PITH_FULL_IMAGE:figures/full_fig_p012_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Completion rate per game when thinking mode is toggled. [PITH_FULL_IMAGE:figures/full_fig_p013_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Per-model completion for SpatialGame1. 0 1 2 3 4 Level 0.0 0.2 0.4 0.6 0.8 1.0 Win Rate local-qwen3-0.6b-vllm local-qwen3-1.7b-vllm local-qwen3-4b-vllm local-qwen3-8b-vllm 0 1 2 3 4 Level 0.0 0.2 0.4 0.6 0.8 1.0 Mean Normalized Reward local-qwen3-0.6b-vllm local-qwen3-1.7b-vllm local-qwen3-4b-vllm local-qwen3-8b-vllm [PITH_FULL_IMAGE:figures/full_fig_p013_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Per-model completion for SpatialGame2. 0 1 2 3 4 Level 0.0 0.2 0.4 0.6 0.8 1.0 Win Rate local-qwen3-0.6b-vllm local-qwen3-1.7b-vllm local-qwen3-4b-vllm local-qwen3-8b-vllm 0 1 2 3 4 Level 0.0 0.2 0.4 0.6 0.8 1.0 Mean Normalized Reward local-qwen3-0.6b-vllm local-qwen3-1.7b-vllm local-qwen3-4b-vllm local-qwen3-8b-vllm [PITH_FULL_IMAGE:figures/full_fig_p013_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Per-model completion for SpatialGame3. 0.00 0.25 0.50 0.75 1.00 Completion Rate 1.00 1.00 1.00 0.63 0.93 0.57 1.000.93 0.97 0.93 0.80 0.70 qwen3-8b 0.00 0.25 0.50 0.75 1.00 Completion Rate 0.97 1.00 0.97 0.80 0.70 0.40 1.00 1.00 1.00 0.87 0.60 0.30 qwen3-4b 0.00 0.25 0.50 0.75 1.00 Completion Rate 1.00 1.00 1.00 0.93 0.87 0.97 0.95 0.97 1.00 0.97 0.69 0.88 qwen3-1.7b spatialgame1 spatialgame2 spatialgame3… view at source ↗
Figure 21
Figure 21. Figure 21: Win rate per game for each model [PITH_FULL_IMAGE:figures/full_fig_p014_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Completion-rate heatmap arranged by game and level difficulty. [PITH_FULL_IMAGE:figures/full_fig_p015_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Win-rate heatmap arranged by model scale and level difficulty. [PITH_FULL_IMAGE:figures/full_fig_p015_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Summary table for the causal prompt experiments. [PITH_FULL_IMAGE:figures/full_fig_p016_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Summary table for the planning horizon comparisons. [PITH_FULL_IMAGE:figures/full_fig_p017_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Summary table for the thinking-mode experiments. [PITH_FULL_IMAGE:figures/full_fig_p018_26.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 9 linked inside Pith

  1. [1]

    Gvgai- llm: Evaluating large language model agents with infinite games,

    Y . Li, C. Lin, M. U. Nasir, P. Bontrager, J. Liu, and J. Togelius, “Gvgai- llm: Evaluating large language model agents with infinite games,”arXiv preprint arXiv:2508.08501, 2025

    Pith/arXiv arXiv 2025

  2. [2]

    From code to play: Benchmarking program search for games using large language models,

    M. Eberhardinger, J. Goodman, A. Dockhorn, D. Perez-Liebana, R. D. Gaina, D. Cakmak, S. Maghsudi, and S. Lucas, “From code to play: Benchmarking program search for games using large language models,” IEEE Transactions on Games, 2025

    2025

  3. [3]

    D. P. Li ´ebana, S. M. Lucas, R. D. Gaina, J. Togelius, A. Khalifa, and J. Liu,General video game artificial intelligence. Springer Nature, 2022

    2022

  4. [4]

    An extensible description language for video games,

    T. Schaul, “An extensible description language for video games,”IEEE Transactions on Computational Intelligence and AI in Games, vol. 6, no. 4, pp. 325–331, 2014

    2014

  5. [5]

    Deep reinforcement learning for general video game ai,

    R. R. Torrado, P. Bontrager, J. Togelius, J. Liu, and D. Perez-Liebana, “Deep reinforcement learning for general video game ai,” in2018 IEEE conference on computational intelligence and games (CIG). IEEE, 2018, pp. 1–8

    2018

  6. [6]

    General video game ai: A multitrack framework for evaluating agents, games, and content generation algorithms,

    D. Perez-Liebana, J. Liu, A. Khalifa, R. D. Gaina, J. Togelius, and S. M. Lucas, “General video game ai: A multitrack framework for evaluating agents, games, and content generation algorithms,”IEEE Transactions on Games, vol. 11, no. 3, pp. 195–214, 2019

    2019

  7. [7]

    Representation learning for grounded spatial reasoning,

    M. Janner, K. Narasimhan, and R. Barzilay, “Representation learning for grounded spatial reasoning,”Transactions of the Association for Computational Linguistics, vol. 6, pp. 49–61, 2018

    2018

  8. [8]

    Sparc and sparp: Spatial reasoning characterization and path generation for understanding spatial reasoning capability of large language models,

    M. I. Rizvi, X. Zhu, and I. Gurevych, “Sparc and sparp: Spatial reasoning characterization and path generation for understanding spatial reasoning capability of large language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 4750–4767

    2024

  9. [9]

    The sifo benchmark: Investigating the sequential instruction following ability of large language models,

    X. Chen, B. Liao, J. Qi, P. Eustratiadis, C. Monz, A. Bisazza, and M. de Rijke, “The sifo benchmark: Investigating the sequential instruction following ability of large language models,” inFindings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 1691–1706

    2024

  10. [10]

    From gameplay traces to game mechanics: Causal induction with large language models,

    M. Jiwatode, A. Dockhorn, and B. Rosenhahn, “From gameplay traces to game mechanics: Causal induction with large language models,”arXiv preprint arXiv:2602.00190, 2026

    arXiv 2026

  11. [11]

    Causal-aware large language models: Enhancing decision-making through learning, adapting and acting,

    W. Chen, J. Zhang, H. Zhu, B. Xu, Z. Hao, K. Zhang, J. Ye, and R. Cai, “Causal-aware large language models: Enhancing decision-making through learning, adapting and acting,”arXiv preprint arXiv:2505.24710, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Bayesian inference-aided large language model agents in infinitely repeated games: A dynamic network view,

    D. Pan, W. Chen, J. Shi, C. Wu, D. Wang, C. S. Hong, and Z. Han, “Bayesian inference-aided large language model agents in infinitely repeated games: A dynamic network view,”IEEE Transactions on Network Science and Engineering, 2025

    2025

  13. [13]

    Optimalthinkingbench: Evaluating over and underthinking in llms,

    P. Aggarwal, S. Kim, J. Lanchantin, S. Welleck, J. Weston, I. Kulikov, and S. Saha, “Optimalthinkingbench: Evaluating over and underthinking in llms,”arXiv preprint arXiv:2508.13141, 2025

    arXiv 2025

  14. [14]

    Smartplay: A benchmark for llms as intelligent agents,

    Y . Wu, X. Tang, T. M. Mitchell, and Y . Li, “Smartplay: A benchmark for llms as intelligent agents,”arXiv preprint arXiv:2310.01557, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Gamebench: Evaluating strate- gic reasoning abilities of llm agents,

    A. Costarelli, M. Allen, R. Hauksson, G. Sodunke, S. Hariharan, C. Cheng, W. Li, J. Clymer, and A. Yadav, “Gamebench: Evaluating strate- gic reasoning abilities of llm agents,”arXiv preprint arXiv:2406.06613, 2024

    Pith/arXiv arXiv 2024

  16. [16]

    Balrog: Bench- marking agentic llm and vlm reasoning on games,

    D. Paglieri, B. Cupiał, S. Coward, U. Piterbarg, M. Wolczyk, A. Khan, E. Pignatelli, Ł. Kuci ´nski, L. Pinto, R. Ferguset al., “Balrog: Bench- marking agentic llm and vlm reasoning on games,”arXiv preprint arXiv:2411.13543, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Measuring gen- eral intelligence with generated games,

    V . Verma, D. Huang, W. Chen, D. Klein, and N. Tomlin, “Measuring gen- eral intelligence with generated games,”arXiv preprint arXiv:2505.07215, 2025

    Pith/arXiv arXiv 2025

  18. [18]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,” inThe eleventh international conference on learning representations, 2022

    2022

  19. [19]

    Generative agents: Interactive simulacra of human behavior,

    J. S. Park, J. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein, “Generative agents: Interactive simulacra of human behavior,” inProceedings of the 36th annual acm symposium on user interface software and technology, 2023, pp. 1–22

    2023

  20. [20]

    V oyager: An open-ended embodied agent with large language models,

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended embodied agent with large language models,”arXiv preprint arXiv:2305.16291, 2023

    Pith/arXiv arXiv 2023

  21. [21]

    Adam: An embodied causal agent in open-world environments,

    S. Yu and C. Lu, “Adam: An embodied causal agent in open-world environments,”arXiv preprint arXiv:2410.22194, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Passive learning of active causal strategies in agents and language models,

    A. Lampinen, S. Chan, I. Dasgupta, A. Nam, and J. Wang, “Passive learning of active causal strategies in agents and language models,” Advances in Neural Information Processing Systems, vol. 36, pp. 1283– 1297, 2023

    2023

  23. [23]

    Curious causality-seeking agents learn meta causal world,

    Z. Zhao, H. Li, H. Zhang, J. Wang, F. Faccio, J. Schmidhuber, and M. Yang, “Curious causality-seeking agents learn meta causal world,” arXiv preprint arXiv:2506.23068, 2025

    arXiv 2025

  24. [24]

    Wohlin, P

    C. Wohlin, P. Runeson, M. H ¨ost, M. C. Ohlsson, B. Regnell, A. Wessl ´en et al.,Experimentation in software engineering. Springer, 2012, vol. 236

    2012

  25. [25]

    Qwen3 technical report,

    Qwen-Team, “Qwen3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  26. [26]

    Qwen/qwen3-8b · hugging face - huggingface.co,

    “Qwen/qwen3-8b · hugging face - huggingface.co,” https://huggingface. co/Qwen/Qwen3-8B#best-practices, [Accessed 15-03-2026]

    2026

  27. [27]

    Wait, wait, wait... why do reasoning models loop?

    C. Pipis, S. Garg, V . Kontonis, V . Shrivastava, A. Krishnamurthy, and D. Papailiopoulos, “Wait, wait, wait... why do reasoning models loop?” arXiv preprint arXiv:2512.12895, 2025. APPENDIXA PROMPT TO TRANSLATEVGDLINTO NATURAL LANGUAGE AND AN EXAMPLE OF GENERATED NATURAL LANGUAGE DESCRIPTION Fig. 12. Prompt used to translate VGDL into natural language Fi...

    arXiv 2025

  28. [28]

    VGDL Game Description for SpatialGame1 B

    > floor 13 14InteractionSet 15avatar wall > stepBack 16goal avatar > killSprite scoreChange=1 17 18TerminationSet 19SpriteCounter stype=goal limit=0 win=True 20SpriteCounter stype=avatar limit=0 win= False 21Timeout limit=6000 win=False Listing 4. VGDL Game Description for SpatialGame1 B. spatialgame2 L0 L1 L2 wwwwww wwwwwww wwwwwwww wA.kew wA..w.w wA...w...

  29. [29]

    VGDL Game Description for SpatialGame2 C

    > floor 17 18InteractionSet 19avatar wall > stepBack 20nokey key > transformTo stype=withkey scoreChange=1 killSecond=True 21exit nokey > stepBack 22exit withkey > killSprite scoreChange=2 23 24TerminationSet 25SpriteCounter stype=exit limit=0 win=True 26SpriteCounter stype=avatar limit=0 win= False 27Timeout limit=6000 win=False Listing 7. VGDL Game Desc...

  30. [30]

    VGDL Game Description for SpatialGame3 APPENDIXD EXTENDEDSPATIALREASONINGVISUALIZATIONS A

    > floor 26 27InteractionSet 28avatar wall door > stepBack 29key avatar > collectResource scoreChange=1 30 31reddoor avatar > killIfOtherHasMore resource=redkey limit=0 scoreChange=1 32avatar reddoor > changeResource resource= redkey value=-1 33 34bluedoor avatar > killIfOtherHasMore resource=bluekey limit=0 scoreChange=1 35avatar bluedoor > changeResource...