Pith. sign in

REVIEW 6 major objections 6 minor 52 references

Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making

T0 review · 6 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that LLM quality should be judged by how models plan, revise, and respect limits inside rule-bound strategic games, and reports process metrics showing that disciplined, selective revision beats frequent self-editing.

desk verdict The process-evaluation framework is worth engaging with, but the reported rankings and the headline OBR–win-rate correlation are not reliable as-is due to arithmetic errors, an unstated parsing pipeline, and a partly circular metric. read the letter →

arxiv 2506.12012 v1 pith:XXAHKAVX submitted 2025-06-13 cs.AI

Xiaopeng Yuan , Xingjian Zhang , Ke Xu , Yifan Xu , Lijun Yu , Jindong Wang , Yushun Dong , Haohan Wang This is my paper
classification cs.AI
keywords LLMevaluationstrategicgamesreasoningprocessesplanningrevisionbehaviorresourceconstraintsprocessmetricsbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that evaluating LLMs by final answers alone misses how they reason. It proposes strategic games—tower defense, auto-battler, and turn-based combat—as closed, rule-based environments in which planning, revision, and resource-constrained decisions can be traced. It defines metrics beyond win rate: over-correction risk rate, correction success rate, improvement slope, and over-budget ratio. Across 4,320 adversarial rounds with 12 models, it finds ChatGPT-o3-mini balances all dimensions (74.7% win rate, 78.6% correction success, +0.041 improvement slope), while Qwen-Plus over-corrects (81.6% over-correction risk rate) and wins only 25.6%; over-correction risk correlates negatively with correction success (r = –0.51, p = 0.093). The paper concludes that process-level measurement reveals model reliability that outcome-only benchmarks hide.

What carries the argument

The central machinery is the closed-loop game simulator paired with four process metrics. In each round, a model emits a free-form strategy proposal, the deterministic simulator executes it and returns rule-based feedback, and the model may optionally revise; the logged sequence is scored by over-correction risk rate (probability of revising immediately after negative feedback), correction success rate (probability a revision improves the result), improvement slope (linear trend of win rate across rounds against matched opponents), and over-budget ratio (fraction of turns exceeding resource limits). These metrics make internal reasoning observable without human annotation.

What would settle it

Recompute the reported metrics from the raw interaction logs while counting every strategy proposal that the simulator's parser rejected or had to repair before execution; if parse failures are frequent or unevenly distributed across models, the over-correction risk, correction success, and over-budget figures measure parsing fidelity rather than the models' planning.

Watch

Extended reading notes

Core claim

The central finding is that process quality splits the models in ways win rate alone does not explain. The o3 family wins by planning soundly, revising rarely but successfully, and never exceeding budget, whereas Qwen models revise impulsively, exceed resource limits, and lose despite high correction effort. The negative correlation between over-correction risk rate and correction success rate, together with the strong negative correlation between over-budget ratio and win rate (r = –0.95, p < 0.001), is offered as evidence that disciplined, selective revision and budget adherence, not correction frequency, drive success. This is the paper's case that what a model does before and between decisions is as diagnostic as the final outcome.

Load-bearing premise

The results assume that the models' free-form strategy texts are translated into game actions without loss or distortion, so the logged moves, rule violations, and revisions reflect the models' own decisions rather than the parser's.

Editorial extensions

If this is right

  • Win rate alone understates model differences: models with similar outcomes can differ sharply in revision discipline and budget adherence, so process metrics should accompany outcome metrics in LLM evaluation.
  • Over-correction is counterproductive; systems that revise less but more accurately achieve higher success, as the negative ORR–CSR correlation indicates.
  • Budget fidelity is a leading indicator of performance: the two models with zero budget violations posted the highest win rates, and over-budget ratio correlates with win rate at r = –0.95.
  • First-mover asymmetries emerge in revision behavior, so balanced move-order evaluation is necessary to separate model capability from positional advantage.
  • Process metrics could guide alignment and training, rewarding selective revision and constraint adherence rather than raw persistence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the process metrics genuinely track reasoning quality, they could be turned into training signals: rewarding low over-correction risk and high correction success might reduce the 'spray-and-pray' revision patterns seen in deployed agents.
  • The framework's reliance on parsing free-form strategy text is a stress point; re-running the benchmark with an independent parser or human-verified strategy translations would show whether the model rankings survive the parsing layer.
  • Because the three games are deterministic, the same rounds could be replayed under perturbed budgets or rule sets to test whether the process metrics generalize to other resource-constrained decision-making settings.
  • The peashooter contamination episode shows that pretraining-memory cues can silently invalidate game benchmarks, suggesting future environments should include a lexical-cue check beyond the redesign described in the paper.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper proposes AdvGameBench, a framework for evaluating LLM reasoning processes by embedding models in three turn-based strategic games (tower defense, auto-battler, turn-based combat). It defines process-level metrics—win rate, over-correction risk rate (ORR), correction success rate (CSR), improvement slope, and over-budget ratio (OBR)—and reports an evaluation of 12 LLMs. The central claim is that ChatGPT-o3-mini achieves the best composite process score (74.7% win rate, 78.6% CSR, +0.041 slope), while Qwen-Plus over-corrects and performs poorly, and that process metrics reveal behaviors that win-rate alone misses. The paper also reports correlations between ORR and other metrics and argues that budget fidelity is a leading indicator of success.

Significance. If the empirical claims were reproducible and the framework were fully specified, AdvGameBench would be a useful step toward process-aware LLM evaluation: the three-game design targets distinct reasoning demands, and the metric suite (with explicit formulas in Appendix B) extends beyond outcome-only benchmarks. The paper explicitly promises public code and uses deterministic simulators, which are commendable. However, in its current form the empirical support is not reliable: the headline OBR–WR correlation is circular under the paper's own win-rate definition, the strategy-to-simulation parser is unspecified, the environment version that produced the reported numbers is unclear, and several internal arithmetic and statistical inconsistencies appear in Table 1 and in the correlations reported across sections. The significance of the proposed direction is real, but the current manuscript does not yet substantiate its central empirical claims.

major comments (6)
  1. [§3.3 and §4.3] The OBR–WR correlation (Fig. 4, r = –0.95, p < 0.001) is circular by construction. The win rate definition states that 'rule violations result in immediate forfeiture' (§3.3), and OBR counts proposals that exceed explicit resource constraints, which is a rule violation. A model that exceeds budget frequently is therefore automatically more likely to lose, independent of any deeper planning or revision quality. The claim that 'OBR ... goes beyond outcome accuracy' (end of §4.3) is not supported by this analysis. Please recompute a win rate that excludes budget-violation forfeitures, or otherwise separate budget compliance from other causes of loss, and re-examine whether the residual correlation remains.
  2. [§3.1 and §3.2] The strategy-to-simulation pipeline is not described. The paper states only that 'the simulator executes both strategies' (Fig. 1 caption, §3.1), but free-form LLM proposals must be parsed into executable unit lists, positions, and skill rotations. No parser grammar, output format, or validation set is provided, and Appendix A gives only game rules. Because parse failures are indistinguishable from planning failures under the paper's own WR/OBR definitions—a malformed proposal could be logged as a rule violation or as a loss—the reported model rankings (e.g., ChatGPT-o3-mini 74.7% vs. Qwen-Plus 25.6%) could reflect parser compatibility rather than reasoning quality. Please specify the exact interface between model output and simulator, the strategy representation, and the handling of unparseable or out-of-spec outputs, and report parse success rates per model.
  3. [§5, 'Hallucation' paragraph] The paper states that the tower-defense environment was redesigned to remove pretraining cues after models produced 'peashooter' outputs, but it never states whether Table 1 and Figures 2–7 were computed on the original or the redesigned environment. If the numbers come from the original version, the acknowledged memory bias undermines their validity; if they come from the redesigned version, the paper must confirm that the redesign did not change game difficulty (unit costs, spawn rates, win conditions) and provide the diff. Without this information, the main results cannot be attributed to either environment. Please state explicitly which environment generated each reported result and provide the redesign details.
  4. [§4.1, Table 1] There are arithmetic inconsistencies in Table 1 that affect the reported rankings. The ChatGPT-4o average win rate is listed as 58.6%, but the three per-game win rates (65.8, 60.8, 59.1) average to 61.9%, and 58.6 is lower than every per-game value. Similarly, the ChatGPT-o3-mini average ORR is listed as 24.5, while the per-game ORRs (25.9, 31.6, 9.0) average to 22.2; the text in the same subsection says 'a relatively low ORR of 49.3%,' which matches neither value. Please recompute all table entries and check every derived average, since the summary statistics are the core of the paper's empirical contribution.
  5. [§4.4, Abstract, and §5] The ORR–CSR correlation is reported with conflicting values across the manuscript. The abstract and §5 state Pearson r = –0.51, p = 0.093; §4.4 states 'lower correction success rates (r = –0.34, p = 0.28)'; and Figure 5(b) shows 'ORR vs CSR (r=-0.404, p=0.192).' These are three different statistics for the same pair of metrics. The reader cannot determine the true relationship, and the abstract's framing of this as a 'finding' is not supported when §4.4 itself notes that none of the correlations reach conventional significance (n = 12). Please unify the statistical reporting, recompute all correlations from the raw data, and report n, confidence intervals, or effect sizes appropriately.
  6. [Abstract and §4] The term 'composite process score' is used to rank ChatGPT-o3-mini first, but no definition or formula is provided anywhere in the paper. Without knowing how WR, ORR, CSR, OBR, and slope are weighted or normalized to form the composite, the claim that a particular model achieves the 'top composite process score' is not checkable. Please define the composite score explicitly, including any weights or normalization steps, and report each model's composite value.
minor comments (6)
  1. [Abstract vs. §5] The abstract reports 4,320 adversarial rounds, while §5 reports 4,752; please reconcile these numbers.
  2. [§5, heading] The section heading 'Hallucation' appears to be a typo; if intended as 'Hallucination,' please correct it.
  3. [§3.3, ORR definition] The name 'Over-Correction Risk Rate' is potentially misleading: the definition measures the frequency of revision after negative feedback, not the frequency of harmful or excessive corrections. Consider renaming it (e.g., 'Revision Rate after Negative Feedback') or clarifying the terminology.
  4. [§4.2 and Figure 3] The text says 'Figure 3 reports improvement slopes' but Figure 3 is captioned 'Win-rate trajectories across five rounds'; please align the figure captions with the text.
  5. [§4.5] The first-mover-advantage values are reported without any uncertainty or significance testing, even though the win-rate differences are within five percentage points for most models; please add error bars or confidence intervals.
  6. [§3.2] The three game descriptions are brief; please add a short example of a strategy proposal and its parsed form to make the evaluation pipeline concrete.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline OBR–win-rate correlation is definitionally forced; the remaining process-metric claims are empirical but their implementation (parser, environment version, composite score) is underspecified.

  1. self definitional [Section 3.3 (Win Rate, Over-Budget Rate) and Section 4.3 (OBR–WR correlation); see also Appendix B.1]
    "Win Rate measures the proportion of matches a model wins out of all played games, with rule violations resulting in immediate forfeiture. ... OBR measures how often a model generates proposals that exceed explicit resource constraints."

    Appendix B.1 defines a rule violation as including budget constraints (V(S)=1 if S 'violates any game rule (including budget constraints)'). Therefore an over-budget proposal is a rule violation, and the paper's own WR definition turns every such violation into an immediate forfeiture. The reported r=-0.95 between OBR and WR is then an algebraic consequence of the two definitions: the more turns a model goes over budget, the more matches it must lose by forfeiture. Calling budget fidelity a 'leading indicator of success' renames this definitional link as an empirical discovery rather than testing it.

full rationale

The only load-bearing result that reduces to its own definitions is the OBR–WR relationship. ORR, CSR, and improvement slope are not forced by the equations in §3.3: ORR conditions on negative feedback, CSR conditions on revision outcomes, and slope is a time trend, so the headline ORR–CSR correlation (r=-0.51, p=0.093) has independent empirical content. I also note, without counting it as circularity, that the paper leaves unstated the parser converting free-form strategy proposals into executable game actions, does not define the 'composite process score' used for the top ranking, and does not state whether Table 1 and Figures 2–7 were produced on the pre- or post-'Hallucation' tower-defense environment; these are reproducibility/external-validity gaps, not internal circularity. No self-citation chain or imported uniqueness theorem appears. Score 6 because one central reported correlation is definitionally forced, which is partial circularity; the rest of the claim stands or falls on empirical measurement.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The framework itself has no fitted model parameters, but the ranking depends on an unspecified composite-score weighting. The central assumptions are about simulator fidelity, lossless parsing, the effectiveness of the post-hoc environment redesign, and generalization to real-world reasoning, none of which are validated externally.

free parameters (1)
  • Composite process score weighting = Not specified
    The abstract ranks models by a 'top composite process score', but the formula and weights for combining WR, CSR, improvement slope, and OBR are never defined. The reported ranking depends on this hidden weighting.
assumptions (4)
  • domain assumption The three game simulators faithfully implement the stated rules and produce rule-verifiable outcomes, including forfeitures for rule violations.
    Section 3.1 asserts a 'closed, deterministic game simulator governed by explicit rules'; no code or test suite is provided, so correctness is assumed.
  • domain assumption Free-form LLM strategy outputs are parsed losslessly into executable game actions.
    Section 3.1 says models 'generate strategies' that the simulator executes, but the parsing pipeline is not described or validated.
  • ad hoc to paper The environment redesign after the 'peashooter' observation removes pretraining-memory cues without changing game difficulty.
    Section 5 'Hallucation' reports the peashooter artifact and claims a redesign, but gives no implementation details or evidence that all reported results were produced on the redesigned environment.
  • domain assumption Performance in these three synthetic game genres generalizes to real-world planning, revision, and resource-constrained decision making.
    The introduction and conclusion claim relevance to deployment settings; no external validation or correlation with real-world tasks is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making." pith.science (2026). https://pith.science/paper/XXAHKAVX

@misc{pith2026250612012,
  author       = {Pith},
  title        = {Pith review of: Tracing LLM Reasoning Processes with Strategic Games: A Framework for Planning, Revision, and Resource-Constrained Decision Making},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XXAHKAVX}},
  note         = {Machine review of arXiv:2506.12012}
}
read the original abstract

Large language models (LLMs) are increasingly used for tasks that require complex reasoning. Most benchmarks focus on final outcomes but overlook the intermediate reasoning steps - such as planning, revision, and decision making under resource constraints. We argue that measuring these internal processes is essential for understanding model behavior and improving reliability. We propose using strategic games as a natural evaluation environment: closed, rule-based systems with clear states, limited resources, and automatic feedback. We introduce a framework that evaluates LLMs along three core dimensions: planning, revision, and resource-constrained decision making. To operationalize this, we define metrics beyond win rate, including overcorrection risk rate, correction success rate, improvement slope, and over-budget ratio. In 4320 adversarial rounds across 12 leading models, ChatGPT-o3-mini achieves the top composite score, with a win rate of 74.7 percent, a correction success rate of 78.6 percent, and an improvement slope of 0.041. By contrast, Qwen-Plus, despite an overcorrection risk rate of 81.6 percent, wins only 25.6 percent of its matches - primarily due to excessive resource use. We also observe a negative correlation between overcorrection risk rate and correction success rate (Pearson r = -0.51, p = 0.093), suggesting that more frequent edits do not always improve outcomes. Our findings highlight the value of assessing not only what LLMs decide but how they arrive at those decisions

Figures

Figures reproduced from arXiv: 2506.12012 by the authors.

Figure 1
Figure 1. This figure illustrates the AdvGameBench evaluation pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Planning performance: slope vs. initial win rate [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 4
Figure 4. Over-Budget Ratio for Each Model [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Correlation analysis between over-correction risk rate (ORR) and four main metrics across [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: First-mover advantage (FMA) across three behavioral dimensions. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]
Figure 7
Figure 7. Figure 7: Model performance metrics To synthesize model performance across reason￾ing dimensions, we constructed a radar chart visualizing five normalized metrics: win rate (WR), correction success rate (CSR), improve￾ment slope, 1 – over-correction risk rate (ORR), and 1 – over…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 23 canonical work pages

  1. [1]

    J., Bethge, M., and Schulz, E

    Akata, E., Schulz, L., Coda-Forno, J., Oh, S. J., Bethge, M., and Schulz, E. (2023). Playing repeated games with large language models. In arXiv preprint arXiv:2305.16867v1

    arXiv 2023

  2. [2]

    Anil, R., Borgeaud, S., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., Millican, K., Silver, D., Johnson, M., Antonoglou, I., Schrittwieser, J., Glaese, A., Chen, J., Pitler, E., Lillicrap, T., Lazaridou, A., Firat, O., Molloy, J., Isard, M., Barham, P. R., Hennigan, T., Lee, B., Viola, F., Reynolds, M., Xu, Y ., Doherty, R., C...

    arXiv 2025

  3. [3]

    The claude 3 model family: Opus, Sonnet, Haiku

    Anthropic (2024). The claude 3 model family: Opus, Sonnet, Haiku. Technical report, Anthropic. Model Card

    work page 2024

  4. [4]

    Bai, J., Bai, S., Chu, Y ., Cui, Z., Dang, K., Deng, X., Fan, Y ., Ge, W., Han, Y ., Huang, F., Hui, B., Ji, L., Li, M., Lin, J., Lin, R., Liu, D., Liu, G., Lu, C., Lu, K., Ma, J., Men, R., Ren, X., Ren, X., Tan, C., Tan, S., Tu, J., Wang, P., Wang, S., Wang, W., Wu, S., Xu, B., Xu, J., Yang, A., Yang, H., Yang, J., Yang, S., Yao, Y ., Yu, B., Yuan, H., Y...

    arXiv 2023

  5. [5]

    N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M

    Chiang, W.-L., Zheng, L., Sheng, Y ., Angelopoulos, A. N., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M. I., Gonzalez, J. E., and Stoica, I. (2024). Chatbot arena: An open platform for evaluating LLMs by human preference. In arXiv preprint arXiv:2403.04132v1

    arXiv 2024

  6. [6]

    Cobbe, K., Kosaraju, V ., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. (2021). Training verifiers to solve math word problems. In arXiv preprint arXiv:2110.14168v2. Version v1 submitted on 27 Oct 2021, v2 (this version) revised 18 Nov 2021

    arXiv 2021

  7. [7]

    Costarelli, A., Allen, M., Hauksson, R., Sodunke, G., Hariharan, S., Cheng, C., Li, W., Clymer, J., and Yadav, A. (2024). GAMEBENCH: Evaluating strategic reasoning abilities of llm agents. In arXiv preprint arXiv:2406.06613v2

    arXiv 2024

  8. [8]

    DeepSeek-AI, Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., and et al. (2024). DeepSeek-V3 Technical Report. In arXiv preprint arXiv:2412.19437

    arXiv 2024

Show all 52 references
  1. [9]

    Duan, J., Zhang, R., Diffenderfer, J., Kailkhura, B., Sun, L., Stengel-Eskin, E., Bansal, M., Chen, T., and Xu, K. (2024). GTBENCH: Uncovering the strategic reasoning limitations of LLMs via game-theoretic evaluations. In arXiv preprint arXiv:2402.12348v2

    2024 arXiv

  2. [10]

    Fan, C., Chen, J., Jin, Y ., and He, H. (2023). Can large language models serve as rational players in game theory? a systematic analysis. In arXiv preprint arXiv:2312.05488v2

    2023 arXiv

  3. [11]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gre...

    2024 arXiv

  4. [12]

    Gupta, A. (2023). Are chatgpt and gpt-4 good poker players? — a pre-flop analysis. In arXiv preprint arXiv:2308.12466v2

    2023 arXiv

  5. [13]

    Hu, L., Li, Q., Xie, A., Jiang, N., Stoica, I., Jin, H., and Zhang, H. (2025). GAMEARENA: Evaluating LLM reasoning through live computer games. In arXiv preprint arXiv:2412.06394v5

    2025 arXiv

  6. [14]

    and Chang, K

    Huang, J. and Chang, K. C.-C. (2023). Towards reasoning in large language models: A survey. In arXiv preprint arXiv:2212.10403v2

    2023 arXiv

  7. [15]

    and et al

    Light, J. and et al. (2023). Avalonbench: Evaluating llms playing the game of avalon. arXiv preprint arXiv:2310.05036

    2023 arXiv

  8. [16]

    Liu, Y ., Li, Z., Liu, P., Xie, Y ., Wu, B., Zhang, Y ., Wang, S., Yu, Y ., Zhao, J., Lu, Z., Gao, Y ., Qiao, Y ., Fan, W., Ye, Y ., Liang, S., and Zhao, Y . (2023). AgentBench: Evaluating LLMs as agents. In arXiv preprint arXiv:2308.03688

    2023 arXiv

  9. [17]

    and Heydari, B

    Lorè, N. and Heydari, B. (2023). Strategic behavior of large language models: Game structure vs. contextual framing. In arXiv preprint arXiv:2309.05898v1

    2023 arXiv

  10. [18]

    Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D., and Lucic, M. (2021). Revisiting the calibration of modern neural networks. In arXiv preprint arXiv:2106.07998v2. Appeared in: 35th Conference on Neural Information Processing Systems (Neur...

    2021 arXiv

  11. [19]

    and Wongkamjan, W

    Nananukul, N. and Wongkamjan, W. (2024). What if red can talk? dynamic dialogue generation using large language models. In arXiv preprint arXiv:2407.20382v1

    2024 arXiv

  12. [20]

    GPT-4 technical report

    OpenAI (2024). GPT-4 technical report. In arXiv preprint arXiv:2303.08774v6

    2024 arXiv

  13. [21]

    and Guven, E

    Renze, M. and Guven, E. (2024). The effect of sampling temperature on problem solving in large language models. In arXiv preprint arXiv:2402.05201v3

    2024 arXiv

  14. [22]

    Sudhakaran, S., González-Duque, M., Freiberger, M., Glanois, C., Najarro, E., and Risi, S. (2023). MarioGPT: Open-ended Text2Level generation through large language models. In arXiv preprint arXiv:2302.05981v3

    2023 arXiv

  15. [23]

    Trivedi, H., Khot, T., Hartmann, M., Manku, R., Dong, V ., Li, E., Gupta, S., Sabharwal, A., and Balasubramanian, N. (2024). AppWorld: A controllable world of apps and people for benchmarking interactive coding agents. In arXiv preprint arXiv:2407.18901v1

    2024 arXiv

  16. [24]

    F., Zhou, X., Liu, S

    Tsai, C. F., Zhou, X., Liu, S. S., Li, J., Mei, H., and Yu, M. (2023). Can large language models play text games well? current state-of-the-art and open questions. In arXiv preprint arXiv:2304.02868v1

    2023 arXiv

  17. [25]

    Wang, G., Xie, Y ., Jiang, Y ., Mandlekar, A., Xiao, C., Zhu, Y ., Fan, L., and Anandkumar, A. (2023a). VOYAGER: An open-ended embodied agent with large language models. In arXiv preprint arXiv:2305.16291v2

    2023 arXiv

  18. [26]

    Wang, S., Long, Z., Fan, Z., Wei, Z., and Huang, X. (2024). Benchmark self-evolving: A multi-agent framework for dynamic LLM evaluation. In arXiv preprint arXiv:2402.11443v1

    2024 arXiv

  19. [27]

    S., Wang, X., Wang, D., Yu, Z., Li, L., and Zhou, H

    Wang, Y ., Yu, D., Dong, L., Bao, F. S., Wang, X., Wang, D., Yu, Z., Li, L., and Zhou, H. (2023b). MINT: Evaluating LLMs in multi-turn interaction with tools and language feedback. In arXiv preprint arXiv:2310.06825

    2023 arXiv

  20. [28]

    Xu, Y ., Wang, S., Li, P., and et al. (2023). Exploring large language models for communication games: An empirical study on werewolf. arXiv preprint arXiv:2309.04658. 11

    2023 arXiv

  21. [29]

    Yang, Z., Li, H., Chen, Y ., Tian, W., Ren, Y ., Su, H., Zhu, J., and Sun, L. (2023a). Large language models play StarCraft II: Benchmarks and a chain of summarization approach. In arXiv preprint arXiv:2310.11432

    2023 arXiv

  22. [30]

    Yang, Z., Li, L., Wang, J., Lin, K., Azarnasab, E., Ahmed, F., Liu, Z., Liu, C., Zeng, M., and Wang, L. (2023b). MM-REACT: Prompting ChatGPT for multimodal reasoning and action. In arXiv preprint arXiv:2303.11381v1

    2023 arXiv

  23. [31]

    Yim, Y ., Chan, C., Shi, T., Deng, Z., Fan, W., Zheng, T., and Song, Y . (2024). Evaluating and enhancing LLMs agent based on theory of mind in Guandan: A multiplayer cooperative game under imperfect information. In arXiv preprint arXiv:2408.02559v1

    2024 arXiv

  24. [32]

    Yu, X., Cheng, H., Liu, X., Roth, D., and Gao, J. (2024). ReEval: Automatic hallucination evaluation for retrieval-augmented large language models via transferable adversarial attacks. In arXiv preprint arXiv:2310.10190v2. Version v1 submitted on 19 Oct 2023, v2 (this version)...

    2024 arXiv

  25. [33]

    Zhang, H., Da, J., Lee, D., Robinson, V ., Wu, C., Song, W., Zhao, T., Raja, P., Zhuang, C., Slack, D., Lyu, Q., Hendryx, S., Kaplan, R., Lunati, M., and Yue, S. (2024a). A careful examination of large language model performance on grade school arithmetic. In arXiv preprint ar...

    2024 arXiv

  26. [34]

    Zhang, Y ., Mao, S., Ge, T., Wang, X., de Wynter, A., Xia, Y ., Wu, W., Song, T., Lan, M., and Wei, F. (2024b). LLM as a mastermind: A survey of strategic reasoning with large language models. In arXiv preprint arXiv:2404.01230v1. A Appendix A.1 Tower defense game A.1.1 Game rules

    2024 arXiv

  27. [35]

    The battlefield consists of 5 rows (corresponding to y-coordinates 0-4)

    Players can purchase characters and place them on the battlefield. The battlefield consists of 5 rows (corresponding to y-coordinates 0-4). The human side can place units in a designated area spanning 11 columns (corresponding to x-coordinates 0-10)

  28. [36]

    Human units are placed on the left side of the battlefield, remain stationary, and attack approaching enemies

    Demons spawn from the right side of the battlefield (x-coordinates 11) and move left. Human units are placed on the left side of the battlefield, remain stationary, and attack approaching enemies

  29. [37]

    Defending units fire bullets or activate skills to attack enemies

    All units attack according to their attack interval, automatically attacking when their cooldown ends. Defending units fire bullets or activate skills to attack enemies. Invading units engage in melee attacks when they come into contact with defending units

  30. [38]

    Placing a new unit in an occupied cell is not allowed

    Each grid cell can only contain one human unit at a time. Placing a new unit in an occupied cell is not allowed

  31. [39]

    If a unit’s health drops to 0, it is eliminated and removed from the battlefield

    When an attack hits, the target takes damage based on the attacker’s power. If a unit’s health drops to 0, it is eliminated and removed from the battlefield

  32. [40]

    If any enemy successfully reaches the left side of the battlefield, the player loses

    If all enemies are eliminated, the player wins. If any enemy successfully reaches the left side of the battlefield, the player loses. A.1.2 Human units Unit Attributes HandgunSoldier Health: 3, Shooting interval: 1000ms, Cost: 100, Damage per shot: 1, No special abilities. Rif...

  33. [41]

    Gold characters cost three times as much as bronze characters, but their stats (attack, health, numerical skill effects, etc.) are twice as high

    At the start of the game, players can purchase all desired characters at once, up to a maximum of 7 characters. Gold characters cost three times as much as bronze characters, but their stats (attack, health, numerical skill effects, etc.) are twice as high. Non-numerical skill...

  34. [42]

    If both sides have the same number of characters, the invader attacks first

    Initiative Determination: The side with more characters attacks first. If both sides have the same number of characters, the invader attacks first

  35. [43]

    Elemental Advantage: Certain elements have an advantage over others, granting a bonus in combat (Fire> Nature, Nature> Water, Water> Earth, Earth> Fire)

  36. [44]

    However, if there are Taunt minions on the opponent’s side, attackers must prioritize attacking them

    Battle Process: Both sides will attack based on their respective target_priority (target priority). However, if there are Taunt minions on the opponent’s side, attackers must prioritize attacking them. The attack order follows a left-to-right sequence. The first minion in the ...

  37. [45]

    If all characters on one side are eliminated, the other side wins

  38. [46]

    A.2.2 Invader units Unit Attributes FireLizard Attack: 2, Health: 2, Cost: 1, Ability: Deals 2 damage to the enemy that killed it upon death

    If both sides are eliminated simultaneously in the same attack resolution, the Invader wins. A.2.2 Invader units Unit Attributes FireLizard Attack: 2, Health: 2, Cost: 1, Ability: Deals 2 damage to the enemy that killed it upon death. WaterElemental Attack: 2, Health: 2, Cost:...

  39. [47]

    Each faction consists of three characters

    This game is a turn-based character battle game divided into two factions: Invader and Defender. Each faction consists of three characters. The Invader faction includes Fire, Water, and Dark elements, while the Defender faction includes Wood, Earth, and Light elements. Charact...

  40. [48]

    In each round, the three Invader characters act first in order, followed by the three Defender characters

    Combat proceeds in rounds. In each round, the three Invader characters act first in order, followed by the three Defender characters. The sequence then repeats in the next round

  41. [49]

    On each turn, a character uses the next skill in their list and continues cycling through them in order

    Each character has three skills that are used in a preset, looping sequence. On each turn, a character uses the next skill in their list and continues cycling through them in order

  42. [50]

    Light and Dark counter each other with 1.5× damage

    The game features an elemental effectiveness system: Fire beats Wood, Wood beats Earth, Earth beats Water, and Water beats Fire (1.2× damage when effective, 0.8× when resisted). Light and Dark counter each other with 1.5× damage. All other combinations deal the standard 1.0× damage

  43. [51]

    A.3.2 Invader skills 15 Skill Name Description Fire Skills flame_splash Deals 12 damage and applies Burning for 2 rounds (1 layer, 5 damage per round)

    If all characters on one side are eliminated, the other side wins. A.3.2 Invader skills 15 Skill Name Description Fire Skills flame_splash Deals 12 damage and applies Burning for 2 rounds (1 layer, 5 damage per round). Cost: 1 residual_warmth Increases the damage of the next f...

  44. [52]

    B.1 Rule violation Rate (RVR) This metric measures how often a model’s initial strategy proposal fails to adhere to the game’s explicit rules, particularly budget constraints

    Cost: 3 B Additional evaluation metrics This section details supplementary metrics used to provide a more granular understanding of LLM behavior in strategic game environments, complementing the core metrics presented in Section 3.4. B.1 Rule violation Rate (RVR) This metric m...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.