Rule Synergy Analysis using LLMs: State of the Art and Implications

Bahar Bateni; Benjamin Pratt; Jim Whitehead

REVIEW 4 major objections 5 minor 17 references

Rule Synergy Analysis using LLMs: State of the Art and Implications

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs can tell when cards do not interact, but miss most harmful interactions.

desk verdict A genuinely new dataset and a plausible negative-synergy result, but unvalidated ground truth and missing baselines keep the empirical claim from being conclusive. read the letter →

arxiv 2508.19484 v1 pith:TUNIW6OE submitted 2025-08-27 cs.CL

Bahar Bateni , Benjamin Pratt , Jim Whitehead This is my paper · ORCID

classification cs.CL

keywords largelanguagemodelsrulesynergycardgamereasoningSlaytheSpirestaticevaluationerrortaxonomytemporal

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether large language models can reason about how game rules interact, using the card game Slay the Spire as a testbed. The authors built a dataset labeling all 5,625 ordered pairs of cards from one card set as positive, negative, or no synergy, and prompted several LLMs to classify each pair from static card text alone. Their central finding is that the models reliably detect pairs with no interaction but largely miss positive and especially negative synergies; no model exceeded 0.17 F1 on the negative-synergy class. The paper also contributes an error taxonomy—timing, game-state, rule, skip, and similarity errors—claimed to explain the persistent failures. If right, the result sets a concrete boundary on using LLMs for static rule evaluation in game design.

What carries the argument

The machinery is the Synergy Analysis Dataset together with a structured chain-of-thought prompt. The dataset forms every ordered pair from 75 Slay the Spire cards into 5,625 instances labeled +1 (positive), -1 (negative), or 0 (no) synergy, where synergy is defined as an additional advantage or disadvantage from playing one card and then the other. The prompt asks the model to recite both card descriptions word for word, state the order of events, analyze the interaction, write a conclusion, and end with a single numeric label; the few-shot examples enforce this format. That structure does two jobs: it turns rule-interaction reasoning into a static classification problem, and it produces reasoning traces from which the paper derives its error taxonomy.

What would settle it

Re-annotate a random sample of card pairs with multiple judges and measure inter-annotator agreement; if agreement is low, or if a scenario-based labeling rule reshuffles the negative class, the reported negative-synergy F1 reflects label instability rather than a fixed model limitation. Alternatively, simulate each pair in a playable Slay the Spire engine and evaluate the models against the simulated ground truth.

Watch

Extended reading notes

Core claim

The paper's central claim is that current large language models, despite strong performance on many reasoning benchmarks, cannot reliably predict emergent rule interactions between pairs of cards when given only card descriptions and game rules. On the new Synergy Analysis Dataset, the best overall accuracy is 0.81, the top positive-synergy F1 is 0.65, and the highest negative-synergy F1 across all evaluated models is 0.17. Because no-synergy pairs make up 81 percent of the data, accuracy overstates competence, and the class-level F1 scores are the meaningful measure. The paper further claims that the failures are systematic: a taxonomy of timing, interruption, continuity, immediacy, game-state, rule, skip, and similarity errors accounts for the mistakes, with timing-related errors among the most common. The intended conclusion is that static rule-interaction reasoning is a genuine gap in current LLM capability, not a prompt-tuning artifact.

Load-bearing premise

The evaluation assumes the 5,625 hand-assigned synergy labels are correct and consistent, but the paper gives no annotation procedure or agreement measure and admits that many card pairs are simultaneously positive and negative, so a different labeling convention could change the measured F1 scores.

Editorial extensions

If this is right

Static game-design tools built on current LLMs should not be used to predict negative card interactions, since the best measured F1 on that class is 0.17.
Accuracy is misleading for this kind of highly imbalanced rule-interaction data; class-level F1 is the metric that reveals model competence.
Temporal reasoning is the binding constraint: timing and immediacy errors persist across prompt refinements, so synergy detection is as much an ordering problem as a rule-application problem.
Prompt engineering and small-scale fine-tuning reduce formatting and skip errors but leave game-state, rule, and similarity errors largely intact, indicating the deficit is not only in instruction following.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A same-card control column (card A with card A) would test whether the diagonal error pattern—identical cards judged synergistic—is a general tendency to confuse additive repetition with emergent interaction.
If the ground-truth labels are re-derived by simulating card pairs in a playable engine, the LLM's low negative-synergy F1 could be compared against a simulation-based gold standard, separating model deficiency from label ambiguity.
The noted cases of simultaneous positive and negative synergy suggest future datasets should replace a single ordinal label with per-scenario or per-mechanic labels; under that richer target, current models might be more capable than this study's F1 implies.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things are worth your time here. First, the Synergy Analysis Dataset: 5,625 labeled card pairs from Slay the Spire, public, with code. Second, the consistent result across five model variants that negative synergies are the hardest class, with F1 scores at 0.12-0.17 while no-synergy F1 reaches 0.5-0.88. The GPT-4o confusion matrix shows the model overpredicts negative synergies (348 predicted vs. 125 actual), so the failure is not a refusal to output the rare class; the models genuinely misclassify actual negative pairs. The error taxonomy (timing, game state, rule, skip, similarity) is concrete, and the appendix examples make the categories easy to understand. That is a real empirical contribution to the LLM game-reasoning niche. The soft spots are concentrated in evaluation methodology, and they are real. The biggest is construct validity of the ground truth. The paper gives no annotation procedure and no inter-annotator agreement, and it admits in Section V-C that many card pairs have both positive and negative synergy, condensed into one label. The prompt resolves conflicts only through 'in most scenarios' and a judgment about 'most significant' effects. If the 125 negative labels are partly a matter of the authors' taste, then a 0.17 F1 measures agreement with that taste, not an objective capability ceiling. The direction of the finding - LLMs miss the nuance - would probably survive, but the exact numbers are not trustworthy as reported. Two smaller gaps: no baselines (majority class, random, human) and no error bars or repeated sampling. The fine-tuned GPT-4o-mini is described as trained on 160 examples 'selected from cases where initial results showed errors,' but the paper never says whether those examples were excluded from the evaluation set. If they leak, the fine-tuning result is uninterpretable. All of these are fixable, and none sinks the dataset or the error analysis. Who this is for: people working on LLMs in games, procedural content generation, and static rule evaluation. The dataset and taxonomy are the lasting value; the empirical claims need stronger support before being cited as established. I would send it to a serious referee, but I would ask for documentation of the label-construction process, a human or heuristic baseline, error bars, and an explicit train/test split for the fine-tuned model. With those, the negative-synergy result would be worth taking as fact.

Referee Report

4 major / 5 minor

Summary. The paper introduces a synergy-detection task for LLMs using pairwise card combinations from the card game Slay the Spire. The authors build a dataset of 5,625 card pairs labeled as positive synergy, negative synergy, or no synergy, and evaluate five LLM configurations (GPT-4o-mini, a fine-tuned GPT-4o-mini, GPT-4o, Gemini 1.0 Pro, and Gemini 1.5 Flash) with a structured prompt. The main empirical claim is that LLMs are good at identifying non-synergistic pairs but perform poorly at detecting positive synergies and especially negative synergies, with the best negative-synergy F1 score at 0.17. The paper also proposes a taxonomy of error types, including timing, game-state, rule, skip, and similarity errors, and discusses implications for automated game design and future work.

Significance. If the evaluation were properly validated, this would be a useful benchmark for a genuinely hard reasoning task: static prediction of rule interactions without a game simulator. The authors contribute a publicly released dataset, prompt-design findings (e.g., requiring verbatim card recitation and named chain-of-thought sections), and a concrete error taxonomy with worked examples. The observation that negative synergies are detected at very low F1 even by strong models is plausible and worth reporting. However, the paper's central claims are currently under-supported because the ground-truth labels lack documented validation and the evaluation lacks the baselines and uncertainty quantification needed to interpret the reported scores.

major comments (4)

[Section IV and Appendix A] The validity of the ground-truth labels is not established. Section IV states that the dataset contains 5,625 pairwise labels but gives no annotation procedure, no inter-annotator agreement, and no adjudication rule. Section V-C explicitly acknowledges that many card pairs present both positive and negative synergies simultaneously, and the prompt in Appendix A resolves this only through phrases such as "in most scenarios" and "which effects are most significant." Section V-H further states that a pair's synergy can depend on other cards, game state, and passing turns, making it hard to attribute an observed effect to the pair alone. Under these conditions, the low negative-synergy F1 ceiling of 0.17 may reflect arbitrary or threshold-dependent labels rather than a limitation of LLM reasoning. I would like to see a documented labeling protocol, inter-annotator agreement on a subset, and a discussion of how conflicting synergies were resolved.
[Section VII, Table II] The claim that LLMs "excel at identifying non-synergistic pairs" is not supported without a majority-class or human baseline. Since 81% of the dataset is no-synergy (Table I), a trivial classifier that always predicts "no synergy" achieves 81% accuracy and a no-synergy F1 of approximately 0.895. The best no-synergy F1 in Table II is 0.88 (GPT-4o), which is below that majority baseline. Accuracy for GPT-4o is 0.81, equal to the majority-class baseline. The paper should report random and majority baselines, human performance, and per-class chance-level F1 values before making claims about what LLMs "excel" at.
[Section VII, Table II] All reported results appear to come from single runs, with no confidence intervals, standard deviations, or significance tests. The differences between models are small relative to what would be expected from 125 negative-synergy examples, and the exclusion of invalid responses (up to 15% for Gemini 1.5 Flash) is not explicitly accounted for in the reported denominators. Without multiple runs or a sensitivity analysis, the paper cannot support comparative statements such as "the best-performing GPT-4o model" or the specific ordering of F1 scores across models. This is a load-bearing issue for the paper's "state of the art" framing.
[Section VIII] The error taxonomy is presented as a main contribution, but it is derived only from GPT-4o outputs and no quantitative distribution of error types is given. The paper says that timing errors are "one of the most common types" and that rule errors are "common," but it provides no counts, rates, or coding reliability. Since the error taxonomy is used in Sections IX-A and IX-B to motivate prompt-design and fine-tuning recommendations, the reader needs to know how often each error type occurs and whether the categories were applied consistently.

minor comments (5)

[Section VIII-C4] The text refers to the card "Demon From" in the Immediacy Error example; this should be "Demon Form."
[Appendix A] In the keyword definition for Strength, "comming" should be "coming."
[Section VI-C] The paper says the expected response format prevents "Skip Error, which we later discuss in Section VIII-F," but elsewhere the error is called "Skip Error" at Section VIII-F; the terminology and section numbering should be checked for consistency.
[Section IX-A] The paper says the word "Synergy" was replaced with "Combo" in the experiments, but the abstract, introduction, and dataset description use "synergy" throughout; a brief note explaining this terminology choice near the dataset definition would improve readability.
[Table II and Section VI] The handling of invalid responses should be clarified: the table reports the percentage of invalid cases, but the paper does not state whether accuracy and F1 are computed only over valid responses or whether invalid responses are counted as errors. This affects the comparability of the reported numbers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark evaluation whose claims are measured against an externally anchored dataset, not derived from its own outputs.

full rationale

This paper does not present a derivation chain; it constructs a dataset of card-pair synergy labels, prompts LLMs with a structured context, and reports precision, recall, and F1 scores. The central claim—that LLMs identify no-synergy pairs more reliably than positive or negative synergies—is an empirical measurement, not an equation that reduces to its inputs. The ground-truth labels are the authors' operationalization of the game's synergy concept; this raises construct-validity concerns (no annotation procedure, no inter-annotator agreement, and the paper's own admission in Section V-C that some pairs present both positive and negative synergies), but a benchmark whose labels are authored by the paper is not circular reasoning. The fine-tuned model is trained on 160 error-selected examples and then evaluated, which is a methodological contamination risk rather than a fitted-input-called-prediction step, and the paper explicitly reports that fine-tuning produced only modest gains. The single self-citation ([12], used to justify omitting card names and to note prior game-playing results) is not load-bearing: the paper's findings do not depend on the truth of that cited result, and no uniqueness theorem, ansatz-via-citation, or renaming of a known result is present. Therefore no significant circularity is found.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted numerical parameters and no new postulated entities. It relies on three domain assumptions: that the hand-authored labels are correct, that the prompt faithfully represents the game, and that static evaluation is a meaningful proxy. These assumptions are plausible but unvalidated.

assumptions (3)

domain assumption The ground-truth synergy labels in the dataset are accurate and reflect the authors' intended synergy definition.
The paper does not report how labels were produced or validated (Section IV). If labels are inconsistent, measured model performance does not measure the intended capability.
domain assumption The prompt's summary of Slay the Spire rules and card effects is a faithful and complete representation of the game.
The appendix prompt (Appendix A) is the only game information given to models. Any inaccuracy or omitted rule shifts the task from the real game to a simplified proxy.
domain assumption Static synergy detection, without a playable game or simulation, is a meaningful proxy for rule-interaction reasoning.
The paper motivates the task as useful for game design tools (Section I) but provides no validation that static predictions match outcomes in actual gameplay.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Rule Synergy Analysis using LLMs: State of the Art and Implications." pith.science (2026). https://pith.science/paper/TUNIW6OE

@misc{pith2026250819484,
  author       = {Pith},
  title        = {Pith review of: Rule Synergy Analysis using LLMs: State of the Art and Implications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TUNIW6OE}},
  note         = {Machine review of arXiv:2508.19484}
}

read the original abstract

Large language models (LLMs) have demonstrated strong performance across a variety of domains, including logical reasoning, mathematics, and more. In this paper, we investigate how well LLMs understand and reason about complex rule interactions in dynamic environments, such as card games. We introduce a dataset of card synergies from the game Slay the Spire, where pairs of cards are classified based on their positive, negative, or neutral interactions. Our evaluation shows that while LLMs excel at identifying non-synergistic pairs, they struggle with detecting positive and, particularly, negative synergies. We categorize common error types, including issues with timing, defining game states, and following game rules. Our findings suggest directions for future research to improve model performance in predicting the effect of rules and their interactions.

Figures

Figures reproduced from arXiv: 2508.19484 by the authors.

**Figure 1.** Visualization of all the positive synergies in the dataset. Each dot represents a card, and the cards are sorted by their effect types. For example, cards [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

**Figure 2.** Error patterns based on our experiments using the GPT-4o model. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Confusion matrix and error rates for each class. [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 10 canonical work pages

[1]

Large language models and games: A survey and roadmap,

R. Gallotta, G. Todd, M. Zammit, S. Earle, A. Liapis, J. Togelius, and G. N. Yannakakis, “Large language models and games: A survey and roadmap,” IEEE Transactions on Games , p. 1–18, 2024. [Online]. Available: http://dx.doi.org/10.1109/TG.2024.3461510

arXiv 2024
[2]

Slay the spire,

M. C. Games, “Slay the spire,” 2019

work page 2019
[3]

Mariogpt: Open-ended text2level generation through large language models,

S. Sudhakaran, M. Gonz ´alez-Duque, C. Glanois, M. Freiberger, E. Na- jarro, and S. Risi, “Mariogpt: Open-ended text2level generation through large language models,” 2023

work page 2023
[4]

Level generation through large language models,

G. Todd, S. Earle, M. U. Nasir, M. C. Green, and J. Togelius, “Level generation through large language models,” in Proceedings of the 18th International Conference on the Foundations of Digital Games , ser. FDG ’23. New York, NY , USA: Association for Computing Machinery,

work page
[5]

Cardistry: Exploring a gpt model workflow as an adapted method of gaminiscing,

B. Lyman, A. Ebrahimi, J. Cox, S. Chan, C. Barney, and B. De Schutter, “Cardistry: Exploring a gpt model workflow as an adapted method of gaminiscing,” in Proceedings of the 19th International Conference on the Foundations of Digital Games , ser. FDG ’24. New York, NY , USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/1...

arXiv 2024
[6]

Crawllm: Theming games with large language models,

M. Zammit, A. Liapis, and G. N. Yannakakis, “Crawllm: Theming games with large language models,” in 2024 IEEE Conference on Games (CoG), 2024, pp. 1–2

work page 2024
[7]

Roborosewater,

M. Milewicz, “Roborosewater,” 2016. [Online]. Available: https: //twitter.com/roborosewater

work page 2016
[8]

Mystical tutor: A magic: The gathering design assistant via denoising sequence-to-sequence learning,

A. Summerville and M. Mateas, “Mystical tutor: A magic: The gathering design assistant via denoising sequence-to-sequence learning,” Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , vol. 12, no. 1, pp. 86–92, Jun

work page

Show all 17 references

[9]

Chaos cards: Creating novel digital card games through grammatical content generation and meta-based card evaluation,

T. Chen and S. Guy, “Chaos cards: Creating novel digital card games through grammatical content generation and meta-based card evaluation,” Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , vol. 16, no. 1, pp. 196–202, Oct. 2...

2020
[10]

Boardgameqa: A dataset for natural language reasoning with contradictory information,

M. Kazemi, Q. Yuan, D. Bhatia, N. Kim, X. Xu, V . Imbrasaite, and D. Ramachandran, “Boardgameqa: A dataset for natural language reasoning with contradictory information,” in Advances in Neural Information Processing Systems , A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hard...

2023
[11]

Tasklama: Probing the complex task understanding of language models,

Q. Yuan, M. Kazemi, X. Xu, I. Noble, V . Imbrasaite, and D. Ramachandran, “Tasklama: Probing the complex task understanding of language models,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 17, pp. 19 468–19 476, Mar. 2024. [Online]. Available: h...

2024
[12]

Language-driven play: Large language models as game-playing agents in slay the spire,

B. Bateni and J. Whitehead, “Language-driven play: Large language models as game-playing agents in slay the spire,” in Proceedings of the 19th International Conference on the Foundations of Digital Games, ser. FDG ’24. New York, NY , USA: Association for Computing Machinery,
[13]

Tram: Benchmarking temporal reasoning for large language models,

Y . Wang and Y . Zhao, “Tram: Benchmarking temporal reasoning for large language models,” 2024. [Online]. Available: https://arxiv.org/abs/ 2310.00835

2024 arXiv
[14]

Towards benchmarking and improving the temporal reasoning capability of large language models,

Q. Tan, H. T. Ng, and L. Bing, “Towards benchmarking and improving the temporal reasoning capability of large language models,” 2023. [Online]. Available: https://arxiv.org/abs/2306.08952

2023 arXiv
[2021]

Available: https://ojs.aaai.org/index.php/AIIDE/article/ view/12851

[Online]. Available: https://ojs.aaai.org/index.php/AIIDE/article/ view/12851
[2023]

Available: https://doi.org/10.1145/3582437.3587211

[Online]. Available: https://doi.org/10.1145/3582437.3587211
[2024]

Available: https://doi.org/10.1145/3649921.3650013

[Online]. Available: https://doi.org/10.1145/3649921.3650013

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Large language models and games: A survey and roadmap,

R. Gallotta, G. Todd, M. Zammit, S. Earle, A. Liapis, J. Togelius, and G. N. Yannakakis, “Large language models and games: A survey and roadmap,” IEEE Transactions on Games , p. 1–18, 2024. [Online]. Available: http://dx.doi.org/10.1109/TG.2024.3461510

arXiv 2024

[2] [2]

Slay the spire,

M. C. Games, “Slay the spire,” 2019

work page 2019

[3] [3]

Mariogpt: Open-ended text2level generation through large language models,

S. Sudhakaran, M. Gonz ´alez-Duque, C. Glanois, M. Freiberger, E. Na- jarro, and S. Risi, “Mariogpt: Open-ended text2level generation through large language models,” 2023

work page 2023

[4] [4]

Level generation through large language models,

G. Todd, S. Earle, M. U. Nasir, M. C. Green, and J. Togelius, “Level generation through large language models,” in Proceedings of the 18th International Conference on the Foundations of Digital Games , ser. FDG ’23. New York, NY , USA: Association for Computing Machinery,

work page

[5] [5]

Cardistry: Exploring a gpt model workflow as an adapted method of gaminiscing,

B. Lyman, A. Ebrahimi, J. Cox, S. Chan, C. Barney, and B. De Schutter, “Cardistry: Exploring a gpt model workflow as an adapted method of gaminiscing,” in Proceedings of the 19th International Conference on the Foundations of Digital Games , ser. FDG ’24. New York, NY , USA: Association for Computing Machinery, 2024. [Online]. Available: https://doi.org/1...

arXiv 2024

[6] [6]

Crawllm: Theming games with large language models,

M. Zammit, A. Liapis, and G. N. Yannakakis, “Crawllm: Theming games with large language models,” in 2024 IEEE Conference on Games (CoG), 2024, pp. 1–2

work page 2024

[7] [7]

Roborosewater,

M. Milewicz, “Roborosewater,” 2016. [Online]. Available: https: //twitter.com/roborosewater

work page 2016

[8] [8]

Mystical tutor: A magic: The gathering design assistant via denoising sequence-to-sequence learning,

A. Summerville and M. Mateas, “Mystical tutor: A magic: The gathering design assistant via denoising sequence-to-sequence learning,” Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , vol. 12, no. 1, pp. 86–92, Jun

work page