Pith. sign in

REVIEW 4 major objections 4 minor 12 references

A benchmark for rules-rich tactical reasoning in D&D combat shows frontier language-model policies win most isolated fights but fail to survive linked adventuring days, revealing a distinct gap in resource-aware tactical discipline.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 04:11 UTC pith:UKVP3Y5M

load-bearing objection A genuinely useful benchmark with a well-designed shared decision stream, but the headline day-track result is not statistically supported by five single-seed episodes. the 4 major comments →

arxiv 2607.29577 v1 pith:UKVP3Y5M submitted 2026-07-31 cs.AI cs.LG

DungeonBench: A Benchmark for Rules-Rich Tactical Reasoning in Dungeons & Dragons Combat

Ismayil Ismayilov , Atakan Kara , Kaan Oktay This is my paper
classification cs.AI cs.LG
keywords tactical reasoningbenchmarkDungeons & Dragonslanguage-model policiesresource managementgame AIaction decompositiondecision stream
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces DungeonBench, a benchmark that turns Dungeons & Dragons 2014 combat into a decision stream of legal choices, and uses it to test whether language-model policies can value tactical options when geometry, timing, resources, and rule interactions matter at once. Its central finding is that full tactical observations do not saturate the benchmark: the same models that win roughly 80 percent of isolated encounters clear only 40 percent of linked encounter days. The Day track links fights through persistent hit points, spell slots, and short-rest timing, so winning a fight can be strategically wrong if it spends a resource needed later. A sympathetic reader should care because this isolates a distinct capability—resource-aware tactical discipline—that single-fight benchmarks do not measure.

Core claim

On the paper's own terms, the discovery is that strong local tactical play does not transfer to linked-day performance. With complete observations, legal options enumerated by an engine, and a fixed heuristic opponent, frontier language-model policies win a large majority of Encounter episodes but clear only a minority of Day episodes. The terminal traces show qualitatively different failure modes—objective losses, wipes, and attrition losses—and the Day track exposes resource budgeting, rest timing, and rule awareness as the binding constraints.

What carries the argument

The central object is the engine-generated decision stream: at every step the benchmark exposes a complete tactical state, a pending decision kind, and an indexed list of legal executable options derived from a 2.5D grid simulator covering a large part of the 2014 SRD combat surface (561 combatants, 229 spells, 275 items). To keep this broad surface manageable, actions are hierarchically decomposed—movement into reachable destinations and consequence-equivalent routes, point-targeted spells into coordinate-wise axis choices, area spells into canonical direction components—so the policy always chooses among legal options rather than emitting free-form commands. The Day track adds intermission

Load-bearing premise

The benchmark's validity rests on the engine resolving the 2014 D&D rules faithfully enough that a policy's failures are failures of tactical judgment rather than artifacts of simulator shortcuts.

What would settle it

Run the Day-track episodes with a second, independent opposing-side planner (or human-played opponents) and check whether day-clear rates change substantially; alternatively, have expert D&D players audit engine-resolved spell and feature resolutions in the Day scenarios for deviations from the 2014 SRD.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Frontier language-model policies achieve roughly 80 percent win rates on isolated encounters but only 40 percent day-clear rates, so the benchmark is not saturated by full observability.
  • Day-track failure traces divide into objective losses, wipes, and attrition losses, meaning binary win/loss is insufficient to diagnose tactical competence.
  • The same decision stream supports heuristic controllers, text-prompted LLM policies, learned option rankers, and masked-action RL agents, so comparisons isolate option valuation rather than command parsing.
  • The Day track makes resource budgeting and rest timing observable and measurable, turning 'win the fight vs. win the day' into a concrete evaluation axis.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because all opposing sides are controlled by a single fixed heuristic planner, the reported win rates may reflect weaknesses of that specific opponent; measuring against a family of opponents would test whether the benchmark measures general tactical competence or opponent-specific exploitation.
  • The authors note hidden-information and multi-agent variants as future work; those variants would test whether full observability itself is responsible for the local-vs-day gap, and whether policies improve when they must infer rather than read enemy traits.
  • A direct extension of the Day track to multiple linked days (a Dungeon track) would stress long-horizon planning across long rests and spell preparation, and would reveal whether the 40 percent day-clear rate decays further with horizon length.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces DungeonBench, a benchmark for tactical reasoning in 2014 D&D SRD combat. It provides an engine that enumerates legal options from a full tactical state, a hierarchical action decomposition for movement/point/direction selections, and two tracks: Encounter (20 scenarios, 3 seeds each) and Day (5 scenarios, 1 seed each). Five frontier language-model policies are evaluated on a shared decision stream with a single fixed heuristic opponent. Reported results show Encounter win rates of 68–83% (best: Gemini 3.1 Pro at 83±5) and Day full-clear rates of 0–40% (best: Gemini 3.1 Pro and GPT-5.5 at 2/5). The central claim is that full tactical observations do not saturate the benchmark and that linked days expose failures in resource budgeting, rest timing, and rule-aware tactical discipline.

Significance. The benchmark design has genuine strengths: complete observability isolates option valuation from perception; hierarchical decomposition is a sensible response to a large, heterogeneous action space; the independent simulator win condition makes the LLM evaluation non-circular; and the shared decision stream for LLM, behavior-cloned, and PPO interfaces is a useful abstraction. If the benchmark and its validation were made public and the empirical claims were supported, DungeonBench could be a valuable resource for studying rule-rich sequential tactical reasoning. However, the headline Day-track conclusion rests on five single-seed episodes, the Encounter-to-Day gap is confounded with horizon length, and the simulator's fidelity to the claimed SRD surface is not validated. The current evaluation is more a pilot than a statistically established result.

major comments (4)
  1. [§4.1, Table 2] The Day-track headline rests on 5 scenarios with 1 fixed seed each. Day-clear rates of 2/5, 2/5, 1/5, 1/5, and 0/5 are binomial samples of size 5; for example, 2/5 has a 95% CI of roughly [2.5%, 85%]. No per-scenario table, no standard errors, and no seed variance are reported, so the gap between Encounter (~83%) and Day (~40%) is not statistically reliable. The claim that linked days expose resource-budgeting and rest-timing failures is not supported by the reported evidence. The authors should add substantially more Day replications/seeds with per-scenario outcomes and confidence intervals, or explicitly reframe the Day results as qualitative/pilot evidence.
  2. [§4.1–4.2] The Encounter-to-Day drop is confounded with decision horizon. Day episodes average 328–391 decisions versus 79–109 for Encounter; under any constant per-decision error rate, full-clear probability falls mechanically. The paper attributes the drop to resource budgeting and rest timing without a control condition, such as the same days with resources reset between encounters or a statistical null model. The attribution is also internally qualified by §4.2, which states that kestrel_approach is an objective-recognition failure ('every model fails the day while preserving the party'), contradicting the unified resource-budgeting narrative in the abstract.
  3. [§3.2] The claim of covering "the vast majority of combat-relevant 2014 SRD content" is unsupported by any fidelity validation. The engine's semantics for 229 spells, 275 items, 561 combatants, and numerous conditions/effects are asserted, but no human-expert adjudication, rule-level test suite, inter-rater reliability, or comparison against known tabletop resolutions is reported. Because every policy outcome is scored by this simulator, a modeling shortcut relative to the actual SRD can be scored as a model failure or success. Please provide a validation protocol (e.g., stratified expert review of rule resolutions, regression tests, or a public test suite) and report its results.
  4. [§4] All opposing sides are controlled by a single 'heuristic planner' whose behavior is not documented beyond that phrase. Results may reflect the planner's specific weaknesses rather than general tactical competence. The paper should include a specification of the planner, and the main comparisons should be supplemented with at least (a) a random/greedy baseline and (b) one alternative opponent policy, to show that the benchmark conclusions are not opponent-specific. A heuristic baseline would also calibrate the claim that 'full tactical observations do not saturate the benchmark.'
minor comments (4)
  1. [§3.4, §4.2] Table numbering is inconsistent: the observation description in §3.4 refers to 'Table 1', but the results Tables 1 and 2 appear in §4.2, and Appendix B renumbers tables (Table 4, Table 5). Please renumber tables consistently throughout.
  2. [§5] The Limitations section does not mention the small Day sample size or the absence of simulator-fidelity validation. These are the most important limitations of the reported evaluation and should be acknowledged explicitly.
  3. [§4.2] The terminal-trace taxonomy (38 wipes, 25 objective failures, 13 depleted losses) is useful, but the counts are pooled across models and scenarios; presenting them per model or with error bars would make the qualitative claims easier to interpret.
  4. [§3.6, Table 7] The scenario IDs are lowercase (e.g., kestrel_approach) in the text but capitalized in Table 7 (e.g., 'Kestrel Approach'). Please unify the naming so scenario references are unambiguous.

Circularity Check

0 steps flagged

No significant circularity: the benchmark evaluation is scored by an independent simulator, and no fitted parameter or load-bearing self-citation is present.

full rationale

The paper's central claims are empirical evaluations, not derivations from fitted inputs. Encounter win rates and Day full-clear rates are produced by an external simulator that executes legal options selected by the model; the paper states that 'the simulator executes the selected option, so comparisons measure tactical option valuation rather than command parsing or action validity.' The Day metric is not used to fit any parameter of the benchmark, and the heuristic opponent is external to the tested policies. No self-citations are load-bearing: the references are to independent prior work, and no uniqueness theorem or ansatz is imported from the authors' own previous papers. The benchmark's rule surface is an input design choice, not a predicted outcome, so 'built to cover the vast majority of combat-relevant 2014 SRD content' is a scoping statement rather than a circular derivation. The limitations noted in the paper—unvalidated SRD fidelity and a single heuristic opponent—are correctness and external-validity risks, not circularity. Likewise, the small-sample Day-track results are a statistical concern, not a reduction of the conclusion to the benchmark's construction. No step in the paper's argument is equivalent to its own inputs by definition.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 2 invented entities

The benchmark's central result rests on the reliability of the new simulator and the representativeness of its single heuristic opponent. The paper does not fit free parameters to LLM scores; the main risks are a lack of released engine/scenario data and unmeasured variance from one opponent policy.

free parameters (1)
  • Encounter seeds = 3 per scenario
    Seeds are an evaluation choice, not a fitted parameter; no model or benchmark parameter is fit to the data. Listed for completeness.
axioms (4)
  • domain assumption The heuristic planner is a single fixed controller that acts for all opposing sides at all levels of play.
    Section 4: 'all opposing sides are controlled by the same heuristic planner.' The entire comparison is conditioned on this opponent policy, and its strength or weaknesses are not characterized.
  • domain assumption The simulator faithfully resolves the 2014 SRD combat rules for the modeled subset.
    Section 3.2 defines the executable scope and admits that effects requiring DM adjudication are excluded. No validation against external play data or rule experts is described.
  • domain assumption A 2.5D grid with height coordinates captures the tactical content of vertical, underwater, and flying combat.
    Section 3.3 adopts the 2.5D grid instead of a continuous space; the paper asserts this preserves tactical effects without proving that all relevant geometry (e.g., squeezing, 3D cones) is faithfully represented.
  • domain assumption Simple decomposition into axis choices (x/y/z, dx/dy/dz) does not bias policy behavior against area spells.
    Section 3.5: area-origin is decomposed into sequential coordinate choices. This changes the decision tree a policy must navigate; no ablation compares it to a flat area-origin option set.
invented entities (2)
  • DungeonBench engine no independent evidence
    purpose: Resolves D&D combat rules, enumerates legal options, and simulates encounters and days.
    The engine is new and is the benchmark itself. The only 'falsifiable handle' outside the paper would be a released engine or a held-out scenario suite that external groups could score against; neither is provided.
  • Scenario bosses 'Tempest Lictor' and 'Black Bell of Ordrune' no independent evidence
    purpose: Created antagonists for day-track scenarios testing boss/lair mechanics.
    Introduced as authored scenario content. Their stats and abilities are in tables but not published as data files.

pith-pipeline@v1.3.0-daily-deepseek · 20086 in / 6491 out tokens · 52839 ms · 2026-08-03T04:11:51.739407+00:00 · methodology

0 comments
read the original abstract

Games and simulators make valuable benchmarks by turning decisions into measurable outcomes, but many current suites under-test rules-rich tactical reasoning: the ability to choose well when geometry, timing, resources, objectives, and rule interactions all matter at once. We introduce DungeonBench, a benchmark for tactical reasoning in Dungeons & Dragons combat, built to cover the vast majority of combat-relevant 2014 System Reference Document content whose effects can be resolved by the simulator while retaining mechanics that simplified combat simulators often abstract away. At each step, DungeonBench exposes a complete tactical observation, a pending decision, and an indexed list of executable options spanning movement, attacks, spells, reactions, objectives, preparation, and scarce resources. The task is to value legal choices whose consequences depend on action economy, creature traits, battlefield geometry, timing windows, and future encounters. DungeonBench has two tracks: Encounter, which evaluates local tactical play in single fights, and Day, which links encounters through persistent hit points, spell slots, consumables, preparation, and short-rest timing, forcing policies to trade off immediate tactical advantage against future survivability. The same engine-generated decision stream supports heuristic controllers, language-model policies, learned option rankers, and masked-action reinforcement-learning agents. We evaluate frontier language-model policies on this shared decision stream. Results show that full tactical observations do not saturate the benchmark: frontier policies often win direct encounters, but linked encounter days expose failures in resource budgeting, rest timing, and rule-aware tactical discipline.

Figures

Figures reproduced from arXiv: 2607.29577 by Atakan Kara, Ismayil Ismayilov, Kaan Oktay.

Figure 1
Figure 1. Figure 1: Overview of DungeonBench. The engine maps tactical state to legal executable options; [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 12 linked inside Pith

  1. [6]

    Openspiel: A framework for reinforcement learning in games.arXiv preprint arXiv:1908.09453,

    Marc Lanctot, Edward Lockhart, Jean-Baptiste Lespiau, Vinicius Zambaldi, Satyaki Upadhyay, Julien Pérolat, Sriram Srinivasan, Finbarr Timbers, Karl Tuyls, Shayegan Omidshafiei, et al. Openspiel: A framework for reinforcement learning in games.arXiv preprint arXiv:1908.09453,

    Pith/arXiv arXiv 1908

  2. [7]

    Craftax: A lightning-fast benchmark for open-ended reinforcement learning.arXiv preprint arXiv:2402.16801,

    Michael Matthews, Michael Beukman, Benjamin Ellis, Mikayel Samvelyan, Matthew Thomas Jackson, Samuel Coward, and Jakob Foerster. Craftax: A lightning-fast benchmark for open-ended reinforcement learning.arXiv preprint arXiv:2402.16801,

    Pith/arXiv arXiv

  3. [8]

    Carlo Romeo and Andrew D

    URLhttps://arxiv.org/abs/2411.13543. Carlo Romeo and Andrew D. Bagdanov. Ntrl: Encounter generation via reinforcement learning for dynamic difficulty adjustment in dungeons and dragons.arXiv preprint arXiv:2506.19530,

    Pith/arXiv arXiv

  4. [9]

    The starcraft multi-agent challenge.arXiv preprint arXiv:1902.04043,

    10 Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim G J Rudner, Chia-Man Hung, Philip H S Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge.arXiv preprint arXiv:1902.04043,

    Pith/arXiv arXiv 1902

  5. [11]

    Zelai Xu, Ruize Zhang, Chao Yu, Huining Yuan, Xiangmin Yi, Shilong Ji, Wenhao Tang, Feng Gao, Wenbo Ding, Xinlei Chen, and Yu Wang

    URLhttps://arxiv.org/abs/2310.01557. Zelai Xu, Ruize Zhang, Chao Yu, Huining Yuan, Xiangmin Yi, Shilong Ji, Wenhao Tang, Feng Gao, Wenbo Ding, Xinlei Chen, and Yu Wang. V olleybots: A testbed for multi-drone volleyball game combining motion control and strategic play.arXiv preprint arXiv:2502.01932,

    Pith/arXiv arXiv

  6. [12]

    choice_id

    URL https://arxiv.org/ abs/2210.03629. Andrew Zhu, Karmanya Aggarwal, Alexander Feng, Lara J. Martin, and Chris Callison-Burch. Fireball: A dataset of dungeons and dragons actual-play with structured game state information. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, pages 4171–4193, 2023a. Andrew Zhu, Lara ...

    Pith/arXiv arXiv 2014

  7. [2019]

    Benchmarking the spectrum of agent capabilities.arXiv preprint arXiv:2109.06780,

    Danijar Hafner. Benchmarking the spectrum of agent capabilities.arXiv preprint arXiv:2109.06780,

    Pith/arXiv arXiv

  8. [2020]

    Ogbinar, and Prospero C

    Joseph Emmanuel DL Dayo, Michel Onasis S. Ogbinar, and Prospero C. Naval Jr. Reinforcement learning environment with llm-controlled adversary in d&d 5th edition combat.arXiv preprint arXiv:2503.15726,

    Pith/arXiv arXiv

  9. [2021]

    Factorio learning environment.arXiv preprint arXiv:2503.09617,

    Jack Hopkins, Mart Bakler, and Akbir Khan. Factorio learning environment.arXiv preprint arXiv:2503.09617,

    Pith/arXiv arXiv

  10. [2023]

    Yue Wu, Xuan Tang, Tom M

    URLhttps://arxiv.org/abs/2305.16291. Yue Wu, Xuan Tang, Tom M. Mitchell, and Yuanzhi Li. Smartplay: A benchmark for llms as intelligent agents,

    Pith/arXiv arXiv

  11. [2024]

    Openai gym.arXiv preprint arXiv:1606.01540,

    Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym.arXiv preprint arXiv:1606.01540,

    Pith/arXiv arXiv

  12. [2025]

    Smacv2: An improved benchmark for cooperative multi-agent reinforce- ment learning.arXiv preprint arXiv:2212.07489,

    Benjamin Ellis, Skander Moalla, Mikayel Samvelyan, Mingfei Sun, Anuj Mahajan, Jakob Foerster, and Shimon Whiteson. Smacv2: An improved benchmark for cooperative multi-agent reinforce- ment learning.arXiv preprint arXiv:2212.07489,

    Pith/arXiv arXiv