Pith. sign in

REVIEW 4 major objections 5 minor 24 references

NTRL: Encounter Generation via Reinforcement Learning for Dynamic Difficulty Adjustment in Dungeons and Dragons

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that a reinforcement-learning agent, NTRL, can generate Dungeons & Dragons combat encounters that are longer, more damaging, and fairer than handbook or human design.

desk verdict A competent but overclaimed RL-DDA application to D&D: the artifact and code are real, the evaluation numbers are largely the reward function talking to itself. read the letter →

arxiv 2506.19530 v2 pith:NQLYESVQ submitted 2025-06-24 cs.AI

Carlo Romeo , Andrew D. Bagdanov This is my paper · ORCID
classification cs.AI
keywords Dungeons&DragonsdynamicdifficultyadjustmentreinforcementlearningencountergenerationcontextualbanditREINFORCEtabletoprole-playinggamescombatbalance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that combat encounter design in Dungeons & Dragons can be automated by a reinforcement-learning agent, NTRL, that picks enemy teams from a party's current statistics. The agent is framed as a contextual bandit and trained with REINFORCE in a simulator, then generates encounters at inference time with no further human tuning. Against both the Dungeon Master's Guide XP-budget heuristic and encounters designed by human DMs, NTRL reports fights roughly 200% longer, post-combat party hit points reduced by 16.67%, and win rates near 70-80% with few total-party kills. If the results transfer to live play, DMs would gain a real-time difficulty-adjustment tool that keeps combat tactically demanding without breaking narrative flow. The main caveat is that all training and evaluation happen in a simulated combat environment with fixed heuristic actors, so the paper's claims are about that environment unless human play confirms them.

What carries the argument

The central machinery is the contextual-bandit formulation of encounter construction, in which a single policy network chooses one enemy at a time and a synergy vector records the running count of selected enemy classes, feeding this count back into the next choice until a STOP action is emitted. Training uses REINFORCE, a policy-gradient algorithm, to maximize a weighted reward $R(p,e)=\alpha\,wp+\beta\,fl+\gamma\,mhp+\delta\,dmg+\lambda\,dth$, with win probability scaled in the thousands and total party kills penalized, so the learned policy optimizes for winnable but draining fights. Dynamic hit-point variations before combat (100%, 75%, 50%, 40%, 30%, 20%, or 10% of original HP, plus small noise) keep the policy from overfitting to full-health parties. The fixed open-source combat simulator, run 100 times per encounter and averaged, converts each candidate encounter into the scalar metrics that drive the reward.

What would settle it

Run a live-play study in which human parties play NTRL-generated encounters against handbook-budgeted ones, recording win rate, average rounds, and post-combat HP; if fights do not last markedly longer while win rate stays near 70-80% and total-party kills stay low, the central claim fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that an encounter generator can learn a difficulty policy rather than follow static XP budgets. NTRL represents each party as a feature matrix of hit points, armor class, core statistics, saving throws, resistances, spell lists, and abilities, perturbs hit points to simulate pre-combat resource loss, and sequentially chooses up to eight enemies from 26 classes using a policy network that also tracks how many of each enemy type have already been selected. A hand-crafted reward combines win probability, fight longevity, missing party HP, total damage dealt, and player deaths while penalizing total party kills; optimizing this reward with REINFORCE yields encounters that last about seven rounds versus three for the DM heuristic, leave the party at roughly 60% hit points instead of 70%, keep win probability near 0.7-0.8, and actually spend fewer XP than the DMG budget in simulation. Against 86 human-designed encounters, NTRL again produces longer fights and lower remaining HP at comparable win rates, though in that setting it spends more XP than the budget, an inversion the authors attribute to the absence of pre-fight HP loss.

Load-bearing premise

The load-bearing premise is that the computer-controlled characters in the simulator fight like real players, so encounters that are hard-but-fair for them will be hard-but-fair for a human party.

Editorial extensions

If this is right

  • A DM could query NTRL mid-session with the party's current hit points and resources and receive an encounter immediately, removing the pre-session XP-budget calculations.
  • Because NTRL spends fewer XP than the DMG budget to achieve the same or greater difficulty in simulations, the learned policy suggests that standard challenge ratings may underestimate optimized enemy synergies.
  • If the reward weights were tuned, the same architecture could target other design goals, such as faster combats, higher lethality, or narrative-appropriate monster themes.
  • The comparative human-DM study indicates that even experienced humans do not consistently reach the tension level of the trained agent, so encounter balancing is a learnable skill that can be automated.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step is testing NTRL in live sessions with human players; the paper's own numbers would hold or fail depending on whether human parties actually sustain 70-80% win rates on the generated encounters.
  • The reward function treats fight length, damage, and deaths as proxies for engagement; one could add player-reported enjoyment or narrative cost as an explicit term, which the current formulation does not model.
  • The XP under-spending result in simulation is an emergent claim that, if replicated, would be a testable hypothesis about CR and XP budgets overestimating difficulty for optimized parties.
  • Because all actors in the simulator use fixed utility-based heuristics, NTRL may be overfitting to predictable opponents; training against adaptive or human-like opponent policies would reveal how much of the difficulty gain is due to exploiting the simulator's own heuristics.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces NTRL, a contextual-bandit REINFORCE agent for generating D&D combat encounters as a dynamic difficulty adjustment tool. Encounters are constructed by sequentially selecting enemies from a pool of 26 classes conditioned on party features and a running synergy vector, and the agent is trained in a turn-based combat simulator. The reward function in Eq. (5) combines win probability, fight longevity, missing party HP, damage dealt, and player deaths, with a penalty for total party kills. Experiments compare NTRL against a DMG-style XP-budget heuristic and a random policy over 10,000 steps, five seeds, and 100 repeated simulations per encounter, plus a comparison with 86 human-DM-designed encounters evaluated in the same simulator. The paper reports a +200% increase in fight longevity, a 16.67% reduction in post-combat party HP, win probabilities around 70-80%, and low TPK rates, and concludes that NTRL improves tactical engagement and fairness.

Significance. If the claims were established, NTRL would be a practical and readily usable contribution to dynamic difficulty adjustment for tabletop RPGs, with instant encounter generation and an open-source implementation. The paper's strengths include a reproducible code release, repeated simulations to reduce dice-roll variance, multiple training seeds, and a real human-DM dataset (86 submissions) for comparison. However, the headline metrics are direct components of the optimized reward function, and all combat outcomes are generated by fixed heuristic AI actors in a simulator, so the current evidence does not establish the central claim about tactical engagement in human play. The contribution is potentially useful, but it needs re-framing and additional validation before the reported benefits can be accepted.

major comments (4)
  1. [§IV.D, Eq. (5), Fig. 2] The headline outcomes reported in Results and Conclusions—longer fights, lower remaining party HP, higher damage, player deaths with low TPK—are the five terms of the reward function in Eq. (5). Because NTRL is trained to maximize exactly this reward, observing that it scores higher than the heuristic baselines on these metrics is an expected consequence of optimization, not independent evidence of encounter quality. The leap from these quantities to 'tactical engagement' requires an external measure of engagement or human-play validation; without such a measure the evaluation is largely circular.
  2. [§IV.B, §V, §VI] All combat evaluations, including the 86 human-designed encounters, are run in the DnDSimulator in which both the party and the enemy are controlled by fixed utility-based heuristics (§IV.B). The authors explicitly acknowledge the gap between simulated and real play, but the conclusion (§VI) nevertheless states the performance gains as established. To support the claim that NTRL improves engagement for human players, the authors should either conduct human playtesting or, at minimum, show that simulator outcomes correlate with human-reported difficulty, fun, or engagement.
  3. [§IV.A, Fig. 2, Fig. 3] Quantitative claims are reported as point estimates without error bars, confidence intervals, or statistical tests, even though training is repeated over five random seeds. The percentage claims (+200% longevity, -16.67% HP) and the win-probability claims are also inconsistent across the manuscript: the abstract and conclusion report 70%, §V reports 75%, and the Figure 3 caption reports 80%. Reporting distributions and effect sizes is necessary before the headline numbers can be taken at face value.
  4. [§III.B, Eq. (5)] The reward weights α, β, γ, δ, and λ are described only qualitatively, and no actual values or selection procedure are given; the HP threshold set and noise range are also under-specified. This makes the sensitivity of the learned behavior to these free parameters impossible to assess and weakens reproducibility. A sensitivity analysis or a table of the chosen values would clarify whether the reported results are robust or tuned to the specific evaluation setup.
minor comments (5)
  1. [Abstract/§III] The acronym NTRL is never clearly expanded; the intended reading appears to be 'E Ncounter GeneraTion via Reinforcement Learning', but this should be stated explicitly at first use.
  2. [References] References [7] and [10] are the same Browne et al. survey; one should be removed or renumbered.
  3. [Figures 2 and 3] The captions and axis labels contain typographic artifacts, such as 'T otal Party Kills' and 'T eam Health'; these should be corrected.
  4. [Algorithm 1] The policy update on line 10 refers to Equation 4, but the return G_t is not defined for the bandit setting and the sampled reward R from line 9 is not used in the update; the learning rule should be stated precisely.
  5. [§III.B] The phrase 'generates of encounters' should read 'generation of encounters'.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline success metrics are terms of the optimized reward; the tactical-engagement claim is inferred rather than independently measured.

  1. self definitional [Eq. (5) (Reward Function, Sec. III.B); Results Sec. IV.D; Conclusions Sec. VI]
    "NTRL (Figure 1) is a contextual bandit [19] agent that is trained in a simulated environment to maximize a hand-crafted reward function that promotes long combats with high damage dealt to the party while avoiding a total annihilation of the players. ... R(p,e ) = α∗wp +β∗fl +γ∗mhp +δ∗dmg +λ∗dth (5) ... Empirical results from training simulations show that NTRL, compared to the DM’s heuristic policy, prolongs combat longevity by 200%, reduces post-fight hit points by 16.67%, and increases tactical engagement, all while maintaining a 70% Win Probability."

    Equation (5) defines the training objective as a weighted sum of exactly the quantities later presented as findings: win probability (wp), fight longevity (fl), missing party HP (mhp), damage dealt (dmg), and player deaths (dth), with TPK penalized. REINFORCE updates the policy to maximize this sum, so a converged agent will, by construction, score highly on these terms relative to uninformed baselines. The conclusion's claims of +200% combat longevity, -16.67% post-fight hit points, and increased tactical engagement are therefore reports of the optimization target itself rather than independent evidence of encounter quality. 'Tactical engagement' is not directly measured; it is inferred from the fl/dmg/mhp terms the reward already rewards.

full rationale

The only load-bearing circular step is the conflation of the reward function with the evaluation results. The agent is trained to maximize Eq. (5), whose explicit components are win probability, fight longevity, missing party HP, damage, and deaths (with TPK discouraged), and the paper then reports these same components as the empirical evidence that NTRL improves encounter quality and promotes tactical engagement. That is a self-definitional element: the claimed successes are the objective being optimized. There is no self-citation chain, imported uniqueness theorem, or ansatz smuggled via citation; the references to automatic playtesting and DDA literature are external. The human-DM comparison in Section V is a legitimate external benchmark, but the human submissions are still evaluated in the same DnDSimulator with fixed utility-based AI combatants, and the paper itself concedes in Section IV.B that 'there will be a gap between the simulated environment and reality due to the decisions made by human players that will not always be optimal and victory-oriented.' That caveat is a validity threat rather than a circularity, but it reinforces that the inference from reward-component improvements to real tactical engagement is unsupported. Because the baseline comparisons and the emergent XP-saving behavior are independent content, the paper is not wholly circular; however, the headline quantitative claims are not independent of the reward definition, so the score is 6 rather than lower.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central results rest on hand-chosen reward weights, the HP-variation scheme, and an unvalidated combat simulator. The paper adds no new physical or mathematical entities; the synergy vector is part of the state encoding, not an independent entity.

free parameters (3)
  • Reward weights alpha, beta, gamma, delta, lambda = Not given in text (described as 'carefully scaled')
    Section III.B, Eq. (5): the reward is R = alpha*wp + beta*fl + gamma*mhp + delta*dmg + lambda*dth. Hand-tuned weights set the trade-off among win probability, fight longevity, missing HP, damage, and deaths, so they directly shape all headline results.
  • HP threshold set and noise range = Thresholds {100%, 75%, 50%, 40%, 30%, 20%, 10%}; noise +/-5%
    Section III.B: dynamic HP variations emulate spent resources. The skewed distribution toward low thresholds affects learned XP-spending behavior and the full-HP human comparison.
  • Evaluation parameters = 10,000 steps, 100 simulations per encounter, 5 seeds, 128 hidden neurons
    Section IV.A and III.B: these are chosen to control variance and model capacity, but no sensitivity analysis is performed.
assumptions (4)
  • domain assumption The DnDSimulator (ref 22) implements D&D 5e combat rules accurately enough for the results to transfer
    All training and evaluation run inside this third-party simulator (Section IV.B). Its rule fidelity and AI behavior are not independently validated.
  • domain assumption The utility-based heuristic AI that controls both the party and enemies is a representative proxy for human tactical decisions
    Section IV.B states every combatant picks actions by a fixed utility score and 'the agents do not learn or adapt between battles.' Human players would not make the same choices, as the paper itself acknowledges.
  • ad hoc to paper The reward function in Eq. (5) is a valid operationalization of engagement, fairness, and strategic depth
    The agent is rewarded for long, damage-heavy fights with low remaining HP and occasional deaths; no external evidence ties these metrics to player experience or to 'strategic depth' as claimed in the abstract.
  • standard math REINFORCE converges to a near-optimal policy given the features, architecture, and reward
    Section III.A/B invokes the policy gradient theorem; no convergence guarantees are provided for this non-convex, finite-sample setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NTRL: Encounter Generation via Reinforcement Learning for Dynamic Difficulty Adjustment in Dungeons and Dragons." pith.science (2026). https://pith.science/paper/NQLYESVQ

@misc{pith2026250619530,
  author       = {Pith},
  title        = {Pith review of: NTRL: Encounter Generation via Reinforcement Learning for Dynamic Difficulty Adjustment in Dungeons and Dragons},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/NQLYESVQ}},
  note         = {Machine review of arXiv:2506.19530}
}
read the original abstract

Balancing combat encounters in Dungeons & Dragons (D&D) is a complex task that requires Dungeon Masters (DM) to manually assess party strength, enemy composition, and dynamic player interactions while avoiding interruption of the narrative flow. In this paper, we propose Encounter Generation via Reinforcement Learning (NTRL), a novel approach that automates Dynamic Difficulty Adjustment (DDA) in D&D via combat encounter design. By framing the problem as a contextual bandit, NTRL generates encounters based on real-time party members attributes. In comparison with classic DM heuristics, NTRL iteratively optimizes encounters to extend combat longevity (+200%), increases damage dealt to party members, reducing post-combat hit points (-16.67%), and raises the number of player deaths while maintaining low total party kills (TPK). The intensification of combat forces players to act wisely and engage in tactical maneuvers, even though the generated encounters guarantee high win rates (70%). Even in comparison with encounters designed by human Dungeon Masters, NTRL demonstrates superior performance by enhancing the strategic depth of combat while increasing difficulty in a manner that preserves overall game fairness.

Figures

Figures reproduced from arXiv: 2506.19530 by the authors.

Figure 1
Figure 1. Human Dungeon Master versus NTRL: A DM, based on his or her skills, must decide what the best level of difficulty is to create a stimulating event. After measuring the party XP budget, the DM must iteratively design an encounter, calculate the XP multiplier if more than one encounter has already been picked, and finally compare the encounter XP budget to that of the party to make these as close as possible. The whol… view at source ↗
Figure 2
Figure 2. Performance comparison of encounter generation policies. Although DM and RND policies are heuristic and do not require any training, both are plotted [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Comparison of NTRL against Human DM and heuristic DM: The Reinforcement Learning-based NTRL policy demonstrates superior optimization of engagement and challenge compared to human Dungeon Masters (Human), heuristic Dungeon Master (DM), and random baseline (RND) methods. NTRL achieves a high win probability of 80% while extending fight longevity beyond all baselines and maximizing rewards collection, ensuring prolong… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 18 canonical work pages

  1. [1]

    Mearls and J

    M. Mearls and J. Crawford, Dungeons & Dragons Player’s Handbook . Wizards of the Coast, 2014

    work page 2014

  2. [2]

    Mearls et al

    M. Mearls et al. , Dungeon Master’s Guide , 5th ed., S. F. Gray et al. , Eds. Wizards of the Coast, 2014

    work page 2014

  3. [3]

    Automatic play-testing of dungeons and dragons combat encounters,

    F. Shyne, “Automatic play-testing of dungeons and dragons combat encounters,” 2023

    work page 2023

  4. [4]

    No player left behind: evolving dungeons and dragons combat to optimize difficulty and player contributions,

    F. Shyne and S. Cooper, “No player left behind: evolving dungeons and dragons combat to optimize difficulty and player contributions,” in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment , vol. 20, no. 1, 2024, pp. 224–234

    work page 2024

  5. [5]

    Dynamic difficulty adjustment approaches in video games: a systematic literature review,

    F. Mortazavi, H. Moradi, and A.-H. Vahabie, “Dynamic difficulty adjustment approaches in video games: a systematic literature review,” Multimedia Tools and Applications , vol. 83, no. 35, pp. 83 227–83 274, 2024

    work page 2024

  6. [6]

    R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction . MIT press, 2018

    2018

  7. [8]

    Automated playtesting with procedural personas through MCTS with evolved heuristics,

    C. Holmg ˚ard, M. C. Green, A. Liapis, and J. Togelius, “Automated playtesting with procedural personas through MCTS with evolved heuristics,” CoRR, vol. abs/1802.06881, 2018. [Online]. Available: http://arxiv.org/abs/1802.06881

    arXiv 2018

  8. [9]

    Automatic playtesting for game parameter tuning via active learning,

    A. Zook, E. Fruchter, and M. O. Riedl, “Automatic playtesting for game parameter tuning via active learning,” 2019. [Online]. Available: https://arxiv.org/abs/1908.01417

    arXiv 2019

Show all 24 references
  1. [10]

    A survey of monte carlo tree search methods,

    C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton, “A survey of monte carlo tree search methods,” IEEE Transactions on Computational Intelligence and AI in games , vol. 4, no. 1, pp. 1–43, 2012

    2012

  2. [11]

    Artificial intelligence methods for automated difficulty and power balance in games,

    S. P. R. A. Reis, “Artificial intelligence methods for automated difficulty and power balance in games,” Ph.D. dissertation, Universidade do Porto (Portugal), 2024

    2024

  3. [12]

    A review of dynamic difficulty adjustment methods for serious games,

    J. C. Lopes and R. P. Lopes, “A review of dynamic difficulty adjustment methods for serious games,” in International Conference on Optimiza- tion, Learning Algorithms and Applications . Springer, 2022, pp. 144– 159. 8

    2022

  4. [13]

    Dynamic difficulty adjustment using deep reinforcement learning: A review,

    T. Zheng, “Dynamic difficulty adjustment using deep reinforcement learning: A review,” Applied and Computational Engineering , vol. 71, pp. 157–162, 2024

    2024

  5. [14]

    Investigating reinforcement learning for dynamic difficulty adjustment,

    T. N. De Oliveira and L. Chaimowicz, “Investigating reinforcement learning for dynamic difficulty adjustment,” in Proceedings of the 22nd Brazilian Symposium on Games and Digital Entertainment , 2023, pp. 66–75

    2023

  6. [15]

    The world of anwin: reinforcement learning in role- playing games,

    G. Q. Gomes, “The world of anwin: reinforcement learning in role- playing games,” Ph.D. dissertation, 2023

    2023

  7. [16]

    A framework for designing reinforcement learning agents with dynamic difficulty adjust- ment in single-player action video games,

    L. Climent, A. Longhi, A. Arbelaez, and M. Mancini, “A framework for designing reinforcement learning agents with dynamic difficulty adjust- ment in single-player action video games,” Entertainment Computing , vol. 50, p. 100686, 2024

    2024

  8. [17]

    Dungeons and DQNs: Toward reinforcement learning agents that play tabletop roleplaying games

    L. J. Martin, S. Sood, and M. O. Riedl, “Dungeons and DQNs: Toward reinforcement learning agents that play tabletop roleplaying games.” in INT/WICED@ AIIDE, 2018

    2018

  9. [18]

    Automated playtesting in videogames,

    P. M. Negr ˜ao, “Automated playtesting in videogames,” Master’s thesis, Universidade NOV A de Lisboa (Portugal), 2020

    2020

  10. [19]

    A contextual-bandit approach to personalized news article recommendation,

    L. Li, W. Chu, J. Langford, and R. E. Schapire, “A contextual-bandit approach to personalized news article recommendation,” in Proceedings of the 19th international conference on World wide web , 2010, pp. 661– 670

    2010

  11. [20]

    Simple statistical gradient-following algorithms for connectionist reinforcement learning,

    R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning , vol. 8, pp. 229–256, 1992

    1992

  12. [21]

    Policy gradi- ent methods for reinforcement learning with function approximation,

    R. S. Sutton, D. McAllester, S. Singh, and Y . Mansour, “Policy gradi- ent methods for reinforcement learning with function approximation,” Advances in neural information processing systems , vol. 12, 1999

    1999

  13. [22]

    DnDSimulator,

    D. K, “DnDSimulator,” https://github.com/DanielK314/DnDSimulator, 2022

    2022

  14. [23]

    Mearls, J

    M. Mearls, J. Crawford, C. Perkins, C. Sims, R. Thompson, P. Lee, R. J. Schwalb, M. Sernett, S. Townshend, J. Wyatt, S. F. Gray, and G. Bilsland, Monster Manual, 5th ed. Wizards of the Coast, 2014

    2014

  15. [24]

    [Online]

    Discord. [Online]. Available: https://discord.com

  16. [25]

    [Online]

    Reddit. [Online]. Available: https://www.reddit.com/

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.