Monte-Carlo Tree Search for Simulation-based Strategy Analysis

Alexander Zook; Brent Harrison; Mark O. Riedl

REVIEW 4 major objections 4 minor 1 cited by

Monte-Carlo Tree Search for Simulation-based Strategy Analysis

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper argues that Monte-Carlo Tree Search agents with tunable rollout budgets can stand in for human players of different skill levels, and that metrics extracted from their playthroughs can reveal game balance and design flaws…

desk verdict A clear proof-of-concept for simulation-based game design metrics whose central skill-proxy assumption is stated but never tested; read it as a method proposal, not as evidence about human play. read the letter →

arxiv 1908.01423 v1 pith:DCBZDHPC submitted 2019-08-04 cs.AI

Alexander Zook , Brent Harrison , Mark O. Riedl This is my paper · ORCID

classification cs.AI

keywords gamedesignsimulationplaytestingplayingMonte-CarloTreeSearchbalanceplayerskillstrategymetrics

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that automated planning agents can replace some human playtesting by simulating players of different skill levels. The skill dial is the number of Monte-Carlo Tree Search rollouts allowed per decision: more rollouts mean deeper strategic lookahead, which the authors use as a proxy for a more skilled human. From the resulting playthroughs they compute metrics at four levels—summary statistics, individual actions, repeated chains, and action spaces—and show in two case studies that these metrics reveal balance in a modified Scrabble game and expose a concrete design flaw in their card game Cardonomicon. The payoff, if the proxy holds, is a fast, cheap, systematic way for designers to probe the strategy space of a game before committing to human playtesting.

What carries the argument

The central machinery is Monte-Carlo Tree Search (MCTS), a sampling-based game-playing algorithm whose computational budget can be tuned. MCTS builds a search tree by repeatedly selecting, expanding, simulating random play, and backpropagating win/loss rewards, and its UCB1 selection policy forces every available move to be tried. The paper treats the number of rollouts—how many times this cycle runs—as the proxy for player skill, creating weak, moderate, and strong agents, and then feeds the resulting playtraces into a four-level metric taxonomy of summaries, atoms, chains, and action spaces.

What would settle it

Compare the same metrics from a human playtesting study in the simplified Scrabble and Cardonomicon against the weak, moderate, and strong agents. If higher-skill humans do not show the same ordering as higher-rollout agents—more wins, longer words, larger action spaces in Scrabble, or catch-up behavior in Cardonomicon—the central skill-proxy claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that a planning agent restricted in search effort can stand in for a human player of a corresponding skill level, so that automated playthroughs can substitute for some human playtesting. In Scrabble, metrics built from these playthroughs show the expected ordering—stronger agents win more often against weaker ones, play longer words, and face a larger action space—demonstrating that the technique can detect skill differences and confirm balance in a 150-point variant of the game. In Cardonomicon, the same metrics expose a concrete flaw: the second player loses far more often at every skill pairing, and the card set lacks synergy chains, indicating a small deck and a missing catch-up mechanic. The authors argue these results show simulated agents can identify both balance and design problems without a human test group.

Load-bearing premise

The load-bearing premise, which the authors openly state, is that the number of MCTS rollouts is a valid proxy for human skill: more computation per move produces behavior like a more skilled human player, and if that mapping fails, the Scrabble and Cardonomicon conclusions do not transfer to humans.

Editorial extensions

If this is right

Game designers can get a first-pass read on balance and strategy diversity for turn-based, fully observable games without recruiting human playtesters.
The four-level metric taxonomy gives designers a way to compare play styles across skill levels, not just win rates.
The same simulated agents can be re-run with modified game parameters, letting designers test tile distributions, card stats, or deck sizes in simulation.
Because the method is game-agnostic, it should transfer to other discrete adversarial games with perfect information, subject to the same skill-proxy caveat.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next step the paper does not take is to validate the skill proxy against human data, for example by comparing word-length and win-rate curves of human players at measured skill levels against the weak, moderate, and strong agents.
The approach could also be inverted: rather than using rollouts to model humans, observed human playtraces could be used to calibrate the rollout budget, turning the method into a measurement instrument for human skill.
The absence of chains in Cardonomicon suggests an automated design loop could search for card effects that create combos, using chain metrics as a fitness signal; this is an extension beyond the paper's case studies.
One implicit consequence is that for games with hidden information, the perfect-information simplification may hide skill dimensions such as bluffing or memory, so extending to imperfect-information games would require a different skill proxy.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

For you: this paper is a usable proof-of-concept for using MCTS to generate playtraces for game design analysis. The genuinely new pieces are treating rollout budget as a skill dial and organizing metrics into four levels—summaries, atoms, chains, action spaces. The case studies are concrete, the visualizations are readable, and the authors are honest in Section 6 that rollouts model only one dimension of skill. But the load-bearing assumption, that rollout count approximates human skill, is not validated. The paper's language repeatedly slides from "agents" to "players"—e.g., "higher skilled agents consistently defeat lower skilled opponents"—and the reader can forget that these are MCTS properties, not human properties.

Credit where it is due: the taxonomy is a useful organizing device for automated playtesting, and the Scrabble results (shorter games, word-length distributions, shrinking action space) show the kind of design-relevant information this pipeline can produce. The Cardonomicon case identifies a plausible second-player disadvantage and some card-usage patterns. The related work on restricted play and action spaces is well chosen, and the method is described well enough to reimplement. For a proof of concept, 100 games per pairing is thin but not disqualifying.

Now the soft spots. First, the skill proxy. The two-move-ahead bound is justified by citing work on recursive reasoning in logic puzzles [4]; that is about deductive reasoning, not strategic game play, and the leap is unjustified. More rollouts generally beat fewer rollouts in MCTS, so the win-rate gradients in Figure 2 are expected and do not validate a human-skill reading. Second, the skill levels are hand-tuned: the authors halve the stronger agents' rollouts after "initial testing" to make differences visible. That is fine for a sensitivity analysis, but it undercuts the claim that levels correspond to principled human skill tiers. Third, there are no error bars or significance tests, so several interpretations (e.g., "stronger agents have more possible actions on a given turn") could be noise. Fourth, the Cardonomicon case is a self-designed game, so the "design flaw" findings are not independent; this is a minor concern for a proof of concept.

Is the central argument intact? Yes, if you read the claim as: MCTS with varying computational budgets can generate diverse playtraces, and those traces can support a metric taxonomy useful for design analysis. That claim holds. The stronger claim about simulating humans of varying skill levels does not, and the authors mostly acknowledge this in the limitations but still lean on it in the case-study interpretations.

Who is this for? Game AI researchers and designers working on automated playtesting or automated game generation. They will get a useful vocabulary and a clear demonstration of one approach. It deserves a serious referee, but the referee should ask for either a human-benchmark validation of the skill proxy or a rewrite that drops the human-simulation framing. My recommendation: send to peer review, conditional on revisions; do not desk reject.

Referee Report

4 major / 4 minor

Summary. The paper proposes using Monte-Carlo Tree Search (MCTS) agents with varying rollout budgets as a proxy for human players of different skill levels, then extracts design metrics from the resulting playthroughs. The authors define a four-level taxonomy of metrics—summaries, atoms, chains, and action spaces—and demonstrate the workflow in two case studies: Scrabble (a modified 150-point version) and Cardonomicon, a self-designed collectible card game. They report that the Scrabble metrics show game balance and skill differentiation, while the Cardonomicon metrics identify a second-player disadvantage and a lack of card synergies. The paper concludes that automated planning agents can simulate human play at varying skill levels and that the extracted metrics can support game design analysis.

Significance. If the central premise is validated, the approach would offer a low-cost complement to human playtesting and a useful taxonomy for organizing strategy analysis in turn-based adversarial games. The metric taxonomy is clearly presented, the two case studies illustrate a complete workflow from playtrace collection to design interpretation, and the authors are explicitly honest in Section 6 about the limitations of using rollouts as a single dimension of skill and about the full-observability restriction. There is no equation-level circularity: the win rates are computed from actual playouts, and the metric values are not fit to the conclusions. However, the paper's main claim—that the agents 'simulate humans of varying skill levels'—is not supported by any evidence connecting rollout budgets to human strategic behavior. The case studies therefore currently characterize MCTS agents with different computational limits, not human players, and the design conclusions rest on an unvalidated analogy.

major comments (4)

[3.1, 5.1] The central premise that MCTS rollout count is a proxy for human player skill is asserted rather than supported. Section 3.1 states 'We use MCTS rollouts as a proxy for player skill' and Section 5.1 calibrates the two-move-ahead upper bound by citing research on recursive reasoning in logic puzzles [4]. No evidence is presented that more rollouts correspond to better human strategic play in Scrabble or Cardonomicon, and the cited work concerns deductive puzzle reasoning, not adversarial game play. This is load-bearing because the abstract and conclusion claim that agents can 'simulate humans of varying skill levels,' and the observed win-rate gradients and action-space differences are interpreted as properties of human skill. As it stands, Figures 2-9 primarily characterize MCTS agents with different computational budgets. A human-subject comparison or an empirically established mapping between search depth and human performance is needed to support the claim.
[5.1] The definition of the skill levels is partly hand-tuned. The authors report that 'Initial testing revealed little difference between the latter two agents; our results report agents that halve the number of rollouts of the two stronger agents as these more clearly illustrate the outcomes of variable player skill.' Because the weak/moderate/strong labels are adjusted until the desired qualitative differences appear, the subsequent comparisons risk confirming the authors' expectations rather than testing a pre-specified model of skill. The manuscript should either derive the rollout budgets from an independent model of human lookahead or report the original budgets and a pre-registered criterion, so that the skill-level manipulation is not circular.
[5.2] The quantitative support for the balance and skill claims is thin. Each pairing uses only 100 games (Section 5.1), and figures such as Figure 2 and the first-turn win-rate statement are reported without error bars or significance tests. For example, the claim that 'higher skilled agents consistently defeat lower skilled opponents' is based on point estimates; the moderate-versus-strong differences in particular appear small enough that sampling noise could change the qualitative conclusion. The authors should report confidence intervals, standard errors, or hypothesis tests for the headline win rates and for the differences in game length and action-space metrics.
[5.3] The Cardonomicon case study's design-flaw conclusions are not validated. Cardonomicon was created by the authors, and the interpretations—for instance, the second-player disadvantage being caused by the lack of a catch-up mechanic, and the absence of chains indicating lack of synergy—are plausible but are never tested by modifying the design and observing whether the metrics change accordingly. Because the central demonstration is that the metrics 'identify potential flaws,' a design-iteration experiment (e.g., adding a catch-up mechanic and showing the second-player win rate moves toward 50%) would substantially strengthen the claim. Without this, the case study is an illustration rather than evidence that the metrics reliably reveal flaws.

minor comments (4)

[3.2] There is a typo in the Scrabble description: 'the the first player to meet or exceed 150 points wins' should read 'the first player.'
[Figures 5-9] The figures showing action-space metrics need more complete captions: it should be stated whether the plotted values are means or medians, over how many games, and what the error characteristics are, since without this information the reader cannot assess the stability of the reported differences.
[5.2, Figure 2] The caption for Figure 2 explains the red/blue coloring but not the ordering of the skill levels on the axes; please state explicitly how the grid corresponds to Player 1 and Player 2 skills.
[6] The limitation about perfect dictionary knowledge in Scrabble is mentioned only in the limitations section; it would help to restate this assumption in Section 5.2 where word-choice behavior is interpreted, so that the 'human skill' framing is not accidentally overstated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reported metrics are computed from MCTS playouts, and the skill proxy is an explicit, unvalidated assumption rather than a derived prediction.

full rationale

The paper's derivation chain is not circular. Agent skill levels are operationalized as MCTS rollout counts in Section 3.1, and all reported metrics—win rates, word-length frequencies, action-space sizes, and card-play frequencies—are computed from the resulting simulated playtraces, not fitted to any human data or target outputs. The rollout bounds are set from approximate branching factors and adjusted after initial testing to display clearer skill differences, but this adjustment is not a fit to the reported dependent variables; the final metrics remain emergent properties of the simulations. The two-move lookahead upper bound is justified by an external citation to reasoning research, not by the authors' own prior work, and the paper's Limitations section explicitly concedes that MCTS rollouts represent only one dimension of player skill. The self-citation to Zook and Riedl (2014) appears only in related work and is not load-bearing. The central weakness—that rollout count is an unvalidated proxy for human skill—is a threat to external validity and a correctness concern, not an equation-level or definitional circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The axioms listed are the load-bearing background assumptions of the demonstration. The most important is the skill proxy: all conclusions about 'weak' and 'strong' players inherit the assumption that rollout count maps to human skill. The remaining axioms concern the mapping from rollout budgets to lookahead depth, the validity of the perfect-information simplifications, and the standard behavior of MCTS with UCB1. Free parameters are the hand-chosen rollout budgets and the modified Scrabble win condition; no parameters were fitted to the metric outputs.

free parameters (3)

Scrabble rollout counts for weak/moderate/strong agents = 50 / 650 / 1250
Chosen from an estimated median branching factor of 50 legal words, representing one-move, intermediate, and two-move lookahead; these define the skill levels used in all Scrabble results.
Cardonomicon rollout counts for weak/moderate/strong agents = 100 / 2500 / 5000
Derived from estimated ~15 play and ~81 attack moves per turn; halved after initial testing because the two strongest agents behaved similarly.
Scrabble win threshold = 150 points
Hand-chosen modification to end games sooner; balance conclusions apply only to this variant and may not transfer to standard Scrabble.

assumptions (5)

domain assumption MCTS rollout counts are a valid proxy for player skill along the lookahead dimension.
Stated in Section 3.1 and used throughout; the paper's limitation section concedes it is only one dimension, and no human validation is provided.
domain assumption Humans typically reason only about two levels of recursive embedding, so two-move lookahead is an appropriate upper bound.
Section 5.1 cites Browne (2013) for this bound and uses it to set strong-agent rollout counts.
domain assumption The branching factor estimates (median 50 Scrabble words, ~100 Cardonomicon moves) are accurate enough to map rollout counts to lookahead depth.
Used in Section 5.1; if the estimates are off, the three named skill levels are not what they claim to be.
domain assumption Perfect information simplifications do not change the design properties under investigation.
Section 3.2 converts both domains to perfect information for MCTS; the balance and skill conclusions assume this abstraction preserves the relevant strategic structure.
domain assumption UCB1 selection with more rollouts monotonically improves play in these domains.
Section 3.1 relies on the standard MCTS property; initial testing found weak vs strong differences but little difference between the top two budgets, so monotonicity is imperfect.

invented entities (1)

Cardonomicon
purpose: A custom collectible card game used as the second case study to demonstrate design-flaw detection.
The game was designed by the authors; its deck and card statistics are not fully specified in the paper, and findings about its imbalance come from the authors' own simulations, so there is no independent benchmark.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Monte-Carlo Tree Search for Simulation-based Strategy Analysis." pith.science (2026). https://pith.science/paper/DCBZDHPC

@misc{pith2026190801423,
  author       = {Pith},
  title        = {Pith review of: Monte-Carlo Tree Search for Simulation-based Strategy Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DCBZDHPC}},
  note         = {Machine review of arXiv:1908.01423}
}

read the original abstract

Games are often designed to shape player behavior in a desired way; however, it can be unclear how design decisions affect the space of behaviors in a game. Designers usually explore this space through human playtesting, which can be time-consuming and of limited effectiveness in exhausting the space of possible behaviors. In this paper, we propose the use of automated planning agents to simulate humans of varying skill levels to generate game playthroughs. Metrics can then be gathered from these playthroughs to evaluate the current game design and identify its potential flaws. We demonstrate this technique in two games: the popular word game Scrabble and a collectible card game of our own design named Cardonomicon. Using these case studies, we show how using simulated agents to model humans of varying skill levels allows us to extract metrics to describe game balance (in the case of Scrabble) and highlight potential design flaws (in the case of Cardonomicon).

Figures

Figures reproduced from arXiv: 1908.01423 by the authors.

**Figure 1.** Diagram of MCTS algorithm steps. tional time and exploration of the full space of play behavior. We choose MCTS as a behavior sampling algorithm for its proven high-level performance, domain generality, and variable computational bounds. Note that other algorithms may serve this need; we leave comparison to alternative algorithms to future work. For simplicity, both of our case study domains have been made perfect… view at source ↗

**Figure 4.** Frequency of the top three-letter words in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 5.** Median number of words that could be played per [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

Figures from the paper (4 more)

**Figure 6.** Figure 6: Number of unique words played per turn by skill. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

**Figure 7.** Figure 7: Win rates for second turn player in Cardonomicon. The x-axis indicates agent strength for the second turn player; the y axis indicates the opposing agent’s strength. use and fixes the deck that players use. These major alterations to the typical structure of a card g…

**Figure 8.** Figure 8: Average number of possible card plays per turn [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

**Figure 9.** Figure 9: Average number of possible attacks per turn based [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Empowering Economic Simulation for Massively Multiplayer Online Games through Generative Agent-Based Modeling
cs.AI 2025-06 conditional novelty 6.0 of 10

LLM-driven agents in a simulated MMO economy reproduce role specialization and price responses to supply and demand, though the price result is partly shaped by what the AI is told.

Reference graph

Works this paper leans on

57 extracted references · 56 canonical work pages · cited by 1 Pith paper

[4]

Selection Choose how to descend a tree of expanded nodes until reaching an unexpanded node

work page
[1]

restricted play

INTRODUCTION Creating a game, from a small-scale indie game to a large AAA title, requires careful consideration of how design deci- sions shape player behavior. Any single design choice—e.g., including a word in Scrabble—has rippling consequences for the space of strategies available to players. While designers directly shape the space of actions availab...

arXiv 1908
[2]

RELA TED WORK Automated game analysis is a growing ﬁeld concerned with developing methods to understand a game design with- out requiring human play [18, 24, 36, 41]. Researchers have developed methods to simulate game systems with little or no human player activity [12], use constraint solving to check for the existence of traces consistent with speculat...

work page
[3]

To do this, we use a simulated agent to sample player be- haviors and then describe playtraces using summary statis- tics and pattern analysis to inform design analysis

METHODOLOGY One goal of this work is to automate player strategy analy- sis. To do this, we use a simulated agent to sample player be- haviors and then describe playtraces using summary statis- tics and pattern analysis to inform design analysis. In this section we review our agent simulation technique, Monte- Carlo Tree Search, and describe the game doma...

work page
[5]

Expansion Expand the set of actions available at an unexpanded node and choose a new node

work page
[6]

Simulation Follow a ﬁxed strategy (usually random choice) for how to act over all remaining unexpanded decisions until reaching the end of the game

work page
[7]

MCTS balances between agents exploring alternative actions and exploiting known good actions

Backpropagation Use the reward from the end game state reached (e.g., win or loss) to update the expected value of the newly expanded node and all of its parent nodes in the tree. MCTS balances between agents exploring alternative actions and exploiting known good actions. Typically, selection uses the UCB1 algorithm, which picks a node using a combina- t...

work page
[8]

We use a simple taxonomy to distinguish classes of design metrics: summaries, atoms, chains, and action spaces

SKILL-BASED DESIGN METRICS In this work we focus on automating the analysis of game designs through design metrics for player actions and how they change based on player skill: e.g., more skilled play- ers may score more points per turn in Scrabble. We use a simple taxonomy to distinguish classes of design metrics: summaries, atoms, chains, and action spa...

work page

Show all 57 references

[9]

con” on a turn was to add an “i

CASE STUDIES To demonstrate how player simulation and our metric tax- onomy can aid in game design evaluation, we performed two case studies. The ﬁrst case study of Scrabble explores how our metrics can evaluate a balanced game. TheScrabble case study veriﬁes that our techniqu...
[10]

Figure 9: Average number of possible attacks per turn based on skill

LIMITA TIONS Our technique has two key limitations: (1) MCTS rollouts Figure 8: Average number of possible card plays per turn based on skill. Figure 9: Average number of possible attacks per turn based on skill. only represent one dimension of player skill and (2) the algo- r...
[11]

CONCLUSION In this paper, we have shown how anytime planning agents can simulate players of various skill levels in turn-based, ad- versarial games. We provide several types of metrics to ana- lyze (human or simulated) player strategies from playtraces— at the levels of summar...
[12]

ACKNOWLEDGMENTS We would also like to thank our reviewers for highlighting additional related work and noting abstraction as a tech- nique to augment sampling approaches
[13]

Andersen, Y.-E

E. Andersen, Y.-E. Liu, E. Apter, F. Boucher-Genesse, and Z. Popovi´ c. Gameplay analysis through state projection. In 5th International Conference on the Foundations of Digital Games , 2010

2010
[14]

Benbassat and M

A. Benbassat and M. Sipper. EvoMCTS: Enhancing MCTS-based players through genetic programming. In IEEE Conference on Computational Intelligence in Games, 2013

2013
[15]

Bj ¨ork and J

S. Bj ¨ork and J. Holopainen. Patterns in Game Design . Cengage Learning, 2005

2005
[16]

C. Browne. Deductive search for logic puzzles. In IEEE Conference on Computational Intelligence in Games, 2013

2013
[17]

Browne and F

C. Browne and F. Maire. Evolutionary game design. IEEE Transactions on Computational Intelligence and AI in Games , 2:1–16, 2010

2010
[18]

C. B. Browne, E. Powley, D. Whitehouse, S. M. Lucas, P. I. Cowling, P. Rohlfshagen, S. Tavener, D. Perez, S. Samothrakis, and S. Colton. A survey of monte carlo tree search methods. IEEE Transactions on Computational Intelligence and AI in Games , 4:1–43, 2012

2012
[19]

T. Cadwell. Counterplay and teamplay in multiplayer game design. In Game Developers Conference, 2013

2013
[20]

Chaslot, J.-T

G. Chaslot, J.-T. Saito, J. W. H. M. Uiterwijk, B. Bouzy, , and H. J. van den Herik. Monte-Carlo strategies for computer Go. In 18th Belgian-Dutch Conference on Artiﬁcial Intelligence , 2006

2006
[21]

D. Cook. The chemistry of game design. website, July 2007

2007
[22]

M. Cook, S. Colton, and J. Gow. Initial results from co-operative co-evolution for automated platformer design. In EvoGAMES, 2012

2012
[23]

P. I. Cowling, C. D. Ward, and E. J. Powley. Ensemble determinization in Monte Carlo Tree Search for the imperfect information card game Magic: The Gathering. IEEE Trans. Computational Intelligence and AI in Games , 4:241–257, 2012

2012
[24]

J. Dormans. Machinations: Elemental feedback structures for game design. In GAMEON-NA, 2009

2009
[25]

Elias, R

G. Elias, R. Garﬁeld, and K. Gutschera. Characteristics of Games . MIT Press, 2012

2012
[26]

J. M. Font, T. Mahlmann, D. Manrique, and J. Togelius. Towards the automatic generation of card games through grammar-guided genetic programming. In 8th International Conference on Foundations of Digital Games, 2013

2013
[27]

Genesereth, N

M. Genesereth, N. Love, and B. Pell. General game playing: Overview of the AAAI competition. AI Magazine, 26(2):62, 2005

2005
[28]

E. J. Jacobsen, R. Greve, and J. Togelius. Monte Mario: Platforming with MCTS. In Genetic and Evolutionary Computation, 2014

2014
[29]

A. Jaﬀe. Understanding Game Balance with Quantitative Methods. PhD thesis, University of Washington, 2013

2013
[30]

A. Jaﬀe, A. Miller, E. Andersen, Y.-E. Liu, A. Karlin, and Z. Popovi´ c. Evaluating competitive game balance with restricted play. In Proceedings of the Eighth Conference on Artiﬁcial Intelligence and Interactive Digital Entertainment, 2012

2012
[31]

R. Koster. A Theory of Fun in Game Design . Paraglyph press, 2nd edition, 2013

2013
[32]

Linehan, G

C. Linehan, G. Bellord, B. Kirman, Z. H. Morford, and B. Roche. Learning curves: Analysing pace and challenge in four successful puzzle games. In CHI PLAY, 2014

2014
[33]

Y.-E. Liu, E. Andersen, R. Snider, S. Cooper, and Z. Popovi´ c. Feature-based projections for eﬀective playtrace analysis. In 6th International Conference on Foundations of Digital Games , 2011

2011
[34]

N. Love, T. Hinrichs, D. Haley, E. Schkufza, and M. Genesereth. General game playing: Game description language speciﬁcation. Technical report, Stanford University, 2008

2008
[35]

Mahlmann, J

T. Mahlmann, J. Togelius, and G. N. Yannakakis. Modelling and evaluation of complex scenarios with the strategy game description language. In IEEE Conference on Computational Intelligence and Games , pages 174–181. IEEE, 2011

2011
[36]

M. J. Nelson. Game metrics without players: Strategies for understanding game artifacts. In 1st Workshop on Artiﬁcial Intelligence in the Game Design Process, 2011

2011
[37]

M. J. Nelson, J. Togelius, C. Browne, and M. Cook. Rules and mechanics. In Procedural Content Generation in Games: A Textbook and an Overview of Current Research. Springer, 2015

2015
[38]

T. S. Nielsen, G. A. Barros, J. Togelius, and M. J. Nelson. General video game evaluation using relative algorithm performance proﬁles. In Applications of Evolutionary Computation. Springer, 2015

2015
[39]

J. C. Osborn and M. Mateas. A game-independent play trace dissimilarity metric. In 9th International Conference on the Foundations of Digital Games , 2014

2014
[40]

J. C. Osborn, B. Samuel, J. A. McCoy, and M. Mateas. Evaluating play trace (dis)similarity metrics. In 10th AAAI Conference on Artiﬁcial Intelligence and Interactive Digital Entertainment , 2014

2014
[41]

Perez, J

D. Perez, J. Togelius, S. Samothrakis, P. Rolhfshagen, and S. M. Lucas. Automated map generation for the physical travelling salesman problem. IEEE Trans. Computational Intelligence and AI in Games , 18:708–720, 2013

2013
[42]

Salen and E

K. Salen and E. Zimmerman. Rules of Play: Game Design Fundamentals. MIT Press, Cambridge Mass., 2003

2003
[43]

Seif El-Nasr, A

M. Seif El-Nasr, A. Drachen, and A. Canossa, editors. Game Analytics. Springer London, 2013

2013
[44]

R. Sifa, A. Drachen, C. Bauckhage, C. Thurau, and A. Canossa. Behavior evolution in Tomb Raider: Underworld. In IEEE Conference on Computational Intelligence and Games , 2013

2013
[45]

Smith and M

A. Smith and M. Mateas. Answer set programming for procedural content generation: A design space approach. IEEE Transactions on Computational Intelligence and AI in Games , 3(3):187–200, 2011

2011
[46]

A. M. Smith. Open problem: Reusable gameplay trace samplers. In 2nd Workshop on Artiﬁcial Intelligence in the Game Design Process , 2013

2013
[47]

A. M. Smith, E. Butler, and Z. Popovi´ c. Quantifying over play: Constraining undesirable solutions in puzzle design. In 8th International Conference on the Foundations of Digital Games , 2013

2013
[48]

A. M. Smith, M. J. Nelson, and M. Mateas. LUDOCORE: A logical game engine for modeling videogames. In IEEE Conference on Computational Intelligence and Games , 2010

2010
[49]

Smith, J

G. Smith, J. Whitehead, and M. Mateas. Tanagra: Reactive planning and constraint solving for mixed-initiative level design. IEEE Transactions on Computational Intelligence and AI in Games , 3(3):201–215, 2011

2011
[50]

Sturtevant

N. Sturtevant. An argument for large-scale breadth-ﬁrst search for game design and content generation via a case study of ﬂing! In 2nd Workshop on Artiﬁcial Intelligence in the Game Design Process , 2013

2013
[51]

Thurau, K

C. Thurau, K. Kersting, and C. Bauckhage. Convex non-negative matrix factorization in the wild. In 9th IEEE Conference on Data Mining , 2009

2009
[52]

Togelius and J

J. Togelius and J. Schmidhuber. An experiment in automatic game design. In IEEE Symposium on Computational Intelligence and Games , 2008

2008
[53]

Togelius, G

J. Togelius, G. Yannakakis, K. Stanley, and C. Browne. Search-based procedural content generation: A taxonomy and survey. IEEE Transactions on Computational Intelligence and AI in Games, 3(3):172–186, 2011

2011
[54]

Tremblay, A

J. Tremblay, A. Borodovski, and C. Verbrugge. I can jump! exploring search algorithms for simulating platformer players. In Experimental AI in Games Workshop, 2014

2014
[55]

Wallner and S

G. Wallner and S. Kriglstein. Visualization-based analysis of gameplay data – a review of literature. Entertainment Computing, 4(3):143 – 155, 2013

2013
[56]

C. D. Ward and P. I. Cowling. Monte Carlo search applied to card selection in Magic: The Gathering. In IEEE Conference on Computational Intelligence and Games, 2009

2009
[57]

Zook and M

A. Zook and M. O. Riedl. Automatic game design via mechanic generation. In Proceedings of the 28th AAAI Conference on Artiﬁcial Intelligence , 2014

2014

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [4]

Selection Choose how to descend a tree of expanded nodes until reaching an unexpanded node

work page

[2] [1]

restricted play

INTRODUCTION Creating a game, from a small-scale indie game to a large AAA title, requires careful consideration of how design deci- sions shape player behavior. Any single design choice—e.g., including a word in Scrabble—has rippling consequences for the space of strategies available to players. While designers directly shape the space of actions availab...

arXiv 1908

[3] [2]

RELA TED WORK Automated game analysis is a growing ﬁeld concerned with developing methods to understand a game design with- out requiring human play [18, 24, 36, 41]. Researchers have developed methods to simulate game systems with little or no human player activity [12], use constraint solving to check for the existence of traces consistent with speculat...

work page

[4] [3]

To do this, we use a simulated agent to sample player be- haviors and then describe playtraces using summary statis- tics and pattern analysis to inform design analysis

METHODOLOGY One goal of this work is to automate player strategy analy- sis. To do this, we use a simulated agent to sample player be- haviors and then describe playtraces using summary statis- tics and pattern analysis to inform design analysis. In this section we review our agent simulation technique, Monte- Carlo Tree Search, and describe the game doma...

work page

[5] [5]

Expansion Expand the set of actions available at an unexpanded node and choose a new node

work page

[6] [6]

Simulation Follow a ﬁxed strategy (usually random choice) for how to act over all remaining unexpanded decisions until reaching the end of the game

work page

[7] [7]

MCTS balances between agents exploring alternative actions and exploiting known good actions

Backpropagation Use the reward from the end game state reached (e.g., win or loss) to update the expected value of the newly expanded node and all of its parent nodes in the tree. MCTS balances between agents exploring alternative actions and exploiting known good actions. Typically, selection uses the UCB1 algorithm, which picks a node using a combina- t...

work page

[8] [8]

We use a simple taxonomy to distinguish classes of design metrics: summaries, atoms, chains, and action spaces

SKILL-BASED DESIGN METRICS In this work we focus on automating the analysis of game designs through design metrics for player actions and how they change based on player skill: e.g., more skilled play- ers may score more points per turn in Scrabble. We use a simple taxonomy to distinguish classes of design metrics: summaries, atoms, chains, and action spa...

work page